[bzip2/lzma] fix for built-in initramfs issue

H. Peter Anvin wrote:
Alain Knaff wrote:
H. Peter Anvin wrote:
I'm thinking that with "preferred" we might just use the first one in
the priority list "lzma, bzip2, gzip, uncompressed" depending on what is
installed in the kernel.

Using the kernel method may definitely not work, especially on
architectures which don't do this style of kernel compression.

-hpa

Ok. For the new change, may I do a diff against the current
x86/lzma-setup , or should I do the whole 5-part dance again?


Go ahead and do an incremental patch at this point.

-hpa

Here it is (see attachment)

Alain


From: Alain Knaff <alain@xxxxxxxx>
Subject: [PATCH] init: fix built-in initramfs vs CONFIG_RD_GZIP

Impact: Consistency

Makes it possible to disable CONFIG_RD_GZIP . In that case, the
built-in initramfs will be compressed by whatever compressor is
available (bzip2 or lzma) or left uncompressed if none is available.

It also removes a couple of warnings which occur when no ramdisk
compression at all is chosen.

It also restores the select ZLIB_INFLATE in drivers/block/Kconfig
which somehow came missing. This is needed to activate compilation of
the stuff in zlib_deflate.

Signed-off-by: Alain Knaff <alain@xxxxxxxx>
---
diff -purN x86.ref/drivers/block/Kconfig x86-builtin-initramfs/drivers/block/Kconfig
--- x86.ref/drivers/block/Kconfig 2009-01-06 08:31:16.000000000 +0100
+++ x86-builtin-initramfs/drivers/block/Kconfig 2009-01-06 21:41:16.000000000 +0100
@@ -363,6 +363,7 @@ config RD_GZIP
default y
depends on BLK_DEV_INITRD=y
select DECOMPRESS_GZIP
+ select ZLIB_INFLATE
help
Support loading of a gzip encoded initial ramdisk or cpio buffer.
If unsure, say Y.
diff -purN x86.ref/init/initramfs.c x86-builtin-initramfs/init/initramfs.c
--- x86.ref/init/initramfs.c 2009-01-06 08:31:16.000000000 +0100
+++ x86-builtin-initramfs/init/initramfs.c 2009-01-06 22:27:26.000000000 +0100
@@ -389,7 +389,7 @@ static int __init write_buffer(char *buf
return len - count;
}

-
+#if defined CONFIG_RD_GZIP || defined CONFIG_RD_BZIP2 || defined CONFIG_RD_LZMA
static int __init flush_buffer(void *bufv, unsigned len)
{
char *buf = (char *) bufv;
@@ -412,6 +412,7 @@ static int __init flush_buffer(void *buf
}
return origLen;
}
+#endif

static unsigned my_inptr; /* index of next byte to be processed in inbuf */

@@ -449,10 +450,12 @@ static char * __init unpack_to_rootfs(ch
continue;
}
this_header = 0;
+#ifdef CONFIG_RD_GZIP
if (!gunzip(buf, len, NULL, flush_buffer, NULL,
&my_inptr, error) &&
message == NULL)
goto ok;
+#endif

#ifdef CONFIG_RD_BZIP2
message = NULL; /* Zero out message, or else cpio will
@@ -473,7 +476,9 @@ static char * __init unpack_to_rootfs(ch
goto ok;
}
#endif
+#if defined CONFIG_RD_GZIP || defined CONFIG_RD_BZIP2 || defined CONFIG_RD_LZMA
ok:
+#endif
if (state != Reset)
error("junk in compressed archive");
this_header = saved_offset + my_inptr;
diff -purN x86.ref/scripts/gen_initramfs_list.sh x86-builtin-initramfs/scripts/gen_initramfs_list.sh
--- x86.ref/scripts/gen_initramfs_list.sh 2009-01-05 01:21:06.000000000 +0100
+++ x86-builtin-initramfs/scripts/gen_initramfs_list.sh 2009-01-06 21:48:19.000000000 +0100
@@ -5,7 +5,7 @@
# Released under the terms of the GNU GPL
#
# Generate a cpio packed initramfs. It uses gen_init_cpio to generate
-# the cpio archive, and gzip to pack it.
+# the cpio archive, and then compresses it.
# The script may also be used to generate the inputfile used for gen_init_cpio
# This script assumes that gen_init_cpio is located in usr/ directory

@@ -16,8 +16,8 @@ usage() {
cat << EOF
Usage:
$0 [-o <file>] [-u <uid>] [-g <gid>] {-d | <cpio_source>} ...
- -o <file> Create gzipped initramfs file named <file> using
- gen_init_cpio and gzip
+ -o <file> Create compressed initramfs file named <file> using
+ gen_init_cpio and compressor depending on the extension
-u <uid> User ID to map to user ID 0 (root).
<uid> is only meaningful if <cpio_source> is a
directory. "squash" forces all files to uid 0.
@@ -225,6 +225,7 @@ cpio_list=
output="/dev/stdout"
output_file=""
is_cpio_compressed=
+compr="gzip -9 -f"

arg="$1"
case "$arg" in
@@ -233,11 +234,15 @@ case "$arg" in
echo "deps_initramfs := \\"
shift
;;
- "-o") # generate gzipped cpio image named $1
+ "-o") # generate compressed cpio image named $1
shift
output_file="$1"
cpio_list="$(mktemp ${TMPDIR:-/tmp}/cpiolist.XXXXXX)"
output=${cpio_list}
+ echo "$output_file" | grep -q "\.gz$" && compr="gzip -9 -f"
+ echo "$output_file" | grep -q "\.bz2$" && compr="bzip2 -9 -f"
+ echo "$output_file" | grep -q "\.lzma$" && compr="lzma -9 -f"
+ echo "$output_file" | grep -q "\.cpio$" && compr="cat"
shift
;;
esac
@@ -274,7 +279,7 @@ while [ $# -gt 0 ]; do
esac
done

-# If output_file is set we will generate cpio archive and gzip it
+# If output_file is set we will generate cpio archive and compress it
# we are carefull to delete tmp files
if [ ! -z ${output_file} ]; then
if [ -z ${cpio_file} ]; then
@@ -287,7 +292,7 @@ if [ ! -z ${output_file} ]; then
if [ "${is_cpio_compressed}" = "compressed" ]; then
cat ${cpio_tfile} > ${output_file}
else
- cat ${cpio_tfile} | gzip -f -9 - > ${output_file}
+ cat ${cpio_tfile} | ${compr} - > ${output_file}
fi
[ -z ${cpio_file} ] && rm ${cpio_tfile}
fi
diff -purN x86.ref/usr/initramfs_data.bz2.S x86-builtin-initramfs/usr/initramfs_data.bz2.S
--- x86.ref/usr/initramfs_data.bz2.S 1970-01-01 01:00:00.000000000 +0100
+++ x86-builtin-initramfs/usr/initramfs_data.bz2.S 2009-01-06 20:54:00.000000000 +0100
@@ -0,0 +1,30 @@
+/*
+ initramfs_data includes the compressed binary that is the
+ filesystem used for early user space.
+ Note: Older versions of "as" (prior to binutils 2.11.90.0.23
+ released on 2001-07-14) dit not support .incbin.
+ If you are forced to use older binutils than that then the
+ following trick can be applied to create the resulting binary:
+
+
+ ld -m elf_i386 --format binary --oformat elf32-i386 -r \
+ -T initramfs_data.scr initramfs_data.cpio.gz -o initramfs_data.o
+ ld -m elf_i386 -r -o built-in.o initramfs_data.o
+
+ initramfs_data.scr looks like this:
+SECTIONS
+{
+ .init.ramfs : { *(.data) }
+}
+
+ The above example is for i386 - the parameters vary from architectures.
+ Eventually look up LDFLAGS_BLOB in an older version of the
+ arch/$(ARCH)/Makefile to see the flags used before .incbin was introduced.
+
+ Using .incbin has the advantage over ld that the correct flags are set
+ in the ELF header, as required by certain architectures.
+*/
+
+.section .init.ramfs,"a"
+.incbin "usr/initramfs_data.cpio.bz2"
+
diff -purN x86.ref/usr/initramfs_data.gz.S x86-builtin-initramfs/usr/initramfs_data.gz.S
--- x86.ref/usr/initramfs_data.gz.S 1970-01-01 01:00:00.000000000 +0100
+++ x86-builtin-initramfs/usr/initramfs_data.gz.S 2009-01-05 01:21:06.000000000 +0100
@@ -0,0 +1,30 @@
+/*
+ initramfs_data includes the compressed binary that is the
+ filesystem used for early user space.
+ Note: Older versions of "as" (prior to binutils 2.11.90.0.23
+ released on 2001-07-14) dit not support .incbin.
+ If you are forced to use older binutils than that then the
+ following trick can be applied to create the resulting binary:
+
+
+ ld -m elf_i386 --format binary --oformat elf32-i386 -r \
+ -T initramfs_data.scr initramfs_data.cpio.gz -o initramfs_data.o
+ ld -m elf_i386 -r -o built-in.o initramfs_data.o
+
+ initramfs_data.scr looks like this:
+SECTIONS
+{
+ .init.ramfs : { *(.data) }
+}
+
+ The above example is for i386 - the parameters vary from architectures.
+ Eventually look up LDFLAGS_BLOB in an older version of the
+ arch/$(ARCH)/Makefile to see the flags used before .incbin was introduced.
+
+ Using .incbin has the advantage over ld that the correct flags are set
+ in the ELF header, as required by certain architectures.
+*/
+
+.section .init.ramfs,"a"
+.incbin "usr/initramfs_data.cpio.gz"
+
diff -purN x86.ref/usr/initramfs_data.lzma.S x86-builtin-initramfs/usr/initramfs_data.lzma.S
--- x86.ref/usr/initramfs_data.lzma.S 1970-01-01 01:00:00.000000000 +0100
+++ x86-builtin-initramfs/usr/initramfs_data.lzma.S 2009-01-06 20:54:13.000000000 +0100
@@ -0,0 +1,30 @@
+/*
+ initramfs_data includes the compressed binary that is the
+ filesystem used for early user space.
+ Note: Older versions of "as" (prior to binutils 2.11.90.0.23
+ released on 2001-07-14) dit not support .incbin.
+ If you are forced to use older binutils than that then the
+ following trick can be applied to create the resulting binary:
+
+
+ ld -m elf_i386 --format binary --oformat elf32-i386 -r \
+ -T initramfs_data.scr initramfs_data.cpio.gz -o initramfs_data.o
+ ld -m elf_i386 -r -o built-in.o initramfs_data.o
+
+ initramfs_data.scr looks like this:
+SECTIONS
+{
+ .init.ramfs : { *(.data) }
+}
+
+ The above example is for i386 - the parameters vary from architectures.
+ Eventually look up LDFLAGS_BLOB in an older version of the
+ arch/$(ARCH)/Makefile to see the flags used before .incbin was introduced.
+
+ Using .incbin has the advantage over ld that the correct flags are set
+ in the ELF header, as required by certain architectures.
+*/
+
+.section .init.ramfs,"a"
+.incbin "usr/initramfs_data.cpio.lzma"
+
diff -purN x86.ref/usr/initramfs_data.S x86-builtin-initramfs/usr/initramfs_data.S
--- x86.ref/usr/initramfs_data.S 2009-01-05 01:21:06.000000000 +0100
+++ x86-builtin-initramfs/usr/initramfs_data.S 2009-01-06 20:54:18.000000000 +0100
@@ -26,5 +26,5 @@ SECTIONS
*/

.section .init.ramfs,"a"
-.incbin "usr/initramfs_data.cpio.gz"
+.incbin "usr/initramfs_data.cpio"

diff -purN x86.ref/usr/Makefile x86-builtin-initramfs/usr/Makefile
--- x86.ref/usr/Makefile 2009-01-05 01:21:06.000000000 +0100
+++ x86-builtin-initramfs/usr/Makefile 2009-01-06 22:23:48.000000000 +0100
@@ -5,14 +5,32 @@
klibcdirs:;
PHONY += klibcdirs

+# Find out "preferred" ramdisk compressor. Order of preference is
+# 1. bzip2 efficient, and likely to be present
+# 2. gzip former default
+# 3. lzma
+# 4. none
+
+# None of the above
+suffix_y =
+
+# Lzma, but no gzip nor bzip2
+suffix_$(CONFIG_RD_LZMA) = .lzma
+
+# Gzip, but no bzip2
+suffix_$(CONFIG_RD_GZIP) = .gz
+
+# Bzip2
+suffix_$(CONFIG_RD_BZIP2) = .bz2
+

# Generate builtin.o based on initramfs_data.o
-obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data.o
+obj-$(CONFIG_BLK_DEV_INITRD) := initramfs_data$(suffix_y).o

-# initramfs_data.o contains the initramfs_data.cpio.gz image.
+# initramfs_data.o contains the compressed initramfs_data.cpio image.
# The image is included using .incbin, a dependency which is not
# tracked automatically.
-$(obj)/initramfs_data.o: $(obj)/initramfs_data.cpio.gz FORCE
+$(obj)/initramfs_data$(suffix_y).o: $(obj)/initramfs_data.cpio$(suffix_y) FORCE

#####
# Generate the initramfs cpio archive
@@ -25,28 +43,28 @@ ramfs-args := \
$(if $(CONFIG_INITRAMFS_ROOT_UID), -u $(CONFIG_INITRAMFS_ROOT_UID)) \
$(if $(CONFIG_INITRAMFS_ROOT_GID), -g $(CONFIG_INITRAMFS_ROOT_GID))

-# .initramfs_data.cpio.gz.d is used to identify all files included
+# .initramfs_data.cpio.d is used to identify all files included
# in initramfs and to detect if any files are added/removed.
# Removed files are identified by directory timestamp being updated
# The dependency list is generated by gen_initramfs.sh -l
-ifneq ($(wildcard $(obj)/.initramfs_data.cpio.gz.d),)
- include $(obj)/.initramfs_data.cpio.gz.d
+ifneq ($(wildcard $(obj)/.initramfs_data.cpio.d),)
+ include $(obj)/.initramfs_data.cpio.d
endif

quiet_cmd_initfs = GEN $@
cmd_initfs = $(initramfs) -o $@ $(ramfs-args) $(ramfs-input)

-targets := initramfs_data.cpio.gz
+targets := initramfs_data.cpio.gz initramfs_data.cpio.bz2 initramfs_data.cpio.lzma initramfs_data.cpio
# do not try to update files included in initramfs
$(deps_initramfs): ;

$(deps_initramfs): klibcdirs
-# We rebuild initramfs_data.cpio.gz if:
-# 1) Any included file is newer then initramfs_data.cpio.gz
+# We rebuild initramfs_data.cpio if:
+# 1) Any included file is newer then initramfs_data.cpio
# 2) There are changes in which files are included (added or deleted)
-# 3) If gen_init_cpio are newer than initramfs_data.cpio.gz
+# 3) If gen_init_cpio are newer than initramfs_data.cpio
# 4) arguments to gen_initramfs.sh changes
-$(obj)/initramfs_data.cpio.gz: $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs
- $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/.initramfs_data.cpio.gz.d
+$(obj)/initramfs_data.cpio$(suffix_y): $(obj)/gen_init_cpio $(deps_initramfs) klibcdirs
+ $(Q)$(initramfs) -l $(ramfs-input) > $(obj)/.initramfs_data.cpio.d
$(call if_changed,initfs)

Relevant Pages

  • Re: speed of bzip2 versus gzip
    ... i never measured it to see if it is 50-60 times slower, but yes, gzip ... I wanted to use bzip2 to compress ... then your goal is to obtain the absolute best compression, ... more reasonable if you care about reasonably timely compression. ...
    (freebsd-questions)
  • Re: speed of bzip2 versus gzip
    ... i never measured it to see if it is 50-60 times slower, but yes, gzip ... I wanted to use bzip2 to compress ... then your goal is to obtain the absolute best compression, ... sys 0m0.257s ...
    (freebsd-questions)
  • Re: Linux Kernel Source Compression
    ... gzip is the winner again. ... usually two to six times more time than lzma, that is, four to twelve ... decompress the faster the smaller the compressed size is, while bzip2 ... gets slower when the compression ratio gets better. ...
    (Linux-Kernel)
  • Re: emulate an end-of-media
    ... the compression ratio on a dump of my home directory ... bzip2 took about 74min to compress, gzip only 11minutes. ...
    (freebsd-hackers)
  • www.CeBeans.com - new program listings - Sep 21 2009
    ... Zipped file with maximum compression. ... The GZip compression/decompression is from ... when a new picture is detected in the folder the program GZips the JPEG ... Free man every round after five. ...
    (microsoft.public.pocketpc)