From: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
Hi,
I've finally managed to act on some idea I shared with a few
folks while in Montreal, namely using unmodified pahole to generate BTF
for each .o right after it is produced, i.e. with this patch:
acme@number:~/git/linux$ git diff
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 1d581ba5df66f2b5..ad9e788910636715 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -240,7 +240,7 @@ cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -
endif
quiet_cmd_cc_o_c = CC $(quiet_modtag) $@
- cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
+ cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< && ${PAHOLE} --btf_encode ${PAHOLE_FLAGS} $@ \
$(cmd_ld_single) \
$(cmd_objtool)
acme@number:~/git/linux$
A kernel built with this ends up with a vmlinux with a .BTF section that
has all the .o .BTF sections concatenated.
This (the series of .BTF concatenated by the unmodified linker) somehow
survives the pre-existing pahole call to generate BTF from DWARF and we
end up with this "BTF archive".
With the minimal set of changes in this series:
tools/lib/bpf/btf.c | 91 ++++++++++++++++++++++++++++++++++++++-------
tools/lib/bpf/btf.h | 3 ++
2 files changed, 81 insertions(+), 13 deletions(-)
With the first patch being just a trivial error handling simplification,
we end up being able to get the same vmlinux.h result from bpftool built
with this libbpf:
acme@number:~/git/bpf-next$ tools/bpf/bpftool/bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c >
+from_archive_combined+dedup_in_libbpf
acme@number:~/git/bpf-next$ tools/bpf/bpftool/bpftool btf dump file ../build/v6.16.0+/vmlinux format c >
+from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
acme@number:~/git/bpf-next$ diff -u from_archive_combined+dedup_in_libbpf from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf | head
acme@number:~/git/bpf-next$ wc -l from_archive_combined+dedup_in_libbpf from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
161588 from_archive_combined+dedup_in_libbpf
161588 from_unmodified_pahole_DWARF2BTF+dedup_in_libbpf
323176 total
acme@number:~/git/bpf-next$
If we use completely unmodified libbpf, bpftool, etc, the "BTF archive"
in the resulting vmlinux .BTF ELF section is still consumable, but just
the first "CU" (the first .o .BTF ELF section) is visible, the one for
init/main.o:
acme@number:~/git/linux$ bpftool version
bpftool v7.5.0
using libbpf v1.5
features: llvm, skeletons
acme@number:~/git/linux$
acme@number:~/git/bpf-next$ bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c | wc -l
11361
acme@number:~/git/linux$ bpftool btf dump file ../build/v6.16.0+/init/main.o format c | wc -l
11361
acme@number:~/git/linux$
Furthermore:
acme@number:~/git/linux$ bpftool btf dump file ../build/v6.16.0+/init/main.o format c > a
acme@number:~/git/linux$ bpftool btf dump file ~/vmlinux-v6.16.0+.btf_archive format c > b
acme@number:~/git/linux$
Each patch has extra explanations of the process.
This is complementary to today's series from Alan Maguire, as we can use
the one liner for the kernel build process to test his series without
requiring installing a toolchain that generates BTF for each .o file
that will result in vmlinux.
Next steps on my side are to:
1. change pahole for when it receives --format_path=btf check if
btf__is_archive(btf) is true, then just replace the current vmlinux .BTF
contents with the raw data in this just loaded BTF, short circuiting
the whole process.
2. the kernel build process should be changed to allow one to ask for
just BTF, not DWARF, and if so, using the above method, strip the DWARF
info after using it to generate BTF.
Then when compilers are producing BTF, we switch to that, falling back
to the above method when a compiler is known to generate buggy BTF.
And also to use in CIs, to compare the output generated by the various
methods in the various components.
3. In 2 we can even use the same scheme we use for parallelizing DWARF
loading when loading all the BTF archive members concatenated in vmlinux
to dedup them.
BTW, this is the size of a vmlinux ELF .BTF section with an BTF archive:
acme@number:~/git/linux$ readelf -SW ../build/v6.16.0+/vmlinux | grep BTF
[15] .BTF PROGBITS ffffffff82ff2000 21f2000 16db5976 00 A 0 0 1
[16] .BTF_ids PROGBITS ffffffff99da8000 18fa8000 001238 00 A 0 0 1
acme@number:~/git/linux$
~365 MiB
While the DWARF for that file is at:
[44] .debug_aranges PROGBITS 0000000000000000 1aa00000 03bfb0 00 0 0 16
[45] .debug_info PROGBITS 0000000000000000 1aa3bfb0 1154b512 00 0 0 1
[46] .debug_abbrev PROGBITS 0000000000000000 2bf874c2 81492f 00 0 0 1
[47] .debug_line PROGBITS 0000000000000000 2c79bdf1 1ec4abd 00 0 0 1
[48] .debug_frame PROGBITS 0000000000000000 2e6608b0 3fd470 00 0 0 8
[49] .debug_str PROGBITS 0000000000000000 2ea5dd20 59bbe8 01 MS 0 0 1
[50] .debug_line_str PROGBITS 0000000000000000 2eff9908 02de43 01 MS 0 0 1
[51] .debug_loclists PROGBITS 0000000000000000 2f02774b 2683cbb 00 0 0 1
[52] .debug_rnglists PROGBITS 0000000000000000 316ab406 4f875b 00 0 0 1
>>> 0x1154b512 + 0x81492f + 0x1ec4abd + 0x3fd470 + 0x59bbe8
341563734
~325 MiB
But then BTF, when dedup'ed gets down to:
acme@number:~/git/linux$ readelf -SW ../build/v6.16.0+.no-btf_archive/vmlinux | grep BTF
[15] .BTF PROGBITS ffffffff82fef000 21ef000 64fb32 00 A 0 0 1
[16] .BTF_ids PROGBITS ffffffff8363f000 283f000 001238 00 A 0 0 1
acme@number:~/git/linux$
~6.3 MiB
And also BTF has some info generated from other sources besides DWARF,
like kfuncs, per cpu, etc.
Also an observation: for distros the optimal way to produce BTF _and_
DWARF seems to be is the one we have now, don't bother generating .BTF
for all .o, just generate DWARF and at the end generate BTF from it 8-)
For developers not needing DWARF and not caring about reproducible
builds then there are other clever tricks to use like go on adding each
generated BTF using the technique in this patchset, i.e. using
btf__add_btf() and trowing away the just generated BTF to then at the end
do the btf__archive_dedup() (also introduced in this patchset) to have
the end result dropped to disk. But I'm getting carried away, sry.
There are many other details that need to be double checked but I think
the current status is good enough for experimentation.
Cheers,
- Arnaldo
Arnaldo Carvalho de Melo (4):
libbpf: Simplify error handling removing needless repeated err checks
libbpf: Check if there is extra data at the end of a BTF
libbpf: Add support for detecting and dedup'ing a BTF archive
libbpf: Check if an ELF .BTF section is an archive and combine/dedup
tools/lib/bpf/btf.c | 91 ++++++++++++++++++++++++++++++++++++++-------
tools/lib/bpf/btf.h | 3 ++
2 files changed, 81 insertions(+), 13 deletions(-)
--
2.50.1
