On 26/04/2025 18:28, Alan Maguire wrote: > On 25/04/2025 21:41, Andrii Nakryiko wrote: >> On Fri, Apr 25, 2025 at 1:36 PM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: >>> >>> On 25/04/2025 18:58, Andrii Nakryiko wrote: >>>> On Fri, Apr 25, 2025 at 10:50 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: >>>>> >>>>> On 25/04/2025 15:50, Alexei Starovoitov wrote: >>>>>> Hi All, >>>>>> >>>>>> Looks like pahole fails to deduplicate BTF when kernel and >>>>>> kernel module are built with gcc-14. >>>>>> I see this issue with various kernel .config-s on bpf and >>>>>> bpf-next trees. >>>>>> I tried pahole 1.28 and the latest master. Same issues. >>>>>> >>>>>> BTF in bpf_testmod.ko built with gcc-14 has 2849 types. >>>>>> When built with gcc-13 it has 454 types. >>>>>> So something is confusing dedup logic. >>>>>> Would be great if dedup experts can take a look, >>>>>> since this dedup issue is breaking a lot of selftests/bpf. >>>>>> >>>>>> Also vmlinux.h generated out of the kernel compiled with gcc-13 >>>>>> and out of the kernel compiled with gcc-14 shows these differences: >>>>>> >>>>>> --- vmlinux13.h 2025-04-24 21:33:50.556884372 -0700 >>>>>> +++ vmlinux14.h 2025-04-24 21:39:10.310488992 -0700 >>>>>> @@ -148815,7 +148815,6 @@ >>>>>> extern int hid_bpf_input_report(struct hid_bpf_ctx *ctx, enum >>>>>> hid_report_type type, u8 *buf, const size_t buf__sz) __weak __ksym; >>>>>> extern void hid_bpf_release_context(struct hid_bpf_ctx *ctx) __weak __ksym; >>>>>> extern int hid_bpf_try_input_report(struct hid_bpf_ctx *ctx, enum >>>>>> hid_report_type type, u8 *buf, const size_t buf__sz) __weak __ksym; >>>>>> -extern bool scx_bpf_consume(u64 dsq_id) __weak __ksym; >>>>>> extern int scx_bpf_cpu_node(s32 cpu) __weak __ksym; >>>>>> extern struct rq *scx_bpf_cpu_rq(s32 cpu) __weak __ksym; >>>>>> extern u32 scx_bpf_cpuperf_cap(s32 cpu) __weak __ksym; >>>>>> @@ -148825,12 +148824,8 @@ >>>>>> extern void scx_bpf_destroy_dsq(u64 dsq_id) __weak __ksym; >>>>>> extern void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 >>>>>> slice, u64 enq_flags) __weak __ksym; >>>>>> extern void scx_bpf_dispatch_cancel(void) __weak __ksym; >>>>>> -extern bool scx_bpf_dispatch_from_dsq(struct bpf_iter_scx_dsq >>>>>> *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak >>>>>> __ksym; >>>>>> -extern void scx_bpf_dispatch_from_dsq_set_slice(struct >>>>>> bpf_iter_scx_dsq *it__iter, u64 slice) __weak __ksym; >>>>>> extern void scx_bpf_dispatch_from_dsq_set_vtime(struct >>>>>> bpf_iter_scx_dsq *it__iter, u64 vtime) __weak __ksym; >>>>>> extern u32 scx_bpf_dispatch_nr_slots(void) __weak __ksym; >>>>>> -extern void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, >>>>>> u64 slice, u64 vtime, u64 enq_flags) __weak __ksym; >>>>>> -extern bool scx_bpf_dispatch_vtime_from_dsq(struct bpf_iter_scx_dsq >>>>>> *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak >>>>>> __ksym; >>>>>> extern void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 >>>>>> slice, u64 enq_flags) __weak __ksym; >>>>>> extern void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 >>>>>> dsq_id, u64 slice, u64 vtime, u64 enq_flags) __weak __ksym; >>>>>> extern bool scx_bpf_dsq_move(struct bpf_iter_scx_dsq *it__iter, >>>>>> struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak __ksym; >>>>>> >>>>>> gcc-14's kernel is clearly wrong. >>>>>> These 5 kfuncs still exist in the kernel. >>>>>> I manually checked there is no if __GNUC__ > 13 in the code. >>>>>> Also: >>>>>> nm bld/vmlinux|grep -w scx_bpf_consume >>>>>> ffffffff8159d4b0 T scx_bpf_consume >>>>>> ffffffff8120ea81 t scx_bpf_consume.cold >>>>>> >>>>>> I suspect the second issue is not related to the dedup problem. >>>>>> All 5 missing kfuncs have ".cold" optimized bodies. >>>>>> But ".cold" maybe a red herring, since >>>>>> nm bld/vmlinux|grep -w scx_bpf_dispatch >>>>>> ffffffff8159d020 T scx_bpf_dispatch >>>>>> ffffffff8120ea0f t scx_bpf_dispatch.cold >>>>>> but this kfunc is present in vmlinux14.h >>>>>> >>>>>> If it makes a difference I have these configs: >>>>>> # CONFIG_DEBUG_INFO_DWARF4 is not set >>>>>> # CONFIG_DEBUG_INFO_DWARF5 is not set >>>>>> # CONFIG_DEBUG_INFO_REDUCED is not set >>>>>> CONFIG_DEBUG_INFO_COMPRESSED_NONE=y >>>>>> # CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set >>>>>> # CONFIG_DEBUG_INFO_SPLIT is not set >>>>>> CONFIG_DEBUG_INFO_BTF=y >>>>>> CONFIG_PAHOLE_HAS_SPLIT_BTF=y >>>>>> CONFIG_DEBUG_INFO_BTF_MODULES=y >>>>> >>>>> thanks for the report! I've just reproduced this now with gcc 14; my >>>>> initial theory was it might be DWARF5-related, but dedup issues occur >>>>> for modules with CONFIG_DEBUG_INFO_DWARF4=y also. I'm seeing task_struct >>>>> duplicates in module BTF among other things, so I will try and dig >>>>> further and report back when I find something. Like you I suspect the >>>> >>>> This is a bizarre case. I have a custom small tool that recursively >>>> traverses two parallel subgraphs of BTF types and prints anything that >>>> differs between them ([0]). (I had to disable distilled BTF to make >>>> use of this, the issue is present both with distilled BTF and >>>> without). >>>> >>>> I see that struct sock both in vmlinux and bpf_testmod.ko are >>>> *IDENTICAL*. There is no difference I could detect. So very weird. I'm >>>> thinking of bisecting, as this didn't happen before with exactly the >>>> same compiler and pahole, so this must be a kernel-side change. >>>> >>>> [0] https://github.com/anakryiko/libbpf-bootstrap/tree/btfdiff-hack >>>> >>> >>> thanks for the pointer to this! My initial suspicion was that we had >>> some sort of dups of slightly-differently-defined primitive types that >>> bubbled up through multiple structs in the module case since the level >>> of duplication is so high; a colleague ran across something like this >>> recently and indeed if I dump vmlinux BTF in C format I see: >>> >>> typedef unsigned char u8___2; >>> >>> ...along with the original u8 definition: >>> >>> typedef unsigned char __u8; >>> typedef __u8 u8; >> >> Are you sure you are not dumping distilled BTF? >> > > nope, that's in vmlinux BTF, originating from crypto/jitterentropy.c > >> This is the commit introducing a regression: >> >> eb0ece16027f ("Merge tag 'mm-stable-2025-03-30-16-52' of >> git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm") >> >> Yes, it's a "merge commit", but there is a lot of code introduced in >> it. Among it: >> >> + /* >> + * Use __typeof_unqual__() when available. >> + * >> + * XXX: Remove test for __CHECKER__ once >> + * sparse learns about __typeof_unqual__(). >> + */ >> + #if CC_HAS_TYPEOF_UNQUAL && !defined(__CHECKER__) >> + # define USE_TYPEOF_UNQUAL 1 >> + #endif >> + >> + /* >> + * Define TYPEOF_UNQUAL() to use __typeof_unqual__() as typeof >> + * operator when available, to return an unqualified type of the exp. >> + */ >> + #if defined(USE_TYPEOF_UNQUAL) >> + # define TYPEOF_UNQUAL(exp) __typeof_unqual__(exp) >> + #else >> + # define TYPEOF_UNQUAL(exp) __typeof__(exp) >> + #endif >> + >> >> >> And that's exactly what causes this divergence. Commenting out that >> USE_TYPEOF_UNQUAL #define fixes issues. >> >> As to why that causes a problem. I suspect __typeof_unqual__() changes >> how GCC generates DWARF information within any given compilation unit >> (CU). Libbpf's BTF dedup relies on a property that compiler won't have >> duplicate definitions of exactly the same type (i.e., DWARF itself >> can't have two `struct blah` definitions), without which it's not >> possible to deduplicate entire clusters of self-referencing BTF types. >> It seems like typeof_unqual breaks this somehow. >> >> We need to compare DWARF with and without TYPEOF_UNQUAL and see what >> the differences are and how we can prevent or accommodate them. > > Great find! As you suspect the handling in BTF btf_dedup_is_equiv() > covers two cases handling multiple instances of objects in DWARF; > duplicate arrays and duplicate structs. In this case however we are for > some reason winding up with multiple copies of structures containing > duplicate pointers in DWARF which have different type ids, which however > both point at the same target type. Adding the following to BTF dedup > accordingly solves the cascade of dedup issues for me: > > diff --git a/src/btf.c b/src/btf.c > index eea99c7..2155dd9 100644 > --- a/src/btf.c > +++ b/src/btf.c > @@ -4379,6 +4379,18 @@ static bool btf_dedup_identical_structs(struct > btf_dedup *d, __u32 id1, __u32 id > return true; > } > > +static bool btf_dedup_identical_ptrs(struct btf_dedup *d, __u32 id1, > __u32 id2) > +{ > + struct btf_type *t1, *t2; > + > + t1 = btf_type_by_id(d->btf, id1); > + t2 = btf_type_by_id(d->btf, id2); > + > + if (btf_kind(t1) != BTF_KIND_PTR || btf_kind(t1) != btf_kind(t2)) > + return false; > + return t1->type == t2->type; > +} > + > /* > * Check equivalence of BTF type graph formed by candidate struct/union > (we'll > * call it "candidate graph" in this description for brevity) to a type > graph > @@ -4511,6 +4523,8 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, > __u32 cand_id, > */ > if (btf_dedup_identical_structs(d, hypot_type_id, cand_id)) > return 1; > + if (btf_dedup_identical_ptrs(d, hypot_type_id, cand_id)) > + return 1; > return 0; > } > > > Now why this new behaviour results from the inclusion of typeof_unqual() > is something I can't explain yet. Requires more digging to understand > exactly what's going on.. > The connection here seems to be around __percpu - annotations; so for example bpf_prog has the "int __percpu *active" field, and with the typeof_unqual() change, duplicate pointer declarations in BTF lead to the dedup issues. As far as DWARF representations go, we can compare good vs bad below. In the good case (no dedup issues) the bpf_prog active member looks like this: <2><155b7b>: Abbrev Number: 4 (DW_TAG_member) <155b7c> DW_AT_name : (indirect string, offset: 0xeaa2c): active <155b80> DW_AT_decl_file : 137 <155b81> DW_AT_decl_line : 1654 <155b83> DW_AT_decl_column : 17 <155b84> DW_AT_type : <0x14decc> <155b88> DW_AT_data_member_location: 40 ...i.e. it is a pointer: <1><14decc>: Abbrev Number: 3 (DW_TAG_pointer_type) <14decd> DW_AT_byte_size : 8 <14dece> DW_AT_type : <0x147ddd> ...which points at an int: <1><147ddd>: Abbrev Number: 216 (DW_TAG_base_type) <147ddf> DW_AT_byte_size : 4 <147de0> DW_AT_encoding : 5 (signed) <147de1> DW_AT_name : int In the bad case, the bpf_prog active member: <2><3d594>: Abbrev Number: 4 (DW_TAG_member) <3d595> DW_AT_name : (indirect string, offset: 0x3b976): active <3d599> DW_AT_decl_file : 125 <3d59a> DW_AT_decl_line : 1654 <3d59c> DW_AT_decl_column : 17 <3d59d> DW_AT_type : <0x4bd05> ...is a pointer: <1><4bd05>: Abbrev Number: 58 (DW_TAG_pointer_type) <4bd06> DW_AT_byte_size : 8 <4bd07> DW_AT_address_class: 2 <4bd08> DW_AT_type : <0x301cd> ...which points at an int <1><301cd>: Abbrev Number: 214 (DW_TAG_base_type) <301cf> DW_AT_byte_size : 4 <301d0> DW_AT_encoding : 5 (signed) <301d1> DW_AT_name : int <301d5> DW_AT_name : int ...but note the the DW_AT_address_class attribute in the latter case and the two DW_AT_name values. We don't use that address attribute in pahole as far as I can see, but it might be enough to cause problems. Alan
