On 25/04/2025 18:58, Andrii Nakryiko wrote: > On Fri, Apr 25, 2025 at 10:50 AM Alan Maguire <alan.maguire@xxxxxxxxxx> wrote: >> >> On 25/04/2025 15:50, Alexei Starovoitov wrote: >>> Hi All, >>> >>> Looks like pahole fails to deduplicate BTF when kernel and >>> kernel module are built with gcc-14. >>> I see this issue with various kernel .config-s on bpf and >>> bpf-next trees. >>> I tried pahole 1.28 and the latest master. Same issues. >>> >>> BTF in bpf_testmod.ko built with gcc-14 has 2849 types. >>> When built with gcc-13 it has 454 types. >>> So something is confusing dedup logic. >>> Would be great if dedup experts can take a look, >>> since this dedup issue is breaking a lot of selftests/bpf. >>> >>> Also vmlinux.h generated out of the kernel compiled with gcc-13 >>> and out of the kernel compiled with gcc-14 shows these differences: >>> >>> --- vmlinux13.h 2025-04-24 21:33:50.556884372 -0700 >>> +++ vmlinux14.h 2025-04-24 21:39:10.310488992 -0700 >>> @@ -148815,7 +148815,6 @@ >>> extern int hid_bpf_input_report(struct hid_bpf_ctx *ctx, enum >>> hid_report_type type, u8 *buf, const size_t buf__sz) __weak __ksym; >>> extern void hid_bpf_release_context(struct hid_bpf_ctx *ctx) __weak __ksym; >>> extern int hid_bpf_try_input_report(struct hid_bpf_ctx *ctx, enum >>> hid_report_type type, u8 *buf, const size_t buf__sz) __weak __ksym; >>> -extern bool scx_bpf_consume(u64 dsq_id) __weak __ksym; >>> extern int scx_bpf_cpu_node(s32 cpu) __weak __ksym; >>> extern struct rq *scx_bpf_cpu_rq(s32 cpu) __weak __ksym; >>> extern u32 scx_bpf_cpuperf_cap(s32 cpu) __weak __ksym; >>> @@ -148825,12 +148824,8 @@ >>> extern void scx_bpf_destroy_dsq(u64 dsq_id) __weak __ksym; >>> extern void scx_bpf_dispatch(struct task_struct *p, u64 dsq_id, u64 >>> slice, u64 enq_flags) __weak __ksym; >>> extern void scx_bpf_dispatch_cancel(void) __weak __ksym; >>> -extern bool scx_bpf_dispatch_from_dsq(struct bpf_iter_scx_dsq >>> *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak >>> __ksym; >>> -extern void scx_bpf_dispatch_from_dsq_set_slice(struct >>> bpf_iter_scx_dsq *it__iter, u64 slice) __weak __ksym; >>> extern void scx_bpf_dispatch_from_dsq_set_vtime(struct >>> bpf_iter_scx_dsq *it__iter, u64 vtime) __weak __ksym; >>> extern u32 scx_bpf_dispatch_nr_slots(void) __weak __ksym; >>> -extern void scx_bpf_dispatch_vtime(struct task_struct *p, u64 dsq_id, >>> u64 slice, u64 vtime, u64 enq_flags) __weak __ksym; >>> -extern bool scx_bpf_dispatch_vtime_from_dsq(struct bpf_iter_scx_dsq >>> *it__iter, struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak >>> __ksym; >>> extern void scx_bpf_dsq_insert(struct task_struct *p, u64 dsq_id, u64 >>> slice, u64 enq_flags) __weak __ksym; >>> extern void scx_bpf_dsq_insert_vtime(struct task_struct *p, u64 >>> dsq_id, u64 slice, u64 vtime, u64 enq_flags) __weak __ksym; >>> extern bool scx_bpf_dsq_move(struct bpf_iter_scx_dsq *it__iter, >>> struct task_struct *p, u64 dsq_id, u64 enq_flags) __weak __ksym; >>> >>> gcc-14's kernel is clearly wrong. >>> These 5 kfuncs still exist in the kernel. >>> I manually checked there is no if __GNUC__ > 13 in the code. >>> Also: >>> nm bld/vmlinux|grep -w scx_bpf_consume >>> ffffffff8159d4b0 T scx_bpf_consume >>> ffffffff8120ea81 t scx_bpf_consume.cold >>> >>> I suspect the second issue is not related to the dedup problem. >>> All 5 missing kfuncs have ".cold" optimized bodies. >>> But ".cold" maybe a red herring, since >>> nm bld/vmlinux|grep -w scx_bpf_dispatch >>> ffffffff8159d020 T scx_bpf_dispatch >>> ffffffff8120ea0f t scx_bpf_dispatch.cold >>> but this kfunc is present in vmlinux14.h >>> >>> If it makes a difference I have these configs: >>> # CONFIG_DEBUG_INFO_DWARF4 is not set >>> # CONFIG_DEBUG_INFO_DWARF5 is not set >>> # CONFIG_DEBUG_INFO_REDUCED is not set >>> CONFIG_DEBUG_INFO_COMPRESSED_NONE=y >>> # CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set >>> # CONFIG_DEBUG_INFO_SPLIT is not set >>> CONFIG_DEBUG_INFO_BTF=y >>> CONFIG_PAHOLE_HAS_SPLIT_BTF=y >>> CONFIG_DEBUG_INFO_BTF_MODULES=y >> >> thanks for the report! I've just reproduced this now with gcc 14; my >> initial theory was it might be DWARF5-related, but dedup issues occur >> for modules with CONFIG_DEBUG_INFO_DWARF4=y also. I'm seeing task_struct >> duplicates in module BTF among other things, so I will try and dig >> further and report back when I find something. Like you I suspect the > > This is a bizarre case. I have a custom small tool that recursively > traverses two parallel subgraphs of BTF types and prints anything that > differs between them ([0]). (I had to disable distilled BTF to make > use of this, the issue is present both with distilled BTF and > without). > > I see that struct sock both in vmlinux and bpf_testmod.ko are > *IDENTICAL*. There is no difference I could detect. So very weird. I'm > thinking of bisecting, as this didn't happen before with exactly the > same compiler and pahole, so this must be a kernel-side change. > > [0] https://github.com/anakryiko/libbpf-bootstrap/tree/btfdiff-hack > thanks for the pointer to this! My initial suspicion was that we had some sort of dups of slightly-differently-defined primitive types that bubbled up through multiple structs in the module case since the level of duplication is so high; a colleague ran across something like this recently and indeed if I dump vmlinux BTF in C format I see: typedef unsigned char u8___2; ...along with the original u8 definition: typedef unsigned char __u8; typedef __u8 u8; However on checking I didn't find any references to the "wrong" u8, so I don't think it is the cause (the definition comes from crypto/jitterentropy.c so as a .c redefinition it's less likely to cause chaos across multiple CUs). Perhaps we should be thinking of cases where "#ifdef MODULE" leads to different structure content, maybe something changed that results in that leaking into core kernel structures like task_struct. Haven't had any luck finding a common culprit across duplicated structures yet.. Alan
