On Thu, Mar 20, 2025 at 01:20:50AM -0700, Jon Pan-Doh wrote: > Proposal > ======== > > When using native AER, spammy devices can flood kernel logs with AER errors > and slow/stall execution. Add per-device per-error-severity ratelimits > for more robust error logging. Allow userspace to configure ratelimits > via sysfs knobs. > > Motivation > ========== > > Several OCP members have issues with inconsistent PCIe error handling, > exacerbated at datacenter scale (myriad of devices). > OCP HW/Fault Management subproject set out to solve this by > standardizing industry: > > - PCIe error handling best practices > - Fault Management/RAS (incl. PCIe errors) > > Exposing PCIe errors/debug info in-band for a userspace daemon (e.g. > rasdaemon) to collect/pass on to repairability services is part of the > roadmap. For the series: Reported-by: Sargun Dhillon <sargun@xxxxxxxx> Acked-by: Paul E. McKenney <paulmck@xxxxxxxxxx> And thank you! I did throw together a quick hack to get things going internally on our fleet, but this series is way better. Thanx, Paul > Background > ========== > > AER error spam has been observed many times, both publicly (e.g. [1], [2], > [3]) and privately. While it usually occurs with correctable errors, it can > happen with uncorrectable errors (e.g. during new HW bringup). > > There have been previous attempts to add ratelimits to AER logs ([4], > [5]). The most recent attempt[5] has many similarities with the proposed > approach. > > Patch organization > ================== > 1-4 AER logging cleanup > 5-7 Ratelimits and sysfs knobs > > Outstanding work > ================ > Cleanup: > - Consolidate aer_print_error() and pci_print_error() path > > Roadmap: > - IRQ ratelimiting > > v4: > - Fix bug where trace not emitted with malformed aer_err_info > - Extend ratelimit to malformed aer_err_info > - Update commit messages with patch motivation > - Squash AER sysfs filename change (Patch 8) > v3: > - Ratelimit aer_print_port_info() (drop Patch 1) > - Add ratelimit enable toggle > - Move trace outside of ratelimit > - Split log level (Patch 2) into two > - More descriptive documentation/sysfs naming > v2: > - Rebased on top of pci/aer (6.14.rc-1) > - Split series into log and IRQ ratelimits (defer patch 5) > - Dropped patch 8 (Move AER sysfs) > - Added log level cleanup patch[6] from Karolina's series > - Fixed bug where dpc errors didn't increment counters > - "X callbacks suppressed" message on ratelimit release -> immediately > - Separate documentation into own patch > > [1] https://bugzilla.kernel.org/show_bug.cgi?id=215027 > [2] https://bugzilla.kernel.org/show_bug.cgi?id=201517 > [3] https://bugzilla.kernel.org/show_bug.cgi?id=196183 > [4] https://lore.kernel.org/linux-pci/20230606035442.2886343-2-grundler@xxxxxxxxxxxx/ > [5] https://lore.kernel.org/linux-pci/cover.1736341506.git.karolina.stolarek@xxxxxxxxxx/ > [6] https://lore.kernel.org/linux-pci/edd77011aafad4c0654358a26b4e538d0c5a321d.1736341506.git.karolina.stolarek@xxxxxxxxxx/ > > Jon Pan-Doh (5): > PCI/AER: Move AER stat collection out of __aer_print_error() > PCI/AER: Rename struct aer_stats to aer_report > PCI/AER: Introduce ratelimit for error logs > PCI/AER: Add ratelimits to PCI AER Documentation > PCI/AER: Add sysfs attributes for log ratelimits > > Karolina Stolarek (2): > PCI/AER: Check log level once and propagate down > PCI/AER: Make all pci_print_aer() log levels depend on error type > > ...es-aer_stats => sysfs-bus-pci-devices-aer} | 34 +++ > Documentation/PCI/pcieaer-howto.rst | 16 +- > drivers/pci/pci-sysfs.c | 1 + > drivers/pci/pci.h | 4 +- > drivers/pci/pcie/aer.c | 271 +++++++++++++----- > drivers/pci/pcie/dpc.c | 3 +- > include/linux/pci.h | 2 +- > 7 files changed, 260 insertions(+), 71 deletions(-) > rename Documentation/ABI/testing/{sysfs-bus-pci-devices-aer_stats => sysfs-bus-pci-devices-aer} (77%) > > -- > 2.49.0.rc1.451.g8f38331e32-goog >
