在 2025/7/15 23:09, Borislav Petkov 写道:
On Tue, Jul 15, 2025 at 09:46:03PM +0800, Shuai Xue wrote:
For the purpose of counting, how about using the cmdline of rasdaemon?
That would mean you have to run rasdaemon on those machines before they
explode and then carve out the rasdaemon db from the coredump (this is
post-mortem analysis).
Rasdaemon is a userspace tool that will collect all hardware error
events reported by the Linux Kernel from several sources (EDAC, MCE,
PCI, ...) into one common framework. And it has been a standard tools
in Alibaba. As far as I know, twitter also use Rasdaemon in its production.
I would love for rasdaemon to log over the network and then other tools can
query those centralized logs but that has its own challenges...
I also prefer collecting rasdaemon data in a centralized data center, as
this is more beneficial for using big data analytics to analyze and
predict errors. At the same time, the centralized side also uses
rasdaemon logs as one of the references for machine operations and
maintenance.
As for rasdaemon itself, it is just a single-node event collector and
database, although it does also print logs. In practice, we use SLS [1]
to collect rasdaemon text logs from individual nodes and parse them on
the central side.
Thanks.
Shuai
[1]https://www.alibabacloud.com/help/en/sls/getting-started
[2]https://blog.x.com/engineering/en_us/topics/infrastructure/2023/how-twitter-uses-rasdaemon-for-hardware-reliability
