GOAL: understand how ECC hardware works & how the event propogates to the operating system (e.g. via interrupts)

reads:

some notes from Andi Kleen:
https://www.halobates.de/

Linux kernel boot messages and seeing if your AMD system has ECC
https://utcc.utoronto.ca/~cks/space/blog/linux/AMDWithECCKernelMessages?showcomments

[PATCH EDACv2 00/12] Add a driver to report Firmware first errors (via GHES)
https://lore.kernel.org/all/cover.1361459782.git.mchehab@redhat.com/

Machine check handling on Linux
https://www.halobates.de/mce.pdf

Ongoing evolution of Linux x86 machine check handling https://halobates.de/mce-lc09-2.pdf

§ ECC hardware:

1. ECC memory controller
2. on-chip (internal) error-correction circuits on DRAM chips
3. EOS memory modules

MCA recovery (since Nehalem-EX CPU)

§ Intel CMCI (corrected machine check interupt)

CMCI via local APIC

INTEL SDM, 15.5 CORRECTED MACHINE CHECK ERROR INTERRUPT

MSRs

    MSR 0x00000179
    IA32_MCG_CAP[10] (BIT 10)

    MCG_CMCI_P (Corrected MC error counting/signaling extension present) flag
    Indicates (when set) that extended state and associated MSRs necessary to
    support the reporting of an interrupt on a corrected MC error event and/or count
    threshold of corrected MC errors, is present. When this bit is set, it does not
    imply this feature is supported across all banks. Software should check the
    availability of the necessary logic on a bank by bank basis when using this
    signaling capability (i.e., bit 30 settable in individual IA32_MCi_CTL2 register)

CORRECTED ERROR FLOW (s)

• mcelog (CMCI loged through /dev/mcelog) + decoding. Deprecated since 4.12

  CONFIG_X86_MCELOG_LEGACY=y
  # enable /dev/mcelog
  

• OR polling

CMCI support in kernel:

88ccbedd9ca85d1aca6a6f99df48dce87b7c02d4
x86, mce, cmci: add CMCI support

b276268631af3a1b0df871e10d19d492f0513d4b
x86, mce, cmci: factor out threshold interrupt handler (the irq handler)

https://wiki.osdev.org/APIC
https://linux-kernel.vger.kernel.narkive.com/pFTxqtbj/patch-0-9-x86-cmci-add-support-for-intel-cmci
https://lore.kernel.org/lkml/20230718210813.291190-1-tony.luck@intel.com/T/

§ OS and Software

x86 booting options


Please see Documentation/arch/x86/x86_64/machinecheck.rst for sysfs runtime tunables.

mce=off
    Disable machine check
mce=no_cmci
    Disable CMCI(Corrected Machine Check Interrupt) that
    Intel processor supports.  Usually this disablement is
    not recommended, but it might be handy if your hardware
    is misbehaving.
    Note that you'll get more problems without CMCI than with
    due to the shared banks, i.e. you might get duplicated
    error logs.
mce=dont_log_ce
    Don't make logs for corrected errors.  All events reported
    as corrected are silently cleared by OS.
    This option will be useful if you have no interest in any
    of corrected errors.
mce=ignore_ce
    Disable features for corrected errors, e.g. polling timer
    and CMCI.  All events reported as corrected are not cleared
    by OS and remained in its error banks.
    Usually this disablement is not recommended, however if
    there is an agent checking/clearing corrected errors
    (e.g. BIOS or hardware monitoring applications), conflicting
    with OS's error handling, and you cannot deactivate the agent,
    then this option will be a help.
mce=no_lmce
    Do not opt-in to Local MCE delivery. Use legacy method
    to broadcast MCEs.
mce=bootlog
    Enable logging of machine checks left over from booting.
    Disabled by default on AMD Fam10h and older because some BIOS
    leave bogus ones.
    If your BIOS doesn't do that it's a good idea to enable though
    to make sure you log even machine check events that result
    in a reboot. On Intel systems it is enabled by default.
mce=nobootlog
    Disable boot machine check logging.
mce=monarchtimeout (number)
    monarchtimeout:
    Sets the time in us to wait for other CPUs on machine checks. 0
    to disable.
mce=bios_cmci_threshold
    Don't overwrite the bios-set CMCI threshold. This boot option
    prevents Linux from overwriting the CMCI threshold set by the
    bios. Without this option, Linux always sets the CMCI
    threshold to 1. Enabling this may make memory predictive failure
    analysis less effective if the bios sets thresholds for memory
    errors since we will not see details for all errors.
mce=recovery
    Force-enable recoverable machine check code paths

nomce (for compatibility with i386)
    same as mce=off

Everything else is in sysfs now.

§ Physical Format, Chips and Memory Rank

---

§ Reference

MCE (Machine Check Exception)

Hardware Specs

• Intel SDM Volume 3, Chapter 15 Machine-Check Architecture

APEI

EINJ

kernel

Misc

• Rowhammer, https://en.wikipedia.org/wiki/Row_hammer

---

§ Glossary

• MCE : Machine Check Exception
• CMCI : Corrected Machine-Check Interrupt
• SEC-DED : Single-Error-Correction, Double-Error-Detection. ECC hardware that is able to correct single bit flip and detect double big flips.

Physical formats:

• SIMM : Single In-line Memory Module
• DIMM : Dual In-line Memory Module
• SO-DIMM : Small Outline DIMM