Definition
Ratio of actual diagnostic accuracy to stated diagnostic confidence across confidence bands, measuring whether the fault-diagnosis system's certainty claims can be trusted for automated remediation decisions.
The Measurement Gap
Measurement Gap and Decision
What FDCI Measures: FDCI is the ratio of actual diagnostic accuracy to stated diagnostic confidence across confidence bands. It answers whether the self-healing system's certainty about root causes corresponds to its actual correctness: perfect calibration means 90% confidence yields 90% accuracy.
Decision Supported:
Can the NOC trust the self-healing system's confidence scores when deciding whether to accept automated remediation or escalate for human investigation? Without FDCI, the NOC either trusts every high-confidence diagnosis equally (dangerous if the model is overconfident on certain fault types) or applies blanket skepticism that defeats the purpose of automation.
Why This Is Unmeasured:
Legacy fault management tracks alarm counts, MTTR, and ticket volumes. None of these assess whether the diagnostic engine's confidence scores are truthful. A system reporting 95% confidence on every diagnosis looks impressive until you discover its actual accuracy on software-configuration faults is 60%. SON platform, fault management system, and network controller all produce confidence scores, but no standard operational metric tests whether those scores correspond to reality. The gap is between "the system says it's sure" and "the system is actually right" (3GPP TS 28.535, Self-Healing Requirements; IEEE Transactions on Network and Service Management, "Calibration of Fault Diagnosis in Autonomic Networks," 2024).
Formula
Plain Language: Group self-healing fault diagnoses into confidence bands (e.g., 90–100%, 80–89%, 70–79%). For each band, compare the percentage of diagnoses that turned out to be correct against the stated confidence level. Perfect calibration means the accuracy in each band matches the confidence label.
Notation:
FDCI(b) = Actual_accuracy(b) / Stated_confidence(b)
FDCI_overall = Σ [n(b) × FDCI(b)] / Σ n(b)
Variable Definitions:
- b = confidence band (e.g., 90–100%, 80–89%, 70–79%)
- Actual_accuracy(b) = percentage of diagnoses in band b confirmed correct by post-resolution analysis or recurrence analysis (percentage, 0–100)
- Stated_confidence(b) = midpoint or average stated confidence for diagnoses in band b (percentage, 0–100)
- n(b) = number of diagnoses in band b (count, dimensionless): used as weight in the overall aggregate
- FDCI_overall = weighted average across all bands, weighted by diagnosis count per band
Units: Dimensionless ratio. 1.0 = perfect calibration. Less than 1.0 = overconfident (system claims higher certainty than it achieves). Greater than 1.0 = underconfident (system is better than it claims). Values below 0.8 at high-confidence bands are the danger zone: the system is acting confidently on wrong diagnoses.
Sign Convention:
Closer to 1.0 is better in both directions. Overconfidence (below 1.0) is more dangerous operationally than underconfidence because it leads to automated remediation of incorrectly diagnosed faults.
Edge Cases:
- Ground truth for diagnosis correctness: if the fault never recurs after remediation, the diagnosis is "probably correct" but not proven. Use recurrence within 14 days as a proxy for incorrect diagnosis. This underestimates errors but is the best available signal without mandatory post-mortem analysis.
- Small sample in high-confidence bands: require a minimum of 30 diagnoses per band per measurement period for statistical reliability.
- Multiple possible root causes: if the diagnosis identified one of several contributing factors, count as partial credit (0.5) rather than binary correct/incorrect.
Proposed formula, pending validation pilots.