Definition
The Spearman rank correlation between a model's top-k contributing-factor rankings in consecutive evaluation windows, measuring whether the model's explanation of its own predictions remains consistent over time.
The Measurement Gap
Dashboards display the model's top contributing factors to support technician decision-making, but no metric checks whether that explanation is internally stable. A model can maintain solid F1, precision, and recall while its internal reasoning shuffles each scoring window: getting the right answer for the wrong reasons. Legacy operational metrics (OEE, PM Compliance, MTBF) say nothing about explanation quality. The failure mode is silent overfitting: importance rankings reorder randomly between windows on similar inputs, indicating the model has not learned durable physical relationships. Downstream KPIs that rely on explanation (Explainability-to-Action Validation, Override Appropriateness, Shared Situational Awareness) become unreliable when the explanation layer is unstable. Concept Drift Velocity captures one of three dimensions of drift; FIS isolates the explanation dimension at finer grain.
Decision Supported:
Can a reliability engineer trust the model's factor importance display this week, or has the explanation layer become unstable enough to require investigation?
Formula
Plain language: Pull the model's top-k contributing factors from this scoring window and from the previous reference window. Compute the Spearman rank correlation between the two ordered lists. A value near 1.0 means the explanation is consistent; values near 0 mean it is unstable.
Notation:
FIS = ρ_s (R_ref, R_now)
Variables:
| Variable | Definition | Unit |
|---|---|---|
| ρ_s | Spearman rank correlation coefficient between two ranked lists | Coefficient (−1.0 to 1.0) |
| R_ref | Ranked importance of the top-k features in the reference window | Ranked list |
| R_now | Ranked importance of the same top-k features in the current evaluation window | Ranked list |
| k | Number of top features tracked (recommended starting point: 10) | count |
Proposed formula, pending validation pilots.