Definition
Change in effective product diversity across AI-influenced customer touchpoints over time, measuring whether recommendation engines are converging on a shrinking set of "winning" products while long-tail items lose visibility.
The Measurement Gap
Recommendation engines optimize for engagement, click-through, or conversion. Products that perform well get recommended more, which makes them perform better, which gets them recommended even more. This self-reinforcing feedback loop systematically concentrates visibility on a narrowing set of "winners" while long-tail products lose the impressions they need to find their audience. Click-through rate and conversion rate on recommendations improve as the engine narrows its focus because it is showing products with proven performance histories, but these metrics cannot detect that the improvement comes at the cost of assortment diversity. Revenue per recommendation session increases as the engine concentrates on best-sellers, masking the decline in long-tail product viability. Catalog breadth (number of active SKUs) measures what is available but not what is visible; a catalog of 50,000 SKUs means nothing if the recommendation engine only surfaces 2,000 of them. Sell-through rate by product tier shows outcome differences but not whether AI-driven visibility concentration caused them. The failure mode is an algorithm that reports improving performance on every metric it optimizes while quietly strangling the long tail that gives the assortment its differentiation and resilience.
Decision Supported:
Is the recommendation engine creating a filter bubble that narrows effective assortment, and should diversity constraints be added to the recommendation algorithm?
Formula
Plain language: Compute the Shannon entropy of product impressions across AI-driven touchpoints (recommendations, search results, homepage features, email) each month. Compare to the same metric at baseline. A falling entropy means the AI is concentrating visibility on fewer products.
Notation:
ADEI = 1 - [H(AI_impressions_t) / H(AI_impressions_t0)]
Variables:
| Variable | Definition | Unit |
|---|---|---|
| ADEI | Assortment Diversity Erosion Index | Index (0 to 1) |
| H(AI_impressions_t) | Shannon entropy of product impression distribution across AI touchpoints in period t; H = -SUM(p_i x log(p_i)), where p_i = share of total impressions for product i | Nats or bits |
| H(AI_impressions_t0) | Shannon entropy at baseline (first measurement period or pre-AI if available) | Nats or bits |
| t | Current measurement period | Month |
| t0 | Baseline measurement period | Month |
Proposed formula, pending validation pilots.