Quality Metrics
Overview
This document defines the quality metrics tracked for Cortex AI.
Search Quality
| Metric |
Definition |
Target |
| MRR (Mean Reciprocal Rank) |
Average of 1/rank of first relevant result |
>0.70 |
| Recall@5 |
% of queries with relevant result in top 5 |
>0.85 |
| Recall@10 |
% of queries with relevant result in top 10 |
>0.92 |
| nDCG (Normalized DCG) |
Ranking quality measure |
>0.75 |
Indexing Quality
| Metric |
Definition |
Target |
| Coverage |
% of files successfully indexed |
>99% |
| Chunk Quality |
Semantic coherence of chunks |
>0.80 |
| Freshness |
Time since last successful index |
<1 hour |
User Experience
| Metric |
Definition |
Target |
| Response Time |
Time to first result |
<500ms |
| Completion Rate |
% of queries that return results |
>95% |
| User Satisfaction |
Based on feedback signals |
>4.0/5.0 |
System Health
| Metric |
Definition |
Target |
| Uptime |
Service availability |
>99.9% |
| Error Rate |
% of failed requests |
<0.1% |
| Queue Depth |
Pending indexing jobs |
<100 |
Monitoring
All metrics are exported to Prometheus and visualized in Grafana.
- Dashboard:
https://grafana.emshvac.co/d/cortex-quality
- Alerts: Configured for metric threshold violations