Persona Evaluation — Ground truth ablation: BGT only (anti_gt)
| Field |
Value |
| Persona |
Nimesh Kulatunga |
| Judge model |
openai/gpt-4.1-mini |
| Embed model |
text-embedding-3-small |
| Rubric draws (k) |
3 |
| Total suggestions |
27 |
| Pipeline mode |
hall_pass=anti_gt (active=True) |
Bucket Distribution
| Bucket |
Count |
% of total |
| Task Critical |
18 |
66.7% |
| Quality Of Life |
4 |
14.8% |
| Noise |
3 |
11.1% |
| Hallucinated |
2 |
7.4% |
Set-Level Diversity Metrics
| Metric |
Value |
Interpretation |
| DPP log-det |
-20.8246 |
Higher = more diverse + high-quality set |
| Cluster coverage |
1.000 |
Fraction of BGT clusters with a task-critical hit |
| ILAD |
0.6331 |
Mean pairwise distance; higher = more diverse |
| Redundancy rate |
0.000 |
Fraction of near-duplicate suggestions (cos > 0.9) |
Composite Score
| Component |
Weight |
Value |
| DPP set score (normalised) |
0.5 |
— |
| Cluster coverage |
0.3 |
1.000 |
| Mean quality (non-hallucinated) |
0.2 |
— |
| Hallucination penalty |
alpha=0.5 |
x 0.9630 |
Composite score: 0.5853
Hallucination Summary
Filter: anti_gt (active). Flagged 2 / 27 suggestions (rate 7.4%). Composite hallucination penalty: 0.9630.
Filtered Hallucinations
| ID |
Failure mode |
Title |
Reasoning |
| 21 |
source_context_confusion |
Flag that Toastmasters Brand Manual is open in ... |
The suggestion misattributes the Toastmasters brand manual viewing to the Bit... |
| 22 |
source_context_confusion |
Extract brand colors (hex), fonts, and logo rul... |
The suggestion fabricates detailed brand color and font extraction linked to ... |
Top 5 Task-Critical Suggestions
| # |
ID |
Quality |
Title |
| 1 |
4 |
0.980 |
Auto-populate research doc with per-org summary stubs |
| 2 |
8 |
0.970 |
Summarise SOB Student Guide pages user read into an appli... |
| 3 |
9 |
0.970 |
Offer a proposal draft outline after 60+ minutes of SOB r... |
| 4 |
7 |
0.947 |
Explain Bitcoin-Fuzz Issues #295 and #490 that user opened |
| 5 |
3 |
0.937 |
Detect that 'Summer of Bitcoin Interested Projects' Googl... |
Top 5 Quality-of-Life Suggestions
| # |
ID |
Quality |
Title |
| 1 |
17 |
0.870 |
Offer to resume GitHub docs task when Canva work wraps |
| 2 |
27 |
0.807 |
Recognise Google Docs as primary note-taking surface for ... |
| 3 |
23 |
0.610 |
Detect WhatsApp Web photo downloads and offer folder orga... |
| 4 |
11 |
0.597 |
Offer to open Google Calendar to block SOB proposal writi... |