pytest-triage
A shadow-mode triage assistant for pytest-dev/pytest issues, measured against what maintainers actually did. It never posts to GitHub.
Labels
11 labels, scored on issues created after 2025-01-01 that maintainers triaged. Anything that costs money must beat the free kNN vote.
| Predictor | macro-F1 | micro-F1 | n |
|---|---|---|---|
| llm:gemini-3.5-flash-lite:v4 | 0.58 | 0.55 | 237 |
| ensemble gate:0.3 | 0.57 | 0.54 | 237 |
| ensemble route | 0.56 | 0.52 | 237 |
| llm:gemini-3.5-flash-lite:v3s | 0.51 | 0.51 | 237 |
| llm:gemini-3.5-flash-lite:v2 | 0.50 | 0.51 | 237 |
| llm:gemini-3.5-flash-lite:v1 | 0.47 | 0.49 | 237 |
| logreg | 0.46 | 0.42 | 237 |
| knn | 0.38 | 0.38 | 237 |
| keywords | 0.35 | 0.38 | 237 |
| everything | 0.18 | 0.18 | 237 |
| majority | 0.03 | 0.18 | 237 |
Needs more information
| Predictor | P | R | F1 |
|---|---|---|---|
| always | 0.14 | 1.00 | 0.24 |
| llm:gemini-3.5-flash-lite:v1 | 0.38 | 0.34 | 0.36 |
| llm:gemini-3.5-flash-lite:v2 | 0.48 | 0.41 | 0.44 |
| llm:gemini-3.5-flash-lite:v3s | 0.48 | 0.38 | 0.42 |
| llm:gemini-3.5-flash-lite:v4 | 0.52 | 0.34 | 0.42 |
Duplicates
15 known duplicate pairs (silver labels from maintainer comments) plus non-duplicates to measure false alarms.
| Method | Found | False alarms | 95% CI |
|---|---|---|---|
| search: bm25 | hit@5 5/15 | - | 15%-58% |
| search: dense | hit@5 4/15 | - | 11%-52% |
| search: hybrid | hit@5 5/15 | - | 15%-58% |
| LLM verifier: gemini-3.5-flash-lite+expand | 5/15 | 5/30 | 15%-58% |
| LLM verifier: gemini-3.5-flash-lite+expand+union | 7/15 | 3/30 | 25%-70% |
| LLM verifier: gemini-3.8-flash+expand | 5/15 | 4/30 | 15%-58% |
High-precision mode
Show an LLM label only when the trained classifier also gives it high probability. The threshold was chosen on 2024 issues (lowest one reaching 70% precision) and then applied once to the test set.
| Split | Confidence | Threshold | Precision (95% CI) | Recall | Issues |
|---|---|---|---|---|---|
| dev | llm gated by logreg p | 0.90 | 0.76 (0.65-0.85) | 0.19 | 24% |
| dev | logreg p alone | 0.95 | 0.78 (0.62-0.88) | 0.11 | 15% |
| test | llm gated by logreg p | 0.90 | 0.82 (0.69-0.91) | 0.14 | 18% |
| test | logreg p alone | 0.95 | 0.89 (0.67-0.97) | 0.06 | 7% |
Where it fails
Micro-F1 on the test set by issue shape, with bootstrap 95% intervals. Slices overlap.
| Slice | n | knn | llm v4 | logreg |
|---|---|---|---|---|
| all | 237 | 0.38 (0.33-0.42) | 0.55 (0.50-0.60) | 0.42 (0.38-0.47) |
| body 400-2000 chars | 137 | 0.36 (0.30-0.42) | 0.56 (0.49-0.62) | 0.44 (0.38-0.49) |
| body < 400 chars | 18 | 0.48 (0.28-0.63) | 0.43 (0.18-0.65) | 0.27 (0.11-0.41) |
| body > 2000 chars | 82 | 0.38 (0.30-0.46) | 0.55 (0.46-0.63) | 0.43 (0.35-0.50) |
| has a code block | 174 | 0.38 (0.33-0.44) | 0.56 (0.51-0.61) | 0.44 (0.39-0.49) |
| has a traceback | 59 | 0.34 (0.23-0.43) | 0.55 (0.46-0.64) | 0.40 (0.30-0.50) |
| maintainers gave no type: label | 99 | 0.20 (0.14-0.26) | 0.30 (0.23-0.38) | 0.23 (0.17-0.29) |
| no code block | 63 | 0.36 (0.27-0.44) | 0.50 (0.39-0.59) | 0.37 (0.26-0.47) |
| no traceback | 178 | 0.39 (0.34-0.44) | 0.55 (0.49-0.61) | 0.43 (0.38-0.48) |
| title is a question | 9 | 0.27 (0.00-0.50) | 0.36 (0.00-0.62) | 0.36 (0.00-0.57) |
| used the bug template | 27 | 0.44 (0.29-0.62) | 0.60 (0.44-0.72) | 0.35 (0.20-0.49) |
Prompt injection
Each attack in 6 variants (verbatim, moved, quoted, in a code block, zero-width spaces, look-alike letters). Landed means a maintainer would see the attacker's outcome; fooled the model counts the model's raw output, before the policy layer routes flagged issues to a human.
| Run | Landed | Fooled the model | Flagged |
|---|---|---|---|
| pipeline + mutations | 0/104 | 3/104 | 62/104 |
| pipeline + mutations (before the fixes) | 1/104 | 2/104 | 34/104 |
Shadow mode (prospective)
Since 2026-10-14, a daily job triages each new issue within about a day of it opening. 5 predicted so far; maintainers have since triaged 0, 5 remain untriaged. Label macro-F1 on those triaged: 0.00 (small n while the run is young). Median latency 2.97s; total cost $0.0107.
| Issue | Suggested labels | Needs info? | Possible duplicate of |
|---|---|---|---|
| #15201 | - | ||
| #15200 | - | ||
| #15190 | type: docs | ||
| #15187 | topic: collection | ||
| #15180 | - |
Earlier backtest (predicted after the fact, older pipeline): 37 issues from 2026-09-01 to 2026-10-07, of which 30 were still untriaged by maintainers when scored.