SREGym Leaderboard
Comparing SRE agents across diagnosis, mitigation, and end-to-end incident resolution on SREGym. Ranked by E2E success rate, requiring both correct root-cause diagnosis and successful mitigation on the same run.
Benchmark
SREGym-Lite-1004 · 17 faultsView cohort
Devin Oncall: 2 attempts per fault (34 runs). Other results: 3 attempts per fault (51 runs).
1 | Devin Oncall* | Claude Opus 5.5 (high) | 100.0 | 100.0 | 100.0 | 76.1 | 141.0 | 1.24M |
2 | CloudThinker* | GPT-6 Astra (low) | 100.0 | 100.0 | 100.0 | 126.2 | 342.0 | 736K |
3 | Devin Oncall* | GPT-6 Astra (max) | 100.0 | 100.0 | 100.0 | 190.8 | 388.4 | 1.11M |
4 | CloudThinker* | GPT-6 Astra (max) | 100.0 | 100.0 | 100.0 | 310.0 | 675.0 | 1.74M |
5 | Devin Oncall* | Claude Opus 5.5 (medium) | 97.1 | 97.1 | 97.1 | 99.7 | 152.4 | 1.26M |
6 | Codex | GPT-6 Astra (max) | 96.1 | 100.0 | 96.1 | 173.1 | 287.6 | 719K |
7 | Devin Oncall* | GPT-6.1 Sol (max) | 94.1 | 100.0 | 94.1 | 129.7 | 270.3 | 1.02M |
8 | CloudThinker* | GPT-6.1 Sol (low) | 94.1 | 98.0 | 92.2 | 152.9 | 273.8 | 564K |
9 | Devin Oncall* | GPT-6 Astra (medium) | 91.2 | 100.0 | 91.2 | 89.2 | 172.4 | 797K |
10 | Codex | GPT-6 Astra (medium) | 100.0 | 90.2 | 90.2 | 92.0 | 138.4 | 377K |
11 | Codex | GPT-6.1 Sol (medium) | 98.0 | 90.2 | 90.2 | 103.8 | 156.6 | 462K |
12 | Codex | GPT-6.1 Sol (max) | 98.0 | 90.2 | 90.2 | 217.4 | 371.6 | 807K |
13 | Devin Oncall* | GPT-6.1 Sol (medium) | 94.1 | 91.2 | 85.3 | 72.1 | 137.3 | 715K |
14 | Devin Oncall* | Claude Fable 5.1 (high) | 97.1 | 88.2 | 85.3 | 164.2 | 278.9 | 1.42M |
15 | Devin Oncall* | Claude Fable 5.1 (medium) | 88.2 | 88.2 | 82.4 | 135.2 | 220.6 | 1.11M |
16 | Codex | GPT-5.6 Sol (max) | 94.1 | 82.4 | 76.5 | 225.1 | 409.7 | 1.54M |
17 | CloudThinker* | Claude Opus 5 | 94.1 | 80.4 | 76.5 | 517.8 | 759.1 | 1.56M |
18 | Claude Code | Claude Opus 5 | 90.2 | 78.4 | 70.6 | 241.8 | 466.2 | 1.86M |
19 | Codex | GPT-5.6 Terra (max) | 82.4 | 74.5 | 62.7 | 233.1 | 444.9 | 1.88M |
20 | Codex | GPT-5.6 Luna (max) | 84.3 | 74.5 | 60.8 | 306.0 | 517.7 | 2.97M |
21 | Codex | GPT-5.6 Sol (medium) | 72.5 | 64.7 | 49.0 | 117.6 | 295.0 | 868K |
22 | Claude Code | Claude Sonnet 5 | 54.9 | 62.7 | 45.1 | 308.6 | 505.2 | 3.37M |
23 | Claude Code | Claude Opus 4.8 | 58.8 | 54.9 | 41.2 | 360.2 | 553.7 | 1.85M |
* Third-party submissions. Results verified by the SREGym team.
Diag. Diagnosis success rate · Mit. Mitigation success rate · E2E End-to-end (both diagnosis and mitigation correct) · TTD Time-to-diagnose (seconds) · TTM Time-to-mitigate (seconds) · Tokens Mean token usage per run