SREGym: Can AI agents resolve production issues? Real-world SRE problems including metastable failures, misconfigurations, and many more. Live system environments. From the University of Illinois at Urbana-Champaign. To submit, open an issue with the submission label at github.com/SREGym/SREGym.
Interactive terminal available on desktop
SREGym-Lite-1004 results
Top results on the curated 17-fault cohort.
1 | Devin Oncall* | Claude Opus 5.5 (high) | 100.0 | 100.0 | 100.0 | 76.1 | 141.0 | 1.24M |
2 | CloudThinker* | GPT-6 Astra (low) | 100.0 | 100.0 | 100.0 | 126.2 | 342.0 | 736K |
3 | Devin Oncall* | GPT-6 Astra (max) | 100.0 | 100.0 | 100.0 | 190.8 | 388.4 | 1.11M |
4 | CloudThinker* | GPT-6 Astra (max) | 100.0 | 100.0 | 100.0 | 310.0 | 675.0 | 1.74M |
5 | Devin Oncall* | Claude Opus 5.5 (medium) | 97.1 | 97.1 | 97.1 | 99.7 | 152.4 | 1.26M |
Devin Oncall: 2 attempts per fault (34 runs). Other results: 3 attempts per fault (51 runs). * Third-party submissions. Results verified by the SREGym team.
Diag. Diagnosis success rate · Mit. Mitigation success rate · E2E End-to-end (both diagnosis and mitigation correct) · TTD Time-to-diagnose (seconds) · TTM Time-to-mitigate (seconds) · Tokens Mean token usage per run