Project Detail
SRE Incident Triage Environment
Deterministic evaluation environment for AI agents performing realistic SRE incident response. Graded on root cause identification, severity assessment, safe mitigation, and execution quality.
Technical Summary
StackPython + FastAPI
Platformapi
RoleDeveloper / Team Member
TechnologiesPython, FastAPI, Pydantic, Docker, OpenEnv, LLM APIs
ContextTeam Project
Team Size4
Key Responsibilities
- Contributed to scenario design and evaluation framework
- Implemented action system and grading logic
- Built inference runner and benchmark tooling
- Deployed with Docker and FastAPI
Project Insights
Designed a deterministic agent evaluation framework modeling real SRE workflows with typed contracts and reward shaping
Technical Concepts
- Deterministic scenarios with easy/medium/hard difficulty progression
- Public and holdout evaluation splits for benchmark integrity
- Typed Pydantic contracts for consistent action interfaces
- Reward shaping for multi-faceted grading (root cause, severity, safety, efficiency)
Architecture & Features
- Alerts, logs, traces, and service metadata as agent inputs
- Deterministic grading based on predefined correct solutions
- Inference runner for testing agent implementations
- Docker deployment for reproducible evaluation
Key Learnings
- Agent evaluation must be deterministic for reliable benchmarking
- Real-world workflows require multi-step reward shaping
- Documented benchmark results achieved mean score 0.99 across six episodes
Outcomes & Impact
- Team qualified for Meta x Hugging Face hackathon national round