Hi all — sharing a project I built and would love technical feedback on, especially from anyone working on incident tooling.
The problem
When a service goes down, an on-call engineer typically has to manually correlate signals across separate tools — metrics dashboards, log search, deployment history, and past-incident records — to figure out what changed. That correlation work is usually the slowest part of incident response, not the fix itself.
What I built
A small FastAPI service that automates the correlation step, not the guessing step:
- Retrieves deployment history, metric trends, and error logs for a given incident
- Computes an evidence-strength signal (strong / moderate / weak) from code — deployment recency, metric delta magnitude, historical match score — before any LLM is involved
- Uses an LLM only to turn already-scored evidence into a readable narrative, with every claim citeable back to a specific log line or metric. It never generates its own confidence number — that’s a deliberate design choice, since a model asked “how confident are you” will produce something plausible-sounding with no calibration behind it
- Similarity search over past incidents via vector embeddings, to answer “has this happened before”
- A real GitHub integration: pulls deployment history from the GitHub REST API, and a webhook receiver with HMAC-SHA256 signature verification for real-time push events
What’s honest about scope
The demo runs against two synthetic services with an injected connection-pool-exhaustion scenario — there’s no live Prometheus/Elasticsearch cluster behind it. Everything downstream of that data existing (correlation, scoring, retrieval, narrative synthesis) is real and tested; the log/metric ingestion is not yet connected to a real production source. 32 tests cover the evidence-scoring logic, the API, and the GitHub integration (including a regression test that the narrative never fabricates a confidence percentage).
Stack: FastAPI, SQLAlchemy, pgvector/Postgres (SQLite for local dev), scikit-learn for embeddings, Anthropic API for narrative synthesis with a deterministic evidence-only fallback when no LLM is configured.
Genuinely interested in critique — particularly from anyone who’s worked on Opsgenie, Statuspage, or JSM’s incident features, since you’d know the failure modes I haven’t hit yet. What would break this against real, messy production data?