At work: Head of Engineering 2024-now
Monitoring and incident response
Broken customer websites are found by the team before customers notice.
Since 2024: one log store for every service, checks on every live customer site, and AI agents that look into incidents and, when allowed, fix them.
- Role
- Creator and main developer
- Years
- 2024-now
- Scope
- Every service and live customer site
- Used by
- Engineering and operations staff
01 The problem
A broken customer website should be found by the team, not by the customer. Errors from every service had to be in one place, and alerts had to point to real problems, not noise.
02 Result
- The team learns about broken sites from checks, not from customers.
- Fewer false alarms, because checks tell real outages from slow work.
- Every AI action on live systems is logged and can be set to watch-only.
03 What I built
- 01
Central log store
Running since 2024. Every service sends its errors and warnings here, and staff search them by service, customer, request or free text.
- 02
Ask the logs in words
Staff can ask the AI "what errors did this service have in the last hour?" and get an answer.
- 03
Chat alerts that stay readable
Errors go to team chat. Repeats are grouped, sends are paced, and an outside check raises an alarm if alerts stop coming in.
- 04
Uptime checks for every site
Checks websites, databases and background work. Each customer site gets its checks when it goes live, and loses them when it goes offline.
- 05
Real-browser site check
Every 5 minutes a real browser opens each customer site, checks that listings show, opens one, and checks that the right page loads.
- 06
AI operations platform
AI agents look into incidents and, when allowed, fix them. It grew from one incident bot into a platform where staff create agents and tools.
04 Risks I handled
- Risk 01
AI fixes, but with limits
An AI that can change live systems is a risk. Each tool has a risk level and a use limit, and the AI must read before it may change anything.
- Risk 02
No alarm for one blip
One network blip should not wake anyone. An incident opens only after several failed checks in a row, and a check that times out counts as a problem with the check itself.
- Risk 03
Slow or stuck?
Busy background work can look like an outage. The check tells slow but steady work from stuck work, and from a deploy in progress.
05 Stack
- Python
- FastAPI
- Celery
- PostgreSQL
- Redis
- Playwright
- Node.js
- Claude
06 How it works
- 01Every service sends logs; checks run on a fixed schedule.
- 02Several failures in a row open an incident and a chat alert.
- 03Open incidents get chat reminders until they are fixed.
- 04The incident closes on its own when checks pass again.
Get in touch.
Hiring an engineering lead, or building a platform or AI agents? Tell me about it.