I got tired of finding out my homelab was broken only when I tried to use it.
I'd sit down to watch something on Jellyfin, hit play, get nothing, and end up opening my laptop instead. Playback would fail, an import would stall, or two services would disagree about the same files. Sometimes Jellyfin could see the media, but ffmpeg would choke when it tried to use it. Other times an import would sit there half-finished until I went digging. After enough nights like that, I stopped treating these as isolated bugs and started looking at how the system recovered from them.
This was never really an uptime problem
For a long time, the homelab was good enough to be useful and bad enough to be annoying.
That is a rough place for a system to live. If everything is down, at least the problem is obvious. My problem was the opposite. Most of the stack usually looked fine. Then I would try to do something normal and find out one part in the middle had gone weird. A service could see files but not use them. An import pipeline would stall for no obvious reason. Permissions between services would drift just enough to break something. A mount would exist, but not in the way the app expected.
None of this was dramatic. That was the issue.
The failures were small, partial, and perfectly timed to waste my evening. They showed up when I wanted to use the homelab, not when I was already in the mood to maintain it.
Why this was fixable
A big reason I could solve this is that the homelab is built in Nix. The machines, services, mounts, reverse proxy config, and deployment flow already live in a repo instead of a pile of manual fixes and old terminal history.
If that state were spread across a few live machines, I could only get faster at noticing and repairing failures. With the system defined in code, a bad user, path, mount, or permission has a durable fix. I can change the definition instead of cleaning up the same live machine again.
The question stopped being how to notice issues faster. I wanted the system to prepare the fix too.
The thing I was missing
I already had logs and alerts. They could tell me that an import failed, but they could not inspect the Nix configuration or prepare a repair.
That is the job I gave an AI agent. It receives the incident, reads the relevant configuration, checks the available logs and live state, and proposes a change in the repo. The agent does not get permission to modify the running homelab. Its output is a branch and a pull request that I can review.
Building the loop
I built a small app that sits between the alerts and the repo.
For observability, I use OpenObserve. That gives me a single place for logs and alerts, and it sends those alerts into the control plane when something goes wrong. From there, the control plane parses the alert, groups repeats, creates an incident, and decides whether it looks like noise or something worth acting on.
If it looks worth acting on, the app starts the agent.
The app opens an isolated worktree and starts an OpenCode run. The prompt includes the incident, service, error details, worktree path, and expected pull request format. The agent can check more logs or inspect live state, but it has one job: identify the cause and change the repo. Without that boundary, it would be too easy for an automated repair to make the homelab harder to trust.
Because the homelab has a declared shape, the agent has something concrete to inspect and change. A permissions mismatch, bad mount, or wrong path becomes a normal code change. Nothing quietly mutates a live machine. I get a branch, a pull request, and the incident context needed to review it.
A lot of the time, that means the system gets from incident to a concrete proposed fix before I need to get involved. I still review and approve the PR, but I am reviewing a repair instead of doing the whole investigation myself.
Where it got annoying
The simple version is alerts in, fixes out.
In practice, it was messier. Once you build a loop like this, the loop itself becomes another system you have to make reliable. It was not enough to get the happy path working. I had to deal with duplicate incidents, stuck runs, retries, follow-ups, merge state, and old worktrees piling up after the fact. The control plane ended up needing its own cleanup and maintenance logic so it did not slowly turn into a second source of mess.
That took longer than I expected, but it is also the reason I trust it now. A self-healing system is only useful if it does not create a fresh problem every time it tries to solve one.
What changed for me
Things still break. The difference is that I am usually reviewing the fix, not discovering the problem for the first time.
Before, a small issue could take over the whole evening because there was nothing between "something is off" and "I need to debug this right now." Now the system groups repeated alerts and starts a repair before I open my laptop.
The result is not simpler, but it interrupts me less. That was the goal. I did not want to become a better on-call engineer for my own house. I wanted to use the homelab without thinking about maintenance most of the time.