Two different things broke in the same week, and from Slack both looked like “the bots are down”. They had nothing to do with each other. They just chose the same week, like relatives.

I’m writing them up together because the order you diagnose them in is the whole lesson.

Symptom

The claims logger stopped posting. The usage-amount bot stopped answering. The error watcher posted one diagnosis and then also went quiet, which felt personal. Three bots, one process, zero information.

Failure 1: the DNS flood

The log was enormous. Opening it, 90% of the lines were one error: the support platform’s API hostname failing to resolve. Not a 500. Not a timeout. Name or service not known, every few seconds, each with a full Python stack trace, each identical to the last.

The machine’s DNS resolver was failing intermittently. Every poll hit it, every failure logged twenty lines, and the retry interval was short. Real errors from the other bots were in there somewhere, buried under an avalanche of the same sentence.

Fixes:

• On resolver failure, back off (30 s, then 60, then 120) instead of retrying immediately like an over-eager puppy.
• Log one line per failure, not a trace. The trace is identical every time and tells you nothing after the first.
• A public resolver as fallback in the machine’s network settings.

Failure 2: the PATH trap

While poking at failure 1, I moved one bot to run as a launchd service instead of from a terminal, because I was “tidying up”. It started, logged “starting”, and then did nothing. No error. No log. Just a bot sitting in the dark.

launchd does not run your shell profile. The PATH it hands a service is minimal to the point of rudeness. The bot called the Claude CLI by its bare name, claude, which the shell could find and launchd could not. The subprocess failed with “command not found”, and the bot’s own error handling swallowed it whole without chewing.

Two fixes, both permanent:

Full paths everywhere. The code resolves the CLI’s absolute path once at startup and never calls the bare name again:

def claude_cli():
    # never call the bare name 'claude' — launchd has no PATH
    for p in (os.environ.get('CLAUDE_CLI'), '/opt/homebrew/bin/claude',
              os.path.expanduser('~/.local/bin/claude'), shutil.which('claude')):
        if p and os.path.exists(p):
            return p
    raise RuntimeError('claude CLI not found; set CLAUDE_CLI')

An explicit PATH in the plist, so anything else the bot shells out to behaves the same under launchd as in a terminal:

<key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin</string>
</dict>

The order to check next time

1. Is the process running? Activity Monitor or launchctl list. If not, it’s launchd, not the bot.
2. Is the log flooding? tail -200 and look for one line repeating. If it is, you have not yet seen the real error. You’ve seen the weather.
3. Does the bot work from a terminal but not as a service? PATH, environment, or login state. Always one of those three. Always.
4. Only now read the bot’s own errors.

I did it in the wrong order and spent a morning reading stack traces that were all the same stack trace. That morning is why this list exists.

One line from me

None of this is hard. It’s the tedious, soul-sucking grunt work nobody wants to do. Hand every bit of it to the bot.

Written by Seungjoo Choi. Who I am and why I run these bots.