dsh-plugin-watchdog

Host-side crash/restart watchdog for dsh. When the dsh server process dies — a crash, OOM, kill -9, or a restart — every running agent is interrupted and the work simply stops. The persisted session logs survive, but nothing picks the work back up until a human reopens the sessions. This plugin closes that gap:

• No browser needed. It is a host-only Cordis plugin; it runs inside the dsh server and works whether the GUI is open, closed, or never opened.
• On restart it checks recently active sessions and resumes the ones whose work was genuinely interrupted — not the ones that were deliberately stopped.
• It picks up where they left off: it resumes the persisted session via ctx.agents.resume, then either re-arms the active goal (the goal-round driver queues the next <goal_round> itself) or wakes the agent with a "continue where you left off" follow-up prompt. Crash-recovery closers (TOOL_OUTCOME_UNKNOWN / TOOL_NOT_STARTED) are already in the resumed history, so the model verifies uncertain side effects before retrying.

Install

The plugin is a host-only Cordis plugin resolved from your dsh profile's package store. The profile's dsh plugin command forwards to pnpm, so any pnpm-supported git spec works.

From GitHub

One block — installs the plugin (and its git-hosted dependency dsh-lib-context-injection) and registers the loader row:

dsh plugin --profile web add github:davidgereb/dsh-plugin-watchdog --config.block-exotic-subdeps=false
PATCH="${DSH_HOME:-$HOME/.dsh}/profiles/web/cordis.patch.yml"
grep -q "name: dsh-plugin-watchdog" "$PATCH" 2>/dev/null || cat >> "$PATCH" <<'EOF'

- insert:
    - id: watchdog
      name: dsh-plugin-watchdog
EOF

pnpm ≥ 11 note. The --config.block-exotic-subdeps=false flag lifts pnpm 11's default ban on git-hosted transitive dependencies — this plugin's dependency dsh-lib-context-injection is resolved that way. No config files to edit.

From a local checkout

# 1. build (regenerates lib/index.js + lib/client.js from src/)
node scripts/build.js
# 2. make the local package resolvable by the profile
dsh plugin --profile web link /path/to/dsh-plugin-watchdog
# 3. same for the lib dependency
dsh plugin --profile web link /path/to/dsh-lib-context-injection

Then add the loader row to the profile patch file ($DSH_HOME/profiles/web/cordis.patch.yml):

# ── Watchdog: resume sessions interrupted by a server crash/restart ──
# Host-only plugin (no browser half). On boot it resumes recently-active
# sessions whose agents were interrupted by an unclean server stop and
# continues their work (goal re-arm or a continue follow-up).
- insert:
    - id: watchdog
      name: dsh-plugin-watchdog

The profile patch is read at server start, so the row takes effect on the next dsh web start — which is also the first real test of the watchdog.

How it decides what to resume

1. It tracks sessions while the server runs

A small persisted registry ($DSH_HOME/storages/watchdog.json) records every session the watchdog sees and when it last did anything:

• every durable session/event touches the session's lastActiveAt;
• agent/status → running (and a periodic poll of ctx.agents.list()) also records lastRunningAt and clears cleanStop.

2. A graceful stop is a deliberate stop

On a graceful shutdown (Ctrl+C / SIGTERM — the plugin's effect disposer and the loader's exit event both run), every tracked session is marked cleanStop. A crash or kill -9 runs neither, so cleanStop stays false.

3. The boot scan resumes only real pending work

After the loader tree settles, the watchdog looks at every tracked session that is recently active (lastActivity within staleMs, default 24 h), not cleanly stopped (unless resumeAfterGracefulShutdown), and not already live. It also does a one-time disk scan of the sessions directory for recently-touched logs it has never seen (so it also helps sessions that were running before the plugin was installed).

For each candidate it reads the session log non-mutatingly (sessionPersistence.inspect) and checks for durable pending work:

Deliberately stopped sessions show none of these: their turns close normally, their inboxes are cleared, and a user-cancelled goal is durably paused by the round driver. One race is covered explicitly: a goal session cancelled right before the crash still reads phase active (the pause mutation had not landed) — if the only signal is the goal and the log's last turn ended in a user-abort, it is treated as deliberately stopped and left alone.

4. Resume mechanics

• ctx.agents.resume({ resumeSessionId }) — the watchdog owns the returned handle and disposes it on unload.
• Goal-only sessions are re-armed (ctx.goals.resume with the live { id, revision }) and left to the goal-round driver; they get no follow-up prompt (the driver queues the next round itself).
• Anything else gets a followup() continuation prompt explaining the interruption (configurable via continuePrompt).
• Subagent sessions are never resumed on their own — their parent drives them.
• Per-boot cap (maxResumePerBoot, default 8) prevents a resume storm; each candidate's failure is contained (logged, next candidate proceeds).

Config (per-entry config: in cordis.patch.yml)

On resume, the watchdog also inspects the log tail for a truncated tool-call block (an unclosed <tool_calls> / <invoke in the last assistant message) and appends a note telling the model to use its normal structured tool-calling format — preventing the "tool calls rendered as chat text" degradation after a mid-call crash.

Settings page (Settings → Plugins → Configurable)

The plugin ships a browser card (like Cost Lens) where you can configure the fallback provider/model and the resume policy without editing the patch:

• Fallback provider / fallback model — used when a resumed session's log has no provider/model to restore.
• Recently-active window (hours) — staleMs as hours.
• Max resumes per boot — maxResumePerBoot.
• Resume after graceful shutdown — resumeAfterGracefulShutdown.
• Only auto-resume when no browser is connected — resumeOnlyWhenNoClient.
• Re-arm active goals — resumeGoals.

Changes are saved by the host to $DSH_HOME/storages/watchdog-settings.json (UI settings override the patch config) and apply live for the fallback model; boot-scan knobs take effect on the next boot.

Architecture

Host half (lib/index.js): the activity registry, clean-shutdown marker, boot scan, resume (with model restoration via setup(agentCtx) + installModelSelection), goal re-arm, quiet follow-up, and the /watchdog/api JSON-RPC settings endpoint (mounted lazily once the webserver exists).

The resume mechanics (route derivation, installModelSelection wrap, and the follow-up message injection) live in the shared dsh-lib-context-injection library, which dsh-plugin-cost-lens also uses for its scheduled/off-peak sends.

Browser half (lib/client.js): the Settings → Plugins card reading/writing /watchdog/api.

Caveats

• Resume is a fresh continuation, not mid-turn time travel. A crash-mid-turn session is closed by the persistence layer's crash repair (synthetic tool/result + turn/end { interrupted }), and the model continues from there — it sees the repair markers and our follow-up prompt, and verifies uncertain side effects before retrying. There is deliberately no partial-turn rewind (that is a persistence-layer limitation, not this plugin's).
• Single-writer sessions. Only one dsh process may write a session log at a time. Do not run two servers against the same $DSH_HOME and expect both to resume the same sessions; the losing resume is caught and logged.
• Deliberate restarts are treated as deliberate stops by default; flip resumeAfterGracefulShutdown if you want graceful restarts to continue too.
• The watchdog only tracks what it has seen. The untracked disk scan covers sessions that ran before install, but it is bounded (scanUntrackedMax) and mtime-based — very large old logs are not scanned.

Testing

# install the git dependency (dsh-lib-context-injection) so lib/index.js resolves:
npm install
node scripts/build.js
node scripts/smoke-test.mjs      # pure event-analysis logic (30 checks)
node scripts/mock-boot-test.mjs  # full apply() orchestration against a mock ctx (20 checks)

Compatibility. Tested against dsh 0.1.0-rc.6 on Node.js v24.19.0 (dsh web profile). Older or newer dsh releases may change the internals this plugin hooks into — check the changelog before upgrading.

⚠️ AI-generated, provided as-is. This project was written with the assistance of an AI. It is provided AS IS without warranty of any kind, express or implied. The author cannot be held responsible for any damage, data loss, or misbehaviour that results from using it. Use at your own risk.