dsh-verify-judge

English | 中文

What it does, in one sentence: when your AI agent says "I'm done", this plugin makes it prove it.

You're running a long task with a goal in DeepSeek Harness. The agent works a few rounds, then announces "done". But done according to whom? By default, according to itself — and models are known to rationalize half-finished work.

With this plugin installed, "done" has a price:

• The agent calls update_goal(complete) to close its goal → the plugin actually runs the project's test suite first. Red tests → the completion is rejected, and the failure output goes back to the agent as "keep fixing" instructions.
• The agent tries to skip the ceremony and just say it's done in chat, then end the turn → the plugin checks again at the turn boundary and sends it back to work.

If everything is genuinely green, the agent never notices the plugin exists. That's the whole UX: invisible when honest, immovable when not.

What it solves

Autonomous goal loops (e.g. dsh's goal mode) let an agent run many rounds by itself — but the moment of "I'm done" was pure self-declaration. With a vague or ambitious objective, the model could stamp complete while tests were still red. verify-judge turns stamping into a gated action: the goal can only close when the workspace's own verification commands exit 0.

Test evidence (real runs, 2026-08-14, headless profile, live API)

Same workspace, same failing test (add implemented as a - b); only the gate differs:

Full-philosophy demo (goal + plan.md acceptance checklist + gate): the agent fixed capitalize, implemented the missing reverseWords, ticked all three checkboxes in plan.md itself, and stamped — the gate re-ran npm test (2 pass / 0 fail) and let the goal close. The receipt on disk (plan.md fully checked) matches the test reality.

Demo

A self-contained, reproducible demo lives in demo/: a workspace with an intentionally failing test, a plan.md acceptance checklist, and run-demo.sh. It walks through the deny → fix → pass loop against the live harness, plus the A/B control (disable the gate and watch the stamp sail through with red tests).

Install

dsh plugin --profile web add github:zriyox/dsh-verify-judge
# or a local checkout:
dsh plugin --profile web add /absolute/path/to/dsh-verify-judge

Restart dsh web (or let the patch watcher reload). From then on, every update_goal(complete) in every session runs the gate.

How it works

model calls update_goal(action: "complete")
  → tools/pre-execute waterfall intercepts the call
  → resolve verification commands:
      config.commands (always wins)
      → else auto-detect in the session workspace:
          package.json with scripts.test → <pm> test (pm from lockfile)
          go.mod → go test ./...
          Cargo.toml → cargo test
  → run each through ctx.shell (inherits the deployment's sandbox)
  → all exit 0 → next() → the stamp proceeds
  → any failure → deny; the model receives the failing command,
    exit code, and an output tail as its fix instruction

Nothing else is touched: non-update_goal calls and non-complete actions delegate via next() unchanged. Disable the row (disabled: true) and behavior returns to stock.

Configuration

Add config to the plugin row (profile or home cordis.patch.yml):

- id: verify-judge
  config:
    commands: ['pnpm test', 'pnpm run lint']  # always wins over auto-detect
    onUndetected: allow                        # allow (default) | deny
    timeoutMs: 300000                          # per-command cap
    outputTailChars: 4000                      # failure tail fed to the model

Development

pnpm install
pnpm run build        # tsc → lib/

Debug without an API key using the harness's mock LLM (pnpm run mock:llm in the deepseek-harness repo), then drive a session that creates a goal and lets the model try to complete it against a workspace with a failing test.

Roadmap

• v2.1: anchor the verification surface at goal-creation time (snapshot commands + hash test files); deny when the suite drifts mid-goal — closes the "edit the tests, then stamp" reward hack.
• v2.2: parse acceptance items from plan.md; judge non-scriptable items with an isolated-context subagent (the main session only receives the verdict table).
• v3: verification receipts — any successful mutating tool call after a pass invalidates the receipt, so the next conclude must re-verify.

Shipped in 0.2.0: the turn-end gate (agent/turn-stopping), per-workspace serialized runs, fail-closed abort handling, runner-error class (no "fix the environment by editing tests" pressure).

License

MIT