English · 中文 · Español · Português · हिन्दी

🤖 dsh-auto-review

Second-model AI approval for DeepSeek Harness — the Codex approvals_reviewer=auto_review / Claude Code auto mode pattern, built as a pure Cordis plugin.

When an agent's action crosses the sandbox boundary, a read-only reviewer subagent decides allow/deny — with a reason — so humans approve nothing while nothing unsafe slips through.

Zero human operations. The request goes to the AI reviewer, the verdict is allow/deny + reason + risk level, and every decision is reconstructable from the session log: approval/asked → autoReview/verdict (or autoReview/rejection for hard disables) → approval/decided.

One real evidence run (real server, real API key, two real model rounds): the AI reviewer allows an escalated workspace write (risk low, 5.2 s), then denies a recursive out-of-workspace delete (risk high, 8.9 s) — the deny reason is fed back to the model, visible in the transcript.

Why a second model instead of rules?

Pattern-based auto-approvers decide before dispatch, with no evidence. dsh-auto-review gives the decision to a reviewer subagent that reads the actual workspace (through its read-only tool face), the already-streamed tool-call arguments (sensitive values redacted), the request reason, and your risk rules — then returns a structured verdict. A deny verdict feeds its reason back to the calling model, so the agent learns why instead of retrying blindly.

✨ Features

How it works

                       approval/request waterfall (answerer chain)
                        │
┌───────────────────────┴──────────────────────┐
│ dsh-auto-review answerer                     │
│  · session enabled?  · policy = ai?         │   no ── next() ──▶ human answerer (UI)
│  · risk rules → toolsPolicy → default       │
└───────────────────────┬──────────────────────┘
                        │ yes
                        ▼
        ┌───────────────────────────────────┐
        │ reviewer subagent (fork, one-shot)│
        │  · toolFilter: read/glob/grep     │
        │  · outputSchema: {decision,       │
        │    reason, riskLevel}             │
        │  · timeout + req.signal abort     │
        └───────────────┬───────────────────┘
                        │ verdict / failure (fail-closed fallback)
                        ▼
 allow → allowed-once        deny → rejected + reason injected into the
                                       denied tool result (callId-linked)
                        │   never → rejected + [auto-review-never] feedback
                        │            (hard disable, no reviewer runs)
                        ▼
 audit: approval/asked → autoReview/verdict | autoReview/rejection
        → approval/decided (session events, log-only, invariant-checked)

Composition order. The answerer runs at its registration position in the waterfall: if a human UI answerer is composed BEFORE the auto-review row, humans answer first and the reviewer only sees what is delegated downstream. Verify with dsh --profile <name> --dump-config and place the auto-review row before your human answerer rows when you want ai-policy tools routed to the reviewer first.

🚀 Quick start

Three install channels; the plugin is a bundle ("dsh": { "bundle": { "patch": "./cordis.patch.yml" } }).

# 1. npm (published artifact, no build step)
dsh plugin --profile web add dsh-auto-review
dsh --profile web               # restart

# 2. npm tarball (built artifacts, offline install)
pnpm pack                       # → dsh-auto-review-<version>.tgz
dsh plugin --profile web add ./dsh-auto-review-<version>.tgz
dsh --profile web               # restart

# 3. git source (pin the commit; self-contained `prepare` builds it)
#    pnpm ≥ 10 blocks lifecycle builds: add the printed allowBuilds key
#    to the profile's pnpm-workspace.yaml first.
dsh plugin --profile web add "github:PerryLink/dsh-auto-review#<commit>"

# 4. local link (development)
dsh plugin --profile web add link:/path/to/dsh-auto-review

Verify:

dsh --profile web --dump-config | grep -A4 'id: auto-review'

Out of the box the shipped patch AI-reviews bash and write; every other tool (including edit — in-place modification) delegates to the human chain. Add edit: ai explicitly if you accept in-place edits without a human in the loop.

⚙️ Configuration

All tunables are Schemastery Config fields (changeable from cordis.yml). An id-targeted override replaces the whole config row — restate every key you need.

Example (annotated full form: fixtures/config/config-full.yaml):

- insert:
    - id: auto-review
      name: dsh-auto-review
      config:
        toolsPolicy:
          overrides: { bash: ai, write: ai }
        riskRules:
          - pattern: '(?i)(rm\s+(-[a-z]+\s+)*/|git\s+push\s+--force)'
            policy: never
          - pattern: 'write'
            policy: never
            field: toolName
        reviewerTimeoutMs: 30000
        fallbackPolicy: delegate
        riskPolicy: { maxAutoAllow: medium, onHighRisk: delegate }
        circuitBreaker: { consecutiveDenies: 3, windowDenies: 6, windowSize: 10, action: delegate }

⌨️ Session command

/auto-review on|off|status|approve [n]

on/off append the durable autoReview/state override (the fold survives restart/resume — replay IS the state) and inject a switch notice the model sees (logged as a user/message event). status reports the effective state, both per-turn budgets (AI verdicts and reviewer failures), a tripped circuit breaker when one is active, and the session's cumulative statistics (allows/denies/fallbacks/never rejects, mean duration, recent verdicts). approve [n] records a single-use autoReview/override for the n-th most recent denial (1 = most recent): the next same-tool review within overrideTtlMs carries the authorization as reviewer context — the reviewer still decides, and the override is consumed by that review regardless of its outcome.

🖥️ Web review panel

In the Web GUI (web profile), the package contributes a session-header action (AI Review) that opens a panel with the session's auto-review state: the switch with on/off buttons (they execute /auto-review on|off), both per-turn budgets, cumulative statistics (including hard-disable rejections), the circuit trip, the recent verdicts, and one-shot approve buttons for recent denials (they execute /auto-review approve [n]).

How it is wired:

• The host registers an autoReview session projection (folded from the log-only autoReview/* events) and serves it through the session-projection channel.
• The browser half is a client module (auto-discovered from the dsh.client declaration) registered on the conversation.session.header.actions seat.
• No extra patch rows are needed: the panel loads whenever the plugin is installed in a profile whose web build provides the session-projection capability (the web profile does). Without that capability the panel reports itself unavailable; the answerer is unaffected.

The panel reads only whole projection values — it never receives the raw session event stream.

🧪 dsh-eval — agent evaluation engine

Beyond the approval reviewer, dsh-auto-review ships dsh-eval: a YAML-driven agent evaluation platform that runs real headless DSH sessions (one isolated agent + scratch workspace per case, the official Minimal persona as the baseline system prompt), collects the tool-call trace from the session event log, and evaluates structured assertions plus an optional second-model review — the same reviewer seam as the approval answerer.

# eval/cases/demo.yaml (abridged)
suite:
  name: my-suite
  cases:
    - id: math-output
      input: Solve 17 × 24 and reply with only the final number, nothing else.
      expect:
        output: { contains: "408" }
    - id: glob-trace
      seedFrom: '.'
      input: Use the glob tool with pattern "src/**" to list the source files…
      expect:
        toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
        results: [{ tool: glob, contains: "index.ts" }]
    - id: review-write
      input: Read src/config.ts, write the default reviewerTimeoutMs into scratch/answer.txt…
      expect:
        output: { contains: "60000" }
      review:
        statement: The agent read the default reviewerTimeoutMs and wrote it to the file.

Run it (a DeepSeek API key must be in the environment):

dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports

CI gate: the process exits 0 only when every case of every suite passed — drop it into a GitHub Action step and failing evaluations fail the build. Each case leaves a replayable session JSONL and a trace JSON beside report.md/report.json; assertion results, token usage, and the review verdict are all written into the report files. The engine never substitutes hardcoded model or timeout defaults, aborts cleanly on SIGINT/SIGTERM, and caps the worker pool at the configured concurrency.

Unlike codex-research (browser-automation agent research), dsh-eval targets harness-level agent evaluation: tool-call-trace assertions against the session event log, second-model review as a supplementary assertion layer, and per-case isolated headless sessions — no browser or Selenium stack.

🔒 Security

• The reviewer runs in a read-only tool face (toolFilter allow-list). It cannot write, edit, run bash, fetch the network, or delegate (maxDepth = its own depth). Its session log is persisted and auditable.
• Sensitive arguments are redacted (key-name matching: token, password, api_key, Authorization, credentials, private keys …) before entering the reviewer prompt; the plugin never executes the reviewed arguments. Redaction is key-based, not content-based — do not AI-review tools whose argument values you cannot afford to show a model.
• Fail closed by default. Every abnormal path (provider missing, capability gaps, start rejection, timeout, non-completed stop reason, missing/malformed verdict, audit-correlation failure) resolves through fallbackPolicy, default rejected — and the rejection feeds an auditable reason back to the model instead of the generic "user rejected" text. allow-once grants unconditionally — it exists only for unattended deployments whose admin accepts that risk.
• Hard disables explain themselves. A never tool or risk rule rejects deterministically AND records a log-only autoReview/rejection event with the matched rule/table entry, then injects a [auto-review-never] marker text into the denied tool result — the model learns the action is hard-disabled instead of retrying it (invariant-checked: marker ⟺ event).
• Rejection circuit breaker. A run of denials in one turn trips the breaker (consecutiveDenies / windowDenies inside windowSize), recorded as a log-only autoReview/circuit event; later requests follow its action (delegate / reject / abort-turn). abort-turn injects a model-visible warning and cancels the agent.
• Reviewer context is presented transcript. contextBudget feeds already-presented session content (messages, tool results) to the reviewer. With the default same-route reviewer model that content stays inside one provider; configure reviewerModel to a different provider only if you accept presenting that transcript to it.
• never is one-way at this layer. A never tool or risk rule rejects before the human chain sees the request — a lockdown knob, not a default.
• The reviewer is a model. Its verdicts are advisory policy, not a security kernel. Prefer human/never rules for irreversible operations.

⚠️ Known limitations

• The reviewer needs a working LLM route (inherited from the session agent by default); without one every review falls back per fallbackPolicy — never a silent grant.
• reviewerTools names must exist as global tools in the profile; an unknown name fails the reviewer child loudly at the earliest point and falls back.
• Risk rules match the request reason, the toolName, or the redacted call arguments per their field; other conditions belong in toolsPolicy.overrides.
• The /auto-review approve override authorizes the next same-tool review, not the exact historical call; a different action on the same tool consumes it.
• The verdict events are log-only; the dedicated Web review panel reads the folded autoReview projection (the raw event stream never reaches browser plugins).
• autoReview/state and autoReview/verdict are appended with the envelope's ignorable: true marker, so any harness build loads the log — readers that do not know the out-of-repo types simply skip those records instead of refusing the session. (rc.6 hosts accept and ignore the marker, keeping the exact pre-marker behavior; sessions written by pre-0.1.1 versions can be repaired with scripts/repair-session-logs.mjs from dsh-permission-rules.)
• The git channel needs the single allowBuilds key the dsh CLI prints for dsh-auto-review itself. The repo ships its own pnpm-workspace.yaml with allowBuilds: { esbuild: true } so the isolated prepare environment does not fail on esbuild's (harmless platform-binary validation) postinstall; typescript + tsdown are regular dependencies so that environment always has the build tools.
• The optional invariant companion (dsh-auto-review/invariant) needs the invariants service (agent-spine compositions such as headless/ACP); the plain web profile does not provide it, so the row ships commented out in the bundle patch.

🏷️ GitHub topics

Recommended when you publish: dsh · dsh-plugin · deepseek-harness · deepseek · cordis · ai-safety · approval · sandbox · subagent · llm

🔗 Related work

• Andy8647/dsh-auto-approval — two-state allow/deny classifier on the tools/pre-execute waterfall with file-log audit. dsh-auto-review deliberately differs: official answerer chain, always delegates what it does not own, read-only second model with a structured verdict, deny reasons fed back to the model, session-log audit.
• ACP automation bridge — one-shot machine decisions for its own ACP-owned agents. dsh-auto-review is session- and tool-policy-scoped for the interactive harness; it never infers durable grants.

🧑‍💻 Development

pnpm install                # node ^22.19 || >=24
pnpm run typecheck          # tsc, src + tests
pnpm test                   # vitest: 190 tests, 14 files
pnpm run build              # tsc declarations + tsdown bundles (lib/, incl. the client bundle)
pnpm run verify:self-contained
pnpm pack                   # publish artifact

Repository layout (plugin-template structure): src/index.ts (plugin contract) · src/config.ts (Schemastery schema + resolution) · src/runtime.ts (answerer, command, deny-reason injection) · src/review.ts (reviewer orchestration, prompt, sanitization) · src/events.ts (session-event vocabulary + folds) · src/projection.ts + src/projection-types.ts (the autoReview session projection) · src/invariant.ts (invariant companion) · src/eval/ (the dsh-eval engine: DSL, runner, assertions, trace, review, reports, CLI) · eval/ (shipped evaluation composition + demo suite) · bin/dsh-eval.mjs (CLI launcher) · src/client/ (browser half: review panel, locales, styles) · test/ · fixtures/.

👥 Contributors

Thanks to everyone who has contributed to dsh-auto-review:

• PerryLink — author and maintainer: approval answerer, reviewer subagent, risk policy and circuit breaker, session-projection review panel, invariant companion, docs, CI/CD and releases.

Want to help? Check the issue templates, the security policy, and AGENTS.md for repo conventions — PRs are welcome in English or Chinese.

PerryLink DSH Plugin Family

This project is one of the 15 DeepSeek Harness plugins maintained by PerryLink. If this one helps you, the others likely will too:

📄 License

Apache License 2.0