dsh-plugin-loud-failure

A DeepSeek Harness plugin that turns silent tool failures into loud ones.

A tool call can exit 0 and still have failed. pandoc drops a glyph and only warns; a pipe swallows a Python traceback; a ; chain hides command not found; NumPy prints a RuntimeWarning and hands back nan. The exit code says success, the model reads success, and the mistake travels downstream. This plugin is one tools/post-execute waterfall listener: it matches the text a tool returned against a rule table and, on a hit, either blocks the result (it becomes an isError result whose content leads with the explanation and keeps the original output) or attaches a notice that lands in the model's next request. It changes no tool and no loop, so it composes with every other post-execute policy and unmounts cleanly.

Contents

• Why
• How it works
• Install
• Configuration
• Built-in rules
• What the model sees
• Verification
• Design notes
• Model Experience
• Known Limitations and Deferred Work
• Development
• License

Why

Every built-in rule comes from a failure that was observed to hide behind a successful exit code:

The harness already records everything the model sees and lets any plugin rewrite a tool result before the model sees it. This plugin uses exactly that seam to make the failure visible at the moment it happens, instead of three steps later.

How it works

flowchart LR
  call["tool/call"] --> pre["tools/pre-execute"] --> exec["tools/execute"] --> post["tools/post-execute<br/>(this plugin)"] --> result["tool/result"]
  post -->|"error rule matched"| block["block: isError result<br/>header + original output"]
  post -->|"context rule matched"| notice["accept + additionalContexts<br/>plugin-sourced notice"]
  post -->|"no match"| next["next()"]

1. At load, the plugin merges the built-in rules with the configured ones (a user rule with a built-in id replaces it) and compiles every pattern. An unusable table (invalid regular expression, duplicate id, stateful flag, empty message) rejects the plugin load with the offending rule id, so a broken config fails at boot, not at the first tool call.
2. On every tools/post-execute, the text blocks of the result (optionally also the JSON of a successful canonical value) are matched against the rules whose when and tools filters apply.
3. If any matching rule has action: error, the listener returns { kind: 'block' } with feedback that begins with a header naming every matched rule and its message, followed by the original content blocks unchanged. The registry turns that into an isError result: the canonical value is gone, so a Code Mode program cannot consume a poisoned value either.
4. Otherwise (only context matches) the listener delegates with next() and appends one UserMessage to the decision's additionalContexts. Its source is { kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary }, so the Web UI shows a collapsed one-line row and the session log records exactly what the model was told.
5. No match: next(). Rules with action: off never run.

Because the listener is registered through ctx.on, it is torn down with the plugin, and a config change under HMR replaces it atomically.

Install

Requires DeepSeek Harness 0.1.0-rc.6 or later (@deepseek-ai/dsh-tools, @deepseek-ai/dsh-llm in the 0.1.x line) and Node.js 20+.

From a release tarball (no build step, no build authorization):

dsh plugin --profile web add https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure/releases/download/v0.1.0/dsh-plugin-loud-failure-0.1.0.tgz
dsh --profile web --dump-config   # shows a "# == dsh-plugin-loud-failure" layer

From GitHub at a pinned commit: a git install fetches source, so pnpm must be allowed to run this package's prepare script (it runs tsc). The first add fails and prints the exact key to allow (for a pinned commit the key includes the codeload URL); append it to the profile's pnpm-workspace.yaml, then rerun:

dsh plugin --profile web add github:Rhymer-Lcy/dsh-plugin-loud-failure#<commit-sha>

# $DSH_HOME/profiles/web/pnpm-workspace.yaml (key copied from pnpm's message)
allowBuilds:
  "dsh-plugin-loud-failure@https://codeload.github.com/Rhymer-Lcy/dsh-plugin-loud-failure/tar.gz/<commit-sha>": true

Treat that authorization as what it is: code from this repository runs on your machine at install time, outside any agent sandbox. Pin the commit.

From a source checkout, without installing:

git clone https://github.com/Rhymer-Lcy/dsh-plugin-loud-failure.git
cd dsh-plugin-loud-failure && pnpm install && pnpm run build

# overlay.yml
- insert:
    - id: loud-failure
      name: /absolute/path/to/dsh-plugin-loud-failure/lib/index.js

dsh web --patch ./overlay.yml

Uninstall with dsh plugin --profile web remove dsh-plugin-loud-failure.

Configuration

The bundle inserts one row, id: loud-failure, with the schema defaults restated. A patch replaces a row's whole config, so an override in your profile's cordis.patch.yml must restate every key it keeps.

A rule:

Disable one built-in rule and add one of your own:

# $DSH_HOME/profiles/web/cordis.patch.yml
- id: loud-failure
  config:
    builtinRules: true
    rules:
      - id: no-such-file-with-success
        pattern: unused
        action: off
        message: unused
      - id: cuda-oom
        pattern: 'CUDA out of memory|torch\.OutOfMemoryError'
        tools: [bash, pwsh, job_output]
        action: error
        message: 'The GPU ran out of memory; whatever this run was supposed to produce is incomplete. Reduce the batch size or free VRAM, then rerun.'
    shellTools: [bash, pwsh, job_output, terminal_read]
    excerptChars: 240
    scanValue: false

Built-in rules

All built-in rules use when: success and apply to shellTools.

The error rules are anchored so that a mention of the phrase does not fire (grep -r "command not found" logs/ is not a hit); the context rules are deliberately broader because a notice is cheap and the model keeps the original result.

What the model sees

A blocked result, exactly as rendered (the second content block is the untouched original output):

[loud-failure] Tool "bash" reported success, but its output matched a rule (pandoc-missing-character) that indicates a silent failure. dsh-plugin-loud-failure marked this result as an error; do not treat the call as successful.

- pandoc-missing-character (error): pandoc/xelatex dropped one or more glyphs; the produced PDF is missing characters even though the exit code was 0. Rewrite the character (for example write SpO2 instead of a subscript digit) or switch to a font that has it, then rebuild and re-check for this warning.
  excerpt: "pandoc thesis.md -o thesis.pdf --pdf-engine=xelatex\n[WARNING] Missing character: There is no ₂ in font Microsoft YaHei/OT!\n[exit code: 0]"

The original tool output follows unchanged.

A notice attached to a successful result (the result itself is unchanged; this arrives as a user-role message with source {"kind":"plugin","plugin":"loud-failure","form":"notice","summary":"loud-failure: bash output matched numpy-runtime-warning"}):

[loud-failure] Notice for tool "bash": its output matched a rule (numpy-runtime-warning) that often means a silent failure. Verify before relying on this result.

- numpy-runtime-warning (context): NumPy raised a RuntimeWarning about NaN, division by zero, or overflow. Downstream numbers may be NaN or inf while the exit code stays 0. Check the affected arrays before using the result.
  excerpt: "python calc.py\ncalc.py:12: RuntimeWarning: invalid value encountered in divide\n[0.5 nan 0.25]"

Verification

pnpm run check runs typecheck, tests, build, and a pack dry-run; CI runs it on Ubuntu and Windows with Node 20 and 22.

• tests/rules.test.ts covers merging, compilation failures (each with its message), filters, statelessness, excerpt bounds, and every built-in rule against a real signature plus the negative cases (clean output, non-shell tools, already-failed results, substring mentions).
• tests/plugin.test.ts mounts the plugin next to the real ToolRuntime and drives tool calls through the complete pipeline: block with original output preserved, notice with the plugin source, precedence of error over context, tool and when filters, built-in override, scanValue, composition with a downstream tools/post-execute listener, clean unmount, and fail-loud on a bad rule table.
• The bundle was also exercised against @deepseek-ai/dsh@0.1.0-rc.6 on Windows: dsh plugin add from a checkout, from the v0.1.0 release tarball URL, and from github:...#<sha> (with the allowBuilds key) all compose, and --dump-config shows the layer; booting dsh --profile web with an overlay that injects an invalid rule fails at boot with RuleConfigError: rule "bad": invalid pattern raised from the installed plugin's apply, and booting with the shipped config serves the Web UI.

Design notes

• tools/post-execute, not tools/result. tools/result only observes the frozen outcome; the point here is to change what the model sees, which is exactly what post-execute is for (replace content, block with feedback, attach context).
• Block keeps the original output. A blocked result would otherwise erase the evidence the model needs to fix the problem. The header comes first so the model reads the verdict before the noise; the original blocks follow unchanged.
• Two severities, no third. error is for signatures that mean the work did not happen; context is for signatures that mean "check before you trust". Anything softer than that is not worth a token.
• Regular expressions, not a model. The signatures are literal strings emitted by tools; a regex is deterministic, cheap, testable, and auditable in the config dump. Semantics belong to the model, which gets the rule message and the excerpt.
• Fail loud at load. A rule table is configuration; a bad one should stop the boot with the rule id, not degrade into a listener that never fires.
• Waterfall etiquette. A blocking decision short-circuits, as a decision-owning policy listener should; a context decision delegates first and enriches whatever came back, so downstream replacements survive.

Model Experience

Request context and condition

What the model sees

Only when a rule matches. For an error match: the tool result becomes isError: true and its content is one text block rendered as in What the model sees, followed by the original content blocks. For a context-only match: the tool result is unchanged and one user-role notice message with source { kind: 'plugin', plugin: 'loud-failure', form: 'notice', summary } is appended after the recorded tool results of that batch. Rule messages are configuration and appear verbatim; the excerpt is bounded by excerptChars.

Token effect

Conditional. Zero when nothing matches. On a match: the header plus, per matched rule, its message and up to excerptChars characters of excerpt. A block adds only that header; the original blocks that follow are the same bytes the model would have received anyway.

KV Cache effect

Append-only. The plugin contributes no system-prompt section and never rewrites earlier history; a block changes the content of the current tool result before it is first recorded, and a notice is appended after that batch's tool results. Neither invalidates any previously reusable prefix.

Known Limitations and Deferred Work

• Text only. The listener inspects text blocks (and, with scanValue, the JSON of a successful value). A failure that leaves no trace in the output, such as a library returning a negative status that the script never prints, cannot be caught here; the pattern for those is to print the value and let a rule catch it.
• Regular expressions can false-positive. The context rules are broad by design; disable one with action: off if it is noisy in your workload. A blocked result still carries the original output, so a false error costs the model one corrective step, not the information.
• Whole-text scan. Very large tool outputs are scanned once per rule; the shipped shell tools already truncate long output, so this has not been a cost in practice, but there is no size cap yet.
• run_code aggregate output. Code Mode sub-dispatches each pass through post-execute and are inspected individually; the outer run_code result (program logs and return value) is inspected as ordinary text when run_code is listed in shellTools, which it is not by default.
• No per-agent scoping. The listener is registered on the plugin context and sees every agent's calls. Scoping to a subagent preset is a Cordis-level exercise not attempted here.

Development

pnpm install
pnpm run check          # typecheck + test + build + pack dry-run
pnpm run test:watch

Source lives in src/ (rules.ts is pure and unit-tested; index.ts is the Cordis adapter); tests in tests/; lib/ is build output produced by tsc and by the prepare script on git installs.

License

MIT