dsh-tool-chaos

English · 简体中文

Deterministic fault injection and evidence-backed resilience tests for the DeepSeek Harness tool pipeline.

dsh-tool-chaos runs an isolated DSH experiment through baseline → dry-run → fault injection → PASS / FAIL / INCONCLUSIVE. It helps plugin and Agent authors verify retry, fallback, cancellation, policy-denial, and untrusted-result behavior before those paths fail unexpectedly.

Status: 0.1.0-next.0 prerelease. DSH is still in developer preview, so pin versions and review the tested compatibility matrix.

Highlights

• Five DSH-native actions: deny, error, delay, abort, and block.
• Reproducible matching and sampling; no runtime randomness.
• Disabled and dry-run by default, with bounded injections and explicit global-wildcard opt-in.
• JSON and Markdown reports with structured decision events and a three-state verdict.
• Native Cordis plugin, zero-runtime-dependency CLI, TypeScript engine API, and Codex/Agent Skill surfaces.

Quick start

The npm prerelease has not been published yet. The currently available path is a source checkout and a locally built tarball:

git clone https://github.com/cyanseek/dsh-tool-chaos.git
cd dsh-tool-chaos
npm ci
npm run build
npm pack --silent

dsh plugin --profile chaos-demo add ./dsh-tool-chaos-0.1.0-next.0.tgz
dsh --profile chaos-demo --dump-config

The effective config should contain tool-chaos and tool-chaos-reporter. Both are inert after installation: the main plugin is disabled and the reporter is disabled.

Generate a complete dry-run overlay:

node bin/dsh-tool-chaos.mjs init .dsh-chaos/retry.patch.yml --preset retry

Review the generated tool matcher, task, and assertion, then run it against the disposable profile:

node bin/dsh-tool-chaos.mjs run \
  --mode dry-run \
  --profile chaos-demo \
  --patch .dsh-chaos/retry.patch.yml \
  --task "Use web_fetch and recover from one transient failure" \
  --expect "expected stable phrase" \
  --no-install \
  --json

A dry-run must show the intended dryRun: true decision while leaving tool execution unchanged. Run a real experiment only after that match is exact.

Install from a pinned GitHub commit

DSH can install this repository directly:

dsh plugin --profile chaos-demo add \
  github:cyanseek/dsh-tool-chaos#<commit-sha>

pnpm 10+ can intentionally block a Git dependency's prepare script. If it does, review the pinned source and copy the exact package key printed by pnpm into that profile's pnpm-workspace.yaml:

allowBuilds:
  dsh-tool-chaos@https://codeload.github.com/cyanseek/dsh-tool-chaos/tar.gz/<commit-sha>: true

Rerun the install and verify the effective config. This approval allows package code to execute during installation; a reviewed tarball remains the simpler option.

To uninstall and verify removal:

dsh plugin --profile chaos-demo remove dsh-tool-chaos
dsh --profile chaos-demo --dump-config

Configuration

Installed defaults cannot inject a fault:

- id: tool-chaos
  config:
    enabled: false
    dryRun: true
    seed: dsh-tool-chaos
    allowGlobalWildcard: false
    rules: []

A bounded preview rule looks like this:

- id: tool-chaos
  config:
    enabled: true
    dryRun: true
    seed: retry-suite-v1
    allowGlobalWildcard: false
    rules:
      - id: first-github-error
        tool: github_*
        action: error
        scope: root
        argumentsPattern: ''
        probability: 1
        afterMatches: 0
        every: 1
        maxInjections: 1
        delayMs: 1000
        message: deterministic GitHub transport failure

tool is an anchored, case-sensitive glob. argumentsPattern runs against stable key-sorted JSON. DSH patch rows replace a row's complete config, so every overlay must restate all keys it relies on.

Fault actions

Each decision emits tool-chaos/decision. The optional reporter converts it to a stable stderr JSONL boundary prefixed with DSH_TOOL_CHAOS_EVENT.

Evidence and verdicts

The CLI writes a JSON source-of-truth report and a Markdown summary. Reports include configuration hashes, bounded process metadata, parsed chaos decisions, assertions, verdict, and a reproduction command. Raw command output is excluded unless explicitly requested.

Exit code alone is never sufficient evidence of resilience.

CLI

dsh-tool-chaos doctor [--profile name] [--json]
dsh-tool-chaos setup [--profile name] [--source npm-or-path] [--codex] [--json]
dsh-tool-chaos init [path] [--preset retry|timeout|nested|result-block|deny] [--force]
dsh-tool-chaos run --patch file --task prompt [--mode baseline|dry-run|experiment]
                   [--expect regex] [--profile name] [--json]
dsh-tool-chaos install-plugin [--profile name] [--source npm-or-path] [--json]
dsh-tool-chaos install-skill [--global | --project path]
dsh-tool-chaos install-codex-plugin [--dir path] [--json]
dsh-tool-chaos verify [--json]
dsh-tool-chaos prompt [--lang en|zh] <requirement...>
dsh-tool-chaos codex [--cwd path] [--json] <requirement...>

Automation should consume --json instead of scraping human-readable output.

Codex and Agent Skill

The repository contains a Codex marketplace plugin and an open Agent Skill. Install the Codex plugin from a checkout:

codex plugin marketplace add /absolute/path/to/dsh-tool-chaos
codex plugin add dsh-tool-chaos@dsh-tool-chaos

Start a new thread and invoke:

$dsh-chaos-test Prove the retry path survives one deterministic GitHub tool error.

See Codex integration for the source-checkout workflow. npx dsh-tool-chaos ... commands become available only after the npm prerelease is published.

Programmatic engine

The dsh-tool-chaos/engine export has no Cordis or DSH runtime import:

import { ChaosEngine, resolveConfig } from 'dsh-tool-chaos/engine'

const engine = new ChaosEngine(resolveConfig({
  enabled: true,
  dryRun: false,
  seed: 'ci-retry-v1',
  rules: [{
    id: 'first-web-error',
    tool: 'web_fetch',
    action: 'error',
    maxInjections: 1,
  }],
}))

const decision = engine.decide('execute', {
  callId: 'call-1',
  rootCallId: 'call-1',
  name: 'web_fetch',
  arguments: { url: 'https://example.test' },
})

See the engine example.

Safety boundaries

• Use disposable profiles and synthetic or read-only test inputs.
• Keep dryRun: true until the exact intended match is visible.
• Exact global * requires allowGlobalWildcard: true.
• Every rule injects at most once by default.
• The runtime plugin does not rewrite arguments, read files, or access the network.
• abort is cooperative; a same-process tool that ignores AbortSignal cannot be hard-killed.
• Never represent skipped or unobservable checks as success.

Production/shared profiles, external mutations, credentials, broad targeting, or privilege changes require separate safeguards and explicit authorization.

Compatibility

Evidence was collected on August 14, 2026 with isolated profiles and synthetic read-only input. See Compatibility for the precise matrix and limitations.

Documentation

• CLI and machine API
• Report schema
• Architecture
• Codex integration
• Compatibility
• Contributing
• Security
• Support
• Changelog

Development

npm ci
npm run sync:skill
npm run check

Compatibility claims must be backed by the exact environment and command that passed.

License

MIT