dsh-plugin-vision

Auxiliary vision for DeepSeek Harness (dsh): analyze images through an external OpenAI-compatible vision endpoint and get a text answer back. Works with any main model — including DeepSeek, whose own API is text-only.

Why

dsh's built-in read_image tool injects the image into the main model's context, which requires the main model to declare image input. The DeepSeek adapter does not (inputModalities: ['text']), so read_image refuses to run with it. This plugin takes the complementary auxiliary vision route: it sends the image to a separate vision model and returns the answer as text. No main-model vision support needed.

How it works

1. Read the image — a local file path or an http(s) URL (size-capped, magic-byte MIME sniffing).
2. Encode it as an inline base64 data URL.
3. POST an OpenAI-compatible chat/completions request with [{type:text}, {type:image_url}] content.
4. Return the vision model's answer (falls back to reasoning_content for thinking models).
5. Transcription cache (new in v0.1.1): each (image, question) pair is transcribed once — keyed by image content sha256 (deduplicated across paste / read_image archive copies / resumed history) plus question sha256 — and later requests reuse the cached text. The vision endpoint sampling temperature is pinned to 0 (temperature) for deterministic output. The result:
   • once an image is in the conversation history, later turns no longer call the vision endpoint again → time-to-first-token (TTFT) stays flat;
   • the transcribed text never changes → the main model's (DeepSeek) prompt-prefix cache is not broken by transcription variance, so the cache hit rate no longer decays.

Two capabilities

1. vision_analyze tool

Send an image (local path or URL) to the configured vision endpoint and get a text answer — works with any main model, including text-only DeepSeek.

2. Image-attachment transcription (pasting images in the Web UI)

The Web UI accepts pasted/dropped images, but the DeepSeek adapter rejects image content (UNSUPPORTED_CONTENT). This plugin hooks agent/pre-step — the documented seam for replacing the messages that enter a step — and, when a user message carries image blocks, calls the vision endpoint to describe them, then swaps each image block for a [User-attached image description] text block before the request reaches the model. DeepSeek only ever sees text, so image attachments just work. When the active model route declares image input, transcription is skipped and native vision takes over.

Toggle with attachImages (default true). Transcription failures degrade to an explicit note instead of blocking the turn.

attachMode controls how the vision model is asked about attached images:

• auto (default): when the user wrote text with the image, that text is passed to the vision model verbatim as its question — "who is this?", "translate the text", "what is wrong on this page?" are answered directly, plus a one-line image summary for follow-ups. Without user text, a generic description is produced.
• describe: always use the generic description prompt, ignoring user text.

3. deepseek-vision provider route (pasting images in the Web UI, root fix)

The Web UI's upload preflight rejects images when the selected model does not declare image input — so pasted images never even reach the agent on the plain DeepSeek route. This plugin registers a deepseek-vision provider: a DeepSeekAdapter subclass that declares image input (preflight passes) and transcribes attached images to text at request time before delegating to the real DeepSeek chat-completions endpoint. The main model is still DeepSeek — same endpoint, same key, same models. Select "DeepSeek (vision via plugin)" in the model picker, then paste/drop images as usual.

Toggle with deepseekVision.enabled (default true); the provider id is deepseekVision.providerId (default deepseek-vision). Transcription goes through the content-hash cache, so once an image is in the history every later turn reuses the cached text instead of calling the vision endpoint again.

No-second-port setup: load the plugin into your existing GUI

The vision API key resolves through the harness credentials service first (the Web UI's stored keys) and falls back to the environment — no export needed in the GUI. Write the plugin into the home-level user patch, which applies to every profile (including the web GUI you already run):

# $DSH_HOME/cordis.patch.yml
- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Store your vision API key in credentials (or export VISION_API_KEY), then restart the GUI once. Use the built lib/index.js — the published CLI loads .ts entries but cannot resolve their .js-suffixed sibling imports from src/.

4. Image archive (auto-save pasted images + index.json)

Every pasted image is automatically saved to ~/.dsh/image-archive/ with a date-numbered name (2026-08-14_120331_001.png), recorded in an index.json manifest (path, sha256, size, source, optional note), and its location is annotated for the model ([图片已存档: …]). Two tools complete the flow:

• image_archive — the agent archives an important image (user preferences, receipts, key data) into a named folder with an optional note: it saves <archiveDir>/<folder>/<name-or-date>.png and updates index.json.
• image_archive_find — search the manifest by name, folder, or note.

Configure with archive.enabled (default true) and archive.dir (default ~/.dsh/image-archive). Deduplicated by attachment id.

Try it

Install the bundle into any profile (the prepare script builds on install):

dsh plugin --profile web add github:Tianbaidi/dsh-plugin-vision

Store the vision API key in your credentials or environment (ALIBABA_CODING_PLAN_API_KEY), then restart the GUI. Paste/drop an image and ask — or use the vision_analyze tool directly.

Prefer a dev overlay instead? Point it at your local checkout:

- insert:
    - id: vision
      name: 'file:///<path-to-plugin>/lib/index.js'

Windows note: plugin paths in overlays must be file:// URLs (file:///D:/...%20...), not bare D:/... paths — the ESM loader rejects the latter as scheme d:.

Configuration

Any OpenAI-compatible vision endpoint works. The defaults are deliberately empty so no provider is assumed; pick one:

Configure per deployment (e.g., your profile's cordis.patch.yml or the plugin row's config):

- id: vision
  name: dsh-plugin-vision
  config:
    baseUrl: https://open.bigmodel.cn/api/paas/v4
    apiKeyEnv: VISION_API_KEY
    model: glm-4.6v-flash
    timeoutMs: 120000

Develop

pnpm install        # installs the published @deepseek-ai peer packages
pnpm typecheck
pnpm test           # 43 vitest cases: MIME sniffing, payload, parsing, source loading, execute, transcription cache

Changelog

v0.1.1 (2026-08-16) — Major fix: TTFT and cache hit rate

Two performance issues in the deepseek-vision provider, located from real session telemetry:

• TTFT inflation: once an image entered the conversation history, every request re-transcribed it synchronously (measured TTFT degraded from ~1s to 22–69s). Added transcriptionCache (image content sha256 + question sha256, persisted to ~/.dsh/vision-transcription-cache.json): each (image, question) pair is transcribed once, later requests reuse the cached text.
• Cache hit rate decay: the non-deterministic transcription text permanently broke the main model's prompt-prefix cache from the image position onward (measured hit rate decayed from 99.8% to 91.1% and kept falling). Added temperature (default 0) and optional seed for deterministic output.
• Added 5 tests covering cache hits, content-hash keying (paste vs read_image archive copy), persistence, eviction, and deterministic payloads.

v0.1.0

Initial release: vision_analyze / vision_reask / image_archive tools, image-attachment transcription, deepseek-vision provider route, image archive.

Known limitations

• Image sources are file paths (resolved against the harness cwd) or plain http(s) URLs. Remote URLs are fetched directly without SSRF hardening — restrict this tool to trusted networks if you use it with URL inputs.
• The image is sent to the configured endpoint as-is; oversized images are rejected rather than downscaled (no Pillow dependency). Compress first for very large screenshots.
• Token costs of the vision call are charged to the configured endpoint's plan.
• The transcription cache is keyed by image content sha256 + question sha256: a new question about the same image triggers one fresh vision call (which is then cached). To force a refresh, remove the matching entry from (or delete) ~/.dsh/vision-transcription-cache.json.
• Once an image is in the conversation history (pasted by the user, a read_image tool result, or a resumed session), every later request carries it; the cache ensures the vision endpoint is only called on the first occurrence, but the transcribed text still occupies tokens in the history.

Publish

This project is a bundle (dsh.bundle.patch). Install with dsh plugin add, share on GitHub with the dsh-plugin topic, or npm publish. See PUBLISH.md in the companion scaffold repo for the full checklist.

License

MIT