dsh-tool-eyes

Local vision "eyes" for DeepSeek Harness (DSH).

Give your text-only agent eyes with two model-facing tools:

• screen — capture the screen (or describe an existing image file) through a local OpenAI-compatible vision endpoint (llama.cpp with --mmproj, LM Studio, Ollama, ...) and return the vision model's text description.
• ocr — extract ALL text verbatim with the Windows built-in OCR engine: zero model, zero GPU, zero cloud, milliseconds.

screen  =  screenshot / image  ->  local VLM  ->  text description (understanding)
ocr     =  screenshot / image  ->  Windows OCR -> verbatim text          (extraction)

Why

DeepSeek's chat-completions line is text-only. Instead of switching your whole conversation to a vision model, keep the text brain and add eyes as tools:

• Private by default — point screen at a local endpoint and images never leave your machine.
• Cheap — a 0.8B–4B local VLM is plenty for describing screens; ocr costs nothing at all.
• Honest by default — the screen prompt tells the VLM to describe only what is visible and never guess app/game/character names unless confirmed by on-screen text (this measurably cuts small-model name hallucination).

Requirements

• Windows 10/11 (the ocr tool uses WinRT OCR; screen uses .NET for capture)
• Node.js >= 22.19, DeepSeek Harness >= 0.1.0-rc.6
• screen additionally needs any OpenAI-compatible VLM endpoint, e.g.:
  • llama.cpp: llama-server -m model.gguf --mmproj mmproj.gguf --port 1235
  • LM Studio (loaded vision model), Ollama, or any OpenAI-compatible gateway

Install

This package is published on GitHub only (not on npm).

dsh plugin --profile web add https://github.com/go-farther-and-farther/dsh-tool-eyes

Then restart dsh web. The screen and ocr tools appear in the agent's toolkit automatically.

Manual install (offline / from source)

Copy this package into the profile's node_modules, then register it in $DSH_HOME/profiles/<profile>/cordis.patch.yml:

- insert:
    - id: tool-eyes
      name: 'dsh-tool-eyes'
      config:
        baseUrl: http://127.0.0.1:1235/v1
        model: ''
        timeoutMs: 180000

Configuration

Plugin config (all optional):

Override in your profile's cordis.patch.yml (id-targeted):

- id: tool-eyes
  name: 'dsh-tool-eyes'
  config:
    baseUrl: http://127.0.0.1:1235/v1
    model: qwen3.5-4b
    timeoutMs: 120000

Usage

In a conversation, the agent can now:

• screen — "what is on my screen?", "describe this image file", with an optional prompt to focus on a region or detail.
• ocr — "read all the text on screen", "transcribe this error dialog".

Both accept an optional image path; without it they capture the screen. The bundled PowerShell scripts can also be run standalone:

powershell -NoProfile -ExecutionPolicy Bypass -File lib\capture.ps1 -Prompt "..." -BaseUrl http://127.0.0.1:1235/v1
powershell -NoProfile -ExecutionPolicy Bypass -File lib\ocr.ps1 -Image C:\path\x.png

Privacy

• ocr is fully local (WinRT OCR, no network).
• screen sends the captured image to the configured baseUrl. Point it at a local endpoint (llama.cpp / LM Studio / Ollama) to keep images on your machine.

Related

• dsh-vision-proxy — automatic transcription of attached images in the chat input (Chatbox-style), so you don't need to give file paths. Pairs well with this plugin.

Development

npm test    # node --test tests/

License

MIT