dsh-fetch-data

🌐 English | 中文

Structured data extractor for DeepSeek Harness — intercepts a page's real data APIs (XHR/fetch JSON) and returns precise fields (comment like-counts, rankings, prices, tables) that text extractors lose when flattening HTML.

read_url answers "what does this page say?" — fetch_data answers "what are the exact numbers/data behind it?"

Why it exists

Real example: on a Xiaoheihe post, read_url returned "60125" with no way to know if it meant 60 likes + 125 favorites or anything else. fetch_data intercepts /bbs/app/link/tree and returns {user, up, content} — unambiguous.

Tool

fetch_data(url, api?, fields?, maxItems?) — capture the page's data APIs and extract fields

Two modes:

1. Structure mode (no fields) — returns the auto-picked endpoint's 2-level structure plus a menu of all JSON APIs (path · size · array?), so the model can re-call with api= to pin one:

   
   页面 25 个 JSON 接口
   选中: /x/web-interface/ranking/v2 (137763B)
   结构: { code: number, message: string, ttl: number, data: { note: string, list: [100] { aid: number, ... } } }

接口清单（可传 api=<路径> 指定其中一个再提取字段）: /x/web-interface/nav · 249B /x/vip/ads/materials · 1140B · 含数组 ...

2. **Field mode** (with `fields`) — extracts exact values, arrays truncated at `maxItems`:

data.list[].title (前5条): 用MC还原《神的随波逐流》 【B萌应援】 WasteTheFallen丨首曝PV＆实机演示：凝视深渊，人性渐泯 ... data.list[].stat.view (前5条): 2428730 10326863 ...

## Auto-pick logic

Ranks captured JSON responses: **largest one containing an array wins** (data endpoints are usually big and array-bearing; tracking/config endpoints are small). Falls back to the largest JSON. Manual `api=` overrides.

## Real-world verification (2026-08-16, v0.1.3)

14-site sweep: **12 OK / 2 static-site expected errors / 0 crashes** — driven by `multi-site.mjs` (committed). Auto-scroll (lazy-load capture) + JSONP parsing verified live.

| Site | Result |
|---|---|
| Bilibili popular ranking | ✅ auto-picked `/x/web-interface/ranking/v2` (139KB, `list[100]`), extracted titles + play counts, 1:1 attributed |
| Juejin feed | ✅ auto-picked `/recommend_api/v1/article/recommend_all_feed`, extracted article titles |
| Weibo | ✅ captured `/ajax/feed/hottimeline` (240KB) + `/ajax/statuses/config` (745KB, oversized config edge case) |
| QQ news | ✅ captured `/getQNChannels` (304KB, 36 APIs) |
| Douban | ✅ captured `/rexxar/api/v2/search/hots` |
| Taobao | ✅ captured 20 JSON APIs incl. mtop config (824KB, oversized-config edge case) |
| JD | ✅ captured `/wp-json/news/list` + `/category/get` |
| Zhihu / Baidu / CSDN / Netease / Xiaoheihe | ✅ captured their JSON endpoints (some are config/menu APIs — use structure mode to pick the data one) |
| example.com / ruanyifeng.com (static) | ✅ clear "no JSON API captured" error, no crash |

- **18 zero-dep assertions** (field-path extractor, pick logic, structure summary, static-asset filter, truncation precision, description-length guard, JSONP parsing) + **22 live-interception assertions** (incl. scroll-triggered lazy capture + JSONP extraction) all green.

## Why it saves tokens

- **Two-phase**: structure mode returns a small "menu" (not the 137KB payload); field mode returns only the requested columns — never dumps the full JSON;
- **Compact fixed cost**: tool description trimmed to ~300 chars (the one thing sent on every call); static schema (KV-cache friendly);
- **Sorted menu**: array-bearing + biggest endpoints listed first, so the model finds the data API immediately; static-asset JSON (Bilibili `/bfs/svg-next/...`) filtered out and the count shown inline;
- Sizes rendered as `136KB` not `139070B`; arrays truncated at `maxItems`; values capped at 200 chars;
- Compact text render; clear one-line errors.

## Architecture (DSH-aligned)

- Browser singleton launched once, **closed via `ctx.effect`** on unload (temporal composability);
- Fresh browser context per call — no cookie/state leakage across calls (avoids anti-bot flakiness);
- `domcontentloaded` + settle wait instead of `networkidle` (heartbeat-polling sites never idle);
- **Auto-scroll** triggers lazy-loaded data APIs (feeds / infinite lists); **JSONP** responses auto-unwrapped;
- Cooperative timeout: `timeoutMs` + `exec.signal`;
- Zero runtime deps beyond Node built-ins; **playwright is the required engine** (it's the interception layer, not an optional enhancement).

## Install

```bash
# playwright is REQUIRED for this plugin (network interception core)
cd <DSH profile dir>
npm i playwright && npx playwright install chromium

# add the plugin
dsh plugin --profile web add github:2672243194/dsh-fetch-data

Boundaries

• Login-walled APIs are not accessible (same as read_url);
• Each site has its own endpoint structure — use the structure-mode menu to discover it;
• Auto-scroll + JSONP cover most lazy-loading/JS-delivered data; remaining gaps (login walls, SSR-only pages) return a clear error, never a guess.

Support

If dsh-fetch-data helps you, give it a ⭐ Star on GitHub. Free and open source (MIT); star count is how I decide where to keep investing.

License

MIT