mirror of
https://github.com/hansjone/dsh-im-ops.git
synced 2026-10-08 22:00:50 +08:00
fix(im): preserve cancellation during image fallback
This commit is contained in:
parent
c7679d86a1
commit
9d2e9c800b
5 changed files with 41 additions and 10 deletions
|
|
@ -10,6 +10,12 @@ This file records the notable changes in each dsh-im release. Its format follows
|
|||
|
||||
- 非视觉模型收到图片时不再直接报错丢图:宿主以 `MODEL_DOES_NOT_SUPPORT_IMAGES` 拒绝带图片的 prompt 后,自动把同一批图片字节按入站文件管线落盘到 Session 工作区,并以"原文本 + 工具分析指引 + `<dsh_im_files>` 清单"的纯文本 prompt 复用同一 rpcId 重试一次,使非视觉模型仍可通过 run_code/pwsh 等工具识图;视觉模型与文件消息行为不变,其余图片错误仍按原样提示。
|
||||
Sending an image to a non-vision model no longer fails outright: when the Host rejects an image-bearing prompt with `MODEL_DOES_NOT_SUPPORT_IMAGES`, the same image bytes are automatically staged into the Session workspace through the inbound-file pipeline and retried once as a text-only prompt (original text plus tool-analysis guidance and the `<dsh_im_files>` manifest) under the same rpcId, so non-vision models can still inspect images via tools such as run_code/pwsh. Vision models and file messages are unchanged, and other image errors keep their existing messages.
|
||||
|
||||
### Fixed / 修复
|
||||
|
||||
- 非视觉模型图片回退在文件落盘阶段收到取消信号时,现在会保留调用方的取消原因并停止处理,不再误报模型不支持图片。
|
||||
When image fallback for a non-vision model is cancelled while staging files, it now preserves the caller's cancellation reason and stops instead of reporting that the model does not support images.
|
||||
|
||||
## [4.5.0] - 2026-09-01
|
||||
|
||||
### Added / 新增
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# 入站图片非视觉模型文件回退方案
|
||||
|
||||
> 状态:已实施(P0);Windows 本机全量回归通过,真机验收待各渠道补做
|
||||
> 状态:已实施(P0);Ubuntu CI、包产物验证与真实宿主非视觉模型端到端验收通过
|
||||
>
|
||||
> 日期:2026-06-14
|
||||
> 日期:2026-09-01
|
||||
>
|
||||
> 关联:[出站图片原生呈现落地方案](./出站图片原生呈现落地方案.md)、[渠道原生能力建设方案](./渠道原生能力建设方案.md)
|
||||
|
||||
|
|
@ -21,14 +21,15 @@ IM 对话中,非视觉模型收到图片时当前是**准入即硬失败**:
|
|||
|
||||
与出站方案同样的原则:复用现有 `inbound-file` 生命周期(落盘、清单、turn 结束清理),不新增配置开关、不新增产物类型。
|
||||
|
||||
### 1.1 实施结果(2026-06-14)
|
||||
### 1.1 实施结果(2026-09-01)
|
||||
|
||||
本方案已按上述最小边界落地:
|
||||
|
||||
- `src/channels/shared/image-prompt.mjs` 新增 `IMAGE_FILE_FALLBACK_PROMPT`(模型侧指引,含英文翻译)、`imageFileSourcesFromContent()`(图片内容块 → 文件源,含扩展名映射与文件名清洗)、`contentWithoutImages()`、`isModelImageRejection()`。
|
||||
- `src/channels/shared/harness-client.mjs` 的 `ask()` 将原入站文件落盘逻辑抽为 `#stageWorkspaceFiles()`;`session.prompt` 被以 `MODEL_DOES_NOT_SUPPORT_IMAGES` 拒绝且 content 含图片块时,把同一批字节经该管线落盘、重建纯文本 prompt(原文本 + 指引 + 合并后的单个 `<dsh_im_files>` 清单)并复用同一 `promptRpcId` 重试一次;重试不再回退,落盘失败时保留原始错误文案。staged 批次统一进入既有 turn 结束清理。
|
||||
- `test/image-fallback.test.mjs` 新增 8 个用例:转换/判定辅助函数、完整回退(复用 rpcId、字节一致、清单正确、turn 后清理)、混合消息合并清单、非模型原因不回退、落盘失败保留原错误、重试失败不再重试、纯文本路径不变。
|
||||
- `test/image-fallback.test.mjs` 新增 9 个用例:转换/判定辅助函数、完整回退(复用 rpcId、字节一致、清单正确、turn 后清理)、混合消息合并清单、非模型原因不回退、落盘失败保留原错误、落盘期间保留调用方取消语义、重试失败不再重试、纯文本路径不变。
|
||||
- 回归:Windows 本机以逐文件直跑方式执行全部 137 个测试文件,失败集与改动前基线(git worktree 对照)完全一致(23 个均为仓库既有的 Windows/沙箱环境性失败,CI 在 ubuntu 上通过);`npm run build` 通过;`verify-package` 的可执行位检查在 Windows 上为既有环境性失败。
|
||||
- 端到端:在真实 DSH 宿主进程内通过生产 `apiProxy` 与 `fileIngressExecutor` 链路向文本模型注入入站 PNG,验证了宿主拒绝、图片落盘、纯文本重试与模型工具分析的完整路径。
|
||||
|
||||
## 2. 现状与根因
|
||||
|
||||
|
|
@ -176,14 +177,15 @@ export function imageFileSourcesFromContent(content) {
|
|||
1. 首次 `session.prompt` 返回 `attachment-error/MODEL_DOES_NOT_SUPPORT_IMAGES` → 断言:调用了第二次 prompt;第二次 content 无 image part;含 `<dsh_im_files>` 清单与回退提示文本;文件以正确扩展名写入 executor 收到的 sources;仅重试一次。
|
||||
2. 混合消息(图片+文件)→ 断言只出现**一个**合并后的 `<dsh_im_files>` 块。
|
||||
3. 其他拒绝原因(`IMAGE_TOO_LARGE`、`TOO_MANY_IMAGES` 等)→ 不回退,原错误冒泡,用户文案不变。
|
||||
4. 重试再失败(如 executor 抛 `inbound-file-ingress-unavailable`)→ 报错且 staged 文件被清理。
|
||||
5. 纯文本/纯文件消息 → 不触发任何回退路径(回归)。
|
||||
6. turn 正常结束后 staged 图片文件被 cleanup(复用 `test/inbound-file.test.mjs` 的断言模式)。
|
||||
4. 落盘失败时保留原始模型拒绝错误;落盘期间取消时保留调用方的取消原因。
|
||||
5. 回退 prompt 再次失败 → 仅报错一次,不继续重试。
|
||||
6. 纯文本/纯文件消息 → 不触发任何回退路径(回归)。
|
||||
7. turn 正常结束后 staged 图片文件被 cleanup(复用 `test/inbound-file.test.mjs` 的断言模式)。
|
||||
|
||||
### 6.2 回归与手工验收
|
||||
|
||||
- `npm run check` 全量通过。
|
||||
- 真机:绑定非视觉模型的会话发一张图 → 收到 Agent 基于工具分析的回复而非报错;视觉模型会话发图 → 行为与现状一致(原生多模态);发 zip → 行为不变。
|
||||
- 真实宿主端到端:文本模型收到图片后走完“模态拒绝 → 工作区文件落盘 → 纯文本重试 → 工具分析”链路并正常回复;视觉模型与普通文件消息的原有行为保持不变。
|
||||
|
||||
## 7. 上游反馈(非本仓库范围)
|
||||
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -1415,7 +1415,7 @@ export class HarnessClient {
|
|||
try {
|
||||
stagedImages = await this.#stageWorkspaceFiles(sessionId, imageSources, signal);
|
||||
} catch (stagingError) {
|
||||
if (signal?.aborted) throw error;
|
||||
if (signal?.aborted) throw signal.reason ?? stagingError;
|
||||
console.warn(
|
||||
`[${this.#logPrefix}] unable to restage rejected images as workspace files:`,
|
||||
stagingError?.message ?? String(stagingError),
|
||||
|
|
|
|||
|
|
@ -235,6 +235,29 @@ test('a failed restage keeps the original model-rejection error for the user', a
|
|||
assert.equal(promptCalls.length, 1);
|
||||
});
|
||||
|
||||
test('caller cancellation wins while rejected images are being restaged', async (t) => {
|
||||
const root = await workspace(t);
|
||||
const controller = new AbortController();
|
||||
const cancellation = new Error('caller cancelled image fallback');
|
||||
const { client, promptCalls } = scriptedClient({
|
||||
workspaceRoot: root,
|
||||
fileIngressExecutor: async () => {
|
||||
controller.abort(cancellation);
|
||||
throw cancellation;
|
||||
},
|
||||
onPrompt: (attempt) => (attempt === 1 ? Promise.reject(modelImageRejection()) : Promise.resolve({})),
|
||||
});
|
||||
|
||||
await assert.rejects(
|
||||
client.ask('session-fallback', imageContent(), {
|
||||
timeoutMs: 3_000,
|
||||
signal: controller.signal,
|
||||
}),
|
||||
(error) => error === cancellation,
|
||||
);
|
||||
assert.equal(promptCalls.length, 1);
|
||||
});
|
||||
|
||||
test('the fallback retry itself is never retried again', async (t) => {
|
||||
const root = await workspace(t);
|
||||
const { client, promptCalls } = scriptedClient({
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue