mirror of
https://github.com/hansjone/oclaw.git
synced 2026-10-09 03:30:48 +08:00
feat(chat): image specialist legacy lane, Responses fixes, ACL + UI attachments
- Image expert: DashScope-style /chat/completions via image_legacy_client; early exit in direct_loop when skill_binding_role is image; shared placeholder helper; docs/IMAGE_SPECIALIST_LANE.md. - Strict attachment ACL: link_attachment_acl on assistant chat_message rows (sqlite_store); chat attachment rate limit when user_id empty; admin chat tests updated. - Admin chat UI: aggregate bubbles render assistant_text attachments (image_ref); WS expand path. - turn_runner: persisted_chat_attachments_nonempty for final_msg selection. - OpenAI Responses transport + agent_messages/agent_core_attempt adjustments; env docs and tests. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
parent
d1bcc4debe
commit
faeb067856
21 changed files with 2516 additions and 331 deletions
|
|
@ -276,18 +276,25 @@ AIA_REPLAY_REASONING_SIGNATURE_POLICY=
|
||||||
OPENAI_IMAGE_MODEL=
|
OPENAI_IMAGE_MODEL=
|
||||||
OPENAI_AUDIO_TRANSCRIPTION_MODEL=
|
OPENAI_AUDIO_TRANSCRIPTION_MODEL=
|
||||||
|
|
||||||
# query_image_attachment / send_ocr_image_messages(三项必填,无缺省模型 id)
|
# query_image_attachment、OCR 降级 send_ocr_image_messages — 仅用 AIA_OCR_*(与图片专家互不继承)
|
||||||
AIA_OCR_MODEL=
|
AIA_OCR_MODEL=
|
||||||
AIA_OCR_BASE_URL=
|
AIA_OCR_BASE_URL=
|
||||||
AIA_OCR_API_KEY=
|
AIA_OCR_API_KEY=
|
||||||
AIA_OCR_CHAT_ENDPOINT=
|
AIA_OCR_CHAT_ENDPOINT=
|
||||||
|
|
||||||
|
# specialist=image → send_legacy_image_messages — 仅用 AIA_IMAGE_EXPERT_*(禁止与 AIA_OCR_* 混用通道)
|
||||||
|
AIA_IMAGE_EXPERT_MODEL=
|
||||||
|
AIA_IMAGE_EXPERT_BASE_URL=
|
||||||
|
AIA_IMAGE_EXPERT_API_KEY=
|
||||||
|
AIA_IMAGE_EXPERT_CHAT_ENDPOINT=
|
||||||
|
|
||||||
|
# Optional; image expert top-level kwargs (parity with DashScope multimodal-gen samples)
|
||||||
|
DASHSCOPE_IMAGE_STREAM=
|
||||||
DASHSCOPE_IMAGE_N=
|
DASHSCOPE_IMAGE_N=
|
||||||
DASHSCOPE_IMAGE_WATERMARK=
|
DASHSCOPE_IMAGE_WATERMARK=
|
||||||
DASHSCOPE_IMAGE_NEGATIVE_PROMPT=
|
DASHSCOPE_IMAGE_NEGATIVE_PROMPT=
|
||||||
DASHSCOPE_IMAGE_PROMPT_EXTEND=
|
DASHSCOPE_IMAGE_PROMPT_EXTEND=
|
||||||
DASHSCOPE_IMAGE_SIZE=
|
DASHSCOPE_IMAGE_SIZE=
|
||||||
DASHSCOPE_IMAGE_MODEL=
|
|
||||||
|
|
||||||
CLOUDFLARE_ACCOUNT_ID=
|
CLOUDFLARE_ACCOUNT_ID=
|
||||||
CLOUDFLARE_API_TOKEN=
|
CLOUDFLARE_API_TOKEN=
|
||||||
|
|
|
||||||
|
|
@ -308,31 +308,68 @@
|
||||||
- 作用:Gemini OpenAI 兼容下 tools 非流式开关
|
- 作用:Gemini OpenAI 兼容下 tools 非流式开关
|
||||||
- 生效:`oclaw/platform/llm/chat_models.py`
|
- 生效:`oclaw/platform/llm/chat_models.py`
|
||||||
|
|
||||||
## OCR / 看多模态专线
|
## OCR / 看图工具与降级专线(``AIA_OCR_*``)
|
||||||
|
|
||||||
用于工具 **`query_image_attachment`**、纯文本模型降级 OCR、**图片专家**单次看图与底层 **`send_ocr_image_messages`**:向 **OpenAI-compatible** 网关发送带图 `messages`(通常为 `POST …/chat/completions`)。旧版 DashScope 形态载荷见 **`send_legacy_image_messages`**(一般不再用于图片专家)。与主聊天模型配置的 Key/Base URL **互不继承**。
|
仅用于:**`query_image_attachment`**、OpenAI-compatible **`send_ocr_image_messages`**、以及 **`openai_chat_completions`** 在纯文本模型上的多模态→OCR 降级。规范为 OpenAI **`image_url` + `messages`** Chat Completions 形态。**与下方「图片专家」专线互相独立**(不配齐不会自动复用另一端)。
|
||||||
|
|
||||||
- `AIA_OCR_BASE_URL`
|
- `AIA_OCR_BASE_URL`
|
||||||
- 默认:无(必填,否则工具判为未配置)
|
- 默认:无(必填,否则工具判为未配置)
|
||||||
- 作用:多模态网关根地址(例如 `https://api.example.com/v1`;实际请求为 `BASE_URL` + `CHAT_ENDPOINT`)
|
- 作用:OCR / 看图工具网关根地址(实际请求为 `BASE_URL` + `CHAT_ENDPOINT`)
|
||||||
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
||||||
|
|
||||||
- `AIA_OCR_API_KEY`
|
- `AIA_OCR_API_KEY`
|
||||||
- 默认:无(必填)
|
- 默认:无(必填)
|
||||||
- 作用:上述网关的 Bearer API Key
|
- 作用:上述网关 Bearer API Key
|
||||||
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
||||||
|
|
||||||
- `AIA_OCR_MODEL`
|
- `AIA_OCR_MODEL`
|
||||||
- 默认:无(必填,**不提供默认模型 id**)
|
- 默认:无(必填,**不提供默认模型 id**)
|
||||||
- 作用:网关上支持图+文的多模态模型名(由你的厂商决定)
|
- 作用:支持 OCR/描述的 vision 模型 id
|
||||||
- 生效:`oclaw/platform/llm/image_ocr_client.py`,`oclaw/runtime/agents/specialist_agent.py`
|
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
||||||
|
|
||||||
- `AIA_OCR_CHAT_ENDPOINT`
|
- `AIA_OCR_CHAT_ENDPOINT`
|
||||||
- 默认:`/chat/completions`
|
- 默认:`/chat/completions`
|
||||||
- 作用:相对 `AIA_OCR_BASE_URL` 的路径
|
- 作用:相对 `AIA_OCR_BASE_URL` 的路径
|
||||||
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
- 生效:`oclaw/platform/llm/image_ocr_client.py`
|
||||||
|
|
||||||
**动态参数:** 代码中也可在调用 `send_ocr_image_messages` / `send_legacy_image_messages` 时传入 `base_url` / `api_key` / `model`,优先级高于环境变量(见 `vision_llm_backend_status` 仅在未传参时使用 env 判断「是否已配置」)。
|
**动态参数:** 调用 `send_ocr_image_messages` 时传入 `base_url` / `api_key` / `model` 优先于上述环境变量;`vision_llm_backend_status` 仅检查 `AIA_OCR_*` 是否在未显式传参时可用。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 图片专家专线(``AIA_IMAGE_EXPERT_*``)
|
||||||
|
|
||||||
|
**路由 specialist=`image`** 时由 **`send_legacy_image_messages`** 调用;可走 native `{"image"}`/`{"text"}` 或 compatible-mode **`image_url` + `text` 块状**载荷。**不使用 `AIA_OCR_*`**,也不在图片专家链路继承主会话模型的 Base URL/API Key。
|
||||||
|
|
||||||
|
- `AIA_IMAGE_EXPERT_BASE_URL`
|
||||||
|
- 默认:无(必填,除非在代码中为 `send_legacy_image_messages(..., base_url=...)` 传入)
|
||||||
|
- 作用:图片专家 HTTP 网关根路径(兼容 OpenAI multimodal 时常见 `https://dashscope.aliyuncs.com/compatible-mode/v1` 等)。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
|
- `AIA_IMAGE_EXPERT_API_KEY`
|
||||||
|
- 默认:无(必填,除非显式传 `api_key`)
|
||||||
|
- 作用:Bearer API Key(可与百炼「图片/多模态」文档中的 Key 一致;与 OCR Key **可不同**)。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
|
- `AIA_IMAGE_EXPERT_MODEL`
|
||||||
|
- 默认:无(在用户界面已选会话模型时,`send_legacy_image_messages` **优先使用该模型的 `model` id**;仅当会话未带模型信息时才读本变量作补全)。
|
||||||
|
- 作用:无 UI 会话模型/`model=` 可传时的专家默认模型 id(与 OCR 所用 VL **无需相同**)。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
|
- `AIA_IMAGE_EXPERT_CHAT_ENDPOINT`
|
||||||
|
- 默认:`/chat/completions`
|
||||||
|
- 作用:相对 `AIA_IMAGE_EXPERT_BASE_URL` 的路径。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
|
- `AIA_IMAGE_EXPERT_REQUEST_EXTRA`
|
||||||
|
- 默认:无
|
||||||
|
- 别名:`AIA_LEGACY_IMAGE_REQUEST_EXTRA`(旧名仍可读)。
|
||||||
|
- 作用:发往图片专家请求的**顶层附加 JSON**。顶层 `model` / `messages` 仍由运行时代码覆盖。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
|
- `DASHSCOPE_IMAGE_STREAM` / `DASHSCOPE_IMAGE_N` / `DASHSCOPE_IMAGE_WATERMARK` / `DASHSCOPE_IMAGE_NEGATIVE_PROMPT` / `DASHSCOPE_IMAGE_PROMPT_EXTEND` / `DASHSCOPE_IMAGE_SIZE`
|
||||||
|
- 默认:均未设置则不附加对应字段。
|
||||||
|
- 作用:与 SDK 示例关键词对齐的零散变量;可被 `AIA_IMAGE_EXPERT_REQUEST_EXTRA`(或别名 JSON)中与同名键覆盖。
|
||||||
|
- 生效:`oclaw/platform/llm/image_legacy_client.py`
|
||||||
|
|
||||||
> **说明:** 历史上曾使用 `AIA_IMAGE_BASE_URL` / `AIA_IMAGE_API_KEY` / `AIA_IMAGE_MODEL` / `AIA_IMAGE_CHAT_ENDPOINT` 作为同一路由;当前实现 **不再读取** 上述变量作 OCR 通道配置,请统一改为 `AIA_OCR_*`。(`AIA_IMAGE_TOOL_RESULT_REPLAY_CAP_CHARS` 等为 **另一用途**,与 OCR 网关无关,仍保留原名。)
|
> **说明:** 历史上曾使用 `AIA_IMAGE_BASE_URL` / `AIA_IMAGE_API_KEY` / `AIA_IMAGE_MODEL` / `AIA_IMAGE_CHAT_ENDPOINT` 作为同一路由;当前实现 **不再读取** 上述变量作 OCR 通道配置,请统一改为 `AIA_OCR_*`。(`AIA_IMAGE_TOOL_RESULT_REPLAY_CAP_CHARS` 等为 **另一用途**,与 OCR 网关无关,仍保留原名。)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -60,13 +60,13 @@
|
||||||
## 2026-05-10 / Unreleased
|
## 2026-05-10 / Unreleased
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- `AIA_OCR_BASE_URL`, `AIA_OCR_API_KEY`, `AIA_OCR_MODEL`, `AIA_OCR_CHAT_ENDPOINT`
|
- `AIA_IMAGE_EXPERT_API_KEY`、`AIA_IMAGE_EXPERT_BASE_URL`、`AIA_IMAGE_EXPERT_MODEL`、`AIA_IMAGE_EXPERT_CHAT_ENDPOINT`:图片专家(`send_legacy_image_messages`)专线,与 **`AIA_OCR_*`** 互不继承。
|
||||||
- 默认值:`CHAT_ENDPOINT` 为 `/chat/completions`,其余无默认(三项主体必填方视为已配置)
|
- `AIA_IMAGE_EXPERT_REQUEST_EXTRA`:图片专家顶层 JSON;旧名 **`AIA_LEGACY_IMAGE_REQUEST_EXTRA`** 仍作别名可读。
|
||||||
- 用途:`query_image_attachment` / `send_ocr_image_messages` 等多模态 HTTP 通道命名(与主 LLM 分离)
|
- `DASHSCOPE_IMAGE_*`(零散变量):由 `image_legacy_client` 映射为请求体顶层字段。
|
||||||
- 影响模块:`oclaw/platform/llm/image_ocr_client.py`、`image_legacy_client.py`,`oclaw/runtime/tools/public/query_image_attachment_tool.py`,`oclaw/runtime/agents/specialist_agent.py`(图片专家走 OCR 多模态 HTTP)
|
- **`AIA_OCR_*`**(四项):仅存 **`query_image_attachment` / OCR 降级** 链路;已与图片专家链路拆分。
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
- `send_ocr_image_messages` / `send_legacy_image_messages` 不再有隐式默认模型 id;未配 `AIA_OCR_MODEL`(且未传 `model`)将直接失败
|
- `send_ocr_image_messages` 未配 `AIA_OCR_MODEL`(且未传 `model`)失败;图片专家 **`send_legacy_image_messages`** 首选 **用户所选会话/专家绑定的模型的 `model`/`base_url`/`api_key`**,缺省时再回落 **`AIA_IMAGE_EXPERT_*`**(不读取 `AIA_OCR_*`);服务端若模型不支持看图则直接报错,不做备用 payload。
|
||||||
|
|
||||||
### Removed(OCR 通道)
|
### Removed(OCR 通道)
|
||||||
- `AIA_IMAGE_BASE_URL` / `AIA_IMAGE_API_KEY` / `AIA_IMAGE_MODEL` / `AIA_IMAGE_CHAT_ENDPOINT` **不再**作为看图/OCR 通道的环境变量读取(须改用 `AIA_OCR_*`)。与附件回放相关的 `AIA_IMAGE_TOOL_RESULT_REPLAY_CAP_CHARS` 等 **不受影响**。
|
- `AIA_IMAGE_BASE_URL` / `AIA_IMAGE_API_KEY` / `AIA_IMAGE_MODEL` / `AIA_IMAGE_CHAT_ENDPOINT` **不再**作为看图/OCR 通道的环境变量读取(须改用 `AIA_OCR_*`)。与附件回放相关的 `AIA_IMAGE_TOOL_RESULT_REPLAY_CAP_CHARS` 等 **不受影响**。
|
||||||
|
|
|
||||||
84
docs/IMAGE_SPECIALIST_LANE.md
Normal file
84
docs/IMAGE_SPECIALIST_LANE.md
Normal file
|
|
@ -0,0 +1,84 @@
|
||||||
|
# 图片专家(Chat UI)专用链路
|
||||||
|
|
||||||
|
本文描述 **Admin `/chat` 选择「图片」专家** 时的端到端路径。这是一条 **与通用 Responses / 主对话模型环路隔离** 的特殊分支:仅在 `skill_binding_role == "image"` 时进入,其它专家或综合模式不受影响。
|
||||||
|
|
||||||
|
> 与本文无关:`docs/GATEWAY_IMAGE_GENERATE.md`(网关 RPC `image.generate`)、OCR 工具使用的 `image_ocr_client` 等。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 触发条件与隔离边界
|
||||||
|
|
||||||
|
| 条件 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| UI / Gateway | 用户在前端选择专家 **`image`**,请求体携带 `skill_binding_role`(或等价字段)为 **`image`**。 |
|
||||||
|
| 入口守卫 | `runtime/direct_loop.py` 中 **`_maybe_image_specialist_legacy_gateway_turn`**:仅当 `skill_binding_role.lower() == "image"` 且未设置禁用开关时执行;否则返回 `None`,后续仍走常规 `run_oclaw_direct_loop`。 |
|
||||||
|
| 禁用开关 | `AIA_IMAGE_SPECIALIST_DISABLE_LEGACY_GATEWAY_LANE=1`:关闭本 Early Return,图片专家改走与普通会话相同的模型/传输栈(用于调试或迁移)。 |
|
||||||
|
|
||||||
|
**不要在本链路外混用**:DashScope 形态的多模态 HTTP、`qwen-image*` 的空兼容响应回退等,均封装在 `platform/llm/image_legacy_client.py` 与 `image_http_common.py`,避免改到 `openai_responses` 的通用逻辑。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 运行时数据流(网关 → 落库)
|
||||||
|
|
||||||
|
1. **`run_oclaw_direct_loop`** 在用户消息落库后立刻调用 **`_maybe_image_specialist_legacy_gateway_turn`**。
|
||||||
|
2. **输入附件**:`collect_legacy_lane_images_from_attachments` 将 UI 附件规范为 `data:` URL 或 HTTP URL(`image_ref` / `input_image` / `image_url` 等)。
|
||||||
|
3. **无图**:直接写入一条 `assistant` / `assistant_text` 提示语并返回,不调用上游。
|
||||||
|
4. **有图**:调用 **`send_legacy_image_messages`**(`/chat/completions` 兼容路径,非 Responses API)。
|
||||||
|
5. **输出解析**:**`legacy_image_turn_bundle`**
|
||||||
|
- 文本可为空;若有生成图则 **`materialize_legacy_response_output_attachments`** 写入本地 blob,产出 **`image_ref`**(或退化为 **`image_url`**)。
|
||||||
|
6. **占位文案**:成功但只有图、无模型正文时,由 **`legacy_image_assistant_body_with_placeholder`**(`image_legacy_client`)写入中英文占位句;网关与 `specialist_agent` 共用,避免两处字符串分叉。
|
||||||
|
7. **持久化**:**`store.add_message(role=assistant, event_type=assistant_text, attachments=…)`** —— 附件以 JSON 形式挂在助手消息上,而非 tool 行。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 与其它入口的差异
|
||||||
|
|
||||||
|
| 场景 | 模块 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| Chat 网关 + 图片专家 | `direct_loop._maybe_image_specialist_legacy_gateway_turn` | 上文主路径;Early Return,不进主 LLM 循环。 |
|
||||||
|
| Specialist 编排临时会话 | `runtime/agents/specialist_agent.py` | 同样调用 `send_legacy_image_messages` / `legacy_image_turn_bundle`,逻辑对齐但不经过同一 Early Return。 |
|
||||||
|
|
||||||
|
两处共用 **`platform/llm/image_legacy_client.py`**,避免分叉实现。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 鉴权与下载(严格 ACL)
|
||||||
|
|
||||||
|
助手消息上的 **`image_ref`** 需在 **`attachment_acl`** 中有记录,才能在 **`AIA_ATTACHMENT_ACL_STRICT=1`** 下通过 **`GET /admin/api/chat/attachments/{id}`**。
|
||||||
|
|
||||||
|
- **`SqliteStore.add_message`** 会在落库后根据会话 **`ui_session_owner`** 对引用型附件执行 **`link_attachment_acl`**(与 tool 结果路径一致)。
|
||||||
|
- 历史数据可用 **`POST /admin/api/chat/admin/attachments/acl/backfill`**(管理员)补齐。详见 **`docs/attachment-acl.md`**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. WebSocket 收口与前端展示
|
||||||
|
|
||||||
|
- **`interfaces/ws/turn_runner.py`**:`final_msg` 从最近一条非空助手消息组装;**`_persisted_chat_attachments_nonempty`** 需识别 **JSON 数组或单个 JSON 对象**,否则纯附件回合会选错行。
|
||||||
|
- **`interfaces/admin/static/chat.js`**:聚合气泡 **`_buildAggregatedAssistantBubble`** 须对 **`assistant_text`** 片段调用 **`renderAttachmentsEl`**(不仅 `tool_result`),否则会出现「只有配文、无图」的现象。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 环境变量(索引)
|
||||||
|
|
||||||
|
详细列表与默认值以 **`docs/ENVIRONMENT_VARIABLES.md`** 为准。与本链路相关的典型前缀:
|
||||||
|
|
||||||
|
- **`AIA_IMAGE_EXPERT_*`** / **`AIA_IMAGE_SPECIALIST_*`**:图片专家 HTTP 基址、端点、模型、请求扩展等(与 **`AIA_OCR_*`** 分离)。
|
||||||
|
- **`AIA_IMAGE_SPECIALIST_DISABLE_LEGACY_GATEWAY_LANE`**:禁用网关侧 legacy 专用线。
|
||||||
|
- **`AIA_ATTACHMENT_ACL_STRICT`**:附件下载是否仅信任 ACL。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 测试与回归
|
||||||
|
|
||||||
|
- **`tests/test_image_legacy_gateway_lane.py`**:legacy 解析与规范化。
|
||||||
|
- **`tests/test_attachment_acl_backfill.py`**:严格 ACL 下助手附件与 ACL 写入。
|
||||||
|
- 修改 **`chat.js`** 聚合或 **`turn_runner`** final 消息逻辑后,应用图片专家跑一轮 **生成图 + 刷新历史** 做冒烟。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 变更原则(避免波及其它链路)
|
||||||
|
|
||||||
|
1. **默认改动范围**:`image_legacy_client.py`、`image_http_common.py`、`direct_loop` 中 **`_maybe_image_specialist_*` 函数体**、`specialist_agent` 中与 legacy image 调用相邻代码、`turn_runner` / `chat.js` 中与 **assistant + attachments** 展示相邻逻辑。
|
||||||
|
2. **勿在** `openai_responses.py` **中为图片专家单独分支**,除非明确要做「非 legacy」通用能力。
|
||||||
|
3. 新增开关优先 **`AIA_IMAGE_*` / `AIA_IMAGE_SPECIALIST_*`**,勿复用 OCR 变量。
|
||||||
|
4. UI 层附件渲染:**assistant_text 与 tool_result** 对称处理引用型附件,避免只修一端。
|
||||||
|
|
@ -30,6 +30,11 @@ Transport selection happens in `oclaw/runtime/agents/factory.py`.
|
||||||
- **Streaming**: output text deltas via `on_token` → WS `chat.delta`
|
- **Streaming**: output text deltas via `on_token` → WS `chat.delta`
|
||||||
- **Key**: profile secret or `OPENAI_API_KEY`
|
- **Key**: profile secret or `OPENAI_API_KEY`
|
||||||
|
|
||||||
|
### Chat UI「图片专家」(绕行本矩阵)
|
||||||
|
|
||||||
|
- **Not** a separate profile transport: when the user selects specialist **`image`** in `/chat`, `runtime/direct_loop.py` takes an early return and calls **`platform/llm/image_legacy_client.send_legacy_image_messages`** (DashScope-style `/chat/completions`), so that turn does **not** use `OpenAIResponsesModel` / chat transports above.
|
||||||
|
- Details, env vars, ACL, and UI hooks: **`docs/IMAGE_SPECIALIST_LANE.md`**.
|
||||||
|
|
||||||
### `anthropic` (Anthropic Messages streaming)
|
### `anthropic` (Anthropic Messages streaming)
|
||||||
- **Transport**: `oclaw/platform/llm/transports/anthropic_messages.py::AnthropicMessagesModel`
|
- **Transport**: `oclaw/platform/llm/transports/anthropic_messages.py::AnthropicMessagesModel`
|
||||||
- **API**: Anthropic `messages.stream` surface (gateway must provide Anthropic-compatible protocol)
|
- **API**: Anthropic `messages.stream` surface (gateway must provide Anthropic-compatible protocol)
|
||||||
|
|
|
||||||
|
|
@ -343,9 +343,10 @@ def _can_access_attachment(store: SqliteStore, ctx: dict[str, Any], *, attachmen
|
||||||
def _rate_limit_attachment_download(*, actor_tenant_id: str, actor_user_id: str) -> bool:
|
def _rate_limit_attachment_download(*, actor_tenant_id: str, actor_user_id: str) -> bool:
|
||||||
tid = str(actor_tenant_id or "").strip()
|
tid = str(actor_tenant_id or "").strip()
|
||||||
uid = str(actor_user_id or "").strip()
|
uid = str(actor_user_id or "").strip()
|
||||||
if not tid or not uid:
|
# Require tenant; allow downloads when user id is unexpectedly empty (rate bucket is tenant-scoped).
|
||||||
|
if not tid:
|
||||||
return False
|
return False
|
||||||
key = f"{tid}:{uid}"
|
key = f"{tid}:{uid or '__actor__'}"
|
||||||
now = time.time()
|
now = time.time()
|
||||||
with _ATT_DOWNLOAD_LOCK:
|
with _ATT_DOWNLOAD_LOCK:
|
||||||
tokens, last = _ATT_DOWNLOAD_BUCKET.get(key, (_ATT_DOWNLOAD_BURST, now))
|
tokens, last = _ATT_DOWNLOAD_BUCKET.get(key, (_ATT_DOWNLOAD_BURST, now))
|
||||||
|
|
|
||||||
|
|
@ -480,34 +480,22 @@ function _appendAssistantTextSegments(inner, rawText, collapsedItems) {
|
||||||
async function _buildAggregatedAssistantBubble(tsIso, items) {
|
async function _buildAggregatedAssistantBubble(tsIso, items) {
|
||||||
const inner = el("div", { class: "chat-msg chat-msg--assistant chat-msg--rich" });
|
const inner = el("div", { class: "chat-msg chat-msg--assistant chat-msg--rich" });
|
||||||
const collapsedItems = [];
|
const collapsedItems = [];
|
||||||
const _inlineImageHtmlFromAttachments = (raw) => {
|
|
||||||
const list = parseAttachments(raw);
|
|
||||||
if (!Array.isArray(list) || !list.length) return "";
|
|
||||||
const blocks = [];
|
|
||||||
for (const att of list) {
|
|
||||||
if (!att || typeof att !== "object") continue;
|
|
||||||
const typ = String(att.type || "").toLowerCase();
|
|
||||||
if (typ === "image" || typ === "input_image") {
|
|
||||||
const b64 = String(att.image_base64 || att.data || "").trim();
|
|
||||||
if (!b64) continue;
|
|
||||||
const mime = String(att.mime || att.mime_type || "image/png").trim() || "image/png";
|
|
||||||
const src = `data:${mime};base64,${b64.replace(/\s/g, "")}`;
|
|
||||||
blocks.push(`<div class="chat-att-wrap"><img class="chat-att-img" src="${escapeHtml(src)}" alt="tool image" /></div>`);
|
|
||||||
} else if (typ === "image_url") {
|
|
||||||
const src = String(att.url || att.image_url || "").trim();
|
|
||||||
if (!src) continue;
|
|
||||||
blocks.push(`<div class="chat-att-wrap"><img class="chat-att-img" src="${escapeHtml(src)}" alt="tool image" /></div>`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return blocks.join("");
|
|
||||||
};
|
|
||||||
for (const it of items || []) {
|
for (const it of items || []) {
|
||||||
const kind = String((it && it.kind) || "");
|
const kind = String((it && it.kind) || "");
|
||||||
const text = String((it && it.text) || "");
|
const text = String((it && it.text) || "");
|
||||||
const hasInlineAtt = !!(it && it.attachments);
|
const parsedAtts = parseAttachments(it && it.attachments);
|
||||||
if (!text.trim() && !(kind === "tool_result" && hasInlineAtt)) continue;
|
const hasInlineAtt = parsedAtts.length > 0;
|
||||||
|
if (
|
||||||
|
!text.trim() &&
|
||||||
|
!(kind === "tool_result" && hasInlineAtt) &&
|
||||||
|
!(kind === "assistant_text" && hasInlineAtt)
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if (kind === "assistant_text") {
|
if (kind === "assistant_text") {
|
||||||
_appendAssistantTextSegments(inner, text, collapsedItems);
|
_appendAssistantTextSegments(inner, text, collapsedItems);
|
||||||
|
const attsAssistant = await renderAttachmentsEl(it && it.attachments);
|
||||||
|
if (attsAssistant) inner.appendChild(attsAssistant);
|
||||||
} else if (kind === "reasoning") {
|
} else if (kind === "reasoning") {
|
||||||
if (adminChatShowToolOutput) collapsedItems.push({ title: t("reasoning.summary"), text });
|
if (adminChatShowToolOutput) collapsedItems.push({ title: t("reasoning.summary"), text });
|
||||||
} else if (kind === "tool_call") {
|
} else if (kind === "tool_call") {
|
||||||
|
|
@ -538,6 +526,7 @@ async function _buildAggregatedAssistantBubble(tsIso, items) {
|
||||||
}
|
}
|
||||||
const hasVisible =
|
const hasVisible =
|
||||||
!!inner.querySelector(".chat-msg__md, .chat-msg__reasoning, .chat-msg__wiki") ||
|
!!inner.querySelector(".chat-msg__md, .chat-msg__reasoning, .chat-msg__wiki") ||
|
||||||
|
!!inner.querySelector(".chat-att-wrap, img.chat-att-img, a.chat-att-ref__link, .chat-att-chip") ||
|
||||||
String(inner.textContent || "").trim().length > 0;
|
String(inner.textContent || "").trim().length > 0;
|
||||||
if (!hasVisible) {
|
if (!hasVisible) {
|
||||||
inner.appendChild(el("div", { class: "muted", text: t("chat.tools.hidden") }));
|
inner.appendChild(el("div", { class: "muted", text: t("chat.tools.hidden") }));
|
||||||
|
|
@ -596,8 +585,14 @@ function _buildRenderRows(msgs) {
|
||||||
if (String(content || "").trim()) {
|
if (String(content || "").trim()) {
|
||||||
agg._items.push({ kind: "reasoning", text: content });
|
agg._items.push({ kind: "reasoning", text: content });
|
||||||
}
|
}
|
||||||
} else if ((eventType === "assistant_text" || eventType === "assistant" || !eventType) && String(content || "").trim()) {
|
} else if (eventType === "assistant_text" || eventType === "assistant" || !eventType) {
|
||||||
agg._items.push({ kind: "assistant_text", text: content });
|
const hasText = !!String(content || "").trim();
|
||||||
|
const attsParsed = parseAttachments(m && m.attachments);
|
||||||
|
if (hasText || attsParsed.length) {
|
||||||
|
const piece = { kind: "assistant_text", text: content };
|
||||||
|
if (attsParsed.length) piece.attachments = attsParsed;
|
||||||
|
agg._items.push(piece);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
const tc = m.tool_calls;
|
const tc = m.tool_calls;
|
||||||
if (tc != null && tc !== "") {
|
if (tc != null && tc !== "") {
|
||||||
|
|
@ -610,9 +605,16 @@ function _buildRenderRows(msgs) {
|
||||||
if (line) agg._items.push({ kind: "tool_call", text: line });
|
if (line) agg._items.push({ kind: "tool_call", text: line });
|
||||||
}
|
}
|
||||||
if (m.attachments) agg.attachments = m.attachments;
|
if (m.attachments) agg.attachments = m.attachments;
|
||||||
} else if (String(content || "").trim()) {
|
} else {
|
||||||
agg._items.push({ kind: "tool_result", text: content, attachments: m.attachments || null });
|
const hasToolText = !!String(content || "").trim();
|
||||||
if (m.attachments) agg.attachments = m.attachments;
|
const toolAtts = parseAttachments(m && m.attachments);
|
||||||
|
if (hasToolText || toolAtts.length) {
|
||||||
|
const piece = { kind: "tool_result", text: content };
|
||||||
|
if (toolAtts.length) piece.attachments = toolAtts;
|
||||||
|
else if (m.attachments) piece.attachments = m.attachments;
|
||||||
|
agg._items.push(piece);
|
||||||
|
if (m.attachments) agg.attachments = m.attachments;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
@ -2235,7 +2237,37 @@ async function renderAttachmentsEl(raw) {
|
||||||
}
|
}
|
||||||
} else if (typ === "image_url") {
|
} else if (typ === "image_url") {
|
||||||
const src = String(att.url || att.image_url || "").trim();
|
const src = String(att.url || att.image_url || "").trim();
|
||||||
const img = el("img", { class: "chat-att-img", src, alt: "" });
|
if (!src) continue;
|
||||||
|
const img = el("img", {
|
||||||
|
class: "chat-att-img",
|
||||||
|
src,
|
||||||
|
alt: String(att.name || "generated image"),
|
||||||
|
decoding: "async",
|
||||||
|
referrerPolicy: "no-referrer",
|
||||||
|
});
|
||||||
|
img.addEventListener(
|
||||||
|
"error",
|
||||||
|
() => {
|
||||||
|
try {
|
||||||
|
const parent = img.parentNode;
|
||||||
|
if (!parent) return;
|
||||||
|
parent.replaceChild(
|
||||||
|
el("a", {
|
||||||
|
class: "chat-att-ref__link",
|
||||||
|
href: src,
|
||||||
|
target: "_blank",
|
||||||
|
rel: "noopener noreferrer",
|
||||||
|
text:
|
||||||
|
currentLang === "zh"
|
||||||
|
? "图片无法嵌入显示(可能被防盗链),点击打开原始链接"
|
||||||
|
: "Image blocked from inline display; open original URL",
|
||||||
|
}),
|
||||||
|
img,
|
||||||
|
);
|
||||||
|
} catch (_) {}
|
||||||
|
},
|
||||||
|
{ once: true },
|
||||||
|
);
|
||||||
wrap.appendChild(img);
|
wrap.appendChild(img);
|
||||||
} else if (typ === "video_ref" || typ === "text_ref" || typ === "binary_ref") {
|
} else if (typ === "video_ref" || typ === "text_ref" || typ === "binary_ref") {
|
||||||
wrap.appendChild(await buildRefCard(att, typ));
|
wrap.appendChild(await buildRefCard(att, typ));
|
||||||
|
|
@ -4026,16 +4058,18 @@ async function renderChatUi() {
|
||||||
const role = String(m.role || "").toLowerCase();
|
const role = String(m.role || "").toLowerCase();
|
||||||
if (requireRole && role !== "assistant") return null;
|
if (requireRole && role !== "assistant") return null;
|
||||||
if (requireContentArray && !Array.isArray(m.content)) return null;
|
if (requireContentArray && !Array.isArray(m.content)) return null;
|
||||||
if (!("content" in m) && !("text" in m)) return null;
|
const hasRootAttachments = m.attachments != null && m.attachments !== "";
|
||||||
|
if (!("content" in m) && !("text" in m) && !hasRootAttachments) return null;
|
||||||
return m;
|
return m;
|
||||||
};
|
};
|
||||||
const _expandAssistantMessageForRender = (message) => {
|
const _expandAssistantMessageForRender = (message) => {
|
||||||
if (!message || typeof message !== "object") return [];
|
if (!message || typeof message !== "object") return [];
|
||||||
|
const mesAttParsed = parseAttachments(message.attachments);
|
||||||
|
const mesAtt = mesAttParsed.length ? mesAttParsed : null;
|
||||||
const base = {
|
const base = {
|
||||||
role: "assistant",
|
role: "assistant",
|
||||||
id: message.id,
|
id: message.id,
|
||||||
timestamp: message.timestamp != null ? message.timestamp : new Date().toISOString(),
|
timestamp: message.timestamp != null ? message.timestamp : new Date().toISOString(),
|
||||||
attachments: message.attachments || null,
|
|
||||||
};
|
};
|
||||||
const out = [];
|
const out = [];
|
||||||
const contentItems = Array.isArray(message.content) ? message.content : [];
|
const contentItems = Array.isArray(message.content) ? message.content : [];
|
||||||
|
|
@ -4061,18 +4095,28 @@ async function renderChatUi() {
|
||||||
...out[lastIdx],
|
...out[lastIdx],
|
||||||
tool_calls: message.tool_calls != null ? message.tool_calls : message.toolCalls,
|
tool_calls: message.tool_calls != null ? message.tool_calls : message.toolCalls,
|
||||||
};
|
};
|
||||||
|
// Avoid duplicating blobs: attach WS/API root attachments to the latest assistant_text row only.
|
||||||
|
if (mesAtt && mesAtt.length) {
|
||||||
|
for (let i = out.length - 1; i >= 0; i -= 1) {
|
||||||
|
if (String(out[i].event_type || "") === "assistant_text") {
|
||||||
|
out[i] = { ...out[i], attachments: mesAtt };
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
const textFallback = decodeEscapedNewlines(
|
const textFallback = decodeEscapedNewlines(
|
||||||
typeof message.content === "string" ? message.content : typeof message.text === "string" ? message.text : "",
|
typeof message.content === "string" ? message.content : typeof message.text === "string" ? message.text : "",
|
||||||
).trim();
|
).trim();
|
||||||
if (textFallback) {
|
if (textFallback || (mesAtt && mesAtt.length)) {
|
||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
...base,
|
...base,
|
||||||
content: textFallback,
|
content: textFallback,
|
||||||
event_type: "assistant_text",
|
event_type: "assistant_text",
|
||||||
tool_calls: message.tool_calls != null ? message.tool_calls : message.toolCalls,
|
tool_calls: message.tool_calls != null ? message.tool_calls : message.toolCalls,
|
||||||
|
...(mesAtt && mesAtt.length ? { attachments: mesAtt } : {}),
|
||||||
},
|
},
|
||||||
];
|
];
|
||||||
}
|
}
|
||||||
|
|
@ -4488,11 +4532,16 @@ ${autoLimit ? `<div style="margin-top:8px;"><span class="muted">auto-added claus
|
||||||
chatStreamSegments.push({ type: "tool", key, title: `${name} ${liveTag}`, body, sqlAudit, images });
|
chatStreamSegments.push({ type: "tool", key, title: `${name} ${liveTag}`, body, sqlAudit, images });
|
||||||
};
|
};
|
||||||
const appendFinalAssistant = async (message, fallbackText) => {
|
const appendFinalAssistant = async (message, fallbackText) => {
|
||||||
|
const wsAttachments =
|
||||||
|
message && typeof message === "object" && message.attachments != null ? message.attachments : null;
|
||||||
const normalized = _normalizeAssistantMessage(message, { requireRole: false, requireContentArray: false });
|
const normalized = _normalizeAssistantMessage(message, { requireRole: false, requireContentArray: false });
|
||||||
if (normalized) {
|
if (normalized) {
|
||||||
const rows = _buildRenderRows(_expandAssistantMessageForRender(normalized));
|
const rows = _buildRenderRows(_expandAssistantMessageForRender(normalized));
|
||||||
const last = rows && rows.length ? rows[rows.length - 1] : null;
|
const last = rows && rows.length ? rows[rows.length - 1] : null;
|
||||||
if (last) {
|
if (last) {
|
||||||
|
if (wsAttachments != null && (last.attachments == null || last.attachments === "")) {
|
||||||
|
last.attachments = wsAttachments;
|
||||||
|
}
|
||||||
if (streamRow && streamRow.parentNode) streamRow.remove();
|
if (streamRow && streamRow.parentNode) streamRow.remove();
|
||||||
await appendMessageRow(messagesEl, last, rowRenderOptions);
|
await appendMessageRow(messagesEl, last, rowRenderOptions);
|
||||||
scrollMessagesToBottom(true);
|
scrollMessagesToBottom(true);
|
||||||
|
|
@ -4502,11 +4551,16 @@ ${autoLimit ? `<div style="margin-top:8px;"><span class="muted">auto-added claus
|
||||||
const t0 = String(fallbackText || "").trim();
|
const t0 = String(fallbackText || "").trim();
|
||||||
if (t0 && !_isSilentReplyStream(t0)) {
|
if (t0 && !_isSilentReplyStream(t0)) {
|
||||||
if (streamRow && streamRow.parentNode) streamRow.remove();
|
if (streamRow && streamRow.parentNode) streamRow.remove();
|
||||||
await appendMessageRow(messagesEl, {
|
await appendMessageRow(
|
||||||
role: "assistant",
|
messagesEl,
|
||||||
content: decodeEscapedNewlines(t0),
|
{
|
||||||
timestamp: new Date().toISOString(),
|
role: "assistant",
|
||||||
}, rowRenderOptions);
|
content: decodeEscapedNewlines(t0),
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
...(wsAttachments != null ? { attachments: wsAttachments } : {}),
|
||||||
|
},
|
||||||
|
rowRenderOptions,
|
||||||
|
);
|
||||||
scrollMessagesToBottom(true);
|
scrollMessagesToBottom(true);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
|
import json
|
||||||
import threading
|
import threading
|
||||||
import uuid
|
import uuid
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
@ -13,6 +14,28 @@ from oclaw.platform.config.paths import db_path
|
||||||
from oclaw.platform.persistence.sqlite_store import SqliteStore
|
from oclaw.platform.persistence.sqlite_store import SqliteStore
|
||||||
|
|
||||||
|
|
||||||
|
def _persisted_chat_attachments_nonempty(raw: Any) -> bool:
|
||||||
|
"""True when chat_message.attachments has at least one JSON object (list or dict or encoded string)."""
|
||||||
|
if raw is None:
|
||||||
|
return False
|
||||||
|
if isinstance(raw, list):
|
||||||
|
return len(raw) > 0
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
return bool(raw)
|
||||||
|
s = str(raw).strip()
|
||||||
|
if not s or s.lower() == "null":
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
parsed = json.loads(s)
|
||||||
|
if isinstance(parsed, list):
|
||||||
|
return len(parsed) > 0
|
||||||
|
if isinstance(parsed, dict):
|
||||||
|
return bool(parsed)
|
||||||
|
except Exception:
|
||||||
|
return bool(s)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
async def run_agent_turn_via_bridge(
|
async def run_agent_turn_via_bridge(
|
||||||
*,
|
*,
|
||||||
conn: Any,
|
conn: Any,
|
||||||
|
|
@ -533,12 +556,13 @@ async def run_agent_turn_via_bridge(
|
||||||
pass
|
pass
|
||||||
elif run_status != "failed":
|
elif run_status != "failed":
|
||||||
try:
|
try:
|
||||||
persisted = store.get_messages(session_id=session_id, limit=12)
|
persisted = store.get_messages(session_id=session_id, limit=64)
|
||||||
for m in reversed(list(persisted or [])):
|
for m in reversed(list(persisted or [])):
|
||||||
if str(getattr(m, "role", "") or "").lower() != "assistant":
|
if str(getattr(m, "role", "") or "").lower() != "assistant":
|
||||||
continue
|
continue
|
||||||
content = str(getattr(m, "content", "") or "")
|
content = str(getattr(m, "content", "") or "")
|
||||||
if not content.strip():
|
atraw = getattr(m, "attachments", None)
|
||||||
|
if not content.strip() and not _persisted_chat_attachments_nonempty(atraw):
|
||||||
continue
|
continue
|
||||||
final_text = content
|
final_text = content
|
||||||
final_msg = {
|
final_msg = {
|
||||||
|
|
@ -547,7 +571,7 @@ async def run_agent_turn_via_bridge(
|
||||||
"content": content,
|
"content": content,
|
||||||
"timestamp": str(getattr(m, "timestamp", "") or ""),
|
"timestamp": str(getattr(m, "timestamp", "") or ""),
|
||||||
"tool_calls": getattr(m, "tool_calls", None),
|
"tool_calls": getattr(m, "tool_calls", None),
|
||||||
"attachments": getattr(m, "attachments", None),
|
"attachments": atraw,
|
||||||
}
|
}
|
||||||
break
|
break
|
||||||
except Exception:
|
except Exception:
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,9 @@ from __future__ import annotations
|
||||||
|
|
||||||
import base64
|
import base64
|
||||||
import io
|
import io
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import time
|
import time
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
|
@ -18,6 +20,32 @@ def join_url(base: str, path: str) -> str:
|
||||||
return f"{b}/{p}"
|
return f"{b}/{p}"
|
||||||
|
|
||||||
|
|
||||||
|
def dashscope_native_multimodal_url_from_compatible_base(base_url: str) -> str | None:
|
||||||
|
"""
|
||||||
|
``qwen-image`` and similar models often return an **empty** OpenAI ``chat/completions`` message on
|
||||||
|
``/compatible-mode/v1``. Native HTTP uses ``/api/v1/services/aigc/multimodal-generation/generation``
|
||||||
|
and returns ``output.choices[].message.content[].image`` URLs.
|
||||||
|
|
||||||
|
Override: ``AIA_IMAGE_EXPERT_DASHSCOPE_NATIVE_URL`` (full URL).
|
||||||
|
Path suffix: ``AIA_IMAGE_EXPERT_DASHSCOPE_NATIVE_PATH`` (default multimodal-generation path).
|
||||||
|
"""
|
||||||
|
explicit = (os.getenv("AIA_IMAGE_EXPERT_DASHSCOPE_NATIVE_URL") or "").strip().rstrip("/")
|
||||||
|
if explicit:
|
||||||
|
return explicit
|
||||||
|
b = (base_url or "").strip().rstrip("/")
|
||||||
|
if not b or "compatible-mode" not in b.lower():
|
||||||
|
return None
|
||||||
|
root = re.sub(r"/compatible-mode/v\d+(?:/.*)?$", "", b, flags=re.IGNORECASE).rstrip("/")
|
||||||
|
if not root or root == b:
|
||||||
|
root = re.sub(r"/compatible-mode/?$", "", b, flags=re.IGNORECASE).rstrip("/")
|
||||||
|
if not root:
|
||||||
|
return None
|
||||||
|
path = (os.getenv("AIA_IMAGE_EXPERT_DASHSCOPE_NATIVE_PATH") or "").strip().lstrip("/")
|
||||||
|
if not path:
|
||||||
|
path = "api/v1/services/aigc/multimodal-generation/generation"
|
||||||
|
return f"{root}/{path}"
|
||||||
|
|
||||||
|
|
||||||
def env_ocr_lane_api_key() -> str:
|
def env_ocr_lane_api_key() -> str:
|
||||||
return (os.getenv("AIA_OCR_API_KEY") or "").strip()
|
return (os.getenv("AIA_OCR_API_KEY") or "").strip()
|
||||||
|
|
||||||
|
|
@ -35,6 +63,25 @@ def env_ocr_lane_chat_endpoint() -> str:
|
||||||
return raw or "/chat/completions"
|
return raw or "/chat/completions"
|
||||||
|
|
||||||
|
|
||||||
|
def env_image_expert_api_key() -> str:
|
||||||
|
"""Bearer key for **image specialist** multimodal/gen HTTP only (never shared with OCR ``AIA_OCR_*``)."""
|
||||||
|
|
||||||
|
return (os.getenv("AIA_IMAGE_EXPERT_API_KEY") or "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def env_image_expert_base_url() -> str:
|
||||||
|
return (os.getenv("AIA_IMAGE_EXPERT_BASE_URL") or "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def env_image_expert_model() -> str:
|
||||||
|
return (os.getenv("AIA_IMAGE_EXPERT_MODEL") or "").strip()
|
||||||
|
|
||||||
|
|
||||||
|
def env_image_expert_chat_endpoint() -> str:
|
||||||
|
raw = (os.getenv("AIA_IMAGE_EXPERT_CHAT_ENDPOINT") or "").strip()
|
||||||
|
return raw or "/chat/completions"
|
||||||
|
|
||||||
|
|
||||||
def is_data_url(s: str) -> bool:
|
def is_data_url(s: str) -> bool:
|
||||||
return s.startswith("data:") and ";base64," in s
|
return s.startswith("data:") and ";base64," in s
|
||||||
|
|
||||||
|
|
@ -97,51 +144,406 @@ def post_with_retry(
|
||||||
raise last_exc
|
raise last_exc
|
||||||
|
|
||||||
|
|
||||||
|
def env_image_expert_download_timeout_sec() -> float:
|
||||||
|
"""Read timeout for DashScope/OSS signed result URLs (official samples use ~300s)."""
|
||||||
|
raw = (os.getenv("AIA_IMAGE_EXPERT_DOWNLOAD_TIMEOUT_SEC") or "").strip()
|
||||||
|
if not raw:
|
||||||
|
return 300.0
|
||||||
|
try:
|
||||||
|
v = float(raw)
|
||||||
|
except ValueError:
|
||||||
|
return 300.0
|
||||||
|
return max(15.0, min(v, 900.0))
|
||||||
|
|
||||||
|
|
||||||
|
def download_http_url_bytes(
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
timeout_sec: float | None = None,
|
||||||
|
user_agent: str | None = None,
|
||||||
|
) -> tuple[bytes, str]:
|
||||||
|
"""Streamed GET with ``raise_for_status`` — matches DashScope OSS download guidance (long reads)."""
|
||||||
|
t = float(timeout_sec) if timeout_sec is not None else env_image_expert_download_timeout_sec()
|
||||||
|
ua = (user_agent or "").strip() or (
|
||||||
|
"Mozilla/5.0 (compatible; oclaw-image-expert/1.0; +https://github.com/)"
|
||||||
|
)
|
||||||
|
req_headers = {"User-Agent": ua, "Accept": "*/*"}
|
||||||
|
connect_cap = min(45.0, max(10.0, t / 10.0))
|
||||||
|
timeout = httpx.Timeout(t, connect=connect_cap)
|
||||||
|
with httpx.Client(timeout=timeout, follow_redirects=True) as client:
|
||||||
|
with client.stream("GET", url, headers=req_headers) as r:
|
||||||
|
r.raise_for_status()
|
||||||
|
ctype = str(r.headers.get("content-type") or "").split(";", 1)[0].strip()
|
||||||
|
parts: list[bytes] = []
|
||||||
|
for chunk in r.iter_bytes(chunk_size=65_536):
|
||||||
|
parts.append(chunk)
|
||||||
|
return b"".join(parts), ctype
|
||||||
|
|
||||||
|
|
||||||
|
def dashscope_multimodal_http_ok(body: dict[str, Any]) -> tuple[bool, str]:
|
||||||
|
"""DashScope HTTP often uses HTTP 200 with a business-level ``code`` for failures."""
|
||||||
|
sc = body.get("status_code")
|
||||||
|
if isinstance(sc, str) and sc.strip().isdigit():
|
||||||
|
sc = int(sc.strip())
|
||||||
|
if isinstance(sc, int) and sc >= 400:
|
||||||
|
return False, str(body.get("message") or body.get("msg") or f"status_code={sc}")
|
||||||
|
code = body.get("code")
|
||||||
|
if code is None:
|
||||||
|
return True, ""
|
||||||
|
if isinstance(code, int):
|
||||||
|
if code == 200:
|
||||||
|
return True, ""
|
||||||
|
msg = str(body.get("message") or body.get("msg") or "").strip()
|
||||||
|
return False, msg or str(code)
|
||||||
|
cs = str(code).strip()
|
||||||
|
if cs == "":
|
||||||
|
return True, ""
|
||||||
|
if cs.lower() in ("success", "ok", "200"):
|
||||||
|
return True, ""
|
||||||
|
msg = str(body.get("message") or body.get("msg") or "").strip()
|
||||||
|
return False, msg or cs
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_response_roots(resp_json: dict[str, Any]) -> list[dict[str, Any]]:
|
||||||
|
"""Collect dict slices that may carry ``choices`` / ``messages`` (DashScope nests under ``output``)."""
|
||||||
|
roots: list[dict[str, Any]] = []
|
||||||
|
seen: set[int] = set()
|
||||||
|
|
||||||
|
def push(d: dict[str, Any]) -> None:
|
||||||
|
i = id(d)
|
||||||
|
if i in seen:
|
||||||
|
return
|
||||||
|
seen.add(i)
|
||||||
|
roots.append(d)
|
||||||
|
|
||||||
|
push(resp_json)
|
||||||
|
nested = resp_json.get("output")
|
||||||
|
if isinstance(nested, dict):
|
||||||
|
push(nested)
|
||||||
|
deeper = nested.get("output")
|
||||||
|
if isinstance(deeper, dict):
|
||||||
|
push(deeper)
|
||||||
|
result = resp_json.get("result")
|
||||||
|
if isinstance(result, dict):
|
||||||
|
push(result)
|
||||||
|
data_obj = resp_json.get("data")
|
||||||
|
if isinstance(data_obj, dict):
|
||||||
|
push(data_obj)
|
||||||
|
return roots
|
||||||
|
|
||||||
|
|
||||||
|
def _append_image_from_any(images: list[str], raw: Any) -> None:
|
||||||
|
if isinstance(raw, str) and raw.strip():
|
||||||
|
s = raw.strip()
|
||||||
|
if s.startswith(("http://", "https://", "data:")):
|
||||||
|
images.append(s)
|
||||||
|
return
|
||||||
|
if isinstance(raw, dict):
|
||||||
|
u = raw.get("url")
|
||||||
|
if isinstance(u, str) and u.strip() and u.strip().startswith(("http://", "https://")):
|
||||||
|
images.append(u.strip())
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_content_part(it: Any, *, text_parts: list[str], images: list[str]) -> None:
|
||||||
|
"""Parse one multimodal element (DashScope shorthand, OpenAI-style typed blocks, etc.)."""
|
||||||
|
if isinstance(it, str) and it.strip():
|
||||||
|
text_parts.append(it.strip())
|
||||||
|
return
|
||||||
|
if not isinstance(it, dict):
|
||||||
|
return
|
||||||
|
typ = str(it.get("type") or "").strip().lower()
|
||||||
|
if typ == "text":
|
||||||
|
tx = it.get("text")
|
||||||
|
if isinstance(tx, str) and tx.strip():
|
||||||
|
text_parts.append(tx.strip())
|
||||||
|
return
|
||||||
|
if typ == "input_text":
|
||||||
|
tx = it.get("text")
|
||||||
|
if isinstance(tx, str) and tx.strip():
|
||||||
|
text_parts.append(tx.strip())
|
||||||
|
return
|
||||||
|
if typ == "image_url":
|
||||||
|
iu = it.get("image_url")
|
||||||
|
if isinstance(iu, dict):
|
||||||
|
_append_image_from_any(images, iu)
|
||||||
|
elif isinstance(iu, str):
|
||||||
|
_append_image_from_any(images, iu)
|
||||||
|
return
|
||||||
|
if typ == "input_image":
|
||||||
|
iu = it.get("image_url")
|
||||||
|
if isinstance(iu, dict):
|
||||||
|
_append_image_from_any(images, iu)
|
||||||
|
elif isinstance(iu, str):
|
||||||
|
_append_image_from_any(images, iu)
|
||||||
|
return
|
||||||
|
if typ == "image" and it.get("image") is not None:
|
||||||
|
_append_image_from_any(images, it.get("image"))
|
||||||
|
return
|
||||||
|
# Untyped / DashScope shorthand blocks
|
||||||
|
tx = it.get("text")
|
||||||
|
if isinstance(tx, str) and tx.strip():
|
||||||
|
text_parts.append(tx.strip())
|
||||||
|
if it.get("image") is not None:
|
||||||
|
_append_image_from_any(images, it.get("image"))
|
||||||
|
ius = it.get("image_url")
|
||||||
|
if isinstance(ius, str):
|
||||||
|
_append_image_from_any(images, ius)
|
||||||
|
elif isinstance(ius, dict):
|
||||||
|
_append_image_from_any(images, ius)
|
||||||
|
uu = it.get("url")
|
||||||
|
if isinstance(uu, str) and uu.strip().startswith(("http://", "https://")):
|
||||||
|
images.append(uu.strip())
|
||||||
|
if isinstance(it.get("b64_json"), str) and str(it.get("b64_json")).strip():
|
||||||
|
images.append(f"data:image/png;base64,{it['b64_json']}")
|
||||||
|
|
||||||
|
|
||||||
|
def _try_parse_content_json_string(raw: str, *, text_parts: list[str], images: list[str]) -> bool:
|
||||||
|
"""Some proxies stringify multimodal ``content`` as JSON."""
|
||||||
|
s = raw.strip()
|
||||||
|
if not s or s[0] not in "[{":
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
parsed = json.loads(s)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
if isinstance(parsed, list):
|
||||||
|
for it in parsed:
|
||||||
|
_parse_content_part(it, text_parts=text_parts, images=images)
|
||||||
|
return True
|
||||||
|
if isinstance(parsed, dict):
|
||||||
|
_parse_content_part(parsed, text_parts=text_parts, images=images)
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _harvest_http_urls_from_value(val: Any, images: list[str], *, depth: int, max_depth: int = 6) -> None:
|
||||||
|
"""Last resort: collect image URLs nested under ``message`` (unknown multimodal block shapes)."""
|
||||||
|
if depth > max_depth or val is None:
|
||||||
|
return
|
||||||
|
if isinstance(val, str):
|
||||||
|
s = val.strip()
|
||||||
|
if s.startswith(("http://", "https://")) and s not in images:
|
||||||
|
images.append(s)
|
||||||
|
return
|
||||||
|
if isinstance(val, dict):
|
||||||
|
for _k, v in val.items():
|
||||||
|
_harvest_http_urls_from_value(v, images, depth=depth + 1, max_depth=max_depth)
|
||||||
|
return
|
||||||
|
if isinstance(val, list):
|
||||||
|
for it in val:
|
||||||
|
_harvest_http_urls_from_value(it, images, depth=depth + 1, max_depth=max_depth)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_choice_message_content(msg: dict[str, Any], *, text_parts: list[str], images: list[str]) -> None:
|
||||||
|
c = msg.get("content")
|
||||||
|
if c is None:
|
||||||
|
_harvest_http_urls_from_value(msg, images, depth=0)
|
||||||
|
return
|
||||||
|
if isinstance(c, str):
|
||||||
|
if c.strip() and _try_parse_content_json_string(c, text_parts=text_parts, images=images):
|
||||||
|
_harvest_http_urls_from_value(msg, images, depth=0)
|
||||||
|
return
|
||||||
|
if c.strip():
|
||||||
|
text_parts.append(c.strip())
|
||||||
|
_harvest_http_urls_from_value(msg, images, depth=0)
|
||||||
|
return
|
||||||
|
if isinstance(c, dict):
|
||||||
|
_parse_content_part(c, text_parts=text_parts, images=images)
|
||||||
|
_harvest_http_urls_from_value(msg, images, depth=0)
|
||||||
|
return
|
||||||
|
if isinstance(c, list):
|
||||||
|
for it in c:
|
||||||
|
_parse_content_part(it, text_parts=text_parts, images=images)
|
||||||
|
_harvest_http_urls_from_value(msg, images, depth=0)
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_messages_array_for_assistant(
|
||||||
|
messages: list[Any], *, text_parts: list[str], images: list[str]
|
||||||
|
) -> None:
|
||||||
|
for m in reversed(messages):
|
||||||
|
if not isinstance(m, dict):
|
||||||
|
continue
|
||||||
|
if str(m.get("role") or "").strip().lower() != "assistant":
|
||||||
|
continue
|
||||||
|
_parse_choice_message_content(m, text_parts=text_parts, images=images)
|
||||||
|
break
|
||||||
|
|
||||||
|
|
||||||
def extract_text_and_images(resp_json: dict[str, Any]) -> tuple[str, list[str]]:
|
def extract_text_and_images(resp_json: dict[str, Any]) -> tuple[str, list[str]]:
|
||||||
text_parts: list[str] = []
|
text_parts: list[str] = []
|
||||||
images: list[str] = []
|
images: list[str] = []
|
||||||
choices = resp_json.get("choices")
|
roots = _extract_response_roots(resp_json)
|
||||||
|
|
||||||
|
for root in roots:
|
||||||
|
ot = root.get("text")
|
||||||
|
if isinstance(ot, str) and ot.strip():
|
||||||
|
text_parts.append(ot.strip())
|
||||||
|
|
||||||
|
choices: list[Any] | None = None
|
||||||
|
for root in roots:
|
||||||
|
ch = root.get("choices")
|
||||||
|
if isinstance(ch, list) and ch:
|
||||||
|
choices = ch
|
||||||
|
break
|
||||||
|
|
||||||
if isinstance(choices, list) and choices:
|
if isinstance(choices, list) and choices:
|
||||||
msg = choices[0].get("message") if isinstance(choices[0], dict) else None
|
for choice in choices:
|
||||||
if isinstance(msg, dict):
|
if not isinstance(choice, dict):
|
||||||
c = msg.get("content")
|
continue
|
||||||
if isinstance(c, str):
|
legacy_txt = choice.get("text")
|
||||||
text_parts.append(c)
|
if isinstance(legacy_txt, str) and legacy_txt.strip():
|
||||||
elif isinstance(c, list):
|
text_parts.append(legacy_txt.strip())
|
||||||
for it in c:
|
msg = choice.get("message")
|
||||||
if not isinstance(it, dict):
|
if isinstance(msg, dict):
|
||||||
continue
|
_parse_choice_message_content(msg, text_parts=text_parts, images=images)
|
||||||
if isinstance(it.get("text"), str):
|
elif not isinstance(msg, dict):
|
||||||
text_parts.append(str(it.get("text")))
|
_harvest_http_urls_from_value(choice, images, depth=0)
|
||||||
elif isinstance(it.get("image"), str):
|
|
||||||
images.append(str(it.get("image")))
|
if not text_parts and not images:
|
||||||
elif isinstance(it.get("image_url"), str):
|
for root in roots:
|
||||||
images.append(str(it.get("image_url")))
|
msgs = root.get("messages")
|
||||||
elif isinstance(it.get("image_url"), dict) and isinstance(it["image_url"].get("url"), str):
|
if isinstance(msgs, list) and msgs:
|
||||||
images.append(str(it["image_url"]["url"]))
|
_parse_messages_array_for_assistant(msgs, text_parts=text_parts, images=images)
|
||||||
elif isinstance(it.get("b64_json"), str):
|
if text_parts or images:
|
||||||
images.append(f"data:image/png;base64,{it['b64_json']}")
|
break
|
||||||
|
|
||||||
if not images:
|
if not images:
|
||||||
data = resp_json.get("data")
|
for root in roots:
|
||||||
if isinstance(data, list):
|
data = root.get("data")
|
||||||
|
if not isinstance(data, list):
|
||||||
|
continue
|
||||||
for it in data:
|
for it in data:
|
||||||
if not isinstance(it, dict):
|
if not isinstance(it, dict):
|
||||||
continue
|
continue
|
||||||
if isinstance(it.get("url"), str):
|
if isinstance(it.get("url"), str) and str(it.get("url")).strip():
|
||||||
images.append(str(it.get("url")))
|
images.append(str(it.get("url")).strip())
|
||||||
if isinstance(it.get("b64_json"), str):
|
if isinstance(it.get("b64_json"), str) and str(it.get("b64_json")).strip():
|
||||||
images.append(f"data:image/png;base64,{it['b64_json']}")
|
images.append(f"data:image/png;base64,{it['b64_json']}")
|
||||||
|
|
||||||
|
if not images:
|
||||||
|
for root in roots:
|
||||||
|
for key in ("results", "artifacts", "task_outputs"):
|
||||||
|
arr = root.get(key)
|
||||||
|
if not isinstance(arr, list):
|
||||||
|
continue
|
||||||
|
for it in arr:
|
||||||
|
if not isinstance(it, dict):
|
||||||
|
continue
|
||||||
|
for ik in ("image", "url", "output_image_url", "image_url"):
|
||||||
|
v = it.get(ik)
|
||||||
|
if isinstance(v, str) and v.strip().startswith(("http://", "https://")):
|
||||||
|
images.append(v.strip())
|
||||||
|
elif isinstance(v, dict) and isinstance(v.get("url"), str):
|
||||||
|
u = str(v.get("url")).strip()
|
||||||
|
if u.startswith(("http://", "https://")):
|
||||||
|
images.append(u)
|
||||||
|
|
||||||
|
# De-dupe URLs while preserving order
|
||||||
|
seen_u: set[str] = set()
|
||||||
|
uniq_images: list[str] = []
|
||||||
|
for u in images:
|
||||||
|
if u not in seen_u:
|
||||||
|
seen_u.add(u)
|
||||||
|
uniq_images.append(u)
|
||||||
|
images = uniq_images
|
||||||
|
|
||||||
return "\n".join([x for x in text_parts if x]).strip(), images
|
return "\n".join([x for x in text_parts if x]).strip(), images
|
||||||
|
|
||||||
|
|
||||||
|
def _diag_fill_choices(out: dict[str, Any], ch: Any, *, prefix: str) -> None:
|
||||||
|
out[f"{prefix}choices_typename"] = type(ch).__name__
|
||||||
|
out[f"{prefix}choices_len"] = len(ch) if isinstance(ch, list) else None
|
||||||
|
if not isinstance(ch, list) or not ch or not isinstance(ch[0], dict):
|
||||||
|
return
|
||||||
|
c0 = ch[0]
|
||||||
|
out[f"{prefix}choice0_keys"] = sorted(c0.keys())[:24]
|
||||||
|
msg = c0.get("message")
|
||||||
|
if isinstance(msg, dict):
|
||||||
|
out[f"{prefix}msg_keys"] = sorted(msg.keys())[:24]
|
||||||
|
c = msg.get("content")
|
||||||
|
out[f"{prefix}content_typename"] = type(c).__name__
|
||||||
|
if isinstance(c, list):
|
||||||
|
out[f"{prefix}content_len"] = len(c)
|
||||||
|
if c:
|
||||||
|
out[f"{prefix}content0_typename"] = type(c[0]).__name__
|
||||||
|
if isinstance(c[0], dict):
|
||||||
|
out[f"{prefix}content0_keys"] = sorted(c[0].keys())[:24]
|
||||||
|
elif isinstance(c, dict):
|
||||||
|
out[f"{prefix}content_keys"] = sorted(c.keys())[:24]
|
||||||
|
elif c is None:
|
||||||
|
out[f"{prefix}content_is_null"] = True
|
||||||
|
|
||||||
|
|
||||||
|
def build_extract_diag_empty(obj: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Compact shape hints when extraction yielded nothing (stderr / chat debug)."""
|
||||||
|
out: dict[str, Any] = {"top_level_keys": sorted(obj.keys())[:40]}
|
||||||
|
ch_top = obj.get("choices")
|
||||||
|
if ch_top is not None:
|
||||||
|
_diag_fill_choices(out, ch_top, prefix="")
|
||||||
|
outp = obj.get("output")
|
||||||
|
if isinstance(outp, dict):
|
||||||
|
out["output_keys"] = sorted(outp.keys())[:40]
|
||||||
|
ch = outp.get("choices")
|
||||||
|
_diag_fill_choices(out, ch, prefix="output_")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def redact_response_for_debug(obj: Any, *, max_chars: int = 2800) -> str:
|
||||||
|
"""JSON preview for error surfaces (truncate; redact long base64 / data URLs)."""
|
||||||
|
|
||||||
|
def _walk(x: Any) -> Any:
|
||||||
|
if isinstance(x, dict):
|
||||||
|
return {str(k): _walk(v) for k, v in x.items()}
|
||||||
|
if isinstance(x, list):
|
||||||
|
return [_walk(v) for v in x[:80]]
|
||||||
|
if isinstance(x, str):
|
||||||
|
s = x
|
||||||
|
if ";base64," in s and s.strip().startswith("data:") and len(s) > 120:
|
||||||
|
h, _, _ = s.partition(";base64,")
|
||||||
|
return f"{h};base64,<redacted {len(s)} chars>"
|
||||||
|
if len(s) > 500:
|
||||||
|
return s[:400] + f"...<truncated {len(s) - 400}>"
|
||||||
|
return s
|
||||||
|
return x
|
||||||
|
|
||||||
|
try:
|
||||||
|
txt = json.dumps(_walk(obj), ensure_ascii=False, default=str)
|
||||||
|
except Exception:
|
||||||
|
txt = str(obj)
|
||||||
|
if len(txt) > max_chars:
|
||||||
|
return txt[: max_chars - 24] + "\n…<truncated>"
|
||||||
|
return txt
|
||||||
|
|
||||||
|
|
||||||
|
def format_extract_diag(diag: dict[str, Any], *, max_chars: int = 900) -> str:
|
||||||
|
try:
|
||||||
|
s = json.dumps(diag, ensure_ascii=False, default=str)
|
||||||
|
except Exception:
|
||||||
|
s = str(diag)
|
||||||
|
if len(s) > max_chars:
|
||||||
|
return s[: max_chars - 20] + "…<truncated>"
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"build_extract_diag_empty",
|
||||||
"compress_data_url_image",
|
"compress_data_url_image",
|
||||||
|
"dashscope_multimodal_http_ok",
|
||||||
|
"dashscope_native_multimodal_url_from_compatible_base",
|
||||||
|
"download_http_url_bytes",
|
||||||
|
"env_image_expert_download_timeout_sec",
|
||||||
|
"redact_response_for_debug",
|
||||||
|
"env_image_expert_api_key",
|
||||||
|
"env_image_expert_base_url",
|
||||||
|
"env_image_expert_chat_endpoint",
|
||||||
|
"env_image_expert_model",
|
||||||
"env_ocr_lane_api_key",
|
"env_ocr_lane_api_key",
|
||||||
"env_ocr_lane_base_url",
|
"env_ocr_lane_base_url",
|
||||||
"env_ocr_lane_chat_endpoint",
|
"env_ocr_lane_chat_endpoint",
|
||||||
"env_ocr_lane_model",
|
"env_ocr_lane_model",
|
||||||
"extract_text_and_images",
|
"extract_text_and_images",
|
||||||
|
"format_extract_diag",
|
||||||
"is_data_url",
|
"is_data_url",
|
||||||
"join_url",
|
"join_url",
|
||||||
"post_with_retry",
|
"post_with_retry",
|
||||||
|
|
|
||||||
|
|
@ -1,28 +1,373 @@
|
||||||
"""Legacy image+text payloads for DashScope-style gateways.
|
"""Legacy image+text payloads for DashScope-style gateways.
|
||||||
|
|
||||||
Uses ``{"image":...}/{"text":...}`` or typed compatible-mode blocks on ``/chat/completions`` only.
|
Uses ``{"image":...}/{"text":...}`` or typed compatible-mode blocks on ``/chat/completions`` only.
|
||||||
Prefer :mod:`oclaw.platform.llm.image_ocr_client` for OpenAI-compatible vision (图片专家已改用该路径).
|
The **image specialist** uses this module from:
|
||||||
|
|
||||||
|
- :mod:`~oclaw.runtime.agents.specialist_agent` (orchestration temp sessions)
|
||||||
|
- :mod:`~oclaw.runtime.direct_loop` when ``skill_binding_role=="image"`` (**gateway /chat UI**), so vision
|
||||||
|
turns never hit :class:`~oclaw.platform.llm.transports.openai_responses.OpenAIResponsesModel` unless explicitly disabled via env.
|
||||||
|
|
||||||
|
Alignment with Alibaba ``dashscope.MultiModalConversation`` examples (**message shape**):
|
||||||
|
``messages = [{"role": "user", "content": [{"image": "<url or data:...>"}, {"text": "..."}]}]``
|
||||||
|
matches our non--``typed`` branch (same ``image`` / ``text`` keys as the SDK doc).
|
||||||
|
|
||||||
|
Lane separation (**not** OCR):
|
||||||
|
- Resolved as ``kwargs …`` from the user's **chosen chat model/profile** first, then ``AIA_IMAGE_EXPERT_*`` when a field is empty.
|
||||||
|
The **`AIA_OCR_*`** variables remain for **`query_image_attachment`** / OCR downgrade only.
|
||||||
|
- Bearer HTTP targets ``BASE_URL`` + ``AIA_IMAGE_EXPERT_CHAT_ENDPOINT`` (default ``/chat/completions``), not ``MultiModalConversation`` SDK.
|
||||||
|
|
||||||
|
SDK-style extras:
|
||||||
|
- Optional top-level fields (``stream``, ``n``, ``watermark``, ``negative_prompt``, ``prompt_extend``, ``size``, …):
|
||||||
|
use ``DASHSCOPE_IMAGE_*`` env vars or JSON in ``AIA_IMAGE_EXPERT_REQUEST_EXTRA`` (alias: ``AIA_LEGACY_IMAGE_REQUEST_EXTRA``).
|
||||||
|
|
||||||
|
Compatibility roots:
|
||||||
|
- For OpenAI-compat multimodal, use ``AIA_IMAGE_EXPERT_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1`` etc.
|
||||||
|
|
||||||
|
For OpenAI-style ``image_url`` chat payloads (tool OCR / multimodal downgrade), use :mod:`oclaw.platform.llm.image_ocr_client`.
|
||||||
|
|
||||||
|
Chat UI **图片专家**端到端说明(与其它链路隔离的变更边界)见仓库内 ``docs/IMAGE_SPECIALIST_LANE.md``.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
|
from oclaw.platform.files.attachment_assets import AttachmentAssetStore
|
||||||
from oclaw.platform.llm.image_http_common import (
|
from oclaw.platform.llm.image_http_common import (
|
||||||
|
build_extract_diag_empty,
|
||||||
compress_data_url_image,
|
compress_data_url_image,
|
||||||
env_ocr_lane_api_key,
|
dashscope_multimodal_http_ok,
|
||||||
env_ocr_lane_base_url,
|
dashscope_native_multimodal_url_from_compatible_base,
|
||||||
env_ocr_lane_model,
|
download_http_url_bytes,
|
||||||
env_ocr_lane_chat_endpoint,
|
env_image_expert_api_key,
|
||||||
|
env_image_expert_base_url,
|
||||||
|
env_image_expert_chat_endpoint,
|
||||||
|
env_image_expert_model,
|
||||||
extract_text_and_images,
|
extract_text_and_images,
|
||||||
|
format_extract_diag,
|
||||||
is_data_url,
|
is_data_url,
|
||||||
join_url,
|
join_url,
|
||||||
post_with_retry,
|
post_with_retry,
|
||||||
|
redact_response_for_debug,
|
||||||
)
|
)
|
||||||
from oclaw.runtime.prompt_templates import render_prompt
|
from oclaw.runtime.prompt_templates import render_prompt
|
||||||
|
|
||||||
|
IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH = (
|
||||||
|
"请根据用户上传的图片作答:描述可见场景、物体与文字;不确定处请标明。"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def collect_legacy_lane_images_from_attachments(
|
||||||
|
attachments: list[dict[str, Any]] | None,
|
||||||
|
*,
|
||||||
|
max_images: int = 3,
|
||||||
|
) -> list[str]:
|
||||||
|
"""Normalize incoming UI/store attachments to URLs/data URLs for :func:`send_legacy_image_messages`."""
|
||||||
|
from oclaw.platform.files.attachment_assets import attachment_id_to_data_url
|
||||||
|
|
||||||
|
cap = max(1, min(int(max_images), 12))
|
||||||
|
out: list[str] = []
|
||||||
|
for att in attachments or []:
|
||||||
|
if not isinstance(att, dict):
|
||||||
|
continue
|
||||||
|
t = str(att.get("type") or "").strip().lower()
|
||||||
|
if t == "image_ref":
|
||||||
|
aid = str(att.get("attachment_id") or "").strip()
|
||||||
|
if not aid:
|
||||||
|
continue
|
||||||
|
data_url = attachment_id_to_data_url(aid, mime=str(att.get("mime") or ""))
|
||||||
|
if data_url:
|
||||||
|
out.append(data_url)
|
||||||
|
elif t in ("input_image", "image"):
|
||||||
|
raw = str(att.get("image_base64") or att.get("data") or "").strip()
|
||||||
|
if raw:
|
||||||
|
mime = str(att.get("mime") or "image/jpeg")
|
||||||
|
if raw.startswith("data:"):
|
||||||
|
out.append(raw)
|
||||||
|
else:
|
||||||
|
out.append(f"data:{mime};base64,{raw}")
|
||||||
|
elif t == "image_url":
|
||||||
|
u = str(att.get("url") or "").strip()
|
||||||
|
if u:
|
||||||
|
out.append(u)
|
||||||
|
if len(out) >= cap:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_legacy_output_image_urls(resp_images: Any, *, max_items: int = 12) -> list[str]:
|
||||||
|
"""Flatten provider ``images`` / content parts to HTTP/data URLs (strings only)."""
|
||||||
|
cap = max(1, min(int(max_items), 24))
|
||||||
|
out: list[str] = []
|
||||||
|
if resp_images is None:
|
||||||
|
return out
|
||||||
|
if isinstance(resp_images, str):
|
||||||
|
s = resp_images.strip()
|
||||||
|
if s:
|
||||||
|
out.append(s)
|
||||||
|
return out[:cap]
|
||||||
|
if not isinstance(resp_images, list):
|
||||||
|
return out
|
||||||
|
for it in resp_images[:cap]:
|
||||||
|
if isinstance(it, str):
|
||||||
|
s = it.strip()
|
||||||
|
if s:
|
||||||
|
out.append(s)
|
||||||
|
continue
|
||||||
|
if not isinstance(it, dict):
|
||||||
|
continue
|
||||||
|
u = it.get("image")
|
||||||
|
if isinstance(u, str) and u.strip():
|
||||||
|
out.append(u.strip())
|
||||||
|
continue
|
||||||
|
u = it.get("url")
|
||||||
|
if isinstance(u, str) and u.strip():
|
||||||
|
out.append(u.strip())
|
||||||
|
continue
|
||||||
|
iu = it.get("image_url")
|
||||||
|
if isinstance(iu, str) and iu.strip():
|
||||||
|
out.append(iu.strip())
|
||||||
|
elif isinstance(iu, dict):
|
||||||
|
u2 = iu.get("url")
|
||||||
|
if isinstance(u2, str) and u2.strip():
|
||||||
|
out.append(u2.strip())
|
||||||
|
return out[:cap]
|
||||||
|
|
||||||
|
|
||||||
|
def materialize_legacy_response_output_attachments(
|
||||||
|
resp_images: Any,
|
||||||
|
*,
|
||||||
|
max_images: int = 3,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
"""Persist remote/base64 model outputs as ``image_ref`` / ``image_url`` rows for chat UI."""
|
||||||
|
cap = max(1, min(int(max_images), 12))
|
||||||
|
produced: list[dict[str, Any]] = []
|
||||||
|
urls = normalize_legacy_output_image_urls(resp_images, max_items=cap)
|
||||||
|
if not urls:
|
||||||
|
return produced
|
||||||
|
store = AttachmentAssetStore()
|
||||||
|
for idx, item in enumerate(urls[:cap], start=1):
|
||||||
|
s = str(item or "").strip()
|
||||||
|
if not s:
|
||||||
|
continue
|
||||||
|
if s.startswith("data:") and ";base64," in s:
|
||||||
|
head, b64 = s.split(";base64,", 1)
|
||||||
|
mime = head.replace("data:", "", 1) or "image/png"
|
||||||
|
try:
|
||||||
|
blob = base64.b64decode(b64.encode("ascii"))
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
meta = store.save_bytes(blob, filename=f"image-output-{idx}.png", mime=mime)
|
||||||
|
produced.append(
|
||||||
|
{
|
||||||
|
"type": "image_ref",
|
||||||
|
"attachment_id": meta.attachment_id,
|
||||||
|
"name": meta.name,
|
||||||
|
"mime": meta.mime,
|
||||||
|
"bytes": meta.bytes,
|
||||||
|
"width": meta.width,
|
||||||
|
"height": meta.height,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
elif s.startswith("http://") or s.startswith("https://"):
|
||||||
|
try:
|
||||||
|
blob, ctype = download_http_url_bytes(s)
|
||||||
|
if blob:
|
||||||
|
mime = (ctype.split(";", 1)[0].strip() if ctype else "") or "image/png"
|
||||||
|
ext = ".png"
|
||||||
|
if mime == "image/jpeg":
|
||||||
|
ext = ".jpg"
|
||||||
|
elif mime == "image/webp":
|
||||||
|
ext = ".webp"
|
||||||
|
elif mime == "image/gif":
|
||||||
|
ext = ".gif"
|
||||||
|
meta = store.save_bytes(blob, filename=f"image-output-{idx}{ext}", mime=mime)
|
||||||
|
produced.append(
|
||||||
|
{
|
||||||
|
"type": "image_ref",
|
||||||
|
"attachment_id": meta.attachment_id,
|
||||||
|
"name": meta.name,
|
||||||
|
"mime": meta.mime,
|
||||||
|
"bytes": meta.bytes,
|
||||||
|
"width": meta.width,
|
||||||
|
"height": meta.height,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
produced.append({"type": "image_url", "url": s, "name": f"image-output-{idx}.png"})
|
||||||
|
return produced
|
||||||
|
|
||||||
|
|
||||||
|
def legacy_image_turn_bundle(resp: dict[str, Any]) -> tuple[bool, str, list[dict[str, Any]]]:
|
||||||
|
"""Interpret ``send_legacy_image_messages`` result for persistence (text-only vision answers allowed)."""
|
||||||
|
ok = bool(resp.get("ok"))
|
||||||
|
text = str(resp.get("text") or "").strip()
|
||||||
|
if not ok:
|
||||||
|
err = str(resp.get("error") or "").strip()
|
||||||
|
return False, f"Image generation failed: {err or 'unknown error'}", []
|
||||||
|
raw_urls = normalize_legacy_output_image_urls(resp.get("images"), max_items=6)
|
||||||
|
imgs = materialize_legacy_response_output_attachments(raw_urls, max_images=3)
|
||||||
|
if not imgs and raw_urls:
|
||||||
|
imgs = [
|
||||||
|
{"type": "image_url", "url": u, "name": f"image-output-{i}.png"}
|
||||||
|
for i, u in enumerate(raw_urls[:3], start=1)
|
||||||
|
if u.startswith(("http://", "https://", "data:"))
|
||||||
|
]
|
||||||
|
if imgs:
|
||||||
|
return True, text, imgs
|
||||||
|
if text:
|
||||||
|
return True, text, []
|
||||||
|
diag = resp.get("extract_diag")
|
||||||
|
hint = ""
|
||||||
|
if isinstance(diag, dict) and diag:
|
||||||
|
hint = format_extract_diag(diag, max_chars=1400)
|
||||||
|
red = resp.get("provider_response_redacted")
|
||||||
|
if isinstance(red, str) and red.strip():
|
||||||
|
hint = f"{hint}\nprovider_json={red.strip()}" if hint else f"provider_json={red.strip()}"
|
||||||
|
base = "Image specialist failed: empty response from provider."
|
||||||
|
out_msg = f"{base} {hint}".strip()
|
||||||
|
if len(out_msg) > 12_000:
|
||||||
|
out_msg = out_msg[:11_980] + "\n…<truncated>"
|
||||||
|
return False, out_msg, []
|
||||||
|
|
||||||
|
|
||||||
|
def legacy_image_assistant_body_with_placeholder(
|
||||||
|
*,
|
||||||
|
lang: str | None,
|
||||||
|
body_text: str,
|
||||||
|
produced: list[dict[str, Any]] | None,
|
||||||
|
) -> str:
|
||||||
|
"""If the model returned images but no visible text, use the standard chat placeholder (ZH/EN).
|
||||||
|
|
||||||
|
Shared by ``direct_loop`` (gateway /chat) and ``specialist_agent`` (temp sessions).
|
||||||
|
"""
|
||||||
|
if str(body_text or "").strip():
|
||||||
|
return str(body_text or "")
|
||||||
|
if produced:
|
||||||
|
return (
|
||||||
|
"Generated image (see attachment below)."
|
||||||
|
if str(lang or "").startswith("en")
|
||||||
|
else "已生成图片(见下方附件)。"
|
||||||
|
)
|
||||||
|
return str(body_text or "")
|
||||||
|
|
||||||
|
|
||||||
|
def _truthy_env(name: str) -> bool:
|
||||||
|
return str(os.getenv(name) or "").strip().lower() in ("1", "true", "yes", "on")
|
||||||
|
|
||||||
|
|
||||||
|
def _redact_payload_for_stderr(obj: Any) -> Any:
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
return {str(k): _redact_payload_for_stderr(v) for k, v in obj.items()}
|
||||||
|
if isinstance(obj, list):
|
||||||
|
return [_redact_payload_for_stderr(x) for x in obj]
|
||||||
|
if isinstance(obj, str):
|
||||||
|
s = obj
|
||||||
|
if ";base64," in s and s.strip().startswith("data:") and len(s) > 120:
|
||||||
|
h, _, _ = s.partition(";base64,")
|
||||||
|
return f"{h};base64,<redacted ~{len(s) - len(h) - 8} chars>"
|
||||||
|
return s
|
||||||
|
return obj
|
||||||
|
|
||||||
|
|
||||||
|
def _stderr_debug_image_legacy(url: str, payload: dict[str, Any]) -> None:
|
||||||
|
if not _truthy_env("AIA_IMAGE_EXPERT_DEBUG_PRINT_PAYLOAD"):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
txt = json.dumps(_redact_payload_for_stderr(dict(payload)), ensure_ascii=False, indent=2, default=str)
|
||||||
|
sys.stderr.write(f"\n[oclaw image_legacy] POST {url}\n{txt}\n\n")
|
||||||
|
sys.stderr.flush()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _extra_request_fields_from_env() -> dict[str, Any]:
|
||||||
|
"""Merge JSON from ``AIA_IMAGE_EXPERT_REQUEST_EXTRA`` (or legacy alias ``AIA_LEGACY_IMAGE_REQUEST_EXTRA``)."""
|
||||||
|
raw = (os.getenv("AIA_IMAGE_EXPERT_REQUEST_EXTRA") or os.getenv("AIA_LEGACY_IMAGE_REQUEST_EXTRA") or "").strip()
|
||||||
|
if not raw:
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
return parsed if isinstance(parsed, dict) else {}
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _dashscope_image_env_kw() -> dict[str, Any]:
|
||||||
|
"""Map `_local/system.env.example` ``DASHSCOPE_IMAGE_*`` vars to multimodal/top-level kwargs (parity with SDK samples)."""
|
||||||
|
out: dict[str, Any] = {}
|
||||||
|
raw_n = (os.getenv("DASHSCOPE_IMAGE_N") or "").strip()
|
||||||
|
if raw_n.isdigit():
|
||||||
|
out["n"] = max(1, min(int(raw_n), 6))
|
||||||
|
|
||||||
|
wm = (os.getenv("DASHSCOPE_IMAGE_WATERMARK") or "").strip().lower()
|
||||||
|
if wm in ("1", "true", "yes", "on"):
|
||||||
|
out["watermark"] = True
|
||||||
|
elif wm in ("0", "false", "no", "off"):
|
||||||
|
out["watermark"] = False
|
||||||
|
|
||||||
|
raw_stream = (os.getenv("DASHSCOPE_IMAGE_STREAM") or "").strip().lower()
|
||||||
|
if raw_stream in ("1", "true", "yes", "on"):
|
||||||
|
out["stream"] = True
|
||||||
|
elif raw_stream in ("0", "false", "no", "off"):
|
||||||
|
out["stream"] = False
|
||||||
|
|
||||||
|
neg = os.getenv("DASHSCOPE_IMAGE_NEGATIVE_PROMPT")
|
||||||
|
if neg is not None:
|
||||||
|
ns = str(neg)
|
||||||
|
if ns.strip():
|
||||||
|
out["negative_prompt"] = ns
|
||||||
|
|
||||||
|
pe = (os.getenv("DASHSCOPE_IMAGE_PROMPT_EXTEND") or "").strip().lower()
|
||||||
|
if pe in ("1", "true", "yes", "on"):
|
||||||
|
out["prompt_extend"] = True
|
||||||
|
elif pe in ("0", "false", "no", "off"):
|
||||||
|
out["prompt_extend"] = False
|
||||||
|
|
||||||
|
size = (os.getenv("DASHSCOPE_IMAGE_SIZE") or "").strip()
|
||||||
|
if size:
|
||||||
|
out["size"] = size
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _openai_compatible_vision_content(images: list[str], prompt_text: str) -> list[dict[str, Any]]:
|
||||||
|
"""DashScope *compatible-mode* / OpenAI Chat Completions vision shape (NOT ``{"image":..., "text":...}``)."""
|
||||||
|
blocks: list[dict[str, Any]] = []
|
||||||
|
for img in images:
|
||||||
|
blocks.append({"type": "image_url", "image_url": {"url": img}})
|
||||||
|
blocks.append({"type": "text", "text": str(prompt_text or "").strip()})
|
||||||
|
return blocks
|
||||||
|
|
||||||
|
|
||||||
|
def _model_triggers_dashscope_native_fallback(model_name: str) -> bool:
|
||||||
|
"""``qwen-image`` on OpenAI-compat ``/chat/completions`` often returns ``message.content=null``."""
|
||||||
|
if str(os.getenv("AIA_IMAGE_EXPERT_DISABLE_DASHSCOPE_NATIVE_FALLBACK") or "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
"on",
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
if str(os.getenv("AIA_IMAGE_EXPERT_FORCE_DASHSCOPE_NATIVE") or "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
"on",
|
||||||
|
):
|
||||||
|
return True
|
||||||
|
m = (model_name or "").strip().lower()
|
||||||
|
needle = (os.getenv("AIA_IMAGE_EXPERT_NATIVE_FALLBACK_MODEL_SUBSTR") or "qwen-image").strip().lower()
|
||||||
|
return bool(needle) and needle in m
|
||||||
|
|
||||||
|
|
||||||
def _http_content_blocks(images: list[str], prompt: str, *, typed: bool) -> list[dict[str, Any]]:
|
def _http_content_blocks(images: list[str], prompt: str, *, typed: bool) -> list[dict[str, Any]]:
|
||||||
prompt_text = str(prompt or "").strip() or render_prompt("image/default_edit_prompt.zh.md", strict=True)
|
prompt_text = str(prompt or "").strip() or render_prompt("image/default_edit_prompt.zh.md", strict=True)
|
||||||
|
|
@ -44,21 +389,21 @@ def send_legacy_image_messages(
|
||||||
api_key: str | None = None,
|
api_key: str | None = None,
|
||||||
base_url: str | None = None,
|
base_url: str | None = None,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
"""Legacy multimodal HTTP (non--OpenAI-``image_url`` schema). Optional; specialists use OCR client."""
|
"""Legacy multimodal HTTP (non--OpenAI-``image_url`` schema); image specialist uses this entry point."""
|
||||||
resolved_base_url = (base_url or env_ocr_lane_base_url()).strip()
|
resolved_base_url = (base_url or env_image_expert_base_url()).strip()
|
||||||
resolved_api_key = (api_key or env_ocr_lane_api_key()).strip()
|
resolved_api_key = (api_key or env_image_expert_api_key()).strip()
|
||||||
model_name = ((model or "").strip() or env_ocr_lane_model())
|
model_name = ((model or "").strip() or env_image_expert_model())
|
||||||
endpoint = env_ocr_lane_chat_endpoint()
|
endpoint = env_image_expert_chat_endpoint()
|
||||||
url = join_url(resolved_base_url, endpoint)
|
url = join_url(resolved_base_url, endpoint)
|
||||||
if not resolved_api_key or not resolved_base_url:
|
if not resolved_api_key or not resolved_base_url:
|
||||||
return {
|
return {
|
||||||
"ok": False,
|
"ok": False,
|
||||||
"error": "missing AIA_OCR_API_KEY or AIA_OCR_BASE_URL (or pass api_key and base_url)",
|
"error": "missing AIA_IMAGE_EXPERT_API_KEY or AIA_IMAGE_EXPERT_BASE_URL (or pass api_key and base_url)",
|
||||||
}
|
}
|
||||||
if not model_name:
|
if not model_name:
|
||||||
return {
|
return {
|
||||||
"ok": False,
|
"ok": False,
|
||||||
"error": "missing AIA_OCR_MODEL (or pass model=...) — no default model id",
|
"error": "missing model (chosen profile/model=… or set AIA_IMAGE_EXPERT_MODEL)",
|
||||||
}
|
}
|
||||||
if not images:
|
if not images:
|
||||||
return {"ok": False, "error": "at least one image input is required"}
|
return {"ok": False, "error": "at least one image input is required"}
|
||||||
|
|
@ -78,22 +423,28 @@ def send_legacy_image_messages(
|
||||||
if not selected:
|
if not selected:
|
||||||
return {"ok": False, "error": "no usable image input (expected URL or data URL)"}
|
return {"ok": False, "error": "no usable image input (expected URL or data URL)"}
|
||||||
|
|
||||||
prefer_typed_http = "compatible-mode" in resolved_base_url.lower()
|
# Compatible-mode expects OpenAI-style ``image_url`` + ``text`` parts; DashScope-native HTTP uses plain ``{"image"}`` blocks.
|
||||||
content_multi = _http_content_blocks(selected, prompt, typed=prefer_typed_http)
|
use_openai_blocks = "compatible-mode" in resolved_base_url.lower()
|
||||||
content_multi_fallback = _http_content_blocks(selected, prompt, typed=not prefer_typed_http)
|
prompt_plain = str(prompt or "").strip() or render_prompt("image/default_edit_prompt.zh.md", strict=True)
|
||||||
|
if use_openai_blocks:
|
||||||
|
content_multi = _openai_compatible_vision_content(selected, prompt_plain)
|
||||||
|
else:
|
||||||
|
content_multi = _http_content_blocks(selected, prompt, typed=False)
|
||||||
|
|
||||||
headers = {
|
headers = {
|
||||||
"Authorization": f"Bearer {resolved_api_key}",
|
"Authorization": f"Bearer {resolved_api_key}",
|
||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
}
|
}
|
||||||
payload_multi = {
|
extra_ds = _dashscope_image_env_kw()
|
||||||
|
extra_json = _extra_request_fields_from_env()
|
||||||
|
extra = {**extra_ds, **extra_json}
|
||||||
|
core_multi = {
|
||||||
"model": model_name,
|
"model": model_name,
|
||||||
"messages": [{"role": "user", "content": content_multi}],
|
"messages": [{"role": "user", "content": content_multi}],
|
||||||
}
|
}
|
||||||
payload_single = {
|
payload_multi = {**extra, **core_multi}
|
||||||
"model": model_name,
|
|
||||||
"messages": [{"role": "user", "content": content_multi_fallback[-2:] if len(content_multi_fallback) >= 2 else content_multi_fallback}],
|
_stderr_debug_image_legacy(url, payload_multi)
|
||||||
}
|
|
||||||
|
|
||||||
with httpx.Client(timeout=float(timeout_sec)) as client:
|
with httpx.Client(timeout=float(timeout_sec)) as client:
|
||||||
try:
|
try:
|
||||||
|
|
@ -102,45 +453,152 @@ def send_legacy_image_messages(
|
||||||
return {"ok": False, "error": f"http request failed: {type(e).__name__}: {e}", "backend_shape": "multi"}
|
return {"ok": False, "error": f"http request failed: {type(e).__name__}: {e}", "backend_shape": "multi"}
|
||||||
|
|
||||||
if r.status_code >= 400:
|
if r.status_code >= 400:
|
||||||
try:
|
return {"ok": False, "error": f"http {r.status_code}: {r.text[:800]}", "backend_shape": "multi"}
|
||||||
r2 = post_with_retry(client, url=url, headers=headers, payload=payload_single)
|
|
||||||
except Exception as e:
|
|
||||||
return {
|
|
||||||
"ok": False,
|
|
||||||
"error": f"http fallback request failed: {type(e).__name__}: {e}",
|
|
||||||
"backend_shape": "single-fallback-failed",
|
|
||||||
}
|
|
||||||
if r2.status_code >= 400:
|
|
||||||
return {
|
|
||||||
"ok": False,
|
|
||||||
"error": f"http {r2.status_code}: {r2.text[:500]}",
|
|
||||||
"backend_shape": "single-fallback-failed",
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
obj2 = r2.json()
|
|
||||||
except Exception:
|
|
||||||
return {"ok": False, "error": f"non-json response: {r2.text[:500]}", "backend_shape": "single"}
|
|
||||||
text, out_images = extract_text_and_images(obj2 if isinstance(obj2, dict) else {})
|
|
||||||
return {
|
|
||||||
"ok": True,
|
|
||||||
"text": text,
|
|
||||||
"images": out_images,
|
|
||||||
"backend_shape": "single",
|
|
||||||
"input_kind": input_kind[-1:] if input_kind else ["data_url"],
|
|
||||||
}
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
obj = r.json()
|
obj = r.json()
|
||||||
except Exception:
|
except Exception:
|
||||||
return {"ok": False, "error": f"non-json response: {r.text[:500]}", "backend_shape": "multi"}
|
return {"ok": False, "error": f"non-json response: {r.text[:500]}", "backend_shape": "multi"}
|
||||||
text, out_images = extract_text_and_images(obj if isinstance(obj, dict) else {})
|
if not isinstance(obj, dict):
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"error": f"expected JSON object from provider, got {type(obj).__name__}",
|
||||||
|
"backend_shape": "multi",
|
||||||
|
"input_kind": input_kind if input_kind else ["data_url"],
|
||||||
|
}
|
||||||
|
body = obj
|
||||||
|
ds_ok, ds_err = dashscope_multimodal_http_ok(body)
|
||||||
|
if not ds_ok:
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"error": ds_err or "provider rejected request (DashScope code/message)",
|
||||||
|
"backend_shape": "multi",
|
||||||
|
"input_kind": input_kind if input_kind else ["data_url"],
|
||||||
|
}
|
||||||
|
text, out_images = extract_text_and_images(body)
|
||||||
|
compat_extract_diag: dict[str, Any] | None = None
|
||||||
|
native_fallback_meta: dict[str, Any] = {}
|
||||||
|
|
||||||
|
if (
|
||||||
|
not str(text or "").strip()
|
||||||
|
and not out_images
|
||||||
|
and use_openai_blocks
|
||||||
|
and _model_triggers_dashscope_native_fallback(model_name)
|
||||||
|
):
|
||||||
|
compat_extract_diag = build_extract_diag_empty(body)
|
||||||
|
native_url = dashscope_native_multimodal_url_from_compatible_base(resolved_base_url)
|
||||||
|
if not native_url:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": False,
|
||||||
|
"hint": "set AIA_IMAGE_EXPERT_DASHSCOPE_NATIVE_URL or use a compatible-mode base_url",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
content_native = _http_content_blocks(selected, prompt, typed=False)
|
||||||
|
param_merge = {**_dashscope_image_env_kw(), **_extra_request_fields_from_env()}
|
||||||
|
param_merge.pop("stream", None)
|
||||||
|
payload_native = {
|
||||||
|
"model": model_name,
|
||||||
|
"input": {"messages": [{"role": "user", "content": content_native}]},
|
||||||
|
"parameters": {"result_format": "message", **param_merge},
|
||||||
|
}
|
||||||
|
_stderr_debug_image_legacy(native_url, payload_native)
|
||||||
|
try:
|
||||||
|
r2 = post_with_retry(client, url=native_url, headers=headers, payload=payload_native)
|
||||||
|
except Exception as e:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"error": f"{type(e).__name__}: {e}",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
if r2.status_code >= 400:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"http_status": int(r2.status_code),
|
||||||
|
"body_head": r2.text[:600],
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
b2 = r2.json()
|
||||||
|
except Exception as e:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"error": f"json: {type(e).__name__}: {e}",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
if isinstance(b2, dict):
|
||||||
|
ok2, err2 = dashscope_multimodal_http_ok(b2)
|
||||||
|
if not ok2:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"dashscope_error": err2 or "business code",
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
t2, im2 = extract_text_and_images(b2)
|
||||||
|
if str(t2 or "").strip() or im2:
|
||||||
|
text, out_images = t2, im2
|
||||||
|
body = b2
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"succeeded": True,
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"succeeded": False,
|
||||||
|
"native_diag": build_extract_diag_empty(b2),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
native_fallback_meta = {
|
||||||
|
"attempted": True,
|
||||||
|
"url": native_url,
|
||||||
|
"error": "native response not a JSON object",
|
||||||
|
}
|
||||||
|
|
||||||
|
extract_diag: dict[str, Any] | None = None
|
||||||
|
provider_response_redacted: str | None = None
|
||||||
|
if not str(text or "").strip() and not out_images:
|
||||||
|
extract_diag = build_extract_diag_empty(body)
|
||||||
|
if compat_extract_diag is not None:
|
||||||
|
extract_diag["openai_compat_empty"] = compat_extract_diag
|
||||||
|
if native_fallback_meta:
|
||||||
|
extract_diag["dashscope_native_fallback"] = native_fallback_meta
|
||||||
|
provider_response_redacted = redact_response_for_debug(body, max_chars=3200)
|
||||||
|
if _truthy_env("AIA_IMAGE_EXPERT_DEBUG_PRINT_PAYLOAD"):
|
||||||
|
try:
|
||||||
|
sys.stderr.write(
|
||||||
|
"\n[oclaw image_legacy] extract_empty extract_diag="
|
||||||
|
+ format_extract_diag(extract_diag, max_chars=4000)
|
||||||
|
+ "\n"
|
||||||
|
+ (provider_response_redacted or "")
|
||||||
|
+ "\n\n"
|
||||||
|
)
|
||||||
|
sys.stderr.flush()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
out_images = normalize_legacy_output_image_urls(out_images, max_items=12)
|
||||||
return {
|
return {
|
||||||
"ok": True,
|
"ok": True,
|
||||||
"text": text,
|
"text": text,
|
||||||
"images": out_images,
|
"images": out_images,
|
||||||
"backend_shape": "multi",
|
"backend_shape": "multi",
|
||||||
"input_kind": input_kind if input_kind else ["data_url"],
|
"input_kind": input_kind if input_kind else ["data_url"],
|
||||||
|
"extract_diag": extract_diag,
|
||||||
|
"provider_response_redacted": provider_response_redacted,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["send_legacy_image_messages"]
|
__all__ = [
|
||||||
|
"IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH",
|
||||||
|
"collect_legacy_lane_images_from_attachments",
|
||||||
|
"legacy_image_assistant_body_with_placeholder",
|
||||||
|
"legacy_image_turn_bundle",
|
||||||
|
"materialize_legacy_response_output_attachments",
|
||||||
|
"normalize_legacy_output_image_urls",
|
||||||
|
"send_legacy_image_messages",
|
||||||
|
]
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@ from __future__ import annotations
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
import uuid
|
import uuid
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
from collections.abc import Callable, Iterable
|
from collections.abc import Callable, Iterable
|
||||||
|
|
@ -12,6 +13,140 @@ from oclaw.platform.llm.transports.base import ChatModel, LLMResponse, LLMToolCa
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def _redact_nested_json_preview(obj: Any, *, max_chars: int = 4000) -> str:
|
||||||
|
"""Best-effort JSON preview for logs (truncate; redact mega data URLs)."""
|
||||||
|
|
||||||
|
try:
|
||||||
|
s = json.dumps(obj, ensure_ascii=False, default=str)
|
||||||
|
except Exception:
|
||||||
|
s = str(obj)
|
||||||
|
|
||||||
|
def _clip_data_urls(t: str) -> str:
|
||||||
|
out_parts: list[str] = []
|
||||||
|
i = 0
|
||||||
|
needle = "data:"
|
||||||
|
while i < len(t):
|
||||||
|
j = t.find(needle, i)
|
||||||
|
if j < 0:
|
||||||
|
out_parts.append(t[i:])
|
||||||
|
break
|
||||||
|
out_parts.append(t[i:j])
|
||||||
|
k = j + len(needle)
|
||||||
|
while k < len(t) and t[k] not in "?;, \n\r\t\"]":
|
||||||
|
k += 1
|
||||||
|
if k < len(t) and t[k:k + 8] == ";base64,":
|
||||||
|
end = k + 8
|
||||||
|
while end < len(t) and t[end] not in "\"}] \n\r\t":
|
||||||
|
end += 1
|
||||||
|
seg = end - (k + 8)
|
||||||
|
out_parts.append(f"data:<redacted base64 ~{seg} chars>")
|
||||||
|
i = end
|
||||||
|
continue
|
||||||
|
out_parts.append(t[j:k])
|
||||||
|
i = k
|
||||||
|
return "".join(out_parts)
|
||||||
|
|
||||||
|
s = _clip_data_urls(s)
|
||||||
|
if len(s) > max_chars:
|
||||||
|
return s[: max_chars - 40] + "\n...<<truncated>>\n..."
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def _deep_redact_for_debug(obj: Any) -> Any:
|
||||||
|
"""Recursive copy for logs: shorten giant ``data:...;base64,...`` and very long strings."""
|
||||||
|
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
return {str(k): _deep_redact_for_debug(v) for k, v in obj.items()}
|
||||||
|
if isinstance(obj, list):
|
||||||
|
return [_deep_redact_for_debug(x) for x in obj]
|
||||||
|
if isinstance(obj, str):
|
||||||
|
s = obj
|
||||||
|
if ";base64," in s and s.strip().startswith("data:") and len(s) > 160:
|
||||||
|
head, _, _tail = s.partition(";base64,")
|
||||||
|
return f"{head};base64,<redacted ~{len(s) - len(head) - 8} b64 chars>"
|
||||||
|
if len(s) > 16_000:
|
||||||
|
return s[:12_000] + f"...<truncated {len(s) - 12_000} chars>"
|
||||||
|
return s
|
||||||
|
return obj
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_int(raw: str | None, default: int, *, max_value: int = 2_000_000) -> int:
|
||||||
|
try:
|
||||||
|
value = int(str(raw or "").strip())
|
||||||
|
except Exception:
|
||||||
|
return default
|
||||||
|
if value < 1:
|
||||||
|
return default
|
||||||
|
return min(value, max_value)
|
||||||
|
|
||||||
|
|
||||||
|
def _log_openai_responses_wire_kwargs(payload: dict[str, Any], *, variant: str, phase: str) -> None:
|
||||||
|
"""Temporary diagnostics: log kwargs passed into ``OpenAI.responses.create`` (after our assembly).
|
||||||
|
|
||||||
|
Enable: ``AIA_RESPONSES_DEBUG_PRINT_PAYLOAD=1`` — emits ``logging.warning`` **and** the same JSON on **stderr**
|
||||||
|
(gateway控制台默认可见,不依赖 logging level)。
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- OpenAI SDK may still apply minor JSON transforms on send; this matches **our** arguments.
|
||||||
|
- Base64 / long ``data:`` URLs are redacted to keep logs readable.
|
||||||
|
"""
|
||||||
|
raw = str(os.getenv("AIA_RESPONSES_DEBUG_PRINT_PAYLOAD") or "").strip().lower()
|
||||||
|
if raw not in ("1", "true", "yes", "on"):
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
dbg = _deep_redact_for_debug(dict(payload))
|
||||||
|
txt = json.dumps(dbg, ensure_ascii=False, indent=2, default=str)
|
||||||
|
cap = _safe_int(os.getenv("AIA_RESPONSES_DEBUG_PRINT_MAX_CHARS"), 120_000, max_value=500_000)
|
||||||
|
if len(txt) > cap:
|
||||||
|
txt = txt[: max(cap - 80, 0)] + "\n...<<truncated>>\n"
|
||||||
|
logger.warning(
|
||||||
|
"openai_responses DEBUG wire kwargs [%s] variant=%s (%d chars):\n%s",
|
||||||
|
phase,
|
||||||
|
variant,
|
||||||
|
len(txt),
|
||||||
|
txt,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
sys.stderr.write(
|
||||||
|
f"\n[oclaw openai_responses DEBUG] phase={phase} variant={variant} chars={len(txt)}\n{txt}\n\n"
|
||||||
|
)
|
||||||
|
sys.stderr.flush()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("openai_responses DEBUG payload serialization failed: %s", exc)
|
||||||
|
|
||||||
|
|
||||||
|
def _openai_sdk_diagnostic_text(exc: BaseException) -> str:
|
||||||
|
"""``str(APIStatusError)`` is often only ``Error code: 400``; validation detail lives in ``body['message']``."""
|
||||||
|
chunks: list[str] = [str(exc)]
|
||||||
|
msg = getattr(exc, "message", None)
|
||||||
|
if isinstance(msg, str) and msg.strip() and msg not in chunks:
|
||||||
|
chunks.append(msg)
|
||||||
|
bod = getattr(exc, "body", None)
|
||||||
|
if isinstance(bod, dict):
|
||||||
|
bm = bod.get("message")
|
||||||
|
if isinstance(bm, str) and bm.strip():
|
||||||
|
chunks.append(bm)
|
||||||
|
err = bod.get("error")
|
||||||
|
if isinstance(err, dict):
|
||||||
|
em = err.get("message")
|
||||||
|
if isinstance(em, str) and em.strip():
|
||||||
|
chunks.append(em)
|
||||||
|
return "\n".join(chunks)
|
||||||
|
|
||||||
|
|
||||||
|
def _is_input_messages_validation_error(exc: BaseException) -> bool:
|
||||||
|
"""Third-party gateways often return 400/422 with Pydantic paths like ``input.messages.0.role``."""
|
||||||
|
m = _openai_sdk_diagnostic_text(exc).lower()
|
||||||
|
if "input.messages" in m:
|
||||||
|
return True
|
||||||
|
if "input should be 'user'" in m and "content" in m:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def _as_dict(obj: Any) -> dict[str, Any] | None:
|
def _as_dict(obj: Any) -> dict[str, Any] | None:
|
||||||
if obj is None:
|
if obj is None:
|
||||||
return None
|
return None
|
||||||
|
|
@ -131,14 +266,56 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
self._client = OpenAI(**kw)
|
self._client = OpenAI(**kw)
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
def _normalize_messages(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
def _strip_leading_system_messages(messages: list[dict[str, Any]]) -> tuple[str, list[dict[str, Any]]]:
|
||||||
|
"""Separate leading ``role=system`` rows for ``instructions=` (Responses API expects user-shaped ``input.messages``)."""
|
||||||
|
msgs = messages or []
|
||||||
|
sys_chunks: list[str] = []
|
||||||
|
i = 0
|
||||||
|
while i < len(msgs):
|
||||||
|
m = msgs[i]
|
||||||
|
if not isinstance(m, dict):
|
||||||
|
break
|
||||||
|
if str(m.get("role") or "").strip().lower() != "system":
|
||||||
|
break
|
||||||
|
c = m.get("content")
|
||||||
|
if isinstance(c, str) and c.strip():
|
||||||
|
sys_chunks.append(c.strip())
|
||||||
|
elif isinstance(c, list):
|
||||||
|
texts: list[str] = []
|
||||||
|
for it in c:
|
||||||
|
if not isinstance(it, dict):
|
||||||
|
continue
|
||||||
|
tt = str(it.get("type") or "").strip().lower()
|
||||||
|
if tt in ("text", "input_text"):
|
||||||
|
xs = str(it.get("text") or "").strip()
|
||||||
|
if xs:
|
||||||
|
texts.append(xs)
|
||||||
|
elif isinstance(it.get("text"), str) and str(it.get("text")).strip():
|
||||||
|
texts.append(str(it.get("text")).strip())
|
||||||
|
if texts:
|
||||||
|
sys_chunks.append("\n".join(texts))
|
||||||
|
i += 1
|
||||||
|
joined = "\n\n".join(sys_chunks).strip()
|
||||||
|
return joined, list(msgs[i:])
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _normalize_messages(
|
||||||
|
messages: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
envelope_openai_message: bool = True,
|
||||||
|
content_chat_completions_parts: bool = False,
|
||||||
|
image_detail_auto: bool = True,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
"""
|
"""
|
||||||
Normalize to the strictest OpenAI-compatible `responses` shape:
|
Build Response input rows used under ``input`` (flat array) or ``input.messages``.
|
||||||
- input.messages[*].role MUST be "user"
|
|
||||||
- input.messages[*].content MUST be a list (content blocks)
|
- **Responses content** (default when ``content_chat_completions_parts=False``): each part is
|
||||||
- text blocks use {"type":"text","text":...}
|
``{"type":"input_text",...}`` / ``{"type":"input_image","image_url":"<str>","detail":"auto"}``
|
||||||
- image blocks use {"type":"image_url","image_url":{"url":"..."}} or {"type":"input_image","image_url":"..."} depending on gateway;
|
(`OpenAI response_input_*` TypedDict surface).
|
||||||
we prefer the common "image_url" block here.
|
- **Chat completions content** (``content_chat_completions_parts=True``): ``text`` /
|
||||||
|
``image_url`` multimodal blocks.
|
||||||
|
- **Envelope** ``envelope_openai_message=True``: ``{"type":"message","role","content":[...]}``;
|
||||||
|
``False``: ``{"role","content":[...]}`` (per some Bailian curls).
|
||||||
"""
|
"""
|
||||||
out: list[dict[str, Any]] = []
|
out: list[dict[str, Any]] = []
|
||||||
for m in messages or []:
|
for m in messages or []:
|
||||||
|
|
@ -147,6 +324,22 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
role = str(m.get("role") or "user").strip().lower() or "user"
|
role = str(m.get("role") or "user").strip().lower() or "user"
|
||||||
content = m.get("content")
|
content = m.get("content")
|
||||||
norm_content: list[dict[str, Any]] = []
|
norm_content: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
def _append_image_part(url_value: str) -> None:
|
||||||
|
if content_chat_completions_parts:
|
||||||
|
norm_content.append({"type": "image_url", "image_url": {"url": url_value}})
|
||||||
|
else:
|
||||||
|
p: dict[str, Any] = {"type": "input_image", "image_url": url_value}
|
||||||
|
if image_detail_auto:
|
||||||
|
p["detail"] = "auto"
|
||||||
|
norm_content.append(p)
|
||||||
|
|
||||||
|
def _append_text_part(txt: str) -> None:
|
||||||
|
if content_chat_completions_parts:
|
||||||
|
norm_content.append({"type": "text", "text": txt})
|
||||||
|
else:
|
||||||
|
norm_content.append({"type": "input_text", "text": txt})
|
||||||
|
|
||||||
if isinstance(content, list):
|
if isinstance(content, list):
|
||||||
for item in content:
|
for item in content:
|
||||||
if isinstance(item, dict) and item.get("type") == "input_image":
|
if isinstance(item, dict) and item.get("type") == "input_image":
|
||||||
|
|
@ -154,39 +347,118 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
b64 = normalize_image_b64_payload(item.get("image_base64") or item.get("data"))
|
b64 = normalize_image_b64_payload(item.get("image_base64") or item.get("data"))
|
||||||
if not b64:
|
if not b64:
|
||||||
continue
|
continue
|
||||||
norm_content.append({"type": "image_url", "image_url": {"url": f"data:{mime};base64,{b64}"}})
|
_append_image_part(f"data:{mime};base64,{b64}")
|
||||||
|
continue
|
||||||
|
if isinstance(item, dict) and str(item.get("type") or "").strip().lower() == "image_url":
|
||||||
|
iu = item.get("image_url")
|
||||||
|
url = ""
|
||||||
|
if isinstance(iu, dict):
|
||||||
|
url = str(iu.get("url") or "").strip()
|
||||||
|
elif isinstance(iu, str):
|
||||||
|
url = iu.strip()
|
||||||
|
if url.startswith(("http://", "https://", "data:")):
|
||||||
|
_append_image_part(url)
|
||||||
continue
|
continue
|
||||||
if isinstance(item, dict) and item.get("type") in ("input_text", "text"):
|
if isinstance(item, dict) and item.get("type") in ("input_text", "text"):
|
||||||
t = str(item.get("text") or "").strip()
|
t = str(item.get("text") or "").strip()
|
||||||
if t:
|
if t:
|
||||||
norm_content.append({"type": "text", "text": t})
|
_append_text_part(t)
|
||||||
continue
|
continue
|
||||||
if isinstance(item, dict):
|
if isinstance(item, dict):
|
||||||
# Best-effort coerce unknown blocks into valid text blocks.
|
itype = str(item.get("type") or "").strip().lower()
|
||||||
|
img_key = item.get("image")
|
||||||
|
if isinstance(img_key, str) and img_key.strip() and itype != "input_image":
|
||||||
|
uu = img_key.strip()
|
||||||
|
if uu.startswith(("http://", "https://", "data:")):
|
||||||
|
_append_image_part(uu)
|
||||||
|
continue
|
||||||
|
if isinstance(item.get("text"), str) and str(item.get("text") or "").strip():
|
||||||
|
_append_text_part(str(item.get("text") or "").strip())
|
||||||
|
continue
|
||||||
s = str(item.get("text") or "").strip()
|
s = str(item.get("text") or "").strip()
|
||||||
if s:
|
if s:
|
||||||
norm_content.append({"type": "text", "text": s})
|
_append_text_part(s)
|
||||||
continue
|
continue
|
||||||
s = str(item or "").strip()
|
s = str(item or "").strip()
|
||||||
if s:
|
if s:
|
||||||
norm_content.append({"type": "text", "text": s})
|
_append_text_part(s)
|
||||||
elif isinstance(content, str):
|
elif isinstance(content, str):
|
||||||
txt = content.strip()
|
txt = content.strip()
|
||||||
if txt:
|
if txt:
|
||||||
norm_content.append({"type": "text", "text": txt})
|
_append_text_part(txt)
|
||||||
elif content is not None:
|
elif content is not None:
|
||||||
s = str(content).strip()
|
s = str(content).strip()
|
||||||
if s:
|
if s:
|
||||||
norm_content.append({"type": "text", "text": s})
|
_append_text_part(s)
|
||||||
|
|
||||||
if not norm_content:
|
if not norm_content:
|
||||||
# Keep shape valid even for empty messages.
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Enforce role=user and preserve other roles via prefix tag.
|
coerced_role = "user"
|
||||||
if role != "user":
|
if role != "user":
|
||||||
prefix = "assistant" if role == "assistant" else ("system" if role == "system" else role)
|
prefix = "assistant" if role == "assistant" else ("system" if role == "system" else role)
|
||||||
norm_content.insert(0, {"type": "text", "text": f"[{prefix}]"})
|
norm_content.insert(
|
||||||
out.append({"role": "user", "content": norm_content})
|
0,
|
||||||
|
OpenAIResponsesModel._role_prefix_part(prefix, content_chat_completions_parts),
|
||||||
|
)
|
||||||
|
|
||||||
|
if envelope_openai_message:
|
||||||
|
out.append({"type": "message", "role": coerced_role, "content": norm_content})
|
||||||
|
else:
|
||||||
|
out.append({"role": coerced_role, "content": norm_content})
|
||||||
|
return out
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _role_prefix_part(prefix: str, chat_parts: bool) -> dict[str, Any]:
|
||||||
|
t = f"[{prefix}]"
|
||||||
|
if chat_parts:
|
||||||
|
return {"type": "text", "text": t}
|
||||||
|
return {"type": "input_text", "text": t}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _responses_input_candidates(
|
||||||
|
msgs: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
flat_responses: bool,
|
||||||
|
prefer_envelope: bool,
|
||||||
|
prefer_chat_parts: bool,
|
||||||
|
) -> list[tuple[str, Any]]:
|
||||||
|
"""Several gateways validate ``input.messages`` differently; try a small deterministic set."""
|
||||||
|
|
||||||
|
def N(env: bool, chat: bool) -> list[dict[str, Any]]:
|
||||||
|
return OpenAIResponsesModel._normalize_messages(
|
||||||
|
msgs,
|
||||||
|
envelope_openai_message=env,
|
||||||
|
content_chat_completions_parts=chat,
|
||||||
|
)
|
||||||
|
|
||||||
|
keys_seen: set[str] = set()
|
||||||
|
out: list[tuple[str, Any]] = []
|
||||||
|
|
||||||
|
def push(tag: str, inp: Any) -> None:
|
||||||
|
try:
|
||||||
|
sk = json.dumps(inp, ensure_ascii=False, sort_keys=True, default=str)
|
||||||
|
except Exception:
|
||||||
|
sk = repr(inp)
|
||||||
|
if sk in keys_seen:
|
||||||
|
return
|
||||||
|
keys_seen.add(sk)
|
||||||
|
out.append((tag, inp))
|
||||||
|
|
||||||
|
if flat_responses:
|
||||||
|
for env in (prefer_envelope, not prefer_envelope):
|
||||||
|
push(f"flat_envelope_{env}", N(env, False))
|
||||||
|
return out
|
||||||
|
|
||||||
|
# Prefer profile/env defaults first (``AIA_RESPONSES_NESTED_CHAT_PARTS`` / envelope toggles), then a fixed
|
||||||
|
# fallback ring so picky gateways still get Chat-shaped ``input.messages`` without dropping multimodal pixels.
|
||||||
|
primary_combo = (prefer_envelope, prefer_chat_parts)
|
||||||
|
fallback_ring = [(False, True), (False, False), (True, False), (True, True)]
|
||||||
|
combos = [primary_combo] + [p for p in fallback_ring if p != primary_combo]
|
||||||
|
for env, chat in combos:
|
||||||
|
norm = N(env, chat)
|
||||||
|
push(f"e{int(env)}c{int(chat)}_messages", {"messages": norm})
|
||||||
|
push(f"e{int(env)}c{int(chat)}_flat_input", norm)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
def chat(
|
def chat(
|
||||||
|
|
@ -196,8 +468,96 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
*,
|
*,
|
||||||
on_token: Optional[Callable[[str], None]] = None,
|
on_token: Optional[Callable[[str], None]] = None,
|
||||||
) -> LLMResponse:
|
) -> LLMResponse:
|
||||||
norm = self._normalize_messages(messages)
|
sys_instructions, rest = OpenAIResponsesModel._strip_leading_system_messages(messages)
|
||||||
# OpenAI-compatible gateways differ: some require input={"messages":[...]} with role=user only.
|
def _env_truthy(name: str) -> bool:
|
||||||
|
return str(os.getenv(name) or "").strip().lower() in ("1", "true", "yes", "on")
|
||||||
|
|
||||||
|
if _env_truthy("AIA_RESPONSES_DEBUG_PRINT_PAYLOAD"):
|
||||||
|
try:
|
||||||
|
sys.stderr.write(
|
||||||
|
"[oclaw openai_responses] chat() ENTERED "
|
||||||
|
f"model={self.model!r} base_url={self.base_url!r} "
|
||||||
|
f"messages={len(messages)} tools={len(tools or [])}\n"
|
||||||
|
)
|
||||||
|
sys.stderr.flush()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Default: nested ``input.messages`` rows use **Responses** parts (``input_text`` / ``input_image``).
|
||||||
|
# Flat Bailian ``input`` array: ``AIA_RESPONSES_INPUT_IS_FLAT_LIST=1`` (optional ``AIA_RESPONSES_FLAT_OPENAI_ENVELOPE``).
|
||||||
|
flat_raw = os.getenv("AIA_RESPONSES_INPUT_IS_FLAT_LIST")
|
||||||
|
if flat_raw is not None and str(flat_raw).strip():
|
||||||
|
flat_responses = _env_truthy("AIA_RESPONSES_INPUT_IS_FLAT_LIST")
|
||||||
|
else:
|
||||||
|
legacy = os.getenv("AIA_RESPONSES_INPUT_USE_MESSAGES_ARRAY")
|
||||||
|
if legacy is None or not str(legacy).strip():
|
||||||
|
flat_responses = False
|
||||||
|
else:
|
||||||
|
flat_responses = not _env_truthy("AIA_RESPONSES_INPUT_USE_MESSAGES_ARRAY")
|
||||||
|
|
||||||
|
# Nested ``input.messages`` defaults to **Responses** content parts (``input_*``) + ``type:message``,
|
||||||
|
# matching OpenAI ``EasyInputMessageParam`` / Model Studio expanded examples. Some proxies wrongly
|
||||||
|
# expect Chat ``text``/``image_url`` parts → ``AIA_RESPONSES_NESTED_CHAT_PARTS=1``.
|
||||||
|
if flat_responses:
|
||||||
|
envelope_openai_message = _env_truthy("AIA_RESPONSES_FLAT_OPENAI_ENVELOPE")
|
||||||
|
nested_chat_parts = False
|
||||||
|
else:
|
||||||
|
envelope_openai_message = True
|
||||||
|
nested_chat_parts = _env_truthy("AIA_RESPONSES_NESTED_CHAT_PARTS")
|
||||||
|
|
||||||
|
def _normalize_batch(msgs: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||||
|
return OpenAIResponsesModel._normalize_messages(
|
||||||
|
msgs,
|
||||||
|
envelope_openai_message=envelope_openai_message,
|
||||||
|
content_chat_completions_parts=nested_chat_parts,
|
||||||
|
)
|
||||||
|
|
||||||
|
msgs_for_norm: list[dict[str, Any]]
|
||||||
|
if rest:
|
||||||
|
msgs_for_norm = rest
|
||||||
|
norm = _normalize_batch(rest)
|
||||||
|
inst_kw: dict[str, Any] = {}
|
||||||
|
if sys_instructions.strip():
|
||||||
|
si = sys_instructions.strip()
|
||||||
|
if len(si) > 80_000:
|
||||||
|
si = si[:80_000] + "\n...[truncated]"
|
||||||
|
inst_kw = {"instructions": si}
|
||||||
|
if not norm:
|
||||||
|
msgs_for_norm = messages
|
||||||
|
norm = _normalize_batch(messages)
|
||||||
|
inst_kw = {}
|
||||||
|
else:
|
||||||
|
msgs_for_norm = messages
|
||||||
|
norm = _normalize_batch(messages)
|
||||||
|
inst_kw = {}
|
||||||
|
|
||||||
|
explicit_variants = os.getenv("AIA_RESPONSES_INPUT_VARIANTS")
|
||||||
|
bu_norm = (self.base_url or "").strip().lower()
|
||||||
|
# CRITICAL: do **not** treat empty ``base_url`` as "official" here. Clients often omit storing the
|
||||||
|
# default URL in profiles while still hitting third-party gateways via env/SDK defaults—but then we
|
||||||
|
# would wrongly disable alternate ``input`` shapes and only ever send ``primary``.
|
||||||
|
host_is_explicit_official_api = bool(bu_norm) and ("api.openai.com" in bu_norm)
|
||||||
|
if explicit_variants is not None and str(explicit_variants).strip():
|
||||||
|
multi_shape = _env_truthy("AIA_RESPONSES_INPUT_VARIANTS")
|
||||||
|
else:
|
||||||
|
multi_shape = not host_is_explicit_official_api
|
||||||
|
|
||||||
|
if flat_responses:
|
||||||
|
primary_input: Any = norm
|
||||||
|
else:
|
||||||
|
primary_input = {"messages": norm}
|
||||||
|
|
||||||
|
input_candidates: list[tuple[str, Any]]
|
||||||
|
if multi_shape and not _env_truthy("AIA_RESPONSES_DISABLE_INPUT_VARIANTS"):
|
||||||
|
input_candidates = OpenAIResponsesModel._responses_input_candidates(
|
||||||
|
msgs_for_norm,
|
||||||
|
flat_responses=flat_responses,
|
||||||
|
prefer_envelope=envelope_openai_message,
|
||||||
|
prefer_chat_parts=nested_chat_parts,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
input_candidates = [("primary", primary_input)]
|
||||||
|
|
||||||
stream_errors: list[str] = []
|
stream_errors: list[str] = []
|
||||||
b = str(self.base_url or "").strip().lower()
|
b = str(self.base_url or "").strip().lower()
|
||||||
force_disable = str(os.getenv("AIA_LLM_THINKING_FORCE_DISABLED") or "").strip().lower() in ("1", "true", "yes", "on")
|
force_disable = str(os.getenv("AIA_LLM_THINKING_FORCE_DISABLED") or "").strip().lower() in ("1", "true", "yes", "on")
|
||||||
|
|
@ -217,26 +577,33 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
reasoning_effort = str(getattr(self, "reasoning_effort", "") or "").strip().lower()
|
reasoning_effort = str(getattr(self, "reasoning_effort", "") or "").strip().lower()
|
||||||
if reasoning_effort not in ("low", "medium", "high"):
|
if reasoning_effort not in ("low", "medium", "high"):
|
||||||
reasoning_effort = ""
|
reasoning_effort = ""
|
||||||
stream_variants: list[dict[str, Any]] = [
|
base_stream_template: dict[str, Any] = {
|
||||||
{
|
**thinking,
|
||||||
**thinking,
|
**inst_kw,
|
||||||
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
||||||
"model": self.model,
|
"model": self.model,
|
||||||
"input": {"messages": norm},
|
"tools": tools or None,
|
||||||
"tools": tools or None,
|
"stream": True,
|
||||||
"stream": True,
|
}
|
||||||
},
|
|
||||||
{
|
|
||||||
**thinking,
|
|
||||||
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
|
||||||
"model": self.model,
|
|
||||||
"input": norm,
|
|
||||||
"tools": tools or None,
|
|
||||||
"stream": True,
|
|
||||||
},
|
|
||||||
]
|
|
||||||
try:
|
try:
|
||||||
for payload in stream_variants:
|
for cand_label, responses_input in input_candidates:
|
||||||
|
stream_payload = {**base_stream_template, "input": responses_input}
|
||||||
|
if _env_truthy("AIA_RESPONSES_LOG_PAYLOAD_SUMMARY"):
|
||||||
|
logger.warning(
|
||||||
|
"openai_responses [%s] payload summary (redacted): %s",
|
||||||
|
cand_label,
|
||||||
|
_redact_nested_json_preview(
|
||||||
|
{
|
||||||
|
"model": self.model,
|
||||||
|
"input": responses_input,
|
||||||
|
"has_instructions": bool(inst_kw.get("instructions")),
|
||||||
|
"tools_n": len(tools or []),
|
||||||
|
"stream": True,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
payload = stream_payload
|
||||||
|
_log_openai_responses_wire_kwargs(payload, variant=str(cand_label), phase="stream")
|
||||||
try:
|
try:
|
||||||
stream = self._client.responses.create(**payload)
|
stream = self._client.responses.create(**payload)
|
||||||
text, tool_calls, final_resp = parse_openai_responses_stream_events(stream, on_token=on_token)
|
text, tool_calls, final_resp = parse_openai_responses_stream_events(stream, on_token=on_token)
|
||||||
|
|
@ -246,6 +613,8 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
text = ot
|
text = ot
|
||||||
if on_token:
|
if on_token:
|
||||||
on_token(text)
|
on_token(text)
|
||||||
|
if cand_label != input_candidates[0][0]:
|
||||||
|
logger.info("openai_responses: succeeded with input variant %s", cand_label)
|
||||||
return LLMResponse(content=text, tool_calls=tool_calls)
|
return LLMResponse(content=text, tool_calls=tool_calls)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
emsg = str(exc)
|
emsg = str(exc)
|
||||||
|
|
@ -258,6 +627,11 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
eb["thinking"] = {"type": "disabled"}
|
eb["thinking"] = {"type": "disabled"}
|
||||||
forced["extra_body"] = eb
|
forced["extra_body"] = eb
|
||||||
forced.pop("reasoning_effort", None)
|
forced.pop("reasoning_effort", None)
|
||||||
|
_log_openai_responses_wire_kwargs(
|
||||||
|
forced,
|
||||||
|
variant=f"{cand_label}_thinking_disabled",
|
||||||
|
phase="stream_retry_thinking",
|
||||||
|
)
|
||||||
stream = self._client.responses.create(**forced)
|
stream = self._client.responses.create(**forced)
|
||||||
text, tool_calls, final_resp = parse_openai_responses_stream_events(stream, on_token=on_token)
|
text, tool_calls, final_resp = parse_openai_responses_stream_events(stream, on_token=on_token)
|
||||||
if (not text.strip()) and final_resp:
|
if (not text.strip()) and final_resp:
|
||||||
|
|
@ -266,31 +640,36 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
text = ot
|
text = ot
|
||||||
if on_token:
|
if on_token:
|
||||||
on_token(text)
|
on_token(text)
|
||||||
|
if cand_label != input_candidates[0][0]:
|
||||||
|
logger.info("openai_responses: succeeded with input variant %s", cand_label)
|
||||||
return LLMResponse(content=text, tool_calls=tool_calls)
|
return LLMResponse(content=text, tool_calls=tool_calls)
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
if _env_truthy("AIA_RESPONSES_LOG_API_ERROR_DETAIL"):
|
||||||
|
eb = getattr(exc, "body", None)
|
||||||
|
if eb is not None:
|
||||||
|
logger.warning(
|
||||||
|
"responses.create stream [%s] body: %s", cand_label, eb
|
||||||
|
)
|
||||||
|
if len(input_candidates) > 1 and _is_input_messages_validation_error(exc):
|
||||||
|
stream_errors.append(f"{cand_label}:{emsg}")
|
||||||
|
continue
|
||||||
stream_errors.append(emsg)
|
stream_errors.append(emsg)
|
||||||
continue
|
break
|
||||||
raise RuntimeError("; ".join(stream_errors) or "responses_stream_all_variants_failed")
|
raise RuntimeError("; ".join(stream_errors) or "responses_stream_all_variants_failed")
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
logger.info("responses stream failed; fallback to non-stream (%s)", exc)
|
logger.info("responses stream failed; fallback to non-stream (%s)", exc)
|
||||||
nonstream_errors: list[str] = []
|
nonstream_errors: list[str] = []
|
||||||
for payload in (
|
base_nonstream = {
|
||||||
{
|
**thinking,
|
||||||
**thinking,
|
**inst_kw,
|
||||||
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
||||||
"model": self.model,
|
"model": self.model,
|
||||||
"input": {"messages": norm},
|
"tools": tools or None,
|
||||||
"tools": tools or None,
|
}
|
||||||
},
|
for cand_label, responses_input in input_candidates:
|
||||||
{
|
payload = {**base_nonstream, "input": responses_input}
|
||||||
**thinking,
|
_log_openai_responses_wire_kwargs(payload, variant=str(cand_label), phase="non_stream")
|
||||||
**({"reasoning_effort": reasoning_effort} if reasoning_effort else {}),
|
|
||||||
"model": self.model,
|
|
||||||
"input": norm,
|
|
||||||
"tools": tools or None,
|
|
||||||
},
|
|
||||||
):
|
|
||||||
try:
|
try:
|
||||||
resp = self._client.responses.create(**payload)
|
resp = self._client.responses.create(**payload)
|
||||||
d = _as_dict(resp) or {}
|
d = _as_dict(resp) or {}
|
||||||
|
|
@ -298,6 +677,8 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
tool_calls = _collect_tool_calls_from_response_dict(d)
|
tool_calls = _collect_tool_calls_from_response_dict(d)
|
||||||
if on_token and text:
|
if on_token and text:
|
||||||
on_token(text)
|
on_token(text)
|
||||||
|
if cand_label != input_candidates[0][0]:
|
||||||
|
logger.info("openai_responses non-stream: succeeded with input variant %s", cand_label)
|
||||||
return LLMResponse(content=text, tool_calls=tool_calls)
|
return LLMResponse(content=text, tool_calls=tool_calls)
|
||||||
except Exception as e2:
|
except Exception as e2:
|
||||||
emsg2 = str(e2)
|
emsg2 = str(e2)
|
||||||
|
|
@ -309,17 +690,36 @@ class OpenAIResponsesModel(ChatModel):
|
||||||
eb["thinking"] = {"type": "disabled"}
|
eb["thinking"] = {"type": "disabled"}
|
||||||
forced["extra_body"] = eb
|
forced["extra_body"] = eb
|
||||||
forced.pop("reasoning_effort", None)
|
forced.pop("reasoning_effort", None)
|
||||||
|
_log_openai_responses_wire_kwargs(
|
||||||
|
forced,
|
||||||
|
variant=f"{cand_label}_thinking_disabled",
|
||||||
|
phase="non_stream_retry_thinking",
|
||||||
|
)
|
||||||
resp = self._client.responses.create(**forced)
|
resp = self._client.responses.create(**forced)
|
||||||
d = _as_dict(resp) or {}
|
d = _as_dict(resp) or {}
|
||||||
text = str(d.get("output_text") or "")
|
text = str(d.get("output_text") or "")
|
||||||
tool_calls = _collect_tool_calls_from_response_dict(d)
|
tool_calls = _collect_tool_calls_from_response_dict(d)
|
||||||
if on_token and text:
|
if on_token and text:
|
||||||
on_token(text)
|
on_token(text)
|
||||||
|
if cand_label != input_candidates[0][0]:
|
||||||
|
logger.info(
|
||||||
|
"openai_responses non-stream: succeeded with input variant %s",
|
||||||
|
cand_label,
|
||||||
|
)
|
||||||
return LLMResponse(content=text, tool_calls=tool_calls)
|
return LLMResponse(content=text, tool_calls=tool_calls)
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
if _env_truthy("AIA_RESPONSES_LOG_API_ERROR_DETAIL"):
|
||||||
|
eb = getattr(e2, "body", None)
|
||||||
|
if eb is not None:
|
||||||
|
logger.warning(
|
||||||
|
"responses.create non-stream [%s] body: %s", cand_label, eb
|
||||||
|
)
|
||||||
|
if len(input_candidates) > 1 and _is_input_messages_validation_error(e2):
|
||||||
|
nonstream_errors.append(f"{cand_label}:{emsg2}")
|
||||||
|
continue
|
||||||
nonstream_errors.append(emsg2)
|
nonstream_errors.append(emsg2)
|
||||||
continue
|
break
|
||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
"openai_responses_request_failed: "
|
"openai_responses_request_failed: "
|
||||||
+ " | ".join([str(exc)] + nonstream_errors[-2:])
|
+ " | ".join([str(exc)] + nonstream_errors[-2:])
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@ from __future__ import annotations
|
||||||
import base64
|
import base64
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import re
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import sys
|
import sys
|
||||||
import hashlib
|
import hashlib
|
||||||
|
|
@ -1436,6 +1437,78 @@ class SqliteStore:
|
||||||
(aid, tid, uid, sid, src, ts),
|
(aid, tid, uid, sid, src, ts),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _attachment_ids_for_acl_from_payload_items(items: list[dict[str, Any]]) -> list[str]:
|
||||||
|
"""Collect stable attachment ids from message attachment JSON (for ACL rows)."""
|
||||||
|
out: list[str] = []
|
||||||
|
seen: set[str] = set()
|
||||||
|
ref_types = {"image_ref", "video_ref", "text_ref", "binary_ref"}
|
||||||
|
relay_re = re.compile(r"^relay://attachments/[^/]+/([a-f0-9]{8,64})$", re.IGNORECASE)
|
||||||
|
for a in items:
|
||||||
|
if not isinstance(a, dict):
|
||||||
|
continue
|
||||||
|
typ = str(a.get("type") or "").strip().lower()
|
||||||
|
aid = str(a.get("attachment_id") or a.get("attachmentId") or "").strip().lower()
|
||||||
|
if typ in ref_types and aid:
|
||||||
|
if aid not in seen:
|
||||||
|
seen.add(aid)
|
||||||
|
out.append(aid)
|
||||||
|
continue
|
||||||
|
if typ != "relay_pointer":
|
||||||
|
continue
|
||||||
|
if not aid:
|
||||||
|
uri = str(a.get("pointer_uri") or "").strip()
|
||||||
|
m = relay_re.match(uri)
|
||||||
|
if m:
|
||||||
|
aid = str(m.group(1) or "").strip().lower()
|
||||||
|
if aid and aid not in seen:
|
||||||
|
seen.add(aid)
|
||||||
|
out.append(aid)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def sync_attachment_acl_from_chat_message_attachments(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
session_id: str,
|
||||||
|
role: str,
|
||||||
|
event_type: str | None,
|
||||||
|
attachments: Any,
|
||||||
|
) -> None:
|
||||||
|
"""Best-effort: link chat_message attachments to session owner so strict ACL downloads work.
|
||||||
|
|
||||||
|
Tool results already link in runtime; assistant rows (image specialist, inbound, etc.)
|
||||||
|
historically did not, which breaks ``AIA_ATTACHMENT_ACL_STRICT=1``.
|
||||||
|
"""
|
||||||
|
sid = str(session_id or "").strip()
|
||||||
|
if not sid or attachments is None:
|
||||||
|
return
|
||||||
|
owner = self.get_ui_session_owner(session_id=sid) or {}
|
||||||
|
tid = str(owner.get("tenant_id") or "").strip()
|
||||||
|
uid = str(owner.get("user_id") or "").strip()
|
||||||
|
if not tid or not uid:
|
||||||
|
return
|
||||||
|
items: list[dict[str, Any]] = []
|
||||||
|
if isinstance(attachments, list):
|
||||||
|
items = [x for x in attachments if isinstance(x, dict)]
|
||||||
|
elif isinstance(attachments, dict):
|
||||||
|
items = [attachments]
|
||||||
|
else:
|
||||||
|
return
|
||||||
|
ids = self._attachment_ids_for_acl_from_payload_items(items)
|
||||||
|
if not ids:
|
||||||
|
return
|
||||||
|
r = str(role or "").strip() or "-"
|
||||||
|
ev = str(event_type or "").strip() or "-"
|
||||||
|
src = f"chat_message:{r}:{ev}"[:240]
|
||||||
|
for aid in ids:
|
||||||
|
self.link_attachment_acl(
|
||||||
|
tenant_id=tid,
|
||||||
|
user_id=uid,
|
||||||
|
session_id=sid,
|
||||||
|
attachment_id=aid,
|
||||||
|
source=src,
|
||||||
|
)
|
||||||
|
|
||||||
def attachment_acl_allows_user(self, *, tenant_id: str, user_id: str, attachment_id: str) -> bool:
|
def attachment_acl_allows_user(self, *, tenant_id: str, user_id: str, attachment_id: str) -> bool:
|
||||||
tid = str(tenant_id or "").strip()
|
tid = str(tenant_id or "").strip()
|
||||||
uid = str(user_id or "").strip()
|
uid = str(user_id or "").strip()
|
||||||
|
|
@ -1922,6 +1995,15 @@ class SqliteStore:
|
||||||
"UPDATE chat_session SET last_message_at = ? WHERE id = ?",
|
"UPDATE chat_session SET last_message_at = ? WHERE id = ?",
|
||||||
(ts, session_id),
|
(ts, session_id),
|
||||||
)
|
)
|
||||||
|
try:
|
||||||
|
self.sync_attachment_acl_from_chat_message_attachments(
|
||||||
|
session_id=str(session_id or "").strip(),
|
||||||
|
role=str(role or ""),
|
||||||
|
event_type=str(event_type or "").strip() or None,
|
||||||
|
attachments=attachments,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
return ChatMessage(
|
return ChatMessage(
|
||||||
id=msg_id,
|
id=msg_id,
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,6 @@
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import Any, Callable, Optional
|
from typing import Any, Callable, Optional
|
||||||
|
|
||||||
|
|
@ -74,36 +75,67 @@ class AttemptRunnerOutput:
|
||||||
outcome: TurnRunOutcome
|
outcome: TurnRunOutcome
|
||||||
|
|
||||||
|
|
||||||
|
def _exception_reason_for_chat(exc: BaseException, *, max_chars: int = 12000) -> str:
|
||||||
|
"""Human-visible attempt failure text (WS/chat reads ``AttemptState.reason``).
|
||||||
|
|
||||||
|
OpenAI Python SDK's ``APIError`` often carries a decoded JSON ``body``; include it so oclaw /chat
|
||||||
|
can show the full upstream validation payload instead of only ``str(exc)``.
|
||||||
|
"""
|
||||||
|
head = f"{type(exc).__name__}:{exc}"
|
||||||
|
blob = ""
|
||||||
|
cur: BaseException | None = exc
|
||||||
|
seen_ids: set[int] = set()
|
||||||
|
depth = 0
|
||||||
|
while cur is not None and depth < 16:
|
||||||
|
if id(cur) in seen_ids:
|
||||||
|
break
|
||||||
|
seen_ids.add(id(cur))
|
||||||
|
depth += 1
|
||||||
|
body = getattr(cur, "body", None)
|
||||||
|
if body is not None:
|
||||||
|
try:
|
||||||
|
blob = json.dumps(body, ensure_ascii=False, indent=2) if isinstance(body, (dict, list)) else str(body)
|
||||||
|
except Exception:
|
||||||
|
blob = str(body)
|
||||||
|
break
|
||||||
|
cur = cur.__cause__
|
||||||
|
out = head if not blob else f"{head}\n\nupstream_json_body:\n{blob}"
|
||||||
|
if len(out) > max_chars:
|
||||||
|
out = out[: max_chars - 24] + "\n...[truncated]"
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _classify_attempt_error(exc: Exception) -> tuple[str, str, bool]:
|
def _classify_attempt_error(exc: Exception) -> tuple[str, str, bool]:
|
||||||
raw = f"{type(exc).__name__}:{exc}"
|
raw = f"{type(exc).__name__}:{exc}"
|
||||||
low = raw.lower()
|
low = raw.lower()
|
||||||
|
reason = _exception_reason_for_chat(exc)
|
||||||
if "relay_envelope_unsupported_version" in low:
|
if "relay_envelope_unsupported_version" in low:
|
||||||
return ("relay_envelope_unsupported_version", raw[:500], False)
|
return ("relay_envelope_unsupported_version", reason, False)
|
||||||
if "relay_envelope_invalid" in low:
|
if "relay_envelope_invalid" in low:
|
||||||
return ("relay_envelope_invalid", raw[:500], False)
|
return ("relay_envelope_invalid", reason, False)
|
||||||
if "interrupted" in low or "cancel" in low or "stopped" in low:
|
if "interrupted" in low or "cancel" in low or "stopped" in low:
|
||||||
return ("control_interrupted", raw[:500], False)
|
return ("control_interrupted", reason, False)
|
||||||
if "api_key" in low or "invalid api key" in low or "unauthorized" in low or "401" in low or "forbidden" in low or "403" in low:
|
if "api_key" in low or "invalid api key" in low or "unauthorized" in low or "401" in low or "forbidden" in low or "403" in low:
|
||||||
return ("auth_invalid_credentials", raw[:500], False)
|
return ("auth_invalid_credentials", reason, False)
|
||||||
if "invalid tool_result sequence" in low or "unexpected tool_use_id" in low:
|
if "invalid tool_result sequence" in low or "unexpected tool_use_id" in low:
|
||||||
return ("tool_replay_protocol_mismatch", raw[:500], True)
|
return ("tool_replay_protocol_mismatch", reason, True)
|
||||||
if "invalid_request" in low or "bad_request" in low or "400" in low:
|
if "invalid_request" in low or "bad_request" in low or "400" in low:
|
||||||
return ("input_invalid_request", raw[:500], False)
|
return ("input_invalid_request", reason, False)
|
||||||
if "context_length" in low or "token limit" in low or "max context" in low:
|
if "context_length" in low or "token limit" in low or "max context" in low:
|
||||||
return ("context_overflow", raw[:500], True)
|
return ("context_overflow", reason, True)
|
||||||
if "tool_loop_guard" in low:
|
if "tool_loop_guard" in low:
|
||||||
return ("tool_loop_guard", raw[:500], False)
|
return ("tool_loop_guard", reason, False)
|
||||||
if "tool_timeout_or_failed" in low or "tool_execution_error" in low:
|
if "tool_timeout_or_failed" in low or "tool_execution_error" in low:
|
||||||
return ("tool_execution_failed", raw[:500], True)
|
return ("tool_execution_failed", reason, True)
|
||||||
if "timeout" in low:
|
if "timeout" in low:
|
||||||
return ("provider_timeout", raw[:500], True)
|
return ("provider_timeout", reason, True)
|
||||||
if "rate" in low and "limit" in low:
|
if "rate" in low and "limit" in low:
|
||||||
return ("provider_rate_limited", raw[:500], True)
|
return ("provider_rate_limited", reason, True)
|
||||||
if "temporary" in low or "temporar" in low:
|
if "temporary" in low or "temporar" in low:
|
||||||
return ("provider_temporary_error", raw[:500], True)
|
return ("provider_temporary_error", reason, True)
|
||||||
if "connection" in low or "network" in low or "503" in low or "502" in low:
|
if "connection" in low or "network" in low or "503" in low or "502" in low:
|
||||||
return ("provider_unavailable", raw[:500], True)
|
return ("provider_unavailable", reason, True)
|
||||||
return ("runtime_unknown_error", raw[:500], False)
|
return ("runtime_unknown_error", reason, False)
|
||||||
|
|
||||||
|
|
||||||
def run_attempt(*, store: Any, data: AttemptRunnerInput) -> AttemptRunnerOutput:
|
def run_attempt(*, store: Any, data: AttemptRunnerInput) -> AttemptRunnerOutput:
|
||||||
|
|
|
||||||
|
|
@ -2,10 +2,9 @@ from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
import time
|
import time
|
||||||
import base64
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import httpx
|
|
||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from dataclasses import dataclass, field
|
from dataclasses import dataclass, field
|
||||||
from typing import Any, Optional
|
from typing import Any, Optional
|
||||||
|
|
@ -14,8 +13,13 @@ from oclaw.runtime.chat.agent import Agent
|
||||||
from oclaw.runtime.chat.agent import GenerationInterrupted
|
from oclaw.runtime.chat.agent import GenerationInterrupted
|
||||||
from oclaw.runtime.agents.network_ops_agent import NetworkOpsAgent
|
from oclaw.runtime.agents.network_ops_agent import NetworkOpsAgent
|
||||||
from oclaw.platform.persistence.sqlite_store import SqliteStore
|
from oclaw.platform.persistence.sqlite_store import SqliteStore
|
||||||
from oclaw.platform.files.attachment_assets import AttachmentAssetStore, attachment_id_to_data_url
|
from oclaw.platform.llm.image_legacy_client import (
|
||||||
from oclaw.platform.llm.image_ocr_client import send_ocr_image_messages
|
IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH,
|
||||||
|
collect_legacy_lane_images_from_attachments,
|
||||||
|
legacy_image_assistant_body_with_placeholder,
|
||||||
|
legacy_image_turn_bundle,
|
||||||
|
send_legacy_image_messages,
|
||||||
|
)
|
||||||
from oclaw.runtime.tools import default_registry
|
from oclaw.runtime.tools import default_registry
|
||||||
from oclaw.runtime.agents.specialists import expert_name_for_specialist
|
from oclaw.runtime.agents.specialists import expert_name_for_specialist
|
||||||
|
|
||||||
|
|
@ -233,56 +237,44 @@ class SpecialistAgentRunner:
|
||||||
try:
|
try:
|
||||||
if step.specialist == "image":
|
if step.specialist == "image":
|
||||||
image_protocol = "messages.content.image"
|
image_protocol = "messages.content.image"
|
||||||
selected_images: list[str] = []
|
selected_images = collect_legacy_lane_images_from_attachments(
|
||||||
for att in parent_task.attachments or []:
|
list(parent_task.attachments or []),
|
||||||
if not isinstance(att, dict):
|
max_images=3,
|
||||||
continue
|
)
|
||||||
t = str(att.get("type") or "").strip().lower()
|
|
||||||
if t == "image_ref":
|
|
||||||
aid = str(att.get("attachment_id") or "").strip()
|
|
||||||
if not aid:
|
|
||||||
continue
|
|
||||||
data_url = attachment_id_to_data_url(aid, mime=str(att.get("mime") or ""))
|
|
||||||
if data_url:
|
|
||||||
selected_images.append(data_url)
|
|
||||||
elif t in ("input_image", "image"):
|
|
||||||
raw = str(att.get("image_base64") or att.get("data") or "").strip()
|
|
||||||
if raw:
|
|
||||||
mime = str(att.get("mime") or "image/jpeg")
|
|
||||||
if raw.startswith("data:"):
|
|
||||||
selected_images.append(raw)
|
|
||||||
else:
|
|
||||||
selected_images.append(f"data:{mime};base64,{raw}")
|
|
||||||
elif t == "image_url":
|
|
||||||
u = str(att.get("url") or "").strip()
|
|
||||||
if u:
|
|
||||||
selected_images.append(u)
|
|
||||||
if len(selected_images) >= 3:
|
|
||||||
break
|
|
||||||
image_input_count = len(selected_images)
|
image_input_count = len(selected_images)
|
||||||
image_input_kind = ["data_url" if s.startswith("data:") else "url" for s in selected_images]
|
image_input_kind = ["data_url" if s.startswith("data:") else "url" for s in selected_images]
|
||||||
if not selected_images:
|
if not selected_images:
|
||||||
output = "Image specialist received no image input."
|
output = "Image specialist received no image input."
|
||||||
ok = False
|
ok = False
|
||||||
else:
|
else:
|
||||||
|
# Use the user's chosen model/session profile (same as specialist routing UI). Wrong model ⇒ upstream HTTP error as-is (no OCR lane, no alternate payload).
|
||||||
_, chosen_model, _ = self._resolve_profile_and_model(step.specialist)
|
_, chosen_model, _ = self._resolve_profile_and_model(step.specialist)
|
||||||
model_name = str(
|
|
||||||
os.getenv("AIA_OCR_MODEL") or getattr(chosen_model, "model", None) or ""
|
|
||||||
).strip() or None
|
|
||||||
api_key = str(getattr(chosen_model, "api_key", "") or "").strip() or None
|
|
||||||
base_url = str(getattr(chosen_model, "base_url", "") or "").strip() or None
|
|
||||||
text_parts = [
|
text_parts = [
|
||||||
str(x).strip()
|
str(x).strip()
|
||||||
for x in (step.objective, step.input_text, parent_task.user_text)
|
for x in (step.objective, step.input_text, parent_task.user_text)
|
||||||
if str(x or "").strip()
|
if str(x or "").strip()
|
||||||
]
|
]
|
||||||
user_text = "\n".join(text_parts) if text_parts else "请根据用户上传的图片作答:描述可见场景、物体与文字;不确定处请标明。"
|
user_text = "\n".join(text_parts) if text_parts else IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH
|
||||||
resp = send_ocr_image_messages(
|
if str(os.getenv("AIA_IMAGE_EXPERT_DEBUG_PRINT_PAYLOAD") or "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
"on",
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
sys.stderr.write(
|
||||||
|
"[oclaw specialist:image] lane=legacy_http → send_legacy_image_messages "
|
||||||
|
"(NOT OpenAIResponsesModel).\n"
|
||||||
|
)
|
||||||
|
sys.stderr.flush()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
resp = send_legacy_image_messages(
|
||||||
images=selected_images,
|
images=selected_images,
|
||||||
prompt=user_text,
|
prompt=user_text,
|
||||||
model=model_name,
|
model=str(getattr(chosen_model, "model", "") or "").strip() or None,
|
||||||
api_key=api_key,
|
api_key=str(getattr(chosen_model, "api_key", "") or "").strip() or None,
|
||||||
base_url=base_url,
|
base_url=str(getattr(chosen_model, "base_url", "") or "").strip() or None,
|
||||||
)
|
)
|
||||||
image_debug_schema = str(resp.get("debug_used_schema") or "").strip()
|
image_debug_schema = str(resp.get("debug_used_schema") or "").strip()
|
||||||
dbg = resp.get("debug_used_debug")
|
dbg = resp.get("debug_used_debug")
|
||||||
|
|
@ -290,91 +282,12 @@ class SpecialistAgentRunner:
|
||||||
image_debug_payload = dbg
|
image_debug_payload = dbg
|
||||||
elif dbg is not None:
|
elif dbg is not None:
|
||||||
image_debug_payload = str(dbg)
|
image_debug_payload = str(dbg)
|
||||||
ok = bool(resp.get("ok"))
|
ok, output, produced_attachments = legacy_image_turn_bundle(resp)
|
||||||
output = str(resp.get("text") or "").strip()
|
output = legacy_image_assistant_body_with_placeholder(
|
||||||
if not ok:
|
lang=self.lang,
|
||||||
err = str(resp.get("error") or "").strip()
|
body_text=output,
|
||||||
output = f"Image generation failed: {err or 'unknown error'}"
|
produced=produced_attachments if ok else None,
|
||||||
elif not output:
|
)
|
||||||
output = "Image processed."
|
|
||||||
# persist output images as attachment assets for UI rendering
|
|
||||||
produced_attachments: list[dict[str, Any]] = []
|
|
||||||
if ok:
|
|
||||||
out_images = resp.get("images")
|
|
||||||
if isinstance(out_images, list):
|
|
||||||
store = AttachmentAssetStore()
|
|
||||||
for idx, item in enumerate(out_images[:3], start=1):
|
|
||||||
s = str(item or "").strip()
|
|
||||||
if not s:
|
|
||||||
continue
|
|
||||||
if s.startswith("data:") and ";base64," in s:
|
|
||||||
head, b64 = s.split(";base64,", 1)
|
|
||||||
mime = head.replace("data:", "", 1) or "image/png"
|
|
||||||
try:
|
|
||||||
blob = base64.b64decode(b64.encode("ascii"))
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
meta = store.save_bytes(
|
|
||||||
blob,
|
|
||||||
filename=f"image-output-{idx}.png",
|
|
||||||
mime=mime,
|
|
||||||
)
|
|
||||||
produced_attachments.append(
|
|
||||||
{
|
|
||||||
"type": "image_ref",
|
|
||||||
"attachment_id": meta.attachment_id,
|
|
||||||
"name": meta.name,
|
|
||||||
"mime": meta.mime,
|
|
||||||
"bytes": meta.bytes,
|
|
||||||
"width": meta.width,
|
|
||||||
"height": meta.height,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
elif s.startswith("http://") or s.startswith("https://"):
|
|
||||||
try:
|
|
||||||
with httpx.Client(timeout=20.0, follow_redirects=True) as client:
|
|
||||||
r = client.get(s)
|
|
||||||
if r.status_code < 400 and r.content:
|
|
||||||
mime = str(r.headers.get("content-type") or "image/png").split(";", 1)[0].strip() or "image/png"
|
|
||||||
ext = ".png"
|
|
||||||
if mime == "image/jpeg":
|
|
||||||
ext = ".jpg"
|
|
||||||
elif mime == "image/webp":
|
|
||||||
ext = ".webp"
|
|
||||||
elif mime == "image/gif":
|
|
||||||
ext = ".gif"
|
|
||||||
meta = store.save_bytes(
|
|
||||||
r.content,
|
|
||||||
filename=f"image-output-{idx}{ext}",
|
|
||||||
mime=mime,
|
|
||||||
)
|
|
||||||
produced_attachments.append(
|
|
||||||
{
|
|
||||||
"type": "image_ref",
|
|
||||||
"attachment_id": meta.attachment_id,
|
|
||||||
"name": meta.name,
|
|
||||||
"mime": meta.mime,
|
|
||||||
"bytes": meta.bytes,
|
|
||||||
"width": meta.width,
|
|
||||||
"height": meta.height,
|
|
||||||
}
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
produced_attachments.append(
|
|
||||||
{
|
|
||||||
"type": "image_url",
|
|
||||||
"url": s,
|
|
||||||
"name": f"image-output-{idx}.png",
|
|
||||||
}
|
|
||||||
)
|
|
||||||
# Treat missing image outputs as failure to avoid false "generated" state.
|
|
||||||
if not produced_attachments:
|
|
||||||
ok = False
|
|
||||||
output = (
|
|
||||||
"Image generation failed: response succeeded but no image output was returned."
|
|
||||||
)
|
|
||||||
self.store.add_message(
|
self.store.add_message(
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
role="assistant",
|
role="assistant",
|
||||||
|
|
|
||||||
|
|
@ -210,8 +210,10 @@ def build_llm_messages(
|
||||||
:func:`~oclaw.runtime.direct_loop._guard_tool_results_for_llm_context`). Omit or leave empty
|
:func:`~oclaw.runtime.direct_loop._guard_tool_results_for_llm_context`). Omit or leave empty
|
||||||
to apply image-blob stripping for every tool row (safe default for callers without turn context).
|
to apply image-blob stripping for every tool row (safe default for callers without turn context).
|
||||||
|
|
||||||
Only the **last** user message may expand attachments into native multimodal ``input_image``;
|
Only the **last** user message may expand attachments into native multimodal ``input_image``
|
||||||
older user attachments are replayed as text metadata only.
|
blocks shaped like ``{"type":"input_image","image_base64","mime"}``; ``OpenAIChatModel`` and
|
||||||
|
``OpenAIResponsesModel`` both accept this shape (see transport normalization code). Older user
|
||||||
|
attachments are replayed as text metadata only.
|
||||||
"""
|
"""
|
||||||
out: list[dict[str, Any]] = [{"role": "system", "content": (system_prompt or "").strip()}]
|
out: list[dict[str, Any]] = [{"role": "system", "content": (system_prompt or "").strip()}]
|
||||||
dropped_unpaired_tool_rows = 0
|
dropped_unpaired_tool_rows = 0
|
||||||
|
|
|
||||||
|
|
@ -1102,6 +1102,101 @@ def _execute_tool_step(
|
||||||
return int((time.perf_counter() - t0) * 1000), results_by_id
|
return int((time.perf_counter() - t0) * 1000), results_by_id
|
||||||
|
|
||||||
|
|
||||||
|
def _maybe_image_specialist_legacy_gateway_turn(
|
||||||
|
*,
|
||||||
|
store: Any,
|
||||||
|
session_id: str,
|
||||||
|
turn_uuid: str,
|
||||||
|
lang: str,
|
||||||
|
model: ChatModel,
|
||||||
|
user_text: str,
|
||||||
|
attachments: list[dict[str, Any]] | None,
|
||||||
|
skill_binding_role: str | None,
|
||||||
|
on_token: Optional[Callable[[str], None]],
|
||||||
|
on_progress: Optional[Callable[[str], None]],
|
||||||
|
) -> TurnRunOutcome | None:
|
||||||
|
"""When the UI selects **image** specialist, skip Responses/chat-model transports.
|
||||||
|
|
||||||
|
Vision/gen HTTP goes through :func:`oclaw.platform.llm.image_legacy_client.send_legacy_image_messages`
|
||||||
|
(``/chat/completions`` lane). Disable with ``AIA_IMAGE_SPECIALIST_DISABLE_LEGACY_GATEWAY_LANE=1``.
|
||||||
|
|
||||||
|
End-to-end notes and safe edit boundaries: ``docs/IMAGE_SPECIALIST_LANE.md``.
|
||||||
|
"""
|
||||||
|
if str(os.getenv("AIA_IMAGE_SPECIALIST_DISABLE_LEGACY_GATEWAY_LANE") or "").strip().lower() in (
|
||||||
|
"1",
|
||||||
|
"true",
|
||||||
|
"yes",
|
||||||
|
"on",
|
||||||
|
):
|
||||||
|
return None
|
||||||
|
if str(skill_binding_role or "").strip().lower() != "image":
|
||||||
|
return None
|
||||||
|
|
||||||
|
from oclaw.platform.llm.image_legacy_client import (
|
||||||
|
IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH,
|
||||||
|
collect_legacy_lane_images_from_attachments,
|
||||||
|
legacy_image_assistant_body_with_placeholder,
|
||||||
|
legacy_image_turn_bundle,
|
||||||
|
send_legacy_image_messages,
|
||||||
|
)
|
||||||
|
|
||||||
|
imgs = collect_legacy_lane_images_from_attachments(attachments)
|
||||||
|
if not imgs:
|
||||||
|
hint_en = "Image specialist received no image input. Attach an image and try again."
|
||||||
|
hint_zh = "图片专家未收到可用的图片输入;请先上传或附上图片后再试。"
|
||||||
|
hint = hint_en if str(lang or "").startswith("en") else hint_zh
|
||||||
|
store.add_message(
|
||||||
|
session_id=session_id,
|
||||||
|
role="assistant",
|
||||||
|
content=hint,
|
||||||
|
turn_uuid=turn_uuid,
|
||||||
|
event_type="assistant_text",
|
||||||
|
)
|
||||||
|
return TurnRunOutcome(
|
||||||
|
final_text=hint,
|
||||||
|
tool_traces=tuple(),
|
||||||
|
handoff_note="image_specialist_legacy_missing_attachment",
|
||||||
|
turn_uuid=turn_uuid,
|
||||||
|
)
|
||||||
|
|
||||||
|
if on_progress:
|
||||||
|
on_progress("oclaw: image specialist (legacy multimodal HTTP)…")
|
||||||
|
|
||||||
|
prompt_plain = str(user_text or "").strip()
|
||||||
|
if not prompt_plain:
|
||||||
|
prompt_plain = IMAGE_SPECIALIST_DEFAULT_PROMPT_ZH
|
||||||
|
|
||||||
|
resp = send_legacy_image_messages(
|
||||||
|
images=imgs,
|
||||||
|
prompt=prompt_plain,
|
||||||
|
model=str(getattr(model, "model", "") or "").strip() or None,
|
||||||
|
api_key=str(getattr(model, "api_key", "") or "").strip() or None,
|
||||||
|
base_url=str(getattr(model, "base_url", "") or "").strip() or None,
|
||||||
|
)
|
||||||
|
ok, body_text, produced = legacy_image_turn_bundle(resp)
|
||||||
|
body_text = legacy_image_assistant_body_with_placeholder(
|
||||||
|
lang=lang,
|
||||||
|
body_text=body_text,
|
||||||
|
produced=produced if ok else None,
|
||||||
|
)
|
||||||
|
store.add_message(
|
||||||
|
session_id=session_id,
|
||||||
|
role="assistant",
|
||||||
|
content=body_text,
|
||||||
|
turn_uuid=turn_uuid,
|
||||||
|
event_type="assistant_text",
|
||||||
|
attachments=(produced or None) if ok else None,
|
||||||
|
)
|
||||||
|
if ok and on_token and body_text:
|
||||||
|
on_token(body_text)
|
||||||
|
return TurnRunOutcome(
|
||||||
|
final_text=body_text,
|
||||||
|
tool_traces=tuple(),
|
||||||
|
handoff_note="image_specialist_legacy_http" if ok else "image_specialist_legacy_upstream_failed",
|
||||||
|
turn_uuid=turn_uuid,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def run_oclaw_direct_loop(
|
def run_oclaw_direct_loop(
|
||||||
*,
|
*,
|
||||||
store: Any,
|
store: Any,
|
||||||
|
|
@ -1149,6 +1244,20 @@ def run_oclaw_direct_loop(
|
||||||
turn_uuid=turn_uuid,
|
turn_uuid=turn_uuid,
|
||||||
event_type="user_text",
|
event_type="user_text",
|
||||||
)
|
)
|
||||||
|
legacy_early = _maybe_image_specialist_legacy_gateway_turn(
|
||||||
|
store=store,
|
||||||
|
session_id=session_id,
|
||||||
|
turn_uuid=turn_uuid,
|
||||||
|
lang=lang,
|
||||||
|
model=model,
|
||||||
|
user_text=str(user_text or ""),
|
||||||
|
attachments=attachments,
|
||||||
|
skill_binding_role=skill_binding_role,
|
||||||
|
on_token=on_token,
|
||||||
|
on_progress=on_progress,
|
||||||
|
)
|
||||||
|
if legacy_early is not None:
|
||||||
|
return legacy_early
|
||||||
|
|
||||||
skill_exec = SkillExecutor(config=ToolExecutionConfig(max_workers=max(1, min(int(max_tool_workers or 8), 32))))
|
skill_exec = SkillExecutor(config=ToolExecutionConfig(max_workers=max(1, min(int(max_tool_workers or 8), 32))))
|
||||||
tool_traces: list[dict[str, Any]] = []
|
tool_traces: list[dict[str, Any]] = []
|
||||||
|
|
|
||||||
106
runtime/operations/scripts/probe_openai_responses_image.py
Normal file
106
runtime/operations/scripts/probe_openai_responses_image.py
Normal file
|
|
@ -0,0 +1,106 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""POST minimal vision payloads to an OpenAI-compatible ``/responses`` endpoint.
|
||||||
|
|
||||||
|
Run locally: loads ``oclaw/_local/system.env`` the same way the gateway does, then POSTs variants.
|
||||||
|
|
||||||
|
Example::
|
||||||
|
|
||||||
|
set OPENAI_BASE_URL=https://...
|
||||||
|
set OPENAI_API_KEY=sk-...
|
||||||
|
set OPENAI_MODEL=qwen-vl-plus
|
||||||
|
python runtime/operations/scripts/probe_openai_responses_image.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||||
|
if str(_REPO_ROOT) not in sys.path:
|
||||||
|
sys.path.insert(0, str(_REPO_ROOT))
|
||||||
|
|
||||||
|
# Same bootstrap as gateway: ``interfaces/http/fastapi_app.py`` calls ``load_system_env()`` so
|
||||||
|
# ``oclaw/_local/system.env`` is merged before reading ``OPENAI_*``.
|
||||||
|
try:
|
||||||
|
from oclaw.platform.config.bootstrap_env import load_system_env
|
||||||
|
|
||||||
|
load_system_env()
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
import httpx
|
||||||
|
except ImportError:
|
||||||
|
print("install httpx: pip install httpx", file=sys.stderr)
|
||||||
|
raise SystemExit(2)
|
||||||
|
|
||||||
|
# 1x1 transparent PNG
|
||||||
|
_PNG_B64 = (
|
||||||
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _variants(model: str, data_uri: str) -> list[tuple[str, dict]]:
|
||||||
|
user_block_resp = {
|
||||||
|
"type": "message",
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "input_text", "text": "What color is this pixel image? One word."},
|
||||||
|
{"type": "input_image", "image_url": data_uri, "detail": "auto"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
user_plain_resp = {
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "input_text", "text": "What color is this pixel image? One word."},
|
||||||
|
{"type": "input_image", "image_url": data_uri, "detail": "auto"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
user_chat = {
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "text", "text": "What color is this pixel image? One word."},
|
||||||
|
{"type": "image_url", "image_url": {"url": data_uri}},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
return [
|
||||||
|
("flat_resp_envelope", {"model": model, "input": [user_block_resp]}),
|
||||||
|
("flat_resp_plain", {"model": model, "input": [user_plain_resp]}),
|
||||||
|
("nested_messages_resp_env", {"model": model, "input": {"messages": [user_block_resp]}}),
|
||||||
|
("nested_messages_resp_plain", {"model": model, "input": {"messages": [user_plain_resp]}}),
|
||||||
|
("nested_messages_chat", {"model": model, "input": {"messages": [user_chat]}}),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
base = (os.getenv("OPENAI_BASE_URL") or "https://api.openai.com/v1").strip().rstrip("/")
|
||||||
|
key = (os.getenv("OPENAI_API_KEY") or "").strip()
|
||||||
|
model = (os.getenv("OPENAI_MODEL") or "gpt-4o-mini").strip()
|
||||||
|
if not key:
|
||||||
|
print("OPENAI_API_KEY is required", file=sys.stderr)
|
||||||
|
raise SystemExit(2)
|
||||||
|
|
||||||
|
url = f"{base}/responses"
|
||||||
|
data_uri = f"data:image/png;base64,{_PNG_B64}"
|
||||||
|
headers = {"Authorization": f"Bearer {key}", "Content-Type": "application/json"}
|
||||||
|
|
||||||
|
print(f"POST {url}", flush=True)
|
||||||
|
for label, body in _variants(model, data_uri):
|
||||||
|
try:
|
||||||
|
r = httpx.post(url, headers=headers, json=body, timeout=120.0)
|
||||||
|
except httpx.HTTPError as exc:
|
||||||
|
print(f"\n=== {label} transport_error {exc}", flush=True)
|
||||||
|
continue
|
||||||
|
tail = (r.text or "")[:2400]
|
||||||
|
print(f"\n=== {label} status={r.status_code}", flush=True)
|
||||||
|
print(tail, flush=True)
|
||||||
|
if r.status_code < 400:
|
||||||
|
print("(first successful variant — use this shape for your gateway)", flush=True)
|
||||||
|
break
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -58,7 +58,7 @@ class AttachmentAclBackfillTests(unittest.TestCase):
|
||||||
def _h(self) -> dict[str, str]:
|
def _h(self) -> dict[str, str]:
|
||||||
return {"authorization": f"Bearer {self.token}"}
|
return {"authorization": f"Bearer {self.token}"}
|
||||||
|
|
||||||
def test_backfill_enables_strict_acl_download(self) -> None:
|
def test_add_message_links_acl_for_strict_download(self) -> None:
|
||||||
store = SqliteStore(str(self.db))
|
store = SqliteStore(str(self.db))
|
||||||
sess = store.create_session_for_user(title="t", tenant_id=self.tenant_id, user_id=self.alice_id)
|
sess = store.create_session_for_user(title="t", tenant_id=self.tenant_id, user_id=self.alice_id)
|
||||||
ast = AttachmentAssetStore()
|
ast = AttachmentAssetStore()
|
||||||
|
|
@ -75,11 +75,12 @@ class AttachmentAclBackfillTests(unittest.TestCase):
|
||||||
prev = os.environ.get("AIA_ATTACHMENT_ACL_STRICT")
|
prev = os.environ.get("AIA_ATTACHMENT_ACL_STRICT")
|
||||||
os.environ["AIA_ATTACHMENT_ACL_STRICT"] = "1"
|
os.environ["AIA_ATTACHMENT_ACL_STRICT"] = "1"
|
||||||
try:
|
try:
|
||||||
# Strict mode: without backfill, this should be forbidden (not avatar).
|
# Strict mode: add_message should have written attachment_acl (not only tool_result rows).
|
||||||
r0 = self.client.get(f"/admin/api/chat/attachments/{aid}", headers=self._h())
|
r0 = self.client.get(f"/admin/api/chat/attachments/{aid}", headers=self._h())
|
||||||
self.assertEqual(r0.status_code, 403, r0.text)
|
self.assertEqual(r0.status_code, 200, r0.text)
|
||||||
|
self.assertTrue(len(r0.content) > 10)
|
||||||
|
|
||||||
# Backfill ACL for tenant, then strict download should succeed.
|
# Backfill remains idempotent.
|
||||||
res = store.backfill_attachment_acl_from_messages(tenant_id=self.tenant_id, limit_messages=5000)
|
res = store.backfill_attachment_acl_from_messages(tenant_id=self.tenant_id, limit_messages=5000)
|
||||||
self.assertTrue(res.get("ok"), res)
|
self.assertTrue(res.get("ok"), res)
|
||||||
r1 = self.client.get(f"/admin/api/chat/attachments/{aid}", headers=self._h())
|
r1 = self.client.get(f"/admin/api/chat/attachments/{aid}", headers=self._h())
|
||||||
|
|
|
||||||
60
tests/test_image_expert_env_lane.py
Normal file
60
tests/test_image_expert_env_lane.py
Normal file
|
|
@ -0,0 +1,60 @@
|
||||||
|
"""Image specialist lane must use ``AIA_IMAGE_EXPERT_*``, not ``AIA_OCR_*``."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from oclaw.platform.llm.image_legacy_client import send_legacy_image_messages
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_fails_when_only_ocr_env_configured(monkeypatch: object) -> None:
|
||||||
|
monkeypatch.setenv("AIA_OCR_API_KEY", "k-ocr")
|
||||||
|
monkeypatch.setenv("AIA_OCR_BASE_URL", "https://ocr.example/v1")
|
||||||
|
monkeypatch.setenv("AIA_OCR_MODEL", "ocr-model")
|
||||||
|
for k in ("AIA_IMAGE_EXPERT_API_KEY", "AIA_IMAGE_EXPERT_BASE_URL", "AIA_IMAGE_EXPERT_MODEL"):
|
||||||
|
monkeypatch.delenv(k, raising=False)
|
||||||
|
out = send_legacy_image_messages(
|
||||||
|
images=["https://example.com/x.png"],
|
||||||
|
prompt="hi",
|
||||||
|
model=None,
|
||||||
|
api_key=None,
|
||||||
|
base_url=None,
|
||||||
|
)
|
||||||
|
assert out.get("ok") is False
|
||||||
|
err = str(out.get("error") or "")
|
||||||
|
assert "AIA_IMAGE_EXPERT" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_accepts_explicit_kwargs_without_expert_env(monkeypatch: object) -> None:
|
||||||
|
"""Explicit api_key/base_url/model override env (for scripts/tests only)."""
|
||||||
|
for k in (
|
||||||
|
"AIA_OCR_API_KEY",
|
||||||
|
"AIA_OCR_BASE_URL",
|
||||||
|
"AIA_IMAGE_EXPERT_API_KEY",
|
||||||
|
"AIA_IMAGE_EXPERT_BASE_URL",
|
||||||
|
):
|
||||||
|
monkeypatch.delenv(k, raising=False)
|
||||||
|
|
||||||
|
recorded: dict[str, object] = {}
|
||||||
|
|
||||||
|
def fake_post(client, *, url: str, headers: dict[str, str], payload: dict[str, object]) -> object:
|
||||||
|
recorded["url"] = url
|
||||||
|
|
||||||
|
class R:
|
||||||
|
status_code = 200
|
||||||
|
|
||||||
|
def json(_self): # noqa: PLR6301
|
||||||
|
return {"choices": [{"message": {"content": [{"text": "ok"}]}}]}
|
||||||
|
|
||||||
|
return R()
|
||||||
|
|
||||||
|
import oclaw.platform.llm.image_legacy_client as mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(mod, "post_with_retry", fake_post)
|
||||||
|
out = send_legacy_image_messages(
|
||||||
|
images=["https://example.com/x.png"],
|
||||||
|
prompt="hello",
|
||||||
|
model="m-special",
|
||||||
|
api_key="explicit-key",
|
||||||
|
base_url="https://gw.example/expert/v1",
|
||||||
|
)
|
||||||
|
assert out.get("ok") is True
|
||||||
|
assert "/chat/completions" in str(recorded.get("url") or "")
|
||||||
248
tests/test_image_legacy_gateway_lane.py
Normal file
248
tests/test_image_legacy_gateway_lane.py
Normal file
|
|
@ -0,0 +1,248 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from oclaw.platform.llm.image_http_common import (
|
||||||
|
dashscope_multimodal_http_ok,
|
||||||
|
dashscope_native_multimodal_url_from_compatible_base,
|
||||||
|
extract_text_and_images,
|
||||||
|
)
|
||||||
|
from oclaw.platform.llm.image_http_common import build_extract_diag_empty
|
||||||
|
from oclaw.platform.llm.image_legacy_client import (
|
||||||
|
collect_legacy_lane_images_from_attachments,
|
||||||
|
legacy_image_assistant_body_with_placeholder,
|
||||||
|
legacy_image_turn_bundle,
|
||||||
|
normalize_legacy_output_image_urls,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_legacy_output_dict_image_parts() -> None:
|
||||||
|
u = "https://dashscope-result-sz.oss-cn-shenzhen.aliyuncs.com/x.png?Expires=1"
|
||||||
|
assert normalize_legacy_output_image_urls([{"image": u}]) == [u]
|
||||||
|
assert normalize_legacy_output_image_urls([{"image_url": {"url": u}}]) == [u]
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_turn_bundle_coerces_dict_images_to_attachments() -> None:
|
||||||
|
ok, text, att = legacy_image_turn_bundle(
|
||||||
|
{"ok": True, "text": "", "images": [{"image": "https://example.invalid/a.png"}]}
|
||||||
|
)
|
||||||
|
assert ok is True
|
||||||
|
assert len(att) == 1
|
||||||
|
assert att[0]["type"] == "image_url"
|
||||||
|
assert att[0]["url"] == "https://example.invalid/a.png"
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashscope_native_url_from_compatible_base() -> None:
|
||||||
|
assert (
|
||||||
|
dashscope_native_multimodal_url_from_compatible_base(
|
||||||
|
"https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||||
|
)
|
||||||
|
== "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
dashscope_native_multimodal_url_from_compatible_base("https://example.com/openai/v1") is None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_collect_legacy_lane_images_image_url() -> None:
|
||||||
|
atts = [{"type": "image_url", "url": "https://example.invalid/x.png"}]
|
||||||
|
assert collect_legacy_lane_images_from_attachments(atts) == ["https://example.invalid/x.png"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_collect_legacy_lane_images_raw_base64() -> None:
|
||||||
|
atts = [{"type": "input_image", "mime": "image/png", "image_base64": "SGVsbG8="}]
|
||||||
|
got = collect_legacy_lane_images_from_attachments(atts)
|
||||||
|
assert len(got) == 1
|
||||||
|
assert got[0].startswith("data:image/png;base64,")
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_turn_bundle_text_only_success() -> None:
|
||||||
|
ok, text, att = legacy_image_turn_bundle({"ok": True, "text": "caption only", "images": []})
|
||||||
|
assert ok is True
|
||||||
|
assert text == "caption only"
|
||||||
|
assert att == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_image_assistant_placeholder_zh_en() -> None:
|
||||||
|
produced = [{"type": "image_ref", "attachment_id": "a" * 64}]
|
||||||
|
assert "附件" in legacy_image_assistant_body_with_placeholder(
|
||||||
|
lang="zh", body_text="", produced=produced
|
||||||
|
)
|
||||||
|
assert "attachment" in legacy_image_assistant_body_with_placeholder(
|
||||||
|
lang="en", body_text="", produced=produced
|
||||||
|
).lower()
|
||||||
|
assert legacy_image_assistant_body_with_placeholder(lang="zh", body_text="x", produced=produced) == "x"
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_turn_bundle_upstream_error() -> None:
|
||||||
|
ok, text, att = legacy_image_turn_bundle({"ok": False, "error": "rate"})
|
||||||
|
assert ok is False
|
||||||
|
assert "rate" in text
|
||||||
|
assert att == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_harvests_nested_https_under_message() -> None:
|
||||||
|
url = "https://cdn.example.invalid/generated.png"
|
||||||
|
text, images = extract_text_and_images(
|
||||||
|
{
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": None,
|
||||||
|
"metadata": {"preview_image": url},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert text == ""
|
||||||
|
assert images == [url]
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_extract_diag_top_level_openai_choices() -> None:
|
||||||
|
d = build_extract_diag_empty(
|
||||||
|
{
|
||||||
|
"choices": [],
|
||||||
|
"model": "x",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert d.get("choices_len") == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_turn_bundle_includes_provider_redacted() -> None:
|
||||||
|
ok, msg, att = legacy_image_turn_bundle(
|
||||||
|
{
|
||||||
|
"ok": True,
|
||||||
|
"text": "",
|
||||||
|
"images": [],
|
||||||
|
"extract_diag": {"choices_len": 0},
|
||||||
|
"provider_response_redacted": '{"choices":[]}',
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert ok is False
|
||||||
|
assert "provider_json=" in msg
|
||||||
|
assert att == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_turn_bundle_empty_ok_response_fails() -> None:
|
||||||
|
ok, text, att = legacy_image_turn_bundle({"ok": True, "text": "", "images": []})
|
||||||
|
assert ok is False
|
||||||
|
assert att == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_and_images_dashscope_output_wrapper() -> None:
|
||||||
|
"""Native multimodal HTTP wraps ``choices`` under ``output`` (not top-level OpenAI shape)."""
|
||||||
|
url = "https://dashscope-result-hz.oss-cn-hangzhou.aliyuncs.com/x.png?Expires=1"
|
||||||
|
payload = {
|
||||||
|
"status_code": 200,
|
||||||
|
"request_id": "959afba6-544e-487e-b58a-6bd9fea97xxx",
|
||||||
|
"code": "",
|
||||||
|
"message": "",
|
||||||
|
"output": {
|
||||||
|
"text": None,
|
||||||
|
"finish_reason": None,
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"finish_reason": "stop",
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": [{"image": url}],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"audio": None,
|
||||||
|
},
|
||||||
|
"usage": {
|
||||||
|
"input_tokens": 0,
|
||||||
|
"output_tokens": 0,
|
||||||
|
"image_count": 1,
|
||||||
|
"width": 2048,
|
||||||
|
"height": 2048,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
text, images = extract_text_and_images(payload)
|
||||||
|
assert text == ""
|
||||||
|
assert images == [url]
|
||||||
|
assert dashscope_multimodal_http_ok(payload)[0] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_and_images_content_dict_not_list() -> None:
|
||||||
|
"""Some gateways return a single object for ``message.content`` instead of an array."""
|
||||||
|
url = "https://dashscope-result-hz.oss-cn-hangzhou.aliyuncs.com/out.png"
|
||||||
|
text, images = extract_text_and_images(
|
||||||
|
{
|
||||||
|
"output": {
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"message": {
|
||||||
|
"role": "assistant",
|
||||||
|
"content": {"image": url},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert text == ""
|
||||||
|
assert images == [url]
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_and_images_messages_fallback() -> None:
|
||||||
|
text, images = extract_text_and_images(
|
||||||
|
{
|
||||||
|
"output": {
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": "x"},
|
||||||
|
{
|
||||||
|
"role": "assistant",
|
||||||
|
"content": [{"image": "https://example.invalid/gen.png"}],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert images == ["https://example.invalid/gen.png"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_and_images_typed_image_url_part() -> None:
|
||||||
|
text, images = extract_text_and_images(
|
||||||
|
{
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"message": {
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "image_url",
|
||||||
|
"image_url": {"url": "https://example.invalid/v.png"},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert images == ["https://example.invalid/v.png"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_dashscope_envelope_rejects_non_success_code() -> None:
|
||||||
|
ok, msg = dashscope_multimodal_http_ok({"code": "InvalidParameter", "message": "bad"})
|
||||||
|
assert ok is False
|
||||||
|
assert "bad" in msg
|
||||||
|
|
||||||
|
|
||||||
|
def test_extract_text_and_images_openai_top_level_unchanged() -> None:
|
||||||
|
text, images = extract_text_and_images(
|
||||||
|
{
|
||||||
|
"choices": [
|
||||||
|
{
|
||||||
|
"message": {
|
||||||
|
"content": [
|
||||||
|
{"type": "text", "text": "hi"},
|
||||||
|
{"image": "https://example.invalid/a.jpg"},
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
assert "hi" in text
|
||||||
|
assert images == ["https://example.invalid/a.jpg"]
|
||||||
160
tests/test_openai_responses_normalize.py
Normal file
160
tests/test_openai_responses_normalize.py
Normal file
|
|
@ -0,0 +1,160 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from openai import BadRequestError
|
||||||
|
|
||||||
|
from oclaw.platform.llm.transports.openai_responses import OpenAIResponsesModel, _is_input_messages_validation_error
|
||||||
|
|
||||||
|
|
||||||
|
def test_input_messages_validation_detects_body_not_str_exc() -> None:
|
||||||
|
"""OpenAI SDK ``str(exc)`` is usually only ``Error code: 400``; gateway detail is in ``body``."""
|
||||||
|
req = httpx.Request("POST", "http://example.invalid/v1/responses")
|
||||||
|
resp = httpx.Response(400, request=req)
|
||||||
|
body = {
|
||||||
|
"message": (
|
||||||
|
"Input should be 'user': input.messages.0.role & Input should be a valid list: "
|
||||||
|
"input.messages.0.content"
|
||||||
|
),
|
||||||
|
"type": "invalid_request_error",
|
||||||
|
"code": "invalid_parameter_error",
|
||||||
|
"param": None,
|
||||||
|
}
|
||||||
|
exc = BadRequestError("Error code: 400", response=resp, body=body)
|
||||||
|
assert "input.messages" not in str(exc).lower()
|
||||||
|
assert _is_input_messages_validation_error(exc) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_strip_leading_system_to_instructions_kw() -> None:
|
||||||
|
msgs = [
|
||||||
|
{"role": "system", "content": "You are helpful."},
|
||||||
|
{"role": "user", "content": "Hi"},
|
||||||
|
]
|
||||||
|
txt, tail = OpenAIResponsesModel._strip_leading_system_messages(msgs)
|
||||||
|
assert txt == "You are helpful."
|
||||||
|
assert tail == [{"role": "user", "content": "Hi"}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_default_responses_parts_and_openai_envelope() -> None:
|
||||||
|
msgs = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "image_url", "image_url": {"url": "https://example.invalid/x.png"}},
|
||||||
|
{"type": "text", "text": "what?"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
out = OpenAIResponsesModel._normalize_messages(msgs)
|
||||||
|
assert len(out) == 1
|
||||||
|
assert out[0]["type"] == "message"
|
||||||
|
assert out[0]["role"] == "user"
|
||||||
|
cc = out[0]["content"]
|
||||||
|
assert isinstance(cc, list)
|
||||||
|
assert any(
|
||||||
|
x.get("type") == "input_image"
|
||||||
|
and isinstance(x.get("image_url"), str)
|
||||||
|
and x.get("detail") == "auto"
|
||||||
|
for x in cc
|
||||||
|
)
|
||||||
|
assert any(x.get("type") == "input_text" and x.get("text") == "what?" for x in cc)
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_nested_chat_parts_opt_in() -> None:
|
||||||
|
msgs = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [{"type": "image_url", "image_url": {"url": "https://example.invalid/x.png"}}],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
out = OpenAIResponsesModel._normalize_messages(
|
||||||
|
msgs, envelope_openai_message=False, content_chat_completions_parts=True
|
||||||
|
)
|
||||||
|
assert "type" not in out[0]
|
||||||
|
assert out[0]["role"] == "user"
|
||||||
|
assert any(x.get("type") == "image_url" for x in out[0]["content"])
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_dashscope_shorthand_image_text_blocks() -> None:
|
||||||
|
msgs = [
|
||||||
|
{"role": "user", "content": [{"image": "https://example.invalid/y.png"}, {"text": "caption"}]},
|
||||||
|
]
|
||||||
|
out = OpenAIResponsesModel._normalize_messages(msgs)
|
||||||
|
assert out[0]["type"] == "message"
|
||||||
|
assert out[0]["role"] == "user"
|
||||||
|
parts = out[0]["content"]
|
||||||
|
assert isinstance(parts, list)
|
||||||
|
imgs = [p for p in parts if isinstance(p, dict) and p.get("type") == "input_image"]
|
||||||
|
assert len(imgs) == 1 and imgs[0].get("detail") == "auto"
|
||||||
|
txts = [p for p in parts if isinstance(p, dict) and p.get("type") == "input_text"]
|
||||||
|
assert any("caption" in str(p.get("text")) for p in txts)
|
||||||
|
|
||||||
|
|
||||||
|
def test_responses_input_candidates_cover_flat_and_nested() -> None:
|
||||||
|
msgs = [{"role": "user", "content": "hi"}]
|
||||||
|
flat = OpenAIResponsesModel._responses_input_candidates(
|
||||||
|
msgs, flat_responses=True, prefer_envelope=True, prefer_chat_parts=False
|
||||||
|
)
|
||||||
|
assert len(flat) >= 1
|
||||||
|
nested = OpenAIResponsesModel._responses_input_candidates(
|
||||||
|
msgs, flat_responses=False, prefer_envelope=True, prefer_chat_parts=False
|
||||||
|
)
|
||||||
|
tags = [t for t, _ in nested]
|
||||||
|
assert any("_messages" in t for t in tags)
|
||||||
|
assert any("_flat_input" in t for t in tags)
|
||||||
|
|
||||||
|
|
||||||
|
def test_responses_input_candidates_primary_combo_first() -> None:
|
||||||
|
msgs = [{"role": "user", "content": "hi"}]
|
||||||
|
nested = OpenAIResponsesModel._responses_input_candidates(
|
||||||
|
msgs, flat_responses=False, prefer_envelope=True, prefer_chat_parts=False
|
||||||
|
)
|
||||||
|
assert nested[0][0] == "e1c0_messages"
|
||||||
|
assert nested[1][0] == "e1c0_flat_input"
|
||||||
|
|
||||||
|
nested2 = OpenAIResponsesModel._responses_input_candidates(
|
||||||
|
msgs, flat_responses=False, prefer_envelope=False, prefer_chat_parts=True
|
||||||
|
)
|
||||||
|
assert nested2[0][0] == "e0c1_messages"
|
||||||
|
assert nested2[1][0] == "e0c1_flat_input"
|
||||||
|
|
||||||
|
|
||||||
|
def test_normalize_agent_messages_style_input_image() -> None:
|
||||||
|
"""Matches ``build_llm_messages`` last-turn multimodal blocks (``input_image`` + ``image_base64``)."""
|
||||||
|
msgs = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "input_image", "image_base64": "SGk=", "mime": "image/png"},
|
||||||
|
{"type": "text", "text": "what is this"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
out = OpenAIResponsesModel._normalize_messages(msgs)
|
||||||
|
assert len(out) == 1
|
||||||
|
assert out[0]["type"] == "message"
|
||||||
|
parts = out[0]["content"]
|
||||||
|
imgs = [p for p in parts if isinstance(p, dict) and p.get("type") == "input_image"]
|
||||||
|
assert len(imgs) == 1
|
||||||
|
assert imgs[0]["image_url"].startswith("data:image/png;base64,")
|
||||||
|
assert imgs[0].get("detail") == "auto"
|
||||||
|
txts = [p for p in parts if isinstance(p, dict) and p.get("type") == "input_text"]
|
||||||
|
assert any(p.get("text") == "what is this" for p in txts)
|
||||||
|
|
||||||
|
chat_parts = OpenAIResponsesModel._normalize_messages(
|
||||||
|
msgs, envelope_openai_message=False, content_chat_completions_parts=True
|
||||||
|
)
|
||||||
|
cp = chat_parts[0]["content"]
|
||||||
|
assert any(
|
||||||
|
isinstance(p, dict) and p.get("type") == "image_url" and "base64" in str(p.get("image_url", {}).get("url"))
|
||||||
|
for p in cp
|
||||||
|
)
|
||||||
|
assert any(p.get("type") == "text" and p.get("text") == "what is this" for p in cp)
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_legacy_compatible_mode_uses_image_url_parts() -> None:
|
||||||
|
from oclaw.platform.llm.image_legacy_client import _openai_compatible_vision_content
|
||||||
|
|
||||||
|
cc = _openai_compatible_vision_content(["data:image/jpeg;base64,SGk="], "go")
|
||||||
|
assert cc[-1]["type"] == "text"
|
||||||
|
assert cc[-1]["text"] == "go"
|
||||||
|
assert cc[0]["type"] == "image_url"
|
||||||
Loading…
Add table
Add a link
Reference in a new issue