diff --git a/docs/biz-state-canonical-fields.md b/docs/biz-state-canonical-fields.md new file mode 100644 index 0000000..3670f0a --- /dev/null +++ b/docs/biz-state-canonical-fields.md @@ -0,0 +1,68 @@ +# 割接对比：规范值契约 + +适用范围：业务对比（`BizCompare*`）和割接监控（`BizMigration*`）。两种模式用同一条身份流水线（`netx_api/biz_state/identity_pipeline.py`），跨厂商对比靠它把两侧设备的原始输出对齐到同一套“规范值”。 + +## 1. 流水线 + +同一行在某一侧（`before` / `after`）依次经过： + +1. **归一化**：接口字段去内部空格 → 厂商内规范名（`VENDOR_CANONICAL`，如华为 `GigabitEthernet`→`GE`）→ 模板别名规则（`iface_normalize_rules`，如 `GE`→`gei`）；键字段按 `field_rules[].normalize`（`lower`/`upper`/`mac`…）；再按 `field_rules[].value_map` 改写状态值。 +2. **行过滤**：`row_filters` 在归一化之后求值，所以按规范值书写；带 `side` 的过滤只作用于该侧。 +3. **端口映射**：只作用于 `before` 侧，键用 before 侧规则归一化、值用 after 侧规则归一化。 +4. **取键**：`scalar_text(value).strip()` 逐字段取值，得到 match key。 + +快照版本 `config_snapshot.version >= 2` 的 run 走完整流水线；`version == 1` 的历史 run 保持旧语义（过滤看原始行、键不归一化、同键按采集顺序配对），回填、检索、导出结果不变。 + +## 2. 规范值 + +比对层只认下面这些取值。设备原始值不同的，用 `value_map` 或 parser 输出规范值解决，**引擎里不写厂商特例**。 + +| metric | 字段 | 规范取值 | +|---|---|---| +| `interface_brief` | `admin` / `phy` / `prot` | `up` / `down`（华为 `*down` 记 `admin=down`） | +| `bgp_peer` | `state` | `established` / `idle` / `active` / `connect` / `opensent` / `openconfirm` | +| `bgp_peer` | `afi` | `ipv4` / `ipv6` / `vpnv4` / `vpnv6` / `evpn` / `vpls` | +| `arp` | `entry_type` | `dynamic` / `static` / `interface` | +| `isis_adjacency` / `ospf_neighbor` | `state` | `up` / `full` / `2way` / `down` / `init` / `idle` | +| `vrrp` | `af` | `ipv4` / `ipv6` | + +### 中兴（已实现）现状 + +中兴 parser 输出设备打印的原值，没有改写： + +- `interface_brief.admin/phy/prot`：`up` / `down`，已是规范值。 +- `arp.entry_type`：由 `age` 推断，`HH:MM:SS` 为 `dynamic`，否则 `static`。 +- `bgp_peer.state`：设备原值，如 `Established`（首字母大写）。与中兴自身对比时一致；与其他厂商对比时在模板里加值映射： + + ```json + {"field": "state", "value_map": {"established": "established", "estab": "established", "*down": "down"}} + ``` + + `value_map` 大小写不敏感匹配，输出为右侧的规范值；两侧都经过同一张表。 + +> 没有改动中兴 parser 的输出值：已采集批次里的值是原值，直接改成小写会让“旧批次 vs 新批次”出现假差异。需要统一时在模板里加 `value_map`。 + +### 其他厂商（待办） + +华为 VRP 的状态 parser（`interface_brief`、`arp`、`bgp_peer`、`isis_adjacency`、`ospf_neighbor`）**尚未实现**：仓库里没有真实输出样本，按规则不凭空编写 TextFSM。收到样本后按 `skills/biz-state/netx-biz-state/SKILL.md` 第 2 节逐项完成，输出与中兴相同的字段名，状态值输出上表的规范值，并给 profile 补 `slice`。 + +目前只有 LLDP 在跨厂商场景下有真实 parser。 + +## 3. 命令切片（`ParseProfile.slice`） + +一个采集命令产出 metric 的哪一部分，由 profile 的 `slice` 声明，例如 `zte.bgp_ipv4_summary` → `{"afi": "ipv4"}`、`zte.vrrp_ipv6` → `{"af": "ipv6"}`。比对表单（sheet）用顶层 `eq` 行过滤声明自己要的切片。 + +- 一侧没有任何命令覆盖该切片 → 该 sheet 状态 `not_collected`，不判业务缺失、不计入通过率。 +- 割接监控创建 HF 采集任务时，取两侧画像命令切片的并集；某侧找不到对应切片的 profile 时返回 `no_profile_for_slice:::`。 +- 没声明 `slice` 的命令视为覆盖整个 metric。 + +## 4. sheet 数据状态 + +| 状态 | 条件 | 通过率 | +|---|---|---| +| `ok` | 两侧采集正常 | 正常计算 | +| `no_data` | 两侧过滤后都是 0 行，命令都成功 | `null`，不计入总通过率 | +| `suspect_parse` | 某侧 `raw_line_count > 5` 但 `row_count == 0`，或全部命令失败 | 失败行照常显示，任务级 `ok = null` | +| `not_collected` | 某侧没有任何命令（或切片）产出该 metric | `null`，不计入总通过率 | + +没有任何命令回执的批次（导入或历史数据）无法判断，按正常处理。 diff --git a/docs/reviews/cutover-optimization-plan-2026-10-12.md b/docs/reviews/cutover-optimization-plan-2026-10-12.md new file mode 100644 index 0000000..cebc1c7 --- /dev/null +++ b/docs/reviews/cutover-optimization-plan-2026-10-12.md @@ -0,0 +1,477 @@ +# 割接对比（业务对比 / 割接监控）问题分析与优化计划 + +日期：2026-10-12。范围：网络管理 → 割接对比下的 5 个子模块：业务监控（采集）、业务对比、割接监控、对比模板、监控模板。 +用途：交给 Cursor 按任务逐项实施；Claude 负责验收。 + +本文基于代码阅读和本地脚本验证，没有连接真实设备，也没有在真实 PostgreSQL 上执行。标了 **[已验证]** 的问题已经用脚本复现（脚本放在会话 scratchpad，结论见各条目）。基线：开始修改前，相关后端回归 **279 项通过**，`tsc -b` 通过，`cutoverDataCache` / `compareTemplateModel` Node 测试通过。 + +--- + +## 0. 结论摘要 + +1. **两种模式共用一套“归一化 + 映射”概念，但各个代码路径的实现不一致。** 割接监控会先对端口映射的键和值做接口归一化，业务对比不会（**[已验证]**：按设备原始名称填写映射，业务对比全部变成“删除 + 新增”）。行过滤用原始值，映射作用在归一化之后。键字段不受 `field_rules` 归一化影响。跨厂商割接最容易在这里产生误报。 +2. **目前只有两种映射：接口类型别名（模板级，两侧同一套规则）和端口映射（before→after，一对一）。** 没有分侧（按厂商）归一化，没有字段值映射（状态枚举、对端 IP、VRF 改名），也无法提前检查映射覆盖率。映射漏配会直接表现为业务“缺失”，用户很难判断是映射问题还是业务问题。 +3. **除 LLDP 外，状态采集解析只实现了中兴。** 华为、思科等的 parser 都是会抛出 `NotImplementedError` 的空壳，且没有注册。内置对比模板（`ZTE status default`）里的过滤和容差也都按中兴设定。所以现在的“跨厂商映射”只对 LLDP 真正有效，其余检查项在跨厂商场景下要么采集不到，要么全量误报。 +4. **判定语义有几处会把正常割接判成失败：** + - 端口预设要求旧端口 `admin=down 且 phy=down`。现场常见的做法是拔纤但不 shutdown，这种情况验收时会判为“未完成/红”（**[已验证]**）。 + - 两侧都是 0 行时通过率是 100%（空对空误判为通过）。 + - 解析不到数据会被当成业务缺失。 + - 新旧两侧的采集命令集不一致时，会被当成业务丢失。 + - 内置模板对 `output_bps` 用绝对值容差 5 bps，对 `rx_power`（dBm）用百分比容差。 +5. **割接监控的运行数据无限增长。** 每个 `BizMigrationRun` 或 `BizCompareRun` 引用的采集批次都会被永久保护，run 和 diff 本身也没有保留策略。60 秒一轮的高频监控跑几天，数据库和磁盘会持续膨胀。 +6. **性能：** 业务对比只要配置了端口映射，所有检查项（包括不含接口字段的 BGP 路由百万行表）都会退回 Python 引擎。割接验收是同步 HTTP 请求，会把路由表全量载入内存。 +7. **引用保护有漏洞：** 被监控模板引用的对比模板、被任务或割接项目引用的端口映射都能被删除。删除后，业务对比会静默地按“无映射”执行；割接监控会直接报错停摆。 + +--- + +## 1. 现状框架梳理（给 Cursor 的背景） + +### 1.1 数据流 + +``` +业务监控任务(biz_state_task, 每台设备) + └─ 采集批次 batch ──> 每条命令 batch_command(parse_status/row_count/raw_line_count) + └─ 解析(ParseProfile → parser → FieldDef) ──> biz_state_metric_row(data_json) / biz_state_lldp_neighbor + │ +对比模板(biz_compare_template.metrics_json = 多个 sheet) + sheet: metric_id, key_fields, iface_fields, compare_fields, display_fields, + row_filters, field_rules(normalize/compare/tolerance), ignore_port_changes + 模板级: iface_normalize_json（接口类型别名，两侧同一套） + │ +┌───────────── 业务对比（BizCompareJob/Run/Diff）─────────────┐ +│ before 批次 vs after 批次（两台设备或同一设备前后） │ +│ 行过滤(原始值) → 接口归一化(两侧) → 端口映射(仅 before) │ +│ → 按 match_key 分组 → 同键按采集顺序 zip 配对 │ +│ → added/removed/changed/unchanged → 通过率 │ +│ 引擎：PG + 可下推 → SQL FULL OUTER JOIN；否则 Python │ +└─────────────────────────────────────────────────────────────┘ +┌───────────── 割接监控（BizMigrationProject/Batch/Run/Diff）──┐ +│ 旧设备: 画像基线 vs 高频当前 → old_kind │ +│ 新设备: 画像基线 vs 高频当前 → new_kind │ +│ 预期集(expect_set: 本批要迁移的端口/对象) + 监控模板(成功/异常规则)│ +│ → dual_verdict: migrated/migrating/anomaly/lost/unfinished… │ +│ → 本批完成(finish_batch)=终验 → 红单(带红继续) │ +│ 高频采集：每个 metric 一对(新/旧)任务，同轮次配对采样 │ +└─────────────────────────────────────────────────────────────┘ +``` + +### 1.2 现有“映射/归一化”层与使用点（不一致的根源） + +| 层 | 归属 | 作用对象 | 业务对比 | 业务对比·实时检索/回填/导出 | 割接监控 | +|---|---|---|---|---|---| +| 接口类型别名 `iface_normalize_json` | 对比模板 | `iface_fields`，两侧同规则，只替换前缀 | 有 | 有（`_run_transform_config`） | 有 | +| 端口映射 `biz_port_mapping` | 任务 / 项目 | `iface_fields`，before→after，精确匹配 + 父接口继承 | **键/值不归一化** | 同左 | **键/值先归一化**（`service.py:1019`） | +| 行过滤 `row_filters` | sheet | **原始值**（归一化之前） | 有（PG 可下推） | 有 | 有（Python） | +| 字段规则 `field_rules.normalize` | sheet | **仅 compare 字段**，键字段不处理 | 有 | 有 | 仅完整性校验 | +| 监控模板 `sheet_overrides` | 监控模板 | 状态语义（up/down 值、成功/异常模式） | — | — | 有 | + +关键文件： +- 引擎：`netx_api/biz_state/compare_engine.py`、`compare_rules.py`、`iface_normalize.py`、`compare_sql.py` +- 业务对比服务：`netx_api/biz_state/compare_service.py`（约 4000 行） +- 割接监控：`netx_api/biz_migration/evaluate.py`、`service.py`、`monitor_templates.py`、`auto_monitor.py` +- 采集：`netx_api/biz_state/collect_runner.py`、`claim.py`、`profiles.py`、`parsers/*` +- 保留：`netx_api/biz_state/retention.py` +- 前端：`web/src/features/cutover/BizComparePage.tsx`、`BizMigrationPage.tsx`、`BizMonitorTemplatesPage.tsx`、`BizStatePage.tsx` +- 内置模板：`netx_api/biz_state/data/default_zte_status_template.json` + +--- + +## 2. 问题清单 + +严重度：**P0** 会产生错误结论或数据损坏；**P1** 高概率误报或明显性能/可用性问题；**P2** 边界问题或体验问题；**P3** 清理项。 + +### 2.1 业务对比（BizCompare） + +| ID | 严重度 | 问题 | 位置 | 影响 | +|---|---|---|---|---| +| B-01 | P0 **[已验证]** | 端口映射的键/值不按模板接口归一化规则转换，必须由用户手工填写“归一化后”的名称；割接监控却会自动归一化。两种模式行为不一致。 | `compare_engine.py:224-260`（`apply_port_map` 用原始 `pmap`），`compare_service.py:2461`、`_run_transform_config:2337` | 模板有 `GE→gei` 规则时，映射 `GE0/1/0/1→gei0/2/0/1` 完全不命中，结果 added=1 / removed=1、`map_ok=False` | +| B-02 | P1 | 任务里只要有端口映射，所有 sheet 都会跳过 SQL 引擎，包括没有接口字段的 BGP/路由表 | `compare_sql.py:165`（`if port_map: return "port_map"`） | 百万行 vpnv4 路由退回 Python，内存和耗时都大幅上升；割接任务几乎都会配映射 | +| B-03 | P1 | 端口映射被删除后，业务对比静默地按“无映射”执行（还会触发自动忽略端口的启发式），删除时也不检查引用 | `delete_mapping:1603`，`_port_map_dict:1612`，`_validate_compare_job:2220` | 结果语义悄悄改变 | +| B-04 | P1 | 对比模板删除只检查 `BizCompareJob` 的引用，不检查 `BizMonitorTemplate.compare_template_id` | `delete_template:1521`，`list_templates:1391` 的 `job_count` | 割接窗口中删掉模板后，`run_evaluate` 报 `compare_template_missing`，监控停摆 | +| B-05 | P1 | 创建向导或任务配置里编辑了映射文本但没点“保存映射”，创建/保存/执行任务时映射文本被丢弃，没有任何提示 | `BizComparePage.tsx:1814 jobConfigBody`、`1895 createJob`、`1997 saveJobConfig`、`2029 runNow` | 用户以为配了映射，实际没生效，产生大量假“缺失/新增” | +| B-06 | P2 | 映射文本解析只认 `, | Tab` 作分隔符；空格分隔或格式不对的行被静默丢弃，没有解析计数或错误提示 | `BizComparePage.tsx:1720 parseMapRows` | 映射少行而不自知 | +| B-07 | P1 | 映射校验只针对“第一个含接口字段的 sheet”，结果以原始 JSON 展示；只报告“映射键未命中”，不报告“before 侧端口没有映射且在 after 侧不存在”（漏配） | `validate_mapping:1810`，`mapping_stats`（`compare_engine.py:100`），`BizComparePage.tsx:2272` | 漏配映射最终表现为业务缺失 | +| B-08 | P1 | `row_filters` 作用在原始行（归一化、映射之前）；跨厂商时，同一条过滤在两侧含义不同 | `_load_metric_rows:1619`（SQL/Python 都在加载时过滤），`_run_sheet:2179` | 例如 `interface contains gei` 会把华为 `GE0/0/1` 整侧滤掉；ARP 的 `age_timer`（HH:MM:SS）只适用于中兴 | +| B-09 | P2 | 键字段不应用 `field_rules.normalize`（如 MAC 格式），跨厂商的 MAC/大小写差异导致配对失败 | `compare_engine.py:96 row_key`，`compare_sql.py _rk_sql` | 键里含 MAC 的 sheet（如 `l2vpn_mac`）全量误报 | +| B-10 | P1 | 两侧都是 0 行时通过率为 100%；解析不到数据（命令执行了但解析 0 行）被当成业务“缺失” | `_enrich_summary:3015`，`compute_pass_rate:210` | 空对空误判为通过；模板不匹配被当成业务丢失 | +| B-11 | P1 | 新旧两侧采集命令集不一致（例如旧设备采了 bgp ipv4 summary，新设备没采），对比结果是“全部缺失” | 没有任何检查 | 采集配置问题被误判为业务问题 | +| B-12 | P2 | 同键多行按采集顺序 zip 配对，CLI 输出顺序变化会产生假“不一致” | `compare_engine.py:331+`、`compare_sql.py` 中 `dup_rn ORDER BY seq` | ECMP/多路径场景误报 | +| B-13 | P2 | 只要提供了映射，`ignore_port_changes` 就被忽略；部分映射时，未映射端口既不按名称匹配也不忽略端口 | `compare_engine.py:224` | 部分映射时策略不可控 | +| B-14 | P2 | 结果表的键列只显示映射后的名称，看不到 before 侧原始端口名 | `BizComparePage.tsx:4161`（`d.key[k] ?? pre[k]`，`pre` 取 `mapped_before`） | 跨厂商时现场人员无法对照原端口 | +| B-15 | P2 | 判定没有方向：状态 down→up、前缀数增加也算失败；流量计数器被当成严格比较项 | `compare_rules.values_equal` | 割接后业务恢复或流量迁入被判为失败 | +| B-16 | P3 | `batch_metric_collect_ok` 是死代码；`_load_metric_rows` 中 LLDP 分支重复过滤；PG 下推时 `before_raw_count` 实际上是过滤后的数量 | `compare_service.py:78`、`1675-1688`、`2199` | 清理 | +| B-17 | P3 | 防止同一任务并发对比的锁是进程内的；`_job_has_active_run` 检查和插入不是原子操作。biz_state worker 多副本时，自动对比有极小概率并发 | `compare_service.py:57`、`2887` | 低概率 | + +### 2.2 割接监控（BizMigration） + +| ID | 严重度 | 问题 | 位置 | 影响 | +|---|---|---|---|---| +| M-01 | P0 **[已验证]** | 端口预设的成功条件要求旧侧 `removed` 或 `admin=down 且 phy=down`。旧端口拔纤但没有 shutdown（admin up / phy down）、新端口已 up 时：窗口内是 `migrating/黄`，**验收时是 `unfinished/红`** | `monitor_templates.py:255-275`（`_success_stateful` 的 `old_down_groups` 用的是 AND） | 正常割接被判为验收失败 | +| M-02 | P1 | 每个 run 引用的 HF 批次都会被永久保护，run 和 diff 本身也没有保留策略 | `retention.py:65-75` | 高频监控期间数据库无限增长 | +| M-03 | P1 | 验收（`finish_batch`）和手动评估是同步 HTTP 请求；路由类 sheet（`bgp_route`/`ip_route`）会把 4 个批次的全量行载入 Python | `biz_migration_router.py` 的 evaluate/finish，`service.py:1148-1163` | 超时、内存暴涨；而且这期间一直持有项目行锁（`with_for_update`，`service.py:940`） | +| M-04 | P1 | 默认监控模板会采集对比模板里的全部 metric，包括只能手动采集的路由表。没有手动采集时，该 sheet 是 `current_missing`，验收**必然**失败（`current_missing` / `coverage_incomplete`） | `resolve_collect_metric_ids:525`、`finish_batch:2358-2371`、`MANUAL_ROUTE_METRICS` | 默认配置跑不通验收 | +| M-05 | P1 | 只要任意 sheet 有重复键，验收就判失败（`duplicate_keys`，同时 `coverage_complete=False`），不管这个 sheet 是否在本批范围内 | `service.py:1299`、`2366` | 例如 ISIS L1/L2 两行同键，就永远过不了验收 | +| M-06 | P1 | 新旧两侧画像任务的命令集可能不同，HF 任务沿用各自画像的命令项，旧侧有、新侧没采的切片会被判为 `lost` | `_portrait_metric_items:1741`、`ensure_highfreq:1919` | 采集配置问题被当成业务丢失 | +| M-07 | P1 | 创建项目时不校验基线批次是否属于该侧画像任务（patch 时会校验） | `create_project:686-691`（对比 `patch_project:744-750`） | 可能拿错设备的基线；直到评估时才报 `baseline_task_mismatch` | +| M-08 | P2 | 不支持同设备割接（板卡更换、同框端口迁移）：`collect_pair_same_device` 直接报错 | `service.py:2016` | 常见场景无法使用 | +| M-09 | P2 | 每个 metric、每台设备一个 HF 任务；15 个检查项就是每分钟每台设备约 15 次 SSH 登录；没有单设备并发上限 | `ensure_highfreq:1943`，`claim.py` | VTY 耗尽、AAA 日志刷屏，进而采集失败，看板大量 `current_missing` | +| M-10 | P2 | 每轮自动评估都重新载入基线全量行（基线不会变） | `run_evaluate.metric_rows:1059` | 白白消耗 CPU/IO | +| M-11 | P2 | 双侧评估里，诊断索引的键串用 `str(v)` 构造，而其他地方用 `scalar_text().strip()`：`None` 变成 `"None"`，`True` 变成 `"True"`，空白不裁剪，预期键匹配不上 | `evaluate.py:759`、`compare_engine.py:273 _key_obj` | **[已验证]**：键字段为 `None` 时，存在且未变化的对象被判为 `unfinished` | +| M-12 | P2 | 用 metric 名称包含 `bgp` / `pw` 来决定是否用端口映射推导范围（字符串启发式） | `evaluate.py:804` | 新增 metric 时行为不可预期，也违反“引擎不写 metric 特例”的原则 | +| M-13 | P2 | 映射允许 N:1 吗？业务对比允许（结果是重复键），割接监控直接报 `mapping_target_ambiguous`；1:N（拆分 breakout）两边都不支持（`before_if` 唯一） | `models/biz_state.py BizPortMappingRow`，`service.py:991/1021` | 端口汇聚、拆分割接无法表达，报错也没说明是哪几行 | +| M-14 | P3 | `project_to_dict`（GET/列表）里会写库（`resolve_project_monitor_template` 持久化默认模板）；`list_projects` 存在 N+1 | `service.py:486-503`、`543` | 只读接口有副作用 | +| M-15 | P3 | 一个键在后续验收中已经变绿，对应红单仍然保持 open，只能人工关闭 | `_persist_red_tickets_from_run:2326` | 红单堆积 | + +### 2.3 内置模板与跨厂商能力 + +| ID | 严重度 | 问题 | 位置 | +|---|---|---|---| +| T-01 | P1 | `interface_detail.output_bps` 用 `numeric` 绝对容差 5（bps），而 `input_bps` 用 5%：相当于对出方向流量做严格比较 | `default_zte_status_template.json` | +| T-02 | P1 | `optical_brief.rx_power` 用 `percent 5`：dBm 是对数单位，-0.5 和 -0.6 已经差 20%。应改为绝对值 dB 容差 | 同上 | +| T-03 | P2 | 流量计数器（bps/util）默认参与比较；割接后流量迁移是预期行为，会产生大量噪声 | 同上 | +| T-04 | P2 | `l2vpn_mac` 键里的 `ac_port` 是接口，但没放进 `iface_fields`，端口映射不生效 | 同上 | +| T-05 | P1 | 华为、思科、华三、Juniper、诺基亚、爱立信的状态 parser 都是空壳（没有注册）；只有 LLDP 跨厂商可用 | `parsers/huawei/__init__.py` 等 | +| T-06 | P2 | 各厂商的状态值不统一（`Established`/`ESTAB`、`UP`/`up`/`*down`、`Full/DR`），而对比层只有 `lower`/`upper`/`mac` 几种归一化，没有枚举别名映射 | `compare_rules.normalize_value` | + +--- + +## 3. 优化方案（分阶段任务，按顺序执行） + +### 通用执行约束（Cursor 每个任务都要遵守） + +1. **引擎里不写 metric 特例**：所有 metric 相关的行为放进模板、sheet 或 profile 声明（遵循 `skills/biz-state/netx-biz-state/SKILL.md` 第 4、5 节）。 +2. **Python 和 SQL 必须一致**：凡是影响配对或判定的改动，要么两个引擎都实现，要么在 `sql_compare_skip_reason` 里让 SQL 自动降级到 Python，并补“同输入两引擎结果一致”的对照测试（SQL 侧至少断言生成的 SQL 和降级原因）。 +3. **历史 run 不变**：新增的执行配置写进 `config_snapshot`，并把 `version` 升到 2。`version==1` 和旧记录的回填、检索、导出保持原行为，不能因为新逻辑改变历史结果。 +4. **数据库结构**：尽量用 JSON 字段扩展，不加列。确实需要加列或表时，同时补 `alembic/versions/` 和 `netx_api/schema_patches.py`（参照现有做法），并写回归测试。 +5. **每个任务**：补单元测试；跑第 5 节的回归命令；前端改动跑 `tsc -b` 和 `vite build`，涉及交互的补 `web/tests/*.browser.mjs` 场景（全部 `/v1` 用固定响应）。 +6. 一个任务一个提交，提交信息写清楚任务 ID（例如 `fix(cutover): B-01 normalize port-map keys`）。 + +--- + +### 阶段 A：正确性快修（P0/P1，改动小，先做） + +#### A1 端口映射键/值统一归一化（B-01） +- 新增函数 `normalize_port_map(port_map, rules) -> (normalized_map, conflicts)`，放在 `iface_normalize.py`：键和值都用 `normalize_iface_name` 处理。归一化后键冲突（两个原始键归一后相同但目标不同）要作为 conflicts 返回。 +- 业务对比：在 `_run_transform_config`（`compare_service.py:2337`）统一处理，覆盖执行、实时检索、回填、导出所有路径。**只对 `config_snapshot.version>=2` 的 run 生效**；`version==1` 和旧记录保持原样。 +- 割接监控：`run_evaluate` 和 `list_baseline_expect_objects` 里的内联归一化改为调用这个函数，消除重复实现。 +- `mapping_stats` 的 `hint` 文案去掉“必须填写归一化后名称”。 +- **验收测试**：模板规则 `GE→gei`、映射写原始名 `GE0/1/0/1→gei0/2/0/1`，新 run 的结果是 `unchanged=1`；同样的数据在 `version=1` 的 run 上回填，结果不变。 + +#### A2 引用保护和缺失检测（B-03、B-04） +- `delete_mapping`：被 `BizCompareJob.mapping_id` 或 `BizMigrationProject.mapping_id` 引用时返回 409 `mapping_in_use`，detail 里带引用列表（任务名/项目名）。 +- `delete_template`：同时检查 `BizMonitorTemplate.compare_template_id`，返回 409 `template_in_use`（detail 区分 job/monitor）。`list_templates` 增加 `monitor_template_count`。 +- `_validate_compare_job`：`job.mapping_id` 非空但映射不存在时返回 400 `mapping_missing`。 +- 前端：删除失败时显示引用对象；模板列表显示两类引用数。 +- **验收测试**：上述 3 种 409/400 的单测；被引用的映射无法删除。 + +#### A3 映射文本不会丢失（B-05、B-06） +- `parseMapRows` 改为返回 `{rows, invalidLines[]}`：两段文本即使用空格分隔也接受（正好两个 token 时）；重复的 before 行要提示。 +- 映射文本相对于已保存版本“有修改”时：创建任务、保存配置、立即执行都要先拦截，提供“保存映射并继续 / 放弃修改”两个选项（或者自动先保存映射再保存任务，二选一，推荐自动保存并提示）。 +- 编辑区下方实时显示“已解析 N 对 / 无效 M 行（行号）”。 +- 保存一个被其他任务或项目共用的映射时，提示“该映射被 X 个任务 / Y 个割接项目使用”（后端 `list_mappings` 返回引用计数）。 +- **验收（浏览器脚本）**：在向导里输入映射后直接点“创建”，请求体里的 `mapping_id` 非空，或者被拦截并出现提示；空格分隔的行能被解析；无效行有提示。 + +#### A4 端口预设成功条件修正（M-01） +- `preset_override_for_metric("interface_brief")` 中旧侧的条件改为三个 OR 组：`[removed]`、`[phy in down]`、`[admin in down]`；新侧保持 `admin up 且 phy up`。 +- 已存在的监控模板：在 `ensure_default_monitor_templates` 中增加一个幂等升级，**只替换结构上和旧预设完全一致的 override**（逐字段对比），用户自定义过的不动，并在 `override_warnings` 里提示“端口成功条件为旧版 AND 语义”。 +- **验收测试**：新增 4 个用例——旧侧 admin up/phy down、admin down/phy up、两者都 down、端口已删除；在窗口内和验收时，新侧 up 都应判为 `migrated/green`。另外，新侧 phy down 时仍判为红。 + +#### A5 创建项目时校验基线归属（M-07） +- `create_project` 复用 `patch_project` 的校验：基线批次的 `task_id` 必须等于对应侧的画像任务，状态必须是 `success` 或 `partial`。抽成公共函数 `_validate_baseline_ownership`。 +- **验收测试**：拿其他设备的批次作为基线创建项目，返回 400 `old_baseline_invalid` / `new_baseline_invalid`。 + +#### A6 键串规范化统一（M-11） +- 在 `compare_engine.py` 新增 `key_tuple(row, fields)`，统一用 `scalar_text(...).strip()`。`_key_obj` 的输出键值和 `build_diff_index_from_compare` 都改用它（`evaluate.py:759` 不再用 `str(v)`）。 +- **验收测试**：键字段为 `None`、`True`、带首尾空格时，预期键能命中，判定为 `ok`/`migrated` 而不是 `unfinished`。 + +#### A7 内置模板参数修正（T-01、T-02、T-04） +- `default_zte_status_template.json`： + - `output_bps` 改为 `percent 5`（和 `input_bps` 对称）。 + - `rx_power` 改为 `numeric`，`tolerance: 2`（单位 dB）。 + - `l2vpn_mac.iface_fields` 加上 `ac_port`。 + - `interface_detail` 的 `input_bps/output_bps/in_util/out_util` 移出 `compare_fields`，只保留在 `display_fields`（见 V3 的方向性规则；在 V3 完成之前先不比较）。 +- `ensure_default_zte_status_template` 增量升级：**只修改值仍等于旧打包值的字段**，用户改过的保留。 +- **验收测试**：旧打包值的模板升级后变成新值；用户自定义过 `rx_power` 规则的模板升级后保持不变。 + +--- + +### 阶段 B：统一的“身份归一化流水线”（跨厂商核心） + +目标：业务对比执行、实时检索、回填、导出、映射校验、割接评估、预期集选择器这 7 条路径，**对同一行算出相同的 match key**，并支持按侧（按厂商）的规则和字段值映射。 + +#### B1 抽出 `IdentityPipeline`（解决“各路径各写一遍”） +- 新文件 `netx_api/biz_state/identity_pipeline.py`： + ```python + @dataclass(frozen=True) + class SideSpec: + iface_rules: list[dict] # 该侧接口归一化（已合并 both + side） + value_maps: dict[str, dict] # field -> {from_lower: to} + @dataclass(frozen=True) + class Pipeline: + before: SideSpec; after: SideSpec + port_map: dict[str, str] # 已归一化（A1） + key_fields, iface_fields, compare_fields, field_rules, row_filters, match_keys + def transform(self, row, side) -> dict # 归一化 → 值映射 →（before 侧）端口映射 + def match_key(self, row, side) -> tuple + def keep(self, row, side) -> bool # 行过滤（在 transform 之后求值，见 B4） + ``` +- 由 `build_pipeline(sheet, snapshot_or_template, port_map, before_vendor, after_vendor)` 统一构造；`compare_rows`、`_live_search_sheet_diffs`、`_load_search_key_groups`、`_hydrate_diff_rows`、`validate_mapping`、`evaluate_metric_dual`、`list_baseline_expect_objects` 全部改用它。 +- **验收测试**：参数化对照测试，用同一份 fixture 断言 7 条路径对同一行给出的 match key 相同（重点覆盖：映射、父接口继承、QinQ、值映射、归一化规则）。 + +#### B2 分侧接口归一化 + 厂商内置规范化 +- `iface_normalize_rules` 每条增加可选字段 `side: "both"|"before"|"after"`（默认 `both`）和 `vendor`（可选，按批次的厂商自动生效）。前端模板编辑器支持选择。 +- 内置“厂商内全称/缩写”规范化表（`iface_normalize.py` 中的 `VENDOR_CANONICAL`）： + - 华为：`GigabitEthernet`↔`GE`、`XGigabitEthernet`↔`XGE`、`100GE`、`Eth-Trunk` + - 思科：`GigabitEthernet`↔`Gi`、`TenGigE`↔`Te`、`HundredGigE`↔`Hu`、`Bundle-Ether`↔`BE` + - 华三：`GigabitEthernet`↔`GE`、`Ten-GigabitEthernet`↔`XGE`、`Bridge-Aggregation`↔`BAGG` + - 中兴：大小写 + - 统一去掉名称中的空格。 + + 只做厂商内同义统一，**不做跨厂商编号转换**（那是端口映射的职责）。模板 sheet 可以用 `builtin_vendor_canonical: false` 关闭。 +- 批次的厂商取 `BizStateBatch.vendor`，取不到再用任务的 vendor，经 `resolve_vendor_key` 解析。 +- `config_snapshot` v2 记录两侧最终生效的规则。 +- **验收测试**：华为 `GigabitEthernet0/0/1` 与 `GE0/0/1` 互认；before 华为、after 中兴，只对 before 生效的规则不影响 after。 + +#### B3 字段值映射（T-06、B-09） +- `field_rules[]` 新增可选字段： + - `value_map: {"estab": "established", "*down": "down", ...}`（大小写不敏感匹配，输出规范值） + - `side: both|before|after` + - `apply_to_key: true|false`（默认：字段在 key_fields 中时为 true） +- `normalize` 的处理顺序：`normalize`（strip/lower/mac…）→ `value_map`。**键字段也要应用**（解决 MAC 作为键时跨厂商对不上的问题）。 +- SQL：`value_map` 作用在键或比较字段上时，`sql_compare_skip_reason` 返回 `value_map`，降级到 Python（以后可以用 `CASE` 下推，本期不做）。 +- `compare_validation.py` 校验 `value_map` 必须是 `str→str` 映射，并且每个 sheet 最多 500 项。 +- 模板编辑器：字段规则行增加“值映射”编辑（每行一对，可以粘贴）。 +- **验收测试**：`state: Estab` 对 `established`，配置值映射后为 unchanged；MAC 作为键字段时，`00e0-fc12-3456` 与 `00e0.fc12.3456` 能够配对；Python 与 SQL 降级原因断言。 + +#### B4 行过滤在归一化之后求值（B-08） +- 新 run（snapshot v2）：先 `transform` 再 `row_filters`。 +- PG 下推只在“过滤涉及的字段都不在 `iface_fields` 中、也没有值映射”时启用；否则 Python 过滤。 +- 支持过滤条件带 `side`（只作用于某一侧），满足“旧侧只看 dynamic ARP”这类需求。 +- 模板编辑器提示“过滤在归一化之后执行，请按规范值书写”。 +- **验收测试**：规则 `GE→gei` 加过滤 `interface contains gei`，before 侧华为 `GE0/0/1` 不再被滤掉；`version=1` 的 run 回放结果不变。 + +#### B5 端口映射覆盖率预检（B-07、B-11、M-06 的基础） +- 新接口：`POST /v1/biz-state/compare/preflight`，请求体 `{template_id, enabled_sheet_ids, mapping_id | mapping_rows, before_batch_id, after_batch_id}`。割接监控复用同一个服务函数：`/v1/biz-migration/projects/{id}/preflight`，传入基线和最新 HF 批次。 +- 按 sheet 返回： + - `before_count / after_count`（过滤后）、`parse` 状态（见 V1） + - `mapped_exact[]`、`mapped_by_parent[]`、`identity_same_name[]`（未映射但两侧同名） + - **`suspect_unmapped[]`**：before 侧接口没有映射，after 侧也不存在 → 疑似漏配 + - `unused_map_keys[]`、`map_targets_missing_on_after[]`、`n_to_1_targets[]` + - `command_coverage`：两侧执行过的 profile_id 集合和差集（见 V2） +- 汇总：映射覆盖率 = 已映射 / (已映射 + 疑似漏配)，以及“可执行 / 有风险”的结论。 +- 业务对比 run 的 summary 也写入同结构的 `mapping_coverage`（按 sheet），结果页可以查看。 +- 前端：映射块里的“校验”按钮改为显示预检表格（取代原始 JSON）；疑似漏配可以一键加入映射编辑区（目标留空，待填）。 +- **验收测试**：构造 3 个端口，分别是精确映射、父接口继承、漏配，预检分类正确；`mapping_rows` 未保存时也能预检。 + +#### B6 SQL 引擎条件放宽（B-02） +- `sql_compare_skip_reason`：`port_map` 只在 `iface_fields ∩ (key_fields ∪ compare_fields)` 非空时才阻止 SQL。 +- **验收测试**：有端口映射的任务，`bgp_route` sheet（`iface_fields=[]`）走 SQL（断言 skip reason 为空），`interface_brief` 仍走 Python。 + +#### B7 映射语义：N:1 与部分映射（B-13、M-13） +- 决策见第 6 节，默认方案： + - **N:1**：允许，但预检和 summary 中必须列出；业务对比按重复键规则处理（V4）；割接监控继续拒绝，但错误信息里列出冲突行。 + - **1:N（breakout）**：本期不支持，在映射编辑器里检测到重复 before 时明确报错，不再只报 `duplicate_before_if`。 + - **部分映射**：新增 sheet 选项 `unmapped_port_policy: "by_name"|"ignore_port"`（默认 `by_name`，即当前行为），让 `ignore_port_changes` 与映射可以组合使用。 +- **验收测试**：N:1 在预检中出现在 `n_to_1_targets`；割接监控的报错包含冲突对。 + +--- + +### 阶段 C：判定语义与数据可信度 + +#### V1 sheet 级数据可信度（B-10） +- `_run_sheet` 和割接评估为每侧补充 `collect_quality`：该 metric 的命令数、`parse_status` 计数、`row_count`、`raw_line_count`、`declared_total`（只读这些列，不读 `raw_text`）。 +- 新增 sheet 状态： + - `no_data`：两侧过滤后都是 0 行，且命令都执行成功 → `pass_rate=null`，不计入总通过率，卡片显示“无数据”。 + - `suspect_parse`：某侧 `raw_line_count > 5` 但 `row_count == 0` → 卡片显示“疑似解析失败”，失败行数照常显示，但给出明确提示。 + - `not_collected`：某侧没有任何命令产生该 metric → 不做判定，`pass_rate=null`。 +- 任务级 `ok`：任一 sheet 处于 `not_collected` / `suspect_parse` 时为 `null`（未判定），不再是 `true`。 +- 前端卡片和 CSV `_summary.csv` 增加状态列。 +- **验收测试**：0 对 0 → `no_data`；before 1000 行、after 有原文但 0 行 → `suspect_parse`；after 侧无命令 → `not_collected`。 + +#### V2 采集命令集对齐（B-11、M-06） +- `ParseProfile` 增加可选字段 `slice: dict[str, str]`（例如 `zte.bgp_ipv4_summary` 的 slice 是 `{"afi": "ipv4"}`），声明这个 profile 产出哪个切片。先给中兴的 `bgp_peer` / `bgp_route` / `vrrp` / `isis` profile 补齐。 +- 判断一个 sheet 是否“被某侧覆盖”：该侧存在 `metric_id` 相同的成功命令，且（sheet 的 row_filters 里有 `eq` 条件时）存在 slice 与之相符的 profile。 +- 业务对比的预检（B5）和执行时都做这个检查：未覆盖的 sheet 标记 `not_collected`（V1）。 +- 割接监控：`ensure_highfreq` 生成 HF 任务项时，取**两侧画像命令项的并集**（按 profile 的 slice 转换成该侧厂商可用的 profile；找不到时返回明确错误 `no_profile_for_slice:::`）。评估时，未覆盖的 sheet 标记为 `not_collected`，不判红。 +- **验收测试**：旧画像有 ipv4 + vpnv4，新画像只有 vpnv4 → HF 任务两侧都有 ipv4 + vpnv4；如果强制只有一侧有，评估结果是 `not_collected`，而不是 `lost`。 + +#### V3 方向性判定（B-15、T-03） +- `field_rules.compare` 新增： + - `not_worse`：配合 `rank: ["up", "down"]`（排在前面表示更好）；after 不差于 before 即通过。 + - `numeric_min`：配合 `tolerance`（百分比），`after >= before*(1-tol%)` 即通过（适用于前缀数、流量“不应明显下降”）。 + - `numeric_max`：对称方向。 +- `explain_diff` 输出方向原因，例如 `worse: up→down`、`drop 37% > 10%`。 +- SQL：先降级（skip reason 为 `directional_compare`），后续再下推。 +- 内置模板：`interface_brief.prot`、`bgp_peer.state` 默认仍为 `eq`（保守）；在模板编辑器里提供这些新模式供选择；流量计数器若要比较，推荐用 `numeric_min`。 +- **验收测试**：`down→up` 用 `not_worse` 判为通过，`up→down` 判为失败；前缀数 1000→1200 用 `numeric_min 5%` 判为通过，1000→900 判为失败。 + +#### V4 重复键配对：先排序再配对（B-12） +- Python：同一个 match key 下，两侧各自按“归一化后的 compare 字段元组, 原始 seq”排序，然后 zip。 +- SQL：`dup_rn` 改为 `row_number() OVER (PARTITION BY rk ORDER BY , seq, id)`。 +- 用 sheet 选项 `duplicate_pairing: "sorted"|"ordered"` 控制，默认 `sorted`，只对 snapshot v2 生效；v1 回放保持 `ordered`。 +- **验收测试**：同键两行的值 `[a, b]` 对 `[b, a]` → 2 个 unchanged（原来是 2 个 changed）；Python 和 SQL 生成 SQL 的断言。 + +#### V5 原始身份展示（B-14） +- diff 增加 `raw_key_before`（before 原始键值，未归一化/映射）。实现方式：在 Python 路径生成；对精简行，在回填时由 `IdentityPipeline` 反推；SQL 路径没有映射，原始键等于 key。 +- 结果表：当原始值与映射后的值不同时，键单元格显示 `cgei-0/2/0/1 ← GE0/0/1`；CSV 增加 `__raw_before` 列（只针对接口键）。 +- **验收（浏览器脚本）**：映射行同时显示两侧端口名。 + +--- + +### 阶段 D：割接监控的健壮性与性能 + +#### H1 run、diff 保留策略 + 批次保护收敛（M-02） +- 新配置项：`biz_migration_auto_run_keep`（默认每个 batch 保留 300 个 auto run）、`biz_compare_auto_run_keep`（默认每个 job 保留 50 个 auto run）。 +- 清理规则： + - 永久保留：`purpose in (acceptance, manual)` 的 run，以及被红单 `run_id` 引用的 run。 + - 其余 auto run：超过数量上限的删除（连同其 diff）。 + - 保护范围：`retention.protected_batch_map` 只保护“仍然存在的 run”所引用的批次，并**加上 `summary.metric_batches` 里的所有批次**（目前只保护首个 binding 的批次）。 +- 清理挂在 `biz_state_scheduler._maybe_purge_retention` 上，每小时执行一次。 +- **验收测试**：插入 400 个 auto run 和 1 个验收 run，清理后剩 300 + 1；被删除 run 所引用的 HF 批次不再受保护，能按 `retention_days` 被清理；验收 run 引用的所有批次（包括 `metric_batches`）都受保护。 + +#### H2 评估和验收改为后台执行（M-03） +- 仿照业务对比的 `enqueue_compare`：`POST /batches/{id}/evaluate` 和 `/finish` 返回 `{run_id, status: "running"}`，在后台线程里执行，`BizMigrationRun.status` 依次经过 `running/success/failed`，`summary.progress` 记录进度。 +- 项目行锁只在“写 run 和 diff”这个短事务里持有，不再覆盖整个计算过程。 +- 重启恢复：参照 `recover_interrupted_compares_on_startup`，把遗留的 `running` 标记为 `cancelled`。 +- 前端：轮询 run 的状态；验收按钮在执行期间不可重复点击。 +- **验收测试**：评估请求立即返回；并发两次评估，第二次返回 409；模拟重启后状态被正确恢复。 + +#### H3 路由类 sheet 的处理（M-04） +- 默认：`MANUAL_ROUTE_METRICS` 对应的 sheet **不参与验收门槛**。没有在本批窗口内手动采集时，显示为 `manual_pending`（灰色），不计入 `current_missing` 或 `coverage_incomplete`；已采集时参与判定。 +- 路由 sheet 的双侧比对改为复用业务对比的 SQL 路径（旧基线 vs 旧当前、新基线 vs 新当前，各执行一次 `run_sql_sheet_compare`，只取计数和失败行），避免在 Python 里载入百万行。 +- **验收测试**：默认监控模板、没有采集路由时，验收不因路由失败；采集了路由时，判定会出现在结果里。 + +#### H4 基线行缓存（M-10） +- 进程内 LRU：键为 `(batch_id, metric_id)`，只缓存基线批次（基线不可变），总行数上限可配置（默认 50 万行）。基线变更或项目删除时失效。 +- **验收测试**：连续两次 auto 评估，第二次不再查询基线行（mock 计数）。 + +#### H5 重复键的处理口径（M-05） +- `duplicate_keys` 只在“本批范围内的键（expect）有重复”时作为验收失败原因；其他 sheet 的重复键作为警告，写进 `accept_summary.warnings`。 +- 卡片上展示具体是哪些重复键（已有 `duplicate_key_list`），并提示“完善 key_fields（例如 ISIS 加上 `lev`）或在 sheet 上启用 `duplicate_pairing: sorted`”。 +- **验收测试**：范围外的 sheet 有重复键时验收可以通过；范围内有重复时验收失败。 + +#### H6 HF 采集合并与单设备并发上限（M-09） +- 决策见第 6 节，默认：**同间隔、同 lane（light/heavy）的 metric 合并为一对任务**（一次 SSH 采多条命令）；heavy 和 manual 的路由表保持独立。 +- 新配置项 `biz_state_max_sessions_per_ne`（默认 2）：`claim_queued_batches` 认领时，同一个 `ne_id` 正在运行的批次达到上限就跳过（成对认领时要两侧都满足）。 +- `_hf_bindings` 的结构保持兼容（`metric_ids` 本来就是列表）。 +- **验收测试**：15 个 light metric、同一间隔 → 每侧 1 个 HF 任务；单设备并发上限生效（3 个同 NE 批次只认领 2 个）。 + +#### H7 同设备割接模式（M-08） +- 当 `old_ne == new_ne` 时允许创建项目：每个 metric 只建一个 HF 任务，新旧两侧共用同一批次（`collect_group_id` 只含一个任务，轮次天然对齐），映射表达同框端口迁移。 +- `evaluate_metric_dual` 不需要改（两侧是同一份行数据，靠映射区分新旧端口）。 +- `enqueue_collect_group` 允许单任务组（`mode=single_ne`）。 +- **验收测试**：同设备项目可以创建；端口 A 迁移到 B（A 变为 down、B 变为 up）判为 `migrated`。 + +#### H8 去掉 metric 名称启发式（M-12） +- `sheet_overrides` 增加 `scope_by_port_map: true|false`；不填时按当前规则推导，并把推导结果写进 override 的 `effective_*` 字段供界面展示。`_map_defines_iface_expect` 不再读取 metric 名称。 +- **验收测试**：一个名称不含 bgp 的 sheet 设置 `scope_by_port_map=false` 后，不会被映射推导出范围。 + +#### H9 其他（M-14、M-15） +- GET 接口不写库：`project_to_dict` 中的默认模板绑定改为只读解析，持久化只在 create/patch 时进行。 +- 红单：在后续验收中已转绿的键标记为 `resolvable=true`（不自动关闭），前端提供“一键关闭可关闭红单”。 + +--- + +### 阶段 E：跨厂商解析（需要你提供真实样本） + +#### P1 华为 VRP 首批状态解析（T-05） +- **前提**：每个命令提供至少 2 份真实输出样本，放到 `tests/fixtures/huawei/`，并做脱敏（不同版本、带子接口和 VRF 的各一份）。**没有样本时 Cursor 不得凭空编写 TextFSM。** +- 顺序：`interface_brief`（`display interface brief`）、`arp`（`display arp all`）、`bgp_peer`（`display bgp peer` / `display bgp vpnv4 all peer`）、`isis_adjacency`（`display isis peer`）、`ospf_neighbor`（`display ospf peer brief`）。 +- 每个 metric 输出**与中兴相同的字段名**（FieldDef 共用），状态值输出**规范枚举**： + - 接口 `admin/phy/prot ∈ {up, down}`；华为的 `*down` 输出为 `admin=down` + - BGP `state ∈ {established, idle, active, connect, opensent, openconfirm}` + - ARP `entry_type ∈ {dynamic, static, interface}` + + 这样对比层的过滤和预设不需要按厂商分叉。 +- 按 `skills/biz-state/netx-biz-state/SKILL.md` 第 2 节逐项完成：TextFSM、`prefer_fsm`、注册、profile（附 `slice`）、单测。 +- 新增“规范值契约”文档：`docs/biz-state-canonical-fields.md`，按 metric 列出字段、取值范围，以及各厂商原始值到规范值的对照。中兴 parser 也要对照这份契约补齐规范化（例如 `entry_type`）。 +- ARP 过滤：内置模板从 `age_timer` 改为 `entry_type eq dynamic`（依赖各厂商 parser 输出规范的 `entry_type`）；中兴 parser 在 `entry_type` 为空时，根据 age 推断出 `dynamic`。 +- **验收**：fixture 单测；中兴→华为的跨厂商对比用例（固定行数据）在配置映射和值映射后通过率为 100%。 + +--- + +### 阶段 F：前端体验（和上面各任务一起做） + +1. 映射编辑器升级为表格模式（保留文本模式）：两列可编辑，支持粘贴 CSV/Excel，逐行显示校验状态（重复、无效、归一化后的值）。“从基线生成”：左列 before 端口、右列 after 端口，可按“相同槽位/端口号”自动推荐。 +2. 预检结果表（B5）：疑似漏配、未使用的键、N:1、命令覆盖差集，每项都可以展开查看明细。 +3. sheet 卡片显示 `no_data` / `suspect_parse` / `not_collected` / `manual_pending` 状态和原因（V1、H3）。 +4. 结果表的键列显示原始端口名（V5）。 +5. 模板编辑器：接口归一化规则增加 side/vendor 选择；字段规则增加值映射和方向性比较；过滤条件可选 side；行过滤处提示“按归一化后的值书写”。 +6. 割接看板：重复键警告、未覆盖切片、预设语义说明（端口成功条件）。 + +--- + +## 4. 建议执行顺序 + +| 顺序 | 任务 | 说明 | +|---|---|---| +| 1 | A1–A7 | 正确性快修，相互独立，可以并行 | +| 2 | B1 → B2 → B3 → B4 | 流水线是 B2–B4 的基础，必须先做 | +| 3 | B5、B6、B7 | 预检依赖流水线 | +| 4 | V1 → V2 → V3 → V4 → V5 | V2 依赖 V1 的状态定义 | +| 5 | H1 → H2 → H3 → H4 → H5 → H6 → H7 → H8 → H9 | H1/H2 优先（生产风险最大） | +| 6 | 阶段 E | 等你提供样本 | +| 7 | 阶段 F | 随各任务交付；最后做一次整体走查 | + +--- + +## 5. 验收方式（Claude 验收时执行） + +### 5.1 回归命令 +```powershell +# 项目根目录：割接对比相关后端回归（基线 279 项通过） +.venv\Scripts\python.exe -m pytest tests/test_biz_state_compare.py tests/test_biz_state_compare_sql.py tests/test_biz_state_compare_search.py tests/test_biz_state_compare_snapshot.py tests/test_biz_state_compare_export.py tests/test_biz_state_compare_recovery.py tests/test_biz_compare_templates.py tests/test_biz_migration_evaluate.py tests/test_biz_migration_evidence.py tests/test_biz_migration_hf_decouple.py tests/test_biz_migration_integrity.py tests/test_iface_normalize.py tests/test_biz_state_monitor_service.py tests/test_biz_state_paired_sampling.py tests/test_biz_state_retention.py -q +# 全量 +.venv\Scripts\python.exe -m pytest -q + +# web 目录 +node node_modules/typescript/bin/tsc -b +node node_modules/vite/bin/vite.js build +node --experimental-strip-types --test tests/cutoverDataCache.test.mjs tests/compareTemplateModel.test.mjs +# 浏览器（先启动 vite :5179） +node tests/bizCompare.browser.mjs +node tests/bizCompareTemplates.browser.mjs +node tests/bizMigration.browser.mjs +node tests/bizState.browser.mjs +``` + +### 5.2 必须新增的测试文件（便于验收定位） +- `tests/test_identity_pipeline.py`：B1 的 7 路径对照、B2、B3、B4 +- `tests/test_compare_preflight.py`：B5、B7、V2 +- `tests/test_compare_quality.py`：V1、V3、V4 +- `tests/test_biz_migration_presets.py`：A4 的 4 个端口场景、H5、H8 +- `tests/test_biz_migration_retention.py`：H1 +- `tests/test_biz_migration_async.py`：H2、H3 +- `tests/test_cutover_references.py`：A2、A5 +- 浏览器：`bizCompare.browser.mjs` 增加 A3、B5、V5 场景；`bizMigration.browser.mjs` 增加 H2、V1 卡片状态场景 + +### 5.3 我会逐条核对的场景 +1. 原始名称映射（A1）：`GE0/1/0/1→gei0/2/0/1` 加规则 `GE→gei`，结果为 unchanged。 +2. 拔纤但未 shutdown（A4）：验收判 `migrated`。 +3. 删除被引用的映射或模板（A2）：返回 409。 +4. 向导中映射未保存（A3）：不会被静默丢弃。 +5. 跨厂商过滤（B4）：华为侧不会被 `contains gei` 整侧滤掉。 +6. MAC 作为键（B3）：不同格式能配对。 +7. 空对空（V1）：显示 `no_data`，不是 100%。 +8. 命令集不一致（V2）：显示 `not_collected`，不是 `lost`。 +9. 同键乱序（V4）：不再出现假“不一致”。 +10. 有映射时 BGP 路由走 SQL（B6）。 +11. auto run 被清理、被引用批次的保护随之释放（H1）。 +12. 验收异步执行、不阻塞请求（H2）；默认模板不因路由未采集而验收失败（H3）。 +13. 历史 run（snapshot v1）的检索、回填、导出结果与改动前一致。 + +--- + +## 6. 需要你确认的决策（Cursor 按“默认”执行，除非你另行指定） + +| # | 决策 | 默认 | +|---|---|---| +| D1 | 业务对比是否允许 N:1 映射 | 允许，但在预检和结果中明确列出；割接监控继续拒绝并说明冲突行 | +| D2 | 1:N（端口拆分 breakout）本期是否支持 | 不支持，明确报错 | +| D3 | 空对空 / 疑似解析失败时，任务级“是否通过” | `null`（未判定），不计入通过率，不当作失败 | +| D4 | 路由表是否参与割接验收门槛 | 未采集时不参与（`manual_pending`），采集后参与 | +| D5 | HF 采集是否按“同间隔 + 同 lane”合并任务 | 合并；每台设备并发 SSH 上限默认 2 | +| D6 | 是否支持同设备割接 | 支持（H7） | +| D7 | auto run 保留数量 | 割接每批 300、业务对比每个任务 50；验收和手动 run 永久保留 | +| D8 | 流量计数器默认是否参与比较 | 不参与；需要时用 `numeric_min` | +| D9 | 华为样本由谁提供、首批覆盖哪些命令 | 由你提供；首批为接口、ARP、BGP、ISIS、OSPF | diff --git a/netx_api/app_startup.py b/netx_api/app_startup.py index 8acc5f5..23d8a0c 100644 --- a/netx_api/app_startup.py +++ b/netx_api/app_startup.py @@ -170,6 +170,14 @@ def run_api_startup() -> None: ) except Exception: _log.exception("startup: biz compare recovery failed") + try: + from .biz_migration.service import recover_interrupted_evaluations_on_startup + + n_eval = recover_interrupted_evaluations_on_startup(db) + if n_eval: + _log.info("startup: cancelled %s interrupted cutover evaluation(s)", n_eval) + except Exception: + _log.exception("startup: cutover evaluation recovery failed") try: from .port_traffic_migrate import backfill_port_traffic_series diff --git a/netx_api/biz_migration/auto_monitor.py b/netx_api/biz_migration/auto_monitor.py index 4cd8174..4a0154c 100644 --- a/netx_api/biz_migration/auto_monitor.py +++ b/netx_api/biz_migration/auto_monitor.py @@ -44,6 +44,11 @@ def try_auto_monitor_for_task(db: Session, task_id: str) -> int: if not batch or proj.status == "done": db.rollback() continue + if db.query(BizMigrationRun.id).filter( + BizMigrationRun.project_id == proj.id, BizMigrationRun.status == "running" + ).first(): + db.rollback() # a manual / acceptance evaluation owns the project right now + continue mt = db.get(BizMonitorTemplate, proj.monitor_template_id) if proj.monitor_template_id else None if not mt: db.rollback() @@ -60,7 +65,8 @@ def try_auto_monitor_for_task(db: Session, task_id: str) -> int: batch.expect_set_json, service._port_map_dict(db, proj.mapping_id), sheets, overrides, defaults, norms, service.resolve_collect_metric_ids(db, proj)] fingerprint = hashlib.sha256(json.dumps(payload, sort_keys=True, ensure_ascii=False).encode()).hexdigest() - last = db.query(BizMigrationRun).filter(BizMigrationRun.batch_id == batch.id).order_by( + last = db.query(BizMigrationRun).filter( + BizMigrationRun.batch_id == batch.id, BizMigrationRun.status == "success").order_by( BizMigrationRun.created_at.desc()).first() if last and (last.summary_json or {}).get("auto_fingerprint") == fingerprint: db.rollback() diff --git a/netx_api/biz_migration/evaluate.py b/netx_api/biz_migration/evaluate.py index f762a49..dd34fc0 100644 --- a/netx_api/biz_migration/evaluate.py +++ b/netx_api/biz_migration/evaluate.py @@ -4,7 +4,7 @@ from __future__ import annotations from typing import Any -from ..biz_state.compare_engine import apply_port_map, compare_rows +from ..biz_state.compare_engine import apply_port_map, compare_rows, key_tuple from ..biz_state.compare_rules import ( effective_compare_fields, field_rule_map, scalar_text, values_equal, ) @@ -756,10 +756,10 @@ def build_diff_index_from_compare(result: dict[str, Any]) -> dict[str, dict[str, continue key = d.get("key") if isinstance(key, dict): - key_list = [str(v) for v in key.values()] + key_list = [scalar_text(v).strip() for v in key.values()] ks = _key_str(key_list) elif isinstance(key, (list, tuple)): - key_list = [str(x) for x in key] + key_list = [scalar_text(x).strip() for x in key] ks = _key_str(key_list) else: ks = str(key or "") @@ -801,17 +801,18 @@ def expect_keys_for_metric( return keys -def _map_defines_iface_expect(metric_id: str, iface_fields: list[str], port_map: dict[str, str]) -> bool: - """Iface sheets: a non-empty port map is the expect scope. +def _map_defines_iface_expect( + iface_fields: list[str], port_map: dict[str, str], scope_by_port_map: bool | None = None, +) -> bool: + """Whether a non-empty port map is this sheet's expect scope. - BGP / PW keep the batch expect picker even if a column is marked iface. + The sheet override decides (``scope_by_port_map``); the built-in presets set it to + false for BGP / pseudowire sheets, which keep the batch expect picker. Unset means + "an interface sheet with a map is scoped by the map". """ if not port_map or not iface_fields: return False - mid = str(metric_id or "").strip().lower() - if "bgp" in mid or "pw" in mid or "pseudowire" in mid: - return False - return True + return True if scope_by_port_map is None else bool(scope_by_port_map) def _iface_values(key_str: str, *, key_fields: list[str], iface_fields: list[str]) -> list[str]: @@ -1022,11 +1023,14 @@ def row_match_key( key_fields: list[str], iface_fields: list[str], rules: list[dict[str, str]] | None, + normalizer: Any = None, ) -> str: from ..biz_state.iface_normalize import apply_iface_normalize data = strip_netx(row) - if iface_fields and rules: + if normalizer is not None: + data = normalizer(dict(data)) + elif iface_fields and rules: data = apply_iface_normalize(dict(data), iface_fields=iface_fields, rules=rules) return KEY_SEP.join(scalar_text(data.get(k)).strip() for k in key_fields) @@ -1037,11 +1041,14 @@ def index_raw_by_match_key( key_fields: list[str], iface_fields: list[str], rules: list[dict[str, str]] | None, + normalizer: Any = None, ) -> dict[str, dict[str, Any]]: """First raw row per internal match key (normalized iface, raw elsewhere).""" idx: dict[str, dict[str, Any]] = {} for raw in raw_rows: - ks = row_match_key(raw, key_fields=key_fields, iface_fields=iface_fields, rules=rules) + ks = row_match_key( + raw, key_fields=key_fields, iface_fields=iface_fields, rules=rules, normalizer=normalizer, + ) if ks and ks not in idx: idx[ks] = raw return idx @@ -1062,6 +1069,7 @@ def iface_lineage( rules: list[dict[str, str]] | None, port_map: dict[str, str] | None, apply_map: bool, + namer: Any = None, ) -> list[dict[str, Any]]: """Per iface field: raw → normalized → (optional) mapped.""" from ..biz_state.iface_normalize import normalize_iface_name, resolve_mapped_iface @@ -1073,7 +1081,7 @@ def iface_lineage( out: list[dict[str, Any]] = [] for f in iface_fields: raw_v = scalar_text(data.get(f)).strip() - norm_v = normalize_iface_name(raw_v, rules) if rules else raw_v + norm_v = namer(raw_v) if namer is not None else (normalize_iface_name(raw_v, rules) if rules else raw_v) mapped = False map_to = "" if apply_map and pmap and norm_v: @@ -1113,6 +1121,7 @@ def evaluate_metric_dual( iface_normalize_rules: list[dict[str, str]] | None = None, previous_expect: dict[str, set[str]] | None = None, allow_mapping_scope: bool = True, + pipeline: Any = None, ) -> dict[str, Any]: """Run old vs old-baseline, new vs new-baseline (or mapped old baseline), dual merge. @@ -1120,10 +1129,12 @@ def evaluate_metric_dual( Board / AI fields ``old_key`` / ``new_key`` and ``old`` / ``new`` stay **raw** device values (A/B), never the post-map BB form. """ - from ..biz_state.iface_normalize import ( - apply_iface_normalize_rows, - normalize_iface_rules, - normalize_iface_name, + from ..biz_state.identity_pipeline import legacy_pipeline + + # Old device = "before" side, new device = "after" side of the identity pipeline. + pipe = pipeline or legacy_pipeline( + key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields, + field_rules=field_rules, port_map=port_map, iface_normalize_rules=iface_normalize_rules, ) sid = str(sheet_id or "").strip() or str(metric_id or "").strip() @@ -1159,48 +1170,44 @@ def evaluate_metric_dual( [dict(r) for r in new_baseline_rows] if new_baseline_rows is not None else None ) - norm_rules = normalize_iface_rules(iface_normalize_rules) def normalized_key(key: str) -> str: parts = key.split(KEY_SEP) if KEY_SEP in key else ([key] if len(key_fields) == 1 else key.split("|")) if len(parts) != len(key_fields): return key - return KEY_SEP.join(normalize_iface_name(p, norm_rules) if f in iface_fields else p + return KEY_SEP.join(pipe.normalize_name(p, "before") if f in iface_fields else p for p, f in zip(parts, key_fields)) + def norm_before(row: dict[str, Any]) -> dict[str, Any]: + return pipe.normalize(row, "before") + + def norm_after(row: dict[str, Any]) -> dict[str, Any]: + return pipe.normalize(row, "after") + expect_keys = {normalized_key(key) for key in expect_keys} previous_keys = {normalized_key(key) for key in previous_keys} - old_baseline_rows = apply_iface_normalize_rows( - [strip_netx(r) for r in raw_old_base], iface_fields=iface_fields, rules=norm_rules - ) - old_current_rows = apply_iface_normalize_rows( - [strip_netx(r) for r in raw_old_cur], iface_fields=iface_fields, rules=norm_rules - ) - new_current_rows = apply_iface_normalize_rows( - [strip_netx(r) for r in raw_new_cur], iface_fields=iface_fields, rules=norm_rules - ) + old_baseline_rows = [norm_before(strip_netx(r)) for r in raw_old_base] + old_current_rows = [norm_before(strip_netx(r)) for r in raw_old_cur] + new_current_rows = [norm_after(strip_netx(r)) for r in raw_new_cur] if raw_new_base is not None: - new_baseline_rows = apply_iface_normalize_rows( - [strip_netx(r) for r in raw_new_base], - iface_fields=iface_fields, - rules=norm_rules, - ) + new_baseline_rows = [norm_after(strip_netx(r)) for r in raw_new_base] else: new_baseline_rows = None old_raw_cur_idx = index_raw_by_match_key( - raw_old_cur, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules + raw_old_cur, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_before ) old_raw_base_idx = index_raw_by_match_key( - raw_old_base, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules + raw_old_base, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_before ) new_raw_cur_idx = index_raw_by_match_key( - raw_new_cur, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules + raw_new_cur, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_after ) new_raw_base_idx = index_raw_by_match_key( raw_new_base or [], key_fields=key_fields, iface_fields=iface_fields, - rules=norm_rules, + rules=None, + normalizer=norm_after, ) old_cmp = compare_rows( @@ -1213,6 +1220,7 @@ def evaluate_metric_dual( field_rules=field_rules, iface_normalize_rules=None, # already applied include_unchanged=True, + duplicate_pairing=pipe.duplicate_pairing, ) old_idx = build_diff_index_from_compare(old_cmp) @@ -1244,15 +1252,16 @@ def evaluate_metric_dual( port_map=None, field_rules=field_rules, include_unchanged=True, + duplicate_pairing=pipe.duplicate_pairing, ) new_idx = build_diff_index_from_compare(new_cmp) rev_map = {v: k for k, v in port_map.items()} - map_scoped = _map_defines_iface_expect(metric_id, iface_fields, port_map) + map_scoped = _map_defines_iface_expect(iface_fields, port_map, ov.get("scope_by_port_map")) # Mapping describes identity; an explicit batch selection defines scope. # Port selection also covers business rows attached to those interfaces. for scope, keys in ((expect, expect_keys), (previous_expect or {}, previous_keys)): - ports = {normalize_iface_name(p, norm_rules) for p in scope.get("_ports") or set()} + ports = {pipe.normalize_name(p, "before") for p in scope.get("_ports") or set()} if ports and not keys and iface_fields: for ks, diff in old_idx.items(): row = diff.get("before") or diff.get("after") or {} @@ -1278,6 +1287,21 @@ def evaluate_metric_dual( scoped.add(canon) expect_keys = scoped + # Duplicate keys only block when they sit inside this wave's scope. + from collections import Counter + + def _dup_keys(rows: list[dict[str, Any]]) -> set[str]: + return {KEY_SEP.join(k) for k, n in Counter(key_tuple(r, key_fields) for r in rows).items() if n > 1} + + in_scope = expect_keys | previous_keys + new_dups = _dup_keys(new_current_rows) | _dup_keys(new_baseline_rows or []) + flagged = ( + _dup_keys(old_baseline_rows) | _dup_keys(old_current_rows) | new_dups + | {_reverse_remap_key_str(k, key_fields=key_fields, iface_fields=iface_fields, rev_map=rev_map) + for k in new_dups} + ) + duplicate_in_expect = sorted(flagged & in_scope)[:64] + canon_keys: list[str] = [] seen: set[str] = set() @@ -1447,9 +1471,10 @@ def evaluate_metric_dual( "iface": iface_lineage( raw_old, iface_fields=iface_fields, - rules=norm_rules, + rules=None, port_map=port_map, apply_map=True, + namer=lambda v: pipe.normalize_name(v, "before"), ), }, "new": { @@ -1457,9 +1482,10 @@ def evaluate_metric_dual( "iface": iface_lineage( raw_new, iface_fields=iface_fields, - rules=norm_rules, + rules=None, port_map=None, apply_map=False, + namer=lambda v: pipe.normalize_name(v, "after"), ), }, "port_map": { @@ -1522,6 +1548,7 @@ def evaluate_metric_dual( return { "metric_id": metric_id, + "effective_scope_by_port_map": map_scoped, "old_summary": old_cmp.get("summary") or {}, "new_summary": new_cmp.get("summary") or {}, "progress_ok": progress_ok, @@ -1536,6 +1563,7 @@ def evaluate_metric_dual( + int((new_cmp.get("summary") or {}).get("duplicate_keys_before") or 0), "duplicate_keys_after": int((old_cmp.get("summary") or {}).get("duplicate_keys_after") or 0) + int((new_cmp.get("summary") or {}).get("duplicate_keys_after") or 0), + "duplicate_in_expect": duplicate_in_expect, "duplicate_key_list": list( dict.fromkeys( list((old_cmp.get("summary") or {}).get("duplicate_key_list") or []) diff --git a/netx_api/biz_migration/monitor_templates.py b/netx_api/biz_migration/monitor_templates.py index 9ebc420..bd72ee0 100644 --- a/netx_api/biz_migration/monitor_templates.py +++ b/netx_api/biz_migration/monitor_templates.py @@ -61,6 +61,18 @@ def _out(row: BizMonitorTemplate, compare_name: str = "", *, db: Session | None return out +def _has_legacy_port_and_group(override: dict[str, Any]) -> bool: + """True when a success rule's old side demands admin down AND phy down.""" + legacy = _port_old_down_groups(legacy_and=True)[0] + for rule in override.get("success") or []: + if not isinstance(rule, dict): + continue + for group in rule.get("old_groups") or []: + if group == legacy: + return True + return False + + def validate_sheet_overrides( db: Session, *, @@ -94,6 +106,9 @@ def validate_sheet_overrides( for ov in overrides: sid = str(ov.get("sheet_id") or "").strip() mid = str(ov.get("metric_id") or "").strip() + if _has_legacy_port_and_group(ov): + warnings.append({"sheet_id": sid, "metric_id": mid, + "reason": "port_success_legacy_and_semantics"}) if sid: if sid not in sheet_ids: warnings.append( @@ -228,9 +243,29 @@ def _success_stateful( } -def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[str, Any]: +_PORT_ADMIN_DOWN = {"type": "value", "field": "admin", "op": "in", "value": ["down"]} +_PORT_PHY_DOWN = {"type": "value", "field": "phy", "op": "in", "value": ["down"]} + + +def _port_old_down_groups(*, legacy_and: bool = False) -> list[list[dict[str, Any]]]: + """Old-side "port is out of service" groups (OR of groups, AND within one). + + A fibre pulled without ``shutdown`` leaves admin up / phy down, so either + signal alone counts. ``legacy_and`` is the pre-A4 form requiring both. + """ + if legacy_and: + return [[dict(_PORT_ADMIN_DOWN), dict(_PORT_PHY_DOWN)]] + return [[dict(_PORT_PHY_DOWN)], [dict(_PORT_ADMIN_DOWN)]] + + +def preset_override_for_metric( + metric_id: str, *, sheet_id: str = "", legacy_port_and: bool = False, +) -> dict[str, Any]: """Default dual-verdict overlay for a compare-template sheet. + ``legacy_port_and`` rebuilds the pre-A4 port preset (old side must be + admin down AND phy down) so stored copies can be recognised and upgraded. + ``metric_id`` is the collected source table; ``sheet_id`` identifies a filtered split (e.g. bgp_peer.vpnv4). Presets key off the source metric. """ @@ -244,6 +279,8 @@ def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[st look = mid.split(".", 1)[0] if "." in mid else mid def _out(body: dict[str, Any]) -> dict[str, Any]: + if _scope_by_map_default_off(look): + body["scope_by_port_map"] = False body["metric_id"] = mid or look if sid and sid != (mid or look): body["sheet_id"] = sid @@ -261,12 +298,7 @@ def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[st "up_values": ["up"], "success": [ _success_stateful( - old_down_groups=[ - [ - {"type": "value", "field": "admin", "op": "in", "value": ["down"]}, - {"type": "value", "field": "phy", "op": "in", "value": ["down"]}, - ] - ], + old_down_groups=_port_old_down_groups(legacy_and=legacy_port_and), new_up_conds=up, ) ], @@ -369,6 +401,8 @@ DEFAULT_MONITOR_TEMPLATE_NAME = "默认割接监控" def ensure_default_monitor_templates(db: Session) -> None: """Seed once when monitor-template table is empty — one template on the full status compare.""" if db.query(BizMonitorTemplate.id).limit(1).first(): + upgrade_legacy_port_overrides(db) + fill_scope_by_port_map(db) return cmp_svc.ensure_default_templates(db) zte = ( @@ -403,9 +437,68 @@ def ensure_default_monitor_templates(db: Session) -> None: db.commit() -def default_port_monitor_template_id(db: Session) -> str: - """Ensure seeds exist and return the default monitor template id.""" - ensure_default_monitor_templates(db) +def _scope_by_map_default_off(name: str) -> bool: + """BGP / pseudowire sheets keep the batch expect picker; a port map does not scope them.""" + low = str(name or "").lower() + return "bgp" in low or "pw" in low or "pseudowire" in low + + +def fill_scope_by_port_map(db: Session) -> int: + """Make the old name-based default explicit on stored overrides (one-off, idempotent).""" + changed = 0 + for row in db.query(BizMonitorTemplate).all(): + out = [] + touched = False + for ov in list(row.sheet_overrides_json or []): + if (isinstance(ov, dict) and "scope_by_port_map" not in ov + and _scope_by_map_default_off(f"{ov.get('metric_id') or ''} {ov.get('sheet_id') or ''}")): + ov = {**ov, "scope_by_port_map": False} + touched = True + out.append(ov) + if touched: + row.sheet_overrides_json = out + row.updated_at = _utcnow() + changed += 1 + if changed: + db.commit() + return changed + + +def upgrade_legacy_port_overrides(db: Session) -> int: + """Rewrite stored port overrides that are exactly the pre-A4 preset (AND semantics). + + Field-by-field equality only: anything a user customised is left alone and + surfaces ``port_success_legacy_and_semantics`` in ``override_warnings``. + """ + changed = 0 + for row in db.query(BizMonitorTemplate).all(): + overrides = list(row.sheet_overrides_json or []) + out: list[Any] = [] + touched = False + for ov in overrides: + if isinstance(ov, dict): + mid = str(ov.get("metric_id") or "").strip() + sid = str(ov.get("sheet_id") or "").strip() + if (mid or sid).split(".", 1)[0] == PORT_METRIC_ID and ov == preset_override_for_metric( + mid, sheet_id=sid, legacy_port_and=True, + ): + out.append(preset_override_for_metric(mid, sheet_id=sid)) + touched = True + continue + out.append(ov) + if touched: + row.sheet_overrides_json = out + row.updated_at = _utcnow() + changed += 1 + if changed: + db.commit() + return changed + + +def default_port_monitor_template_id(db: Session, *, seed: bool = True) -> str: + """Return the default monitor template id, seeding the built-ins first unless ``seed=False``.""" + if seed: + ensure_default_monitor_templates(db) for name in ( DEFAULT_MONITOR_TEMPLATE_NAME, "ZTE 状态割接监控", diff --git a/netx_api/biz_migration/service.py b/netx_api/biz_migration/service.py index d895809..fc965c8 100644 --- a/netx_api/biz_migration/service.py +++ b/netx_api/biz_migration/service.py @@ -3,6 +3,9 @@ from __future__ import annotations import hashlib +import logging +import threading +from collections import OrderedDict from collections import Counter from datetime import datetime, timezone from typing import Any @@ -13,6 +16,7 @@ from sqlalchemy import and_, insert, or_ from sqlalchemy.orm import Session from ..biz_state.compare_rules import apply_row_filters, scalar_text +from ..config import settings from ..biz_state.compare_service import ( _load_metric_rows, _port_map_dict, @@ -44,6 +48,44 @@ from .evaluate import ( ) PURPOSE_CUTOVER_HF = "cutover_hf" +_log = logging.getLogger("netx.biz_migration") + + +class BaselineRowCache: + """Process-local LRU of immutable baseline rows, bounded by total row count.""" + + def __init__(self, max_rows: int = 500_000) -> None: + self.max_rows = max_rows + self._items: "OrderedDict[tuple[str, str], list[dict[str, Any]]]" = OrderedDict() + self._rows = 0 + self._lock = threading.Lock() + + def get_or_load(self, batch_id: str, metric_id: str, loader: Any) -> list[dict[str, Any]]: + key = (batch_id, metric_id) + with self._lock: + hit = self._items.get(key) + if hit is not None: + self._items.move_to_end(key) + return hit + rows = loader() + if self.max_rows > 0 and len(rows) <= self.max_rows: + with self._lock: + if key not in self._items: + self._items[key] = rows + self._rows += len(rows) + while self._rows > self.max_rows and self._items: + _, evicted = self._items.popitem(last=False) + self._rows -= len(evicted) + return rows + + def invalidate(self, batch_id: str = "") -> None: + with self._lock: + for key in [k for k in self._items if not batch_id or k[0] == batch_id]: + self._rows -= len(self._items.pop(key)) + + +BASELINE_ROWS = BaselineRowCache(int(getattr(settings, "biz_migration_baseline_cache_rows", 500_000))) + MANUAL_ROUTE_METRICS = frozenset({"ip_route", "ipv6_route", "bgp_route"}) @@ -301,6 +343,22 @@ def _metric_sample_ok(db: Session, batch: BizStateBatch, metric_id: str) -> bool return bool(statuses) and all(s in ("ok", "ok_aux") for s in statuses) +def _uncovered_slice_sides( + db: Session, sheet: dict[str, Any], metric_id: str, old_batch_id: str, new_batch_id: str, +) -> list[str]: + from ..biz_state.compare_quality import covers_slice, side_receipts, sheet_slice + + wanted = sheet_slice(sheet) + if not wanted: + return [] + out = [] + for side, bid in (("old", old_batch_id), ("new", new_batch_id)): + receipts = side_receipts(db, bid, metric_id) + if receipts and not covers_slice(receipts, wanted): + out.append(side) + return out + + def _current_pair_for_metric(db: Session, proj: BizMigrationProject, mid: str, pin_old: str = "", pin_new: str = "") -> tuple[BizStateBatch | None, BizStateBatch | None, dict[str, Any]]: old_tid, new_tid = (_hf_task_id_for_metric(proj, side, mid) for side in ("old", "new")) @@ -308,6 +366,20 @@ def _current_pair_for_metric(db: Session, proj: BizMigrationProject, mid: str, new_task = db.get(BizStateTask, new_tid) if new_tid else None gid = old_task.collect_group_id if old_task else "" meta: dict[str, Any] = {"collect_group_id": gid, "manual_only": bool(old_task and old_task.collect_manual_only)} + if gid and old_tid and old_tid == new_tid and not (pin_old or pin_new): + # Same device on both sides: one task, one batch, trivially the same round. + latest = db.query(BizStateBatch).filter( + BizStateBatch.collect_group_id == gid, BizStateBatch.task_id == old_tid).order_by( + BizStateBatch.queued_at.desc(), BizStateBatch.started_at.desc(), BizStateBatch.id.desc()).first() + if latest is None: + return None, None, {**meta, "sample_issues": ["collect_pair_missing"]} + meta.update({"collect_round_id": latest.collect_round_id, "old_collect_status": latest.status, + "new_collect_status": latest.status}) + if latest.status in ("running", "queued"): + return None, None, {**meta, "sample_issues": ["collect_pair_pending"]} + if not _metric_sample_ok(db, latest, mid): + return None, None, {**meta, "sample_issues": ["collect_pair_failed"]} + return latest, latest, meta if gid and not (pin_old or pin_new): if not new_task or new_task.collect_group_id != gid: return None, None, {**meta, "sample_issues": ["collect_pair_invalid"]} @@ -483,19 +555,26 @@ def _monitor_template_brief(db: Session, template_id: str) -> dict[str, Any]: return mon_tpl._out(row, cmp_name, db=db) # noqa: SLF001 -def resolve_project_monitor_template(db: Session, proj: BizMigrationProject) -> BizMonitorTemplate: - """Return monitor template for project; seed default port template if unbound.""" - mon_tpl.ensure_default_monitor_templates(db) +def resolve_project_monitor_template( + db: Session, proj: BizMigrationProject, *, persist: bool = True, +) -> BizMonitorTemplate: + """Return monitor template for project; seed default port template if unbound. + + ``persist=False`` is for read-only callers (GET): it never seeds, upgrades or binds, + it only reports which template the project would use. + """ + if persist: + mon_tpl.ensure_default_monitor_templates(db) tid = str(getattr(proj, "monitor_template_id", None) or "").strip() row = db.get(BizMonitorTemplate, tid) if tid else None if row: return row - default_id = mon_tpl.default_port_monitor_template_id(db) + default_id = mon_tpl.default_port_monitor_template_id(db, seed=persist) row = db.get(BizMonitorTemplate, default_id) if default_id else None if not row: raise HTTPException(status_code=400, detail="monitor_template_required") # Persist default on first use so UI shows binding - if not tid: + if not tid and persist: proj.monitor_template_id = row.id proj.updated_at = utcnow_naive() db.commit() @@ -522,12 +601,12 @@ def resolve_evaluate_sheets( return [port_sheet_def()], overrides, defaults, [] -def resolve_collect_metric_ids(db: Session, proj: BizMigrationProject) -> list[str]: +def resolve_collect_metric_ids(db: Session, proj: BizMigrationProject, *, persist: bool = True) -> list[str]: """Project-level override wins; else monitor template; else sheet metric ids.""" override = _project_collect_override(proj) if override: return override - mt = resolve_project_monitor_template(db, proj) + mt = resolve_project_monitor_template(db, proj, persist=persist) collect = [str(x).strip() for x in (mt.collect_metric_ids_json or []) if str(x).strip()] if collect: return collect @@ -545,7 +624,7 @@ def project_to_dict(db: Session, p: BizMigrationProject) -> dict[str, Any]: collect_override = _project_collect_override(p) mt_brief = _monitor_template_brief(db, getattr(p, "monitor_template_id", None) or "") # Same resolution as ensure/evaluate (empty → compare sheets) - effective_collect = resolve_collect_metric_ids(db, p) + effective_collect = resolve_collect_metric_ids(db, p, persist=False) return { "id": p.id, "name": p.name, @@ -597,11 +676,79 @@ def batch_to_dict(b: BizMigrationBatch) -> dict[str, Any]: } +def _preload_project_refs(db: Session, rows: list[BizMigrationProject]) -> list[Any]: + """Load everything ``project_to_dict`` looks up with a handful of IN queries. + + ``Session.get`` answers from the identity map afterwards, so a list of N projects + costs a constant number of queries instead of ~10 per project. The identity map is + weak: the caller must keep the returned objects alive while it serializes. + """ + keep: list[Any] = [] + def ids(*attrs: str) -> set[str]: + out: set[str] = set() + for p in rows: + for a in attrs: + v = getattr(p, a, None) + if v: + out.add(str(v)) + return out + + task_ids = ids("old_task_id", "new_task_id", "old_hf_task_id", "new_hf_task_id") + for side in ("old", "new"): + for p in rows: + task_ids.update(b["task_id"] for b in _hf_bindings(p, side) if b.get("task_id")) + if task_ids: + keep.append(db.query(BizStateTask).filter(BizStateTask.id.in_(task_ids)).all()) + batch_ids = ids("old_baseline_batch_id", "new_baseline_batch_id") + if batch_ids: + keep.append(db.query(BizStateBatch).filter(BizStateBatch.id.in_(batch_ids)).all()) + template_ids = ids("monitor_template_id") + if template_ids: + monitors = db.query(BizMonitorTemplate).filter(BizMonitorTemplate.id.in_(template_ids)).all() + keep.append(monitors) + compare_ids = {m.compare_template_id for m in monitors if m.compare_template_id} + if compare_ids: + keep.append(db.query(BizCompareTemplate).filter(BizCompareTemplate.id.in_(compare_ids)).all()) + return keep + + def list_projects(db: Session) -> list[dict[str, Any]]: rows = db.query(BizMigrationProject).order_by(BizMigrationProject.created_at.desc()).all() + preloaded = _preload_project_refs(db, rows) # noqa: F841 — keeps the identity map populated return [project_to_dict(db, p) for p in rows] +def _reject_ambiguous_mapping( + port_map: dict[str, str], *, conflicts: list[dict[str, str]] | None = None, + raw: dict[str, str] | None = None, +) -> None: + """Cutover monitoring needs a 1:1 map. Say which rows collide, not just "ambiguous".""" + by_target: dict[str, list[str]] = {} + for before, after in port_map.items(): + by_target.setdefault(after, []).append(before) + n_to_1 = [{"target": t, "sources": sorted(src)} for t, src in sorted(by_target.items()) if len(src) > 1] + collided = bool(conflicts) or (raw is not None and len(port_map) != len(raw)) + if not n_to_1 and not collided: + return + parts = [f"{', '.join(c['sources'])} -> {c['target']}" for c in n_to_1] + parts += [f"{c['dropped_from']} vs {c['kept_from']}" for c in (conflicts or [])] + raise HTTPException(status_code=400, detail={ + "code": "mapping_target_ambiguous", + "message": "mapping_target_ambiguous: " + "; ".join(parts), + "n_to_1": n_to_1, "key_conflicts": list(conflicts or []), + }) + + +def _validate_baseline_ownership(db: Session, side: str, batch_id: str, task_id: str) -> None: + """A baseline batch must be a finished sample of that side's portrait task.""" + bid = str(batch_id or "").strip() + if not bid: + return + sample = db.get(BizStateBatch, bid) + if not sample or sample.task_id != str(task_id or "").strip() or sample.status not in ("success", "partial"): + raise HTTPException(status_code=400, detail=f"{side}_baseline_invalid") + + def create_project(db: Session, body: dict[str, Any]) -> dict[str, Any]: """One-shot cutover project: NE + monitor template + metrics + schedule + baseline + mapping. @@ -689,6 +836,8 @@ def create_project(db: Session, body: dict[str, Any]) -> dict[str, Any]: raise HTTPException(status_code=404, detail="old_baseline_not_found") if new_baseline and not db.get(BizStateBatch, new_baseline): raise HTTPException(status_code=404, detail="new_baseline_not_found") + _validate_baseline_ownership(db, "old", old_baseline, old_task_id) + _validate_baseline_ownership(db, "new", new_baseline, new_task_id) p = BizMigrationProject( id=uuid4().hex, @@ -744,10 +893,7 @@ def patch_project(db: Session, project_id: str, body: dict[str, Any]) -> dict[st for side in ("old", "new"): bid = str(body.get(f"{side}_baseline_batch_id", getattr(p, f"{side}_baseline_batch_id")) or "").strip() tid = str(body.get(f"{side}_task_id", getattr(p, f"{side}_task_id")) or "").strip() - if bid: - sample = db.get(BizStateBatch, bid) - if not sample or sample.task_id != tid or sample.status not in ("success", "partial"): - raise HTTPException(status_code=400, detail=f"{side}_baseline_invalid") + _validate_baseline_ownership(db, side, bid, tid) if "name" in body and body["name"] is not None: p.name = str(body["name"]).strip() or p.name if "note" in body and body["note"] is not None: @@ -763,11 +909,13 @@ def patch_project(db: Session, project_id: str, body: dict[str, Any]) -> dict[st bid = str(body["old_baseline_batch_id"] or "").strip() if bid and not db.get(BizStateBatch, bid): raise HTTPException(status_code=404, detail="batch_not_found") + BASELINE_ROWS.invalidate(p.old_baseline_batch_id) p.old_baseline_batch_id = bid if "new_baseline_batch_id" in body and body["new_baseline_batch_id"] is not None: bid = str(body["new_baseline_batch_id"] or "").strip() if bid and not db.get(BizStateBatch, bid): raise HTTPException(status_code=404, detail="batch_not_found") + BASELINE_ROWS.invalidate(p.new_baseline_batch_id) p.new_baseline_batch_id = bid if "monitor_template_id" in body and body["monitor_template_id"] is not None: mid = str(body["monitor_template_id"] or "").strip() @@ -857,6 +1005,8 @@ def delete_project(db: Session, project_id: str) -> dict[str, Any]: db.delete(b) db.delete(p) db.commit() + BASELINE_ROWS.invalidate(p.old_baseline_batch_id) + BASELINE_ROWS.invalidate(p.new_baseline_batch_id) return {"ok": True} @@ -932,12 +1082,23 @@ def run_evaluate( purpose: str = "manual", _commit: bool = True, _metric_ids: set[str] | None = None, + _run_id: str = "", ) -> dict[str, Any]: + """Evaluate a wave. + + Synchronous callers (auto monitor, tests) hold the project row lock for the whole + computation. With ``_run_id`` (background evaluate/finish) the caller already + claimed the project through a ``running`` run row, so the lock is only taken for the + short write at the end. + """ mb = db.get(BizMigrationBatch, batch_id) if not mb: raise HTTPException(status_code=404, detail="batch_not_found") - # PostgreSQL serializes evaluate/finish for this project across API workers. - proj = db.query(BizMigrationProject).filter(BizMigrationProject.id == mb.project_id).with_for_update().populate_existing().first() + proj_query = db.query(BizMigrationProject).filter(BizMigrationProject.id == mb.project_id) + if not _run_id: + # PostgreSQL serializes evaluate/finish for this project across API workers. + proj_query = proj_query.with_for_update().populate_existing() + proj = proj_query.first() if not proj: raise HTTPException(status_code=404, detail="project_not_found") db.refresh(mb) @@ -988,8 +1149,7 @@ def run_evaluate( port_map = _port_map_dict(db, proj.mapping_id) if proj.mapping_id and not db.get(BizPortMapping, proj.mapping_id): raise HTTPException(status_code=400, detail="mapping_missing") - if len(set(port_map.values())) != len(port_map): - raise HTTPException(status_code=400, detail="mapping_target_ambiguous") + _reject_ambiguous_mapping(port_map) expect = parse_expect_set(mb.expect_set_json if isinstance(mb.expect_set_json, dict) else {}) previous_expect: dict[str, set[str]] = {} for prior in db.query(BizMigrationBatch).filter( @@ -1014,12 +1174,15 @@ def run_evaluate( from ..biz_state.compare_validation import validate_template_body validate_template_body({"metrics": sheets, "iface_normalize_rules": iface_norm}) - from ..biz_state.iface_normalize import normalize_iface_name + from ..biz_state.compare_service import _batch_vendor_key + from ..biz_state.identity_pipeline import build_pipeline + from ..biz_state.iface_normalize import normalize_port_map - normalized_map = {normalize_iface_name(k, iface_norm): normalize_iface_name(v, iface_norm) - for k, v in port_map.items()} - if len(normalized_map) != len(port_map) or len(set(normalized_map.values())) != len(normalized_map): - raise HTTPException(status_code=400, detail="mapping_target_ambiguous") + raw_port_map = dict(port_map) + old_vendor = _batch_vendor_key(db, proj.old_baseline_batch_id) + new_vendor = _batch_vendor_key(db, proj.new_baseline_batch_id) + normalized_map, map_conflicts = normalize_port_map(port_map, iface_norm) + _reject_ambiguous_mapping(normalized_map, conflicts=map_conflicts, raw=port_map) port_map = normalized_map known_sheets = {sheet_key(s) for s in sheets} known_metrics = {str(s.get("metric_id") or "") for s in sheets} @@ -1044,33 +1207,43 @@ def run_evaluate( scope_ids.add("_ports") uncovered_scope = sorted({k for scope in (expect, previous_expect) for k, values in scope.items() if values and k not in scope_ids}) - config_snapshot = {"sheets": sheets, "sheet_overrides": sheet_overrides, + config_snapshot = {"identity_version": 2, "vendors": {"old": old_vendor, "new": new_vendor}, + "sheets": sheets, "sheet_overrides": sheet_overrides, "defaults": defaults, "iface_normalize": iface_norm, "port_map": port_map, "expect_set": mb.expect_set_json, "previous_expect": {k: sorted(v) for k, v in previous_expect.items()}, "baseline_ids": [proj.old_baseline_batch_id, proj.new_baseline_batch_id], "collect_metric_ids": sorted(collect_ids), "window_active": window_active, "bindings": [_hf_bindings(proj, "old"), _hf_bindings(proj, "new")]} - prior_run = db.query(BizMigrationRun).filter(BizMigrationRun.batch_id == batch_id).order_by( + prior_run = db.query(BizMigrationRun).filter( + BizMigrationRun.batch_id == batch_id, BizMigrationRun.status == "success").order_by( BizMigrationRun.created_at.desc()).first() if _metric_ids is not None else None reusable = (prior_run.summary_json or {}).get("config_snapshot") == config_snapshot if prior_run else False prior_cards = {sheet_key(c): c for c in (prior_run.summary_json or {}).get("sheet_cards", [])} if reusable else {} pair_cache: dict[str, tuple[BizStateBatch | None, BizStateBatch | None, dict[str, Any]]] = {} + baseline_ids = {proj.old_baseline_batch_id, proj.new_baseline_batch_id} + def metric_rows(bid: str, mid: str) -> list[dict[str, Any]]: key = (bid, mid) if key not in rows_cache: - rows_cache[key] = _load_metric_rows(db, batch_id=bid, metric_id=mid) + if bid in baseline_ids: + # Baselines never change: share them across the minute-by-minute evaluations. + rows_cache[key] = BASELINE_ROWS.get_or_load( + bid, mid, lambda: _load_metric_rows(db, batch_id=bid, metric_id=mid)) + else: + rows_cache[key] = _load_metric_rows(db, batch_id=bid, metric_id=mid) return rows_cache[key] - for sheet in sheets: + for sheet_index, sheet in enumerate(sheets, start=1): mid = str(sheet.get("metric_id") or "").strip() sid = sheet_key(sheet) key_fields = list(sheet.get("key_fields") or []) if not mid or not key_fields: continue + if _run_id: + _report_eval_progress(_run_id, sheet_index, len(sheets), sheet_title(sheet)) iface_fields = list(sheet.get("iface_fields") or []) compare_fields = list(sheet.get("compare_fields") or []) - row_filters = list(sheet.get("row_filters") or []) field_rules = list(sheet.get("field_rules") or []) sheet_ov = override_for_sheet(sheet_overrides, sheet_id=sid, metric_id=mid) if sheet_ov.get("skip_dual"): @@ -1111,6 +1284,16 @@ def run_evaluate( pair_cache[mid] = _current_pair_for_metric(db, proj, mid, pin_old, pin_new) old_cur_batch, new_cur_batch, pair_meta = pair_cache[mid] # Do NOT fall back to another HF task's batch — empty/wrong metric → false red + if (not old_cur_batch or not new_cur_batch) and mid in MANUAL_ROUTE_METRICS: + # Route tables are collected by hand; until someone does, they are not part of the gate. + sheet_cards.append({ + "metric_id": mid, "sheet_id": sid, "title": sheet_title(sheet), + "progress_ok": 0, "progress_total": 0, "anomaly": 0, "anomaly_in_expect": 0, + "old_summary": {}, "new_summary": {}, "new_baseline_mode": "manual_pending", + "new_baseline_missing": False, "manual_pending": True, "collect_skipped": False, + "current_missing": False, **pair_meta, + }) + continue if not old_cur_batch or not new_cur_batch: sheet_cards.append( { @@ -1134,6 +1317,19 @@ def run_evaluate( continue old_cur_mid = old_cur_batch.id new_cur_mid = new_cur_batch.id + # A slice (address family) one device never collected is a collection gap, not a lost business. + uncovered = _uncovered_slice_sides(db, sheet, mid, old_cur_mid, new_cur_mid) + if uncovered: + sheet_cards.append({ + "metric_id": mid, "sheet_id": sid, "title": sheet_title(sheet), + "progress_ok": 0, "progress_total": 0, "anomaly": 0, "anomaly_in_expect": 0, + "old_summary": {}, "new_summary": {}, "new_baseline_mode": "not_collected", + "new_baseline_missing": False, "not_collected": True, "data_state": "not_collected", + "data_state_reasons": [f"{side}:slice_not_collected" for side in uncovered], + "collect_skipped": False, "current_missing": False, + "old_batch_id": old_cur_mid, "new_batch_id": new_cur_mid, **pair_meta, + }) + continue timing = {**pair_meta, **_sample_timing(proj, mid, old_cur_batch, new_cur_batch, check_age=acceptance or not (pin_old and pin_new))} baseline_missing = [side for side in ("old", "new") if not _metric_sample_ok( @@ -1145,22 +1341,15 @@ def run_evaluate( "baseline_missing_sides": baseline_missing, **timing}) continue - old_base = apply_row_filters( - metric_rows(proj.old_baseline_batch_id, mid), - row_filters, - ) - old_now = apply_row_filters( - metric_rows(old_cur_mid, mid), - row_filters, - ) - new_base_rows = apply_row_filters( - metric_rows(proj.new_baseline_batch_id, mid), - row_filters, - ) - new_now = apply_row_filters( - metric_rows(new_cur_mid, mid), - row_filters, + # Old device = "before", new device = "after"; filters see normalized rows. + pipe = build_pipeline( + sheet, iface_rules=iface_norm, port_map=raw_port_map, + before_vendor=old_vendor, after_vendor=new_vendor, ) + old_base = [raw for raw, _ in pipe.prepare(metric_rows(proj.old_baseline_batch_id, mid), "before")] + old_now = [raw for raw, _ in pipe.prepare(metric_rows(old_cur_mid, mid), "before")] + new_base_rows = [raw for raw, _ in pipe.prepare(metric_rows(proj.new_baseline_batch_id, mid), "after")] + new_now = [raw for raw, _ in pipe.prepare(metric_rows(new_cur_mid, mid), "after")] one = evaluate_metric_dual( metric_id=mid, @@ -1172,7 +1361,7 @@ def run_evaluate( old_current_rows=old_now, new_baseline_rows=new_base_rows, new_current_rows=new_now, - port_map=port_map if iface_fields else {}, + port_map=pipe.port_map if iface_fields else {}, expect=expect, window_active=window_active, acceptance=acceptance, @@ -1182,6 +1371,7 @@ def run_evaluate( iface_normalize_rules=iface_norm, previous_expect=previous_expect, allow_mapping_scope=False, + pipeline=pipe, ) # A target snapshot taken before the source's disappearance cannot prove # loss. Wait for the next target receipt; final acceptance stays blocked. @@ -1226,6 +1416,7 @@ def run_evaluate( "duplicate_key_list": list(one.get("duplicate_key_list") or []), "duplicate_keys_before": int(one.get("duplicate_keys_before") or 0), "duplicate_keys_after": int(one.get("duplicate_keys_after") or 0), + "duplicate_in_expect": list(one.get("duplicate_in_expect") or []), **timing, "awaiting_peer": awaiting, "verdict_counts": dict(Counter(r["verdict"] for r in one["rows"] if r.get("verdict"))), @@ -1253,10 +1444,12 @@ def run_evaluate( for verdict, count in card.get("verdict_counts", {}).items(): verdict_counts[verdict] = verdict_counts.get(verdict, 0) + count active_cards = [ - c for c in sheet_cards if not c.get("collect_skipped") and not c.get("current_missing") + c for c in sheet_cards + if not c.get("collect_skipped") and not c.get("current_missing") and not c.get("manual_pending") + and not c.get("not_collected") ] run = BizMigrationRun( - id=uuid4().hex, + id=_run_id or uuid4().hex, project_id=proj.id, batch_id=mb.id, old_batch_id=old_cur, @@ -1294,9 +1487,31 @@ def run_evaluate( "verdict_counts": verdict_counts, "anomaly_outside_expect": sum(int(c.get("anomaly") or 0) - int(c.get("anomaly_in_expect") or 0) for c in active_cards), "steady_outside": sum(int(c.get("steady_outside") or 0) for c in active_cards), - "duplicate_keys": sum(int(c.get("duplicate_keys_before") or 0) + int(c.get("duplicate_keys_after") or 0) for c in active_cards), + # Only duplicate keys inside the wave's scope fail an acceptance; the rest are warnings. + "duplicate_keys": sum(len(c.get("duplicate_in_expect") or []) for c in active_cards), + "duplicate_warnings": [ + {"sheet_id": c.get("sheet_id"), "title": c.get("title"), + "count": int(c.get("duplicate_keys_before") or 0) + int(c.get("duplicate_keys_after") or 0), + "examples": list(c.get("duplicate_key_list") or [])[:5]} + for c in active_cards + if (c.get("duplicate_keys_before") or c.get("duplicate_keys_after")) + and not c.get("duplicate_in_expect") + ], + "manual_pending_metrics": [ + str(c.get("title") or c.get("sheet_id") or c.get("metric_id") or "") + for c in sheet_cards if c.get("manual_pending") + ], + "not_collected_metrics": [ + str(c.get("title") or c.get("sheet_id") or c.get("metric_id") or "") + for c in sheet_cards if c.get("not_collected") + ], "uncovered_scope": uncovered_scope, - "coverage_complete": bool(enabled_sheets) and not uncovered_scope and not any(c.get("current_missing") or c.get("awaiting_peer") or c.get("duplicate_keys_before") or c.get("duplicate_keys_after") for c in sheet_cards if not c.get("collect_skipped")), + "coverage_complete": bool(enabled_sheets) and not uncovered_scope and not any( + c.get("current_missing") or c.get("awaiting_peer") + or c.get("duplicate_in_expect") + for c in sheet_cards + if not c.get("collect_skipped") and not c.get("manual_pending") and not c.get("not_collected") + ), "config_snapshot": config_snapshot, "window_active": window_active, "expect_ports": sorted(expect.get("_ports") or ()), @@ -1305,7 +1520,15 @@ def run_evaluate( }, message="", ) - db.add(run) + if _run_id: + # Claim the project only for the write; the computation above ran unlocked. + db.query(BizMigrationProject).filter(BizMigrationProject.id == proj.id).with_for_update().first() + existing = db.get(BizMigrationRun, _run_id) + if existing is None or existing.status != "running": + raise HTTPException(status_code=409, detail="evaluation_cancelled") + run = db.merge(run) + else: + db.add(run) db.flush() for card in sheet_cards: card.setdefault("source_run_id", run.id) @@ -1546,13 +1769,17 @@ def list_baseline_expect_objects(db: Session, project_id: str) -> dict[str, Any] mt = resolve_project_monitor_template(db, p) sheets, sheet_overrides, _defaults, _norm = resolve_evaluate_sheets(db, mt) collect_ids = set(resolve_collect_metric_ids(db, p)) - port_map = _port_map_dict(db, p.mapping_id) + raw_port_map = _port_map_dict(db, p.mapping_id) out_sheets: list[dict[str, Any]] = [] - from ..biz_state.iface_normalize import apply_iface_normalize, resolve_mapped_iface, normalize_iface_name + from ..biz_state.compare_service import _batch_vendor_key + from ..biz_state.identity_pipeline import build_pipeline + from ..biz_state.iface_normalize import normalize_port_map, resolve_mapped_iface from .evaluate import KEY_SEP rows_cache: dict[str, list[dict[str, Any]]] = {} - port_map = {normalize_iface_name(k, _norm): normalize_iface_name(v, _norm) for k, v in port_map.items()} + old_vendor = _batch_vendor_key(db, p.old_baseline_batch_id) + new_vendor = _batch_vendor_key(db, p.new_baseline_batch_id) + port_map, _ = normalize_port_map(raw_port_map, _norm) for sheet in sheets: mid = str(sheet.get("metric_id") or "").strip() sid = sheet_key(sheet) @@ -1571,14 +1798,13 @@ def list_baseline_expect_objects(db: Session, project_id: str) -> dict[str, Any] iface_fields = [str(k) for k in (sheet.get("iface_fields") or []) if str(k).strip()] if mid not in rows_cache: rows_cache[mid] = _load_metric_rows(db, batch_id=p.old_baseline_batch_id, metric_id=mid) - rows_raw = apply_row_filters( - rows_cache[mid], - list(sheet.get("row_filters") or []), + pipe = build_pipeline( + sheet, iface_rules=_norm, port_map=raw_port_map, + before_vendor=old_vendor, after_vendor=new_vendor, ) items: list[dict[str, Any]] = [] seen_keys: set[str] = set() - for r in rows_raw: - normalized = apply_iface_normalize(dict(r), iface_fields=iface_fields, rules=_norm) + for r, normalized in pipe.prepare(rows_cache[mid], "before"): keys = [scalar_text(normalized.get(f)).strip() for f in key_fields] if not any(keys): continue @@ -1588,7 +1814,7 @@ def list_baseline_expect_objects(db: Session, project_id: str) -> dict[str, Any] seen_keys.add(key_str) mapped = "" if iface_fields and key_fields and key_fields[0] in iface_fields: - mapped = resolve_mapped_iface(keys[0], port_map) if port_map else "" + mapped = resolve_mapped_iface(keys[0], pipe.port_map) if pipe.port_map else "" items.append( { "key": key_str, @@ -1617,6 +1843,55 @@ def list_baseline_expect_objects(db: Session, project_id: str) -> dict[str, Any] } +def preflight_project(db: Session, project_id: str) -> dict[str, Any]: + """Mapping coverage + baseline data quality of a cutover project, before the window. + + Judged on the two portrait baselines (full collections of both devices), so a + missing/ambiguous mapping row shows up before monitoring starts. + """ + from ..biz_state.compare_preflight import preflight + from ..biz_state.compare_service import _batch_vendor_key + from ..biz_state.iface_normalize import normalize_port_map + + p = db.get(BizMigrationProject, project_id) + if not p: + raise HTTPException(status_code=404, detail="project_not_found") + if not p.old_baseline_batch_id or not p.new_baseline_batch_id: + raise HTTPException(status_code=400, detail="baseline_required") + mt = resolve_project_monitor_template(db, p) + sheets, sheet_overrides, _defaults, iface_norm = resolve_evaluate_sheets(db, mt) + collect_ids = set(resolve_collect_metric_ids(db, p)) + scoped = [ + s for s in sheets + if str(s.get("metric_id") or "") and (not collect_ids or str(s.get("metric_id")) in collect_ids) + and not override_for_sheet(sheet_overrides, sheet_id=sheet_key(s), metric_id=str(s.get("metric_id"))).get("skip_dual") + ] + raw_map = _port_map_dict(db, p.mapping_id) + cache: dict[tuple[str, str], list[dict[str, Any]]] = {} + + def load(side: str, sheet: dict[str, Any]) -> list[dict[str, Any]]: + bid = p.old_baseline_batch_id if side == "before" else p.new_baseline_batch_id + key = (bid, str(sheet["metric_id"])) + if key not in cache: + cache[key] = _load_metric_rows(db, batch_id=bid, metric_id=key[1]) + return cache[key] + + report = preflight( + db, sheets=scoped, iface_rules=iface_norm, raw_port_map=raw_map, + before_batch_id=p.old_baseline_batch_id, after_batch_id=p.new_baseline_batch_id, + before_vendor=_batch_vendor_key(db, p.old_baseline_batch_id), + after_vendor=_batch_vendor_key(db, p.new_baseline_batch_id), + load_rows=load, + ) + normalized, _ = normalize_port_map(raw_map, iface_norm) + # Monitoring needs 1:1; the report lists the rows that collide. + if report["mapping"]["n_to_1_targets"] or report["mapping"]["conflicts"]: + report["risk_reasons"] = sorted({*report["risk_reasons"], "mapping_target_ambiguous"}) + report["verdict"] = "risk" + report["mapping"]["pairs_normalized"] = len(normalized) + return report + + def _catalog_item_for_metric(*, vendor: str, device_type: str, metric_id: str) -> dict[str, Any]: from ..biz_state.profiles import profiles_for_vendor from ..lldp_shared import resolve_vendor_key @@ -1758,6 +2033,96 @@ def _portrait_metric_items(db: Session, portrait: BizStateTask | None, mid: str) return out +def _group_metrics(metric_ids: list[str], by_metric: dict[str, int], default_iv: int) -> list[tuple[int, list[str]]]: + """Collect groups: one task pair (one SSH login per device) per interval + lane. + + Manual route tables stay on their own; heavy dumps never share a task with light + status tables. + """ + from ..biz_state.profiles import metric_lane + + groups: list[tuple[int, list[str]]] = [] + buckets: dict[tuple[int, str], list[str]] = {} + for mid in metric_ids: + iv = max(60, by_metric.get(mid) or default_iv) + if mid in MANUAL_ROUTE_METRICS: + groups.append((iv, [mid])) + continue + key = (iv, metric_lane(mid)) + if key not in buckets: + buckets[key] = [] + groups.append((iv, buckets[key])) + buckets[key].append(mid) + return groups + + +def _group_id(project_id: str, metrics: list[str]) -> str: + return "cutover:" + project_id + ":" + hashlib.sha256("+".join(sorted(metrics)).encode()).hexdigest()[:16] + + +def _slice_key(item: dict[str, Any]) -> tuple[tuple[str, str], ...]: + from ..biz_state.profiles import get_profile + + prof = get_profile(str(item.get("source_profile_id") or "")) + return tuple(sorted((prof.slice or {}).items())) if prof is not None else () + + +def _union_slice_items( + metric_id: str, old_items: list[dict[str, Any]], new_items: list[dict[str, Any]], + old_vendor: str, new_vendor: str, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + """Both devices must collect the same slices (afi / address family) of a metric. + + A slice one portrait collects and the other does not would otherwise look like + business loss. The missing side gets this vendor's profile for the slice, or a + clear ``no_profile_for_slice`` error. + """ + from ..biz_state.profiles import get_profile, profiles_for_vendor + from ..lldp_shared import resolve_vendor_key + + if not old_items or not new_items: + return old_items, new_items + + def fill(items: list[dict[str, Any]], other: list[dict[str, Any]], vendor: str) -> list[dict[str, Any]]: + have = {_slice_key(i) for i in items} + out = list(items) + vkey = resolve_vendor_key(vendor, "") or vendor + for want in [i for i in other if _slice_key(i) not in have and _slice_key(i)]: + wanted_slice = dict(_slice_key(want)) + profile = next((p for p in profiles_for_vendor(vkey, kind="collect") + if p.metric_id == metric_id and dict(p.slice or {}) == wanted_slice), None) + if profile is None: + label = ",".join(f"{k}={v}" for k, v in sorted(wanted_slice.items())) + raise HTTPException( + status_code=400, detail=f"no_profile_for_slice:{metric_id}:{label}:{vkey or vendor or 'unknown'}") + names = {ph.name for ph in profile.placeholders} + source = get_profile(str(want.get("source_profile_id") or "")) + copied = [b for b in want.get("bindings") or [] if b.get("placeholder") in names] + if any(getattr(ph, "required", True) for ph in profile.placeholders) and not copied: + raise HTTPException( + status_code=400, detail=f"metric_needs_bindings:{metric_id}:" + ",".join(sorted(names))) + out.append({"source_profile_id": profile.profile_id, "kind": "catalog", "enabled": True, + "title": profile.title or (source.title if source else metric_id), "bindings": copied}) + have.add(_slice_key(out[-1])) + return out + + return fill(old_items, new_items, old_vendor), fill(new_items, old_items, new_vendor) + + +def _group_items( + db: Session, portrait: BizStateTask | None, template_fields: dict[str, Any], metrics: list[str], + portrait_items: dict[str, list[dict[str, Any]]], +) -> list[dict[str, Any]]: + """Every metric of a merged task keeps its own portrait-configured commands.""" + out: list[dict[str, Any]] = [] + for mid in metrics: + items = portrait_items.get(mid) or [_catalog_item_for_metric( + vendor=str(template_fields.get("vendor") or ""), + device_type=str(template_fields.get("device_type") or ""), metric_id=mid)] + out.extend(items) + return out + + def _ensure_side_highfreq( db: Session, *, @@ -1940,7 +2305,7 @@ def ensure_highfreq( if not metric_ids: metric_ids = [PORT_METRIC_ID] by_metric = _metric_interval_map(proj) - groups = [(max(60, by_metric.get(mid) or iv_default), [mid]) for mid in metric_ids] + groups = _group_metrics(metric_ids, by_metric, iv_default) want_status = _hf_window_status(proj) old_portrait = db.get(BizStateTask, proj.old_task_id) if proj.old_task_id else None @@ -1953,11 +2318,9 @@ def ensure_highfreq( for binding in _hf_bindings(proj, side): task = db.get(BizStateTask, binding["task_id"]) if task: - mids = binding.get("metric_ids") or metric_ids - for mid in mids: + for mid in binding.get("metric_ids") or metric_ids: if mid in metric_ids: out.setdefault(mid, task) - break return out old_existing = _existing_by_metric("old") @@ -1983,14 +2346,26 @@ def ensure_highfreq( old_any_created = False new_any_created = False + # Same physical device on both sides (board swap, same-chassis port move): one task + # per group serves as "old" and "new"; the port map tells the two ports apart. + same_device = (old_fields.get("source"), old_fields.get("ne_id")) == (new_fields.get("source"), new_fields.get("ne_id")) + for g_iv, g_metrics in groups: - mid = g_metrics[0] - gid = "cutover:" + proj.id + ":" + hashlib.sha256(mid.encode()).hexdigest()[:16] - old_candidate, new_candidate = old_existing.get(mid), new_existing.get(mid) + manual_only = any(m in MANUAL_ROUTE_METRICS for m in g_metrics) + gid = _group_id(proj.id, g_metrics) + old_candidate = next((old_existing[m] for m in g_metrics if m in old_existing), None) + new_candidate = next((new_existing[m] for m in g_metrics if m in new_existing), None) if old_candidate and old_candidate.collect_group_id not in ("", gid): old_candidate = None if new_candidate and new_candidate.collect_group_id not in ("", gid): new_candidate = None + old_by_metric = {m: _portrait_metric_items(db, old_portrait, m) for m in g_metrics} + new_by_metric = {m: _portrait_metric_items(db, new_portrait, m) for m in g_metrics} + if not same_device: + for m in g_metrics: + old_by_metric[m], new_by_metric[m] = _union_slice_items( + m, old_by_metric[m], new_by_metric[m], + str(old_fields.get("vendor") or ""), str(new_fields.get("vendor") or "")) old_task, old_created = _ensure_side_highfreq( db, existing_hf=old_candidate, @@ -2000,36 +2375,35 @@ def ensure_highfreq( retention_days=ret, metric_ids=g_metrics, status="paused", - configured_items=_portrait_metric_items(db, old_portrait, mid), + configured_items=_group_items(db, old_portrait, old_fields, g_metrics, old_by_metric), ) - new_task, new_created = _ensure_side_highfreq( - db, - existing_hf=new_candidate, - template_fields=new_fields, - project_name=proj.name, - interval_sec=g_iv, - retention_days=ret, - metric_ids=g_metrics, - status="paused", - configured_items=_portrait_metric_items(db, new_portrait, mid), - ) - if old_task.ne_id == new_task.ne_id: + if same_device: + new_task, new_created = old_task, False + else: + new_task, new_created = _ensure_side_highfreq( + db, + existing_hf=new_candidate, + template_fields=new_fields, + project_name=proj.name, + interval_sec=g_iv, + retention_days=ret, + metric_ids=g_metrics, + status="paused", + configured_items=_group_items(db, new_portrait, new_fields, g_metrics, new_by_metric), + ) + if not same_device and old_task.ne_id == new_task.ne_id: raise HTTPException(status_code=400, detail="collect_pair_same_device") - for task in (old_task, new_task): + for task in {old_task.id: old_task, new_task.id: new_task}.values(): task.collect_group_id = gid - task.collect_manual_only = mid in MANUAL_ROUTE_METRICS + task.collect_manual_only = manual_only task.status = "paused" if task.collect_manual_only else want_status db.commit() old_any_created = old_any_created or old_created new_any_created = new_any_created or new_created - old_bindings.append( - {"task_id": old_task.id, "metric_ids": list(g_metrics), "interval_sec": g_iv, - "collect_group_id": gid, "manual_only": mid in MANUAL_ROUTE_METRICS} - ) - new_bindings.append( - {"task_id": new_task.id, "metric_ids": list(g_metrics), "interval_sec": g_iv, - "collect_group_id": gid, "manual_only": mid in MANUAL_ROUTE_METRICS} - ) + binding = {"metric_ids": list(g_metrics), "interval_sec": g_iv, "collect_group_id": gid, + "manual_only": manual_only} + old_bindings.append({"task_id": old_task.id, **binding}) + new_bindings.append({"task_id": new_task.id, **binding}) proj = db.get(BizMigrationProject, project_id) if not proj: @@ -2165,6 +2539,7 @@ def _red_ticket_to_dict(t: BizMigrationRedTicket) -> dict[str, Any]: "new_status": t.new_status, "detail": detail, "evidence": dict(detail.get("evidence") or {}), + "resolvable": bool(detail.get("resolvable")), "status": t.status, "carried_to_batch_id": t.carried_to_batch_id, "note": t.note, @@ -2323,16 +2698,51 @@ def _persist_red_tickets_from_run( created_or_updated.append(t) seen.add(ident) - # Tickets open on this batch but no longer red → leave as open (operator resolves). - # Carried tickets for keys not in this run stay carried. + # Tickets whose key is now green stay open (an operator decides) but are flagged so the + # UI can offer a one-click close. Keys absent from this run are left alone. + pending = {ident: t for ident, t in by_id.items() if ident not in seen and t.status in ("open", "carried")} + if pending: + metrics = {ident[0] for ident in pending} + greens: dict[tuple[str, str, str], str] = {} + for g in db.query(BizMigrationDiff).filter( + BizMigrationDiff.run_id == run_id, BizMigrationDiff.color == "green", + BizMigrationDiff.metric_id.in_(metrics), + ): + gk = g.key_json if isinstance(g.key_json, dict) else {} + mk = str(gk.get("match_old_key") or gk.get("key_str") or gk.get("old_key") or "").strip() + greens[(str(g.metric_id or ""), str(gk.get("sheet_id") or g.metric_id or ""), mk)] = str(g.verdict or "") + for ident, t in pending.items(): + verdict = greens.get(ident) + if verdict is not None: + t.detail_json = {**dict(t.detail_json or {}), "resolvable": {"run_id": run_id, "verdict": verdict}} return created_or_updated +def resolve_resolvable_red_tickets(db: Session, project_id: str) -> dict[str, Any]: + """Close every ticket the latest acceptance saw turn green (explicit operator action).""" + if not db.get(BizMigrationProject, project_id): + raise HTTPException(status_code=404, detail="project_not_found") + closed = [] + for t in db.query(BizMigrationRedTicket).filter( + BizMigrationRedTicket.project_id == project_id, + BizMigrationRedTicket.status.in_(("open", "carried")), + ): + mark = (t.detail_json or {}).get("resolvable") if isinstance(t.detail_json, dict) else None + if mark: + t.status = "resolved" + t.resolved_at = utcnow_naive() + t.note = (t.note or f"auto-closed: green in run {mark.get('run_id', '')}")[:500] + closed.append(t.id) + db.commit() + return {"closed": len(closed), "ids": closed, "open_red_count": count_open_red_tickets(db, project_id)} + + def finish_batch( db: Session, batch_id: str, *, mark_done: bool = False, + _run_id: str = "", ) -> dict[str, Any]: """本批完成：关窗 → 终验 → 验收小结 → 红单留痕（不硬卡下一批）.""" mb = db.get(BizMigrationBatch, batch_id) @@ -2346,7 +2756,9 @@ def finish_batch( # acceptance=True closes the logical window. Persist batch closure only after # evaluation succeeds, so an API/collection failure keeps the batch operable. - run = run_evaluate(db, batch_id=batch_id, acceptance=True, purpose="acceptance", _commit=False) + run = run_evaluate( + db, batch_id=batch_id, acceptance=True, purpose="acceptance", _commit=False, _run_id=_run_id, + ) summary = dict(run.get("summary") or {}) progress = dict(summary.get("progress") or {}) anomaly = int(summary.get("anomaly") or 0) @@ -2381,6 +2793,11 @@ def finish_batch( "anomaly": anomaly, "verdict_counts": dict(summary.get("verdict_counts") or {}), "expect_ports": list(summary.get("expect_ports") or []), + "warnings": { + "duplicate_keys_outside_scope": list(summary.get("duplicate_warnings") or []), + "manual_pending_metrics": list(summary.get("manual_pending_metrics") or []), + "not_collected_metrics": list(summary.get("not_collected_metrics") or []), + }, "run_id": run.get("id"), "old_batch_id": run.get("old_batch_id"), "new_batch_id": run.get("new_batch_id"), @@ -2418,6 +2835,123 @@ def finish_batch( } +_EVAL_THREADS_LOCK = threading.Lock() + + +def _report_eval_progress(run_id: str, index: int, total: int, title: str) -> None: + """Progress visible to pollers: separate short transaction, never fatal.""" + from ..db import SessionLocal + + try: + s = SessionLocal() + try: + run = s.get(BizMigrationRun, run_id) + if run is not None and run.status == "running": + summary = dict(run.summary_json or {}) + summary["progress"] = {"phase": "evaluating", "sheet_index": index, + "sheet_total": total, "sheet_title": title} + run.summary_json = summary + s.commit() + finally: + s.close() + except Exception: # noqa: BLE001 — progress is best effort + _log.debug("evaluate progress write failed run=%s", run_id, exc_info=True) + + +def _claim_evaluation( + db: Session, batch_id: str, *, acceptance: bool, purpose: str, +) -> BizMigrationRun: + """Create the ``running`` run that owns the project. 409 when one is already in flight.""" + mb = db.get(BizMigrationBatch, batch_id) + if not mb: + raise HTTPException(status_code=404, detail="batch_not_found") + proj = db.query(BizMigrationProject).filter(BizMigrationProject.id == mb.project_id).with_for_update().first() + if not proj: + raise HTTPException(status_code=404, detail="project_not_found") + db.refresh(mb) + if acceptance and mb.status not in ("active", "review"): + raise HTTPException(status_code=409, detail="monitor_window_closed") + if not proj.old_baseline_batch_id: + raise HTTPException(status_code=400, detail="old_baseline_required") + if not proj.new_baseline_batch_id: + raise HTTPException(status_code=400, detail="new_baseline_required") + busy = db.query(BizMigrationRun.id).filter( + BizMigrationRun.project_id == proj.id, BizMigrationRun.status == "running").first() + if busy: + raise HTTPException(status_code=409, detail="evaluation_in_progress") + run = BizMigrationRun( + id=uuid4().hex, project_id=proj.id, batch_id=batch_id, purpose=purpose, status="running", + summary_json={"progress": {"phase": "queued", "sheet_index": 0, "sheet_total": 0}}, + message="queued", + ) + db.add(run) + db.commit() + db.refresh(run) + return run + + +def _spawn_evaluation(run_id: str, work: Any, label: str) -> None: + from ..db import SessionLocal + + def _bg() -> None: + s = SessionLocal() + try: + work(s) + except Exception as exc: # noqa: BLE001 — recorded on the run for the poller + s.rollback() + detail = getattr(exc, "detail", None) + _log.warning("background %s failed run=%s: %s", label, run_id, detail or exc) + run = s.get(BizMigrationRun, run_id) + if run is not None and run.status == "running": + run.status = "failed" + run.message = str(detail if detail is not None else exc)[:1000] + s.commit() + finally: + s.close() + + threading.Thread(target=_bg, name=f"biz-mig-{label}-{run_id[:8]}", daemon=True).start() + + +def enqueue_evaluate( + db: Session, batch_id: str, *, old_batch_id: str = "", new_batch_id: str = "", +) -> dict[str, Any]: + """Start a manual evaluation in the background; poll ``GET /runs/{id}`` for the result.""" + run = _claim_evaluation(db, batch_id, acceptance=False, purpose="manual") + _spawn_evaluation( + run.id, + lambda s: run_evaluate(s, batch_id=batch_id, old_batch_id=old_batch_id, new_batch_id=new_batch_id, + purpose="manual", _run_id=run.id), + "evaluate", + ) + return run_to_dict(db, run) + + +def enqueue_finish(db: Session, batch_id: str, *, mark_done: bool = False) -> dict[str, Any]: + """Final acceptance in the background. The wave closes only if the evaluation succeeds.""" + mb = db.get(BizMigrationBatch, batch_id) + if not mb: + raise HTTPException(status_code=404, detail="batch_not_found") + if mb.status not in ("active", "review"): + raise HTTPException(status_code=400, detail=f"batch_not_finishable:{mb.status}") + run = _claim_evaluation(db, batch_id, acceptance=True, purpose="acceptance") + _spawn_evaluation( + run.id, lambda s: finish_batch(s, batch_id, mark_done=mark_done, _run_id=run.id), "finish", + ) + return {"run_id": run.id, "status": "running", "run": run_to_dict(db, run), + "batch": batch_to_dict(db.get(BizMigrationBatch, batch_id))} + + +def recover_interrupted_evaluations_on_startup(db: Session) -> int: + """A restart kills background evaluations; release the projects they were holding.""" + rows = db.query(BizMigrationRun).filter(BizMigrationRun.status == "running").all() + for r in rows: + r.status = "cancelled" + r.message = "interrupted_by_restart" + if rows: + db.commit() + return len(rows) + + def _validate_batch_status(db: Session, project_id: str, batch_id: str, status: str) -> None: if status not in ("pending", "active", "review", "done"): raise HTTPException(status_code=400, detail="batch_status_invalid") @@ -2544,9 +3078,10 @@ def board(db: Session, batch_id: str, run_id: str = "") -> dict[str, Any]: if not run or run.batch_id != batch_id: raise HTTPException(status_code=404, detail="run_not_found") else: + # Latest *finished* run; a run still being computed has nothing to show yet. run = ( db.query(BizMigrationRun) - .filter(BizMigrationRun.batch_id == batch_id) + .filter(BizMigrationRun.batch_id == batch_id, BizMigrationRun.status == "success") .order_by(BizMigrationRun.created_at.desc()) .first() ) diff --git a/netx_api/biz_migration/validation.py b/netx_api/biz_migration/validation.py index 881a3b3..998761a 100644 --- a/netx_api/biz_migration/validation.py +++ b/netx_api/biz_migration/validation.py @@ -23,6 +23,8 @@ def validate_monitor_rules(defaults: Any, overrides: Any) -> None: if ident in seen: fail(path + ".duplicate") seen.add(ident) + if ov.get("scope_by_port_map") is not None and not isinstance(ov["scope_by_port_map"], bool): + fail(path + ".scope_by_port_map") for key in ("status_fields", "up_values", "down_values"): if key in ov and (not isinstance(ov[key], list) or any(not str(v).strip() for v in ov[key])): fail(path + "." + key) diff --git a/netx_api/biz_migration_router.py b/netx_api/biz_migration_router.py index 52508f8..8104a4c 100644 --- a/netx_api/biz_migration_router.py +++ b/netx_api/biz_migration_router.py @@ -151,6 +151,12 @@ def api_baseline_ports(project_id: str, db: Session = Depends(get_db)): return svc.list_baseline_ports(db, project_id) +@router.get("/projects/{project_id}/preflight") +@router.post("/projects/{project_id}/preflight") +def api_project_preflight(project_id: str, db: Session = Depends(get_db)): + return svc.preflight_project(db, project_id) + + @router.get("/projects/{project_id}/baseline-expect") def api_baseline_expect(project_id: str, db: Session = Depends(get_db)): """Multi-metric baseline keys for expect-set picking (from monitor template sheets).""" @@ -222,8 +228,13 @@ def api_finish_batch( mark_done: bool = False, db: Session = Depends(get_db), ): - """本批完成：终验 + 小结 + 红单（允许带红继续）.""" - return svc.finish_batch(db, batch_id, mark_done=mark_done) + """本批完成：终验 + 小结 + 红单（允许带红继续）. Runs in the background; poll the run.""" + return svc.enqueue_finish(db, batch_id, mark_done=mark_done) + + +@router.post("/projects/{project_id}/red-tickets/resolve-resolvable") +def api_resolve_resolvable(project_id: str, db: Session = Depends(get_db)): + return svc.resolve_resolvable_red_tickets(db, project_id) @router.get("/projects/{project_id}/red-tickets") @@ -258,9 +269,10 @@ def api_patch_batch(batch_id: str, body: BatchPatchIn, db: Session = Depends(get @router.post("/batches/{batch_id}/evaluate") def api_evaluate(batch_id: str, body: EvaluateIn | None = None, db: Session = Depends(get_db)): payload = body.model_dump() if body else {} - return svc.run_evaluate( + # Returns the ``running`` run at once; poll GET /runs/{id} until ``status`` settles. + return svc.enqueue_evaluate( db, - batch_id=batch_id, + batch_id, old_batch_id=str(payload.get("old_batch_id") or ""), new_batch_id=str(payload.get("new_batch_id") or ""), ) diff --git a/netx_api/biz_state/claim.py b/netx_api/biz_state/claim.py index 2a592c0..d45b6ec 100644 --- a/netx_api/biz_state/claim.py +++ b/netx_api/biz_state/claim.py @@ -2,6 +2,8 @@ from __future__ import annotations +from collections import defaultdict + import logging from datetime import datetime, timedelta from typing import Any @@ -25,6 +27,11 @@ def max_concurrent_tasks() -> int: return max(1, int(getattr(settings, "biz_state_max_concurrent_tasks", 16) or 16)) +def max_sessions_per_ne() -> int: + """Concurrent SSH collections allowed per network element (>= 1).""" + return max(1, int(getattr(settings, "biz_state_max_sessions_per_ne", 2) or 2)) + + def enqueue_collect(task_id: str, *, manual: bool = False) -> dict[str, Any]: """Create a queued batch and mark task collect_running. Does not open SSH. @@ -135,15 +142,16 @@ def enqueue_collect_group(group_id: str, *, manual: bool = False, BizStateTask.id).with_for_update().populate_existing().all() result: dict[str, Any] = {"ok": False, "queued": False, "task_id": requested_task_id, "collect_group_id": group_id} - if len(tasks) != 2 or len({t.ne_id for t in tasks}) != 2: + # One task = a single device used as both sides (same-device cutover); two tasks = a pair. + if len(tasks) not in (1, 2) or len({t.ne_id for t in tasks}) != len(tasks): return {**result, "reason": "collect_pair_invalid"} - if max_concurrent_tasks() < 2: + if max_concurrent_tasks() < len(tasks): return {**result, "reason": "collect_pair_capacity"} from ..cli_budget import clamp_cli_workers worker_cap = clamp_cli_workers(int(getattr(settings, "biz_state_worker_collect_threads", None) or getattr(settings, "biz_state_dispatch_workers", 8) or 8)) - if worker_cap < 2: + if worker_cap < len(tasks): return {**result, "reason": "collect_pair_capacity"} if any(t.collect_running for t in tasks): return {**result, "reason": "collect_pair_busy"} @@ -230,13 +238,12 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]: if slots <= 0: return [] - busy_nes = { - str(r[0] or "").strip() - for r in db.query(BizStateBatch.ne_id) - .filter(BizStateBatch.status == "running") - .all() - if str(r[0] or "").strip() - } + # Running collections per NE; an NE accepts new ones until it reaches its session cap. + per_ne_cap = max_sessions_per_ne() + ne_running: dict[str, int] = defaultdict(int) + for (running_ne,) in db.query(BizStateBatch.ne_id).filter(BizStateBatch.status == "running").all(): + if str(running_ne or "").strip(): + ne_running[str(running_ne).strip()] += 1 # Manual route rounds yield to fresh monitoring, but must eventually # run even when a busy device always has regular metrics queued. @@ -266,14 +273,19 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]: peers = db.query(BizStateBatch).filter( BizStateBatch.collect_round_id == batch.collect_round_id, ).with_for_update(skip_locked=pg).all() - if len(peers) != 2 or any(p.status != "queued" for p in peers): + expected = db.query(BizStateTask.id).filter( + BizStateTask.collect_group_id == batch.collect_group_id).count() if batch.collect_group_id else 2 + if len(peers) != max(1, expected) or any(p.status != "queued" for p in peers): continue group = peers visited.update(p.id for p in group) if len(group) > slots - len(claimed_rows): continue - nes = {str(p.ne_id or "").strip() for p in group} - if nes & busy_nes: + wanted: dict[str, int] = defaultdict(int) + for p in group: + if str(p.ne_id or "").strip(): + wanted[str(p.ne_id).strip()] += 1 + if any(ne_running[ne] + n > per_ne_cap for ne, n in wanted.items()): continue now = _utcnow() for member in group: @@ -285,7 +297,8 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]: if task: task.last_collect_started_at = now claimed_rows.append(member) - busy_nes.update(nes - {""}) + for ne, n in wanted.items(): + ne_running[ne] += n if not claimed_rows: db.rollback() diff --git a/netx_api/biz_state/compare_engine.py b/netx_api/biz_state/compare_engine.py index 87af097..27953ff 100644 --- a/netx_api/biz_state/compare_engine.py +++ b/netx_api/biz_state/compare_engine.py @@ -93,8 +93,17 @@ def apply_port_map( return out +def key_tuple(row: Mapping[str, Any], fields: Sequence[str]) -> tuple[str, ...]: + """The one canonical match-key form: ``scalar_text(...).strip()`` per field. + + Every place that builds, stores or looks up a key string must go through + this (``None`` → ``""``, ``True`` → ``"true"``, surrounding blanks trimmed). + """ + return tuple(scalar_text(row.get(k)).strip() for k in fields) + + def row_key(row: dict[str, Any], key_fields: list[str]) -> tuple[str, ...]: - return tuple(scalar_text(row.get(k)).strip() for k in key_fields) + return key_tuple(row, key_fields) def mapping_stats( @@ -109,8 +118,8 @@ def mapping_stats( ``hit_before``: map key appears as a normalized before iface, or as the parent of a before subinterface (so main-port-only maps still validate). - Map keys are matched against **normalized** iface values (same pipeline as - compare). Enter mapping keys in normalized form. + Map keys are matched against **normalized** iface values; callers pass a + port map already normalized with the template iface rules. """ before_ifaces: set[str] = set() after_ifaces: set[str] = set() @@ -156,7 +165,7 @@ def mapping_stats( "miss_after": miss_after, "unused_before_keys": unused, "ok": not miss_before and not miss_after, - "hint": "map keys must match normalized iface names (post iface_normalize)", + "hint": "map keys/values are normalized with the template iface rules before matching", } @@ -174,6 +183,10 @@ def compare_rows( include_unchanged: bool = False, unchanged_limit: int | None = None, compact_unchanged: bool = False, + pipeline: Any = None, + match_keys_override: Sequence[str] | None = None, + prepared: tuple[list, list] | None = None, + duplicate_pairing: str = "ordered", ) -> dict[str, Any]: """Return summary + diffs list. @@ -200,6 +213,11 @@ def compare_rows( - ``True``: force drop iface from match key when a non-empty candidate exists. - ``False``: never drop iface from match key. + ``pipeline`` (``identity_pipeline.Pipeline``) owns normalization, filtering, + port mapping and key building; when omitted a legacy pipeline is derived + from the loose arguments (callers have already filtered the rows). + ``match_keys_override`` pins the match key (re-running a stored result). + ``summary.duplicate`` is always 0. ``duplicate_keys_*`` count match keys that appear more than once on a side (diagnostic only). @@ -208,32 +226,40 @@ def compare_rows( """ if not key_fields: raise ValueError("key_fields required") - pmap = dict(port_map or {}) - rules = field_rule_map(field_rules) - norm_rules = normalize_iface_rules(iface_normalize_rules) - iface_list = [str(f) for f in (iface_fields or []) if str(f).strip()] + if pipeline is None: + from .identity_pipeline import legacy_pipeline + + pipeline = legacy_pipeline( + key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields, + field_rules=field_rules, port_map=port_map, iface_normalize_rules=iface_normalize_rules, + ignore_port_changes=ignore_port_changes, duplicate_pairing=duplicate_pairing, + ) + pmap = dict(pipeline.port_map) + rules = field_rule_map(field_rules if field_rules is not None else pipeline.field_rules) + iface_list = list(pipeline.iface_fields) iface_set = set(iface_list) - before_norm = apply_iface_normalize_rows( - before_rows, iface_fields=iface_list, rules=norm_rules - ) if norm_rules and iface_list else before_rows - after_norm = apply_iface_normalize_rows( - after_rows, iface_fields=iface_list, rules=norm_rules - ) if norm_rules and iface_list else after_rows + if prepared is not None: + before_pairs, after_pairs = prepared + else: + before_pairs = pipeline.prepare(before_rows, "before") + after_pairs = pipeline.prepare(after_rows, "after") + before_rows = [orig for orig, _ in before_pairs] + after_rows = [orig for orig, _ in after_pairs] + before_norm = [norm for _, norm in before_pairs] + after_norm = [norm for _, norm in after_pairs] ignore_ports = False - # No map → optionally ignore port renames by dropping iface from match key. - if not pmap and iface_set: - candidate = [k for k in key_fields if k not in iface_set] - if not candidate: - match_keys = list(key_fields) - elif ignore_port_changes is True: - match_keys = candidate - ignore_ports = True - elif ignore_port_changes is False: - match_keys = list(key_fields) - else: - # Auto heuristic (legacy default) + if match_keys_override is not None: + match_keys = [str(k) for k in match_keys_override] + ignore_ports = match_keys != list(key_fields) + else: + # No map → optionally ignore port renames by dropping iface from match key. + policy = pipeline.ignore_port_changes + unique = None + if not pmap and iface_set and policy is None: + candidate = [k for k in key_fields if k not in iface_set] + def unique_keys(rows: list[dict[str, Any]]) -> bool: seen: set[tuple[str, ...]] = set() for row in rows: @@ -243,23 +269,20 @@ def compare_rows( seen.add(key) return True - if unique_keys(before_norm) and unique_keys(after_norm): - match_keys = candidate - ignore_ports = True - else: - match_keys = list(key_fields) - else: - match_keys = list(key_fields) + unique = bool(candidate) and unique_keys(before_norm) and unique_keys(after_norm) + match_keys = pipeline.effective_match_keys(unique_without_iface=unique) + ignore_ports = match_keys != list(key_fields) before_groups: dict[tuple[str, ...], list[tuple[dict[str, Any], dict[str, Any]]]] = ( defaultdict(list) ) after_groups: dict[tuple[str, ...], list[dict[str, Any]]] = defaultdict(list) for orig, norm in zip(before_rows, before_norm): - mapped = apply_port_map(norm, iface_fields=iface_list, port_map=pmap) if iface_list else norm - before_groups[row_key(mapped, match_keys)].append((orig, mapped)) - for r in after_norm: - after_groups[row_key(r, match_keys)].append(r) + mapped, k = pipeline.key_of(norm, "before", match_keys) + before_groups[k].append((orig, mapped)) + for norm in after_norm: + after_groups[pipeline.key_of(norm, "after", match_keys)[1]].append(norm) + sort_pairs = pipeline.duplicate_pairing == "sorted" diffs: list[dict[str, Any]] = [] added = removed = changed = unchanged = 0 @@ -271,7 +294,7 @@ def compare_rows( sample = _StratifiedSample(limit_n) if include_unchanged and limit_n is not None else None def _key_obj(row: dict[str, Any]) -> dict[str, Any]: - return {f: row.get(f, "") for f in key_fields} + return dict(zip(key_fields, key_tuple(row, key_fields))) def _row_id(row: dict[str, Any] | None) -> str: if not isinstance(row, dict): @@ -325,6 +348,10 @@ def compare_rows( multi_before_keys.append(k) if len(a_list) > 1: multi_after_keys.append(k) + if sort_pairs and (len(b_list) > 1 or len(a_list) > 1): + # Same-key rows pair by value, not by CLI output order (ECMP, multipath). + b_list = sorted(b_list, key=lambda pair: pipeline.pair_sort_key(pair[1])) + a_list = sorted(a_list, key=pipeline.pair_sort_key) n = max(len(b_list), len(a_list)) for i in range(n): if i >= len(b_list): diff --git a/netx_api/biz_state/compare_preflight.py b/netx_api/biz_state/compare_preflight.py new file mode 100644 index 0000000..c9c6081 --- /dev/null +++ b/netx_api/biz_state/compare_preflight.py @@ -0,0 +1,194 @@ +"""Pre-flight check of a port mapping against the two batches it will be applied to. + +The same function backs the compare "validate mapping" button and the cutover +project pre-flight, so both modes judge mapping coverage the same way. It answers +before anyone runs a comparison: + +- which *before* interfaces are mapped (exact / by parent), unmapped but present on + the *after* side by the same name, or **suspect** (not mapped, not on *after*) +- which map keys / targets match nothing, which targets receive several sources +- whether each sheet's two sides were actually collected (``data_state``) +""" + +from __future__ import annotations + +from typing import Any, Callable, Mapping, Sequence + +from sqlalchemy.orm import Session + +from .compare_quality import assess_sheet, command_coverage, collect_quality, side_receipts +from .identity_pipeline import build_pipeline +from .iface_normalize import normalize_port_map, resolve_mapped_iface + +LIST_CAP = 200 +# Million-row tables are never loaded just to look at interface names. +HEAVY_METRICS = frozenset({"bgp_route", "ip_route", "ipv6_route", "l2vpn_mac", "evpn_mac", "interface_detail"}) + +RowsLoader = Callable[[str, Mapping[str, Any]], list[dict[str, Any]]] + + +def _parents(name: str) -> list[str]: + parts = name.split(".") + return [".".join(parts[:i]) for i in range(len(parts) - 1, 0, -1)] + + +def _classify(name: str, pmap: Mapping[str, str], after: set[str]) -> str: + if name in pmap: + return "mapped_exact" + if any(p in pmap for p in _parents(name)): + return "mapped_by_parent" + if name in after: + return "identity_same_name" + return "suspect_unmapped" + + +def _present(name: str, names: set[str]) -> bool: + """``name`` exists, or is the parent of an existing sub-interface.""" + if name in names: + return True + prefix = name + "." + return any(n.startswith(prefix) for n in names) + + +def _top_level(names: Sequence[str]) -> list[str]: + """Drop sub-interfaces whose parent is already in the list (noise).""" + pool = set(names) + return sorted(n for n in pool if not any(p in pool for p in _parents(n))) + + +def sheet_mapping_coverage( + pipe: Any, + before_pairs: Sequence[tuple[Mapping[str, Any], Mapping[str, Any]]], + after_pairs: Sequence[tuple[Mapping[str, Any], Mapping[str, Any]]], +) -> dict[str, Any]: + """Classify every *before* interface of a sheet against the map and the *after* side. + + ``before_pairs`` / ``after_pairs`` are ``pipeline.prepare`` outputs. The result + carries two private sets (``_after_ifaces``, ``_used_keys``) for aggregation. + """ + b_if = {str(n.get(f) or "").strip() for _, n in before_pairs for f in pipe.iface_fields} + a_if = {str(n.get(f) or "").strip() for _, n in after_pairs for f in pipe.iface_fields} + b_if.discard("") + a_if.discard("") + buckets: dict[str, list[str]] = { + "mapped_exact": [], "mapped_by_parent": [], "identity_same_name": [], "suspect_unmapped": [], + } + for name in sorted(b_if): + buckets[_classify(name, pipe.port_map, a_if)].append(name) + used = {k for name in b_if for k in pipe.port_map if name == k or name.startswith(k + ".")} + missing_targets = sorted({ + resolve_mapped_iface(n, pipe.port_map) for n in buckets["mapped_exact"] + buckets["mapped_by_parent"] + if not _present(resolve_mapped_iface(n, pipe.port_map), a_if) + }) + suspects = _top_level(buckets["suspect_unmapped"]) + return { + "counts": {k: len(v) for k, v in buckets.items()}, + "mapped_exact": buckets["mapped_exact"][:LIST_CAP], + "mapped_by_parent": buckets["mapped_by_parent"][:LIST_CAP], + "identity_same_name": buckets["identity_same_name"][:LIST_CAP], + "suspect_unmapped": suspects[:LIST_CAP], + "suspect_unmapped_total": len(suspects), + "mapped_target_missing_on_after": missing_targets[:LIST_CAP], + "_after_ifaces": a_if, + "_used_keys": used, + } + + +def preflight( + db: Session, + *, + sheets: Sequence[Mapping[str, Any]], + iface_rules: Sequence[Mapping[str, Any]] | None, + raw_port_map: Mapping[str, str] | None, + before_batch_id: str, + after_batch_id: str, + before_vendor: str = "", + after_vendor: str = "", + load_rows: RowsLoader, +) -> dict[str, Any]: + """Build the pre-flight report. ``load_rows(side, sheet)`` returns raw rows.""" + raw_map = dict(raw_port_map or {}) + out_sheets: list[dict[str, Any]] = [] + used_keys: set[str] = set() + after_all: set[str] = set() + totals = {"mapped": 0, "suspect": 0, "same_name": 0} + risk_reasons: list[str] = [] + full_pmap: dict[str, str] = {} + conflicts: list[dict[str, str]] = [] + + for sheet in sheets: + sid = str(sheet.get("sheet_id") or sheet.get("metric_id") or "") + mid = str(sheet.get("metric_id") or "") + title = str(sheet.get("title") or sid) + pipe = build_pipeline( + sheet, iface_rules=iface_rules, port_map=raw_map, + before_vendor=before_vendor, after_vendor=after_vendor, + ) + if not full_pmap and pipe.port_map: + full_pmap = dict(pipe.port_map) + heavy = mid in HEAVY_METRICS + card: dict[str, Any] = {"sheet_id": sid, "metric_id": mid, "title": title} + if heavy: + b_n = collect_quality(side_receipts(db, before_batch_id, mid))["row_count"] + a_n = collect_quality(side_receipts(db, after_batch_id, mid))["row_count"] + card.update(skipped="heavy_table", before_count=b_n, after_count=a_n) + else: + b_pairs = pipe.prepare(load_rows("before", sheet), "before") + a_pairs = pipe.prepare(load_rows("after", sheet), "after") + card.update(before_count=len(b_pairs), after_count=len(a_pairs)) + if pipe.iface_fields: + cov = sheet_mapping_coverage(pipe, b_pairs, a_pairs) + after_all |= cov.pop("_after_ifaces") + used_keys |= cov.pop("_used_keys") + card.update(cov) + totals["mapped"] += cov["counts"]["mapped_exact"] + cov["counts"]["mapped_by_parent"] + totals["same_name"] += cov["counts"]["identity_same_name"] + totals["suspect"] += cov["suspect_unmapped_total"] + card.update(assess_sheet( + db, sheet, before_batch_id=before_batch_id, after_batch_id=after_batch_id, + before_rows=int(card.get("before_count") or 0), after_rows=int(card.get("after_count") or 0), + )) + if card["data_state"] in ("not_collected", "suspect_parse"): + risk_reasons.append(f"{sid}:{card['data_state']}") + out_sheets.append(card) + + if raw_map and not full_pmap: + full_pmap, conflicts = normalize_port_map(raw_map, iface_rules) + elif raw_map: + _, conflicts = normalize_port_map(raw_map, iface_rules) + sources: dict[str, list[str]] = {} + for k, v in full_pmap.items(): + sources.setdefault(v, []).append(k) + n_to_1 = [{"target": t, "sources": sorted(src)} for t, src in sorted(sources.items()) if len(src) > 1] + unused = sorted(k for k in full_pmap if k not in used_keys) + targets_missing = sorted(v for v in set(full_pmap.values()) if after_all and not _present(v, after_all)) + + judged = totals["mapped"] + totals["suspect"] + coverage = { + "mapped": totals["mapped"], "suspect_unmapped": totals["suspect"], + "identity_same_name": totals["same_name"], + "rate": round(totals["mapped"] / judged * 100, 1) if judged else None, + } + mapping = { + "pairs": len(full_pmap), "conflicts": conflicts[:LIST_CAP], + "unused_map_keys": unused[:LIST_CAP], "unused_map_keys_total": len(unused), + "map_targets_missing_on_after": targets_missing[:LIST_CAP], + "map_targets_missing_on_after_total": len(targets_missing), + "n_to_1_targets": n_to_1[:LIST_CAP], + } + if totals["suspect"]: + risk_reasons.append("suspect_unmapped") + if unused: + risk_reasons.append("unused_map_keys") + if targets_missing: + risk_reasons.append("map_targets_missing_on_after") + if conflicts: + risk_reasons.append("map_conflicts") + return { + "verdict": "risk" if risk_reasons else "ready", + "risk_reasons": sorted(set(risk_reasons)), + "coverage": coverage, + "mapping": mapping, + "sheets": out_sheets, + "command_coverage": command_coverage(db, before_batch_id=before_batch_id, after_batch_id=after_batch_id), + } diff --git a/netx_api/biz_state/compare_quality.py b/netx_api/biz_state/compare_quality.py new file mode 100644 index 0000000..30a3551 --- /dev/null +++ b/netx_api/biz_state/compare_quality.py @@ -0,0 +1,198 @@ +"""Collection quality of a compare side: is a missing row a business loss or a data problem? + +Compare results are only as good as the two batches behind them. This module +reads *receipts* (``BizStateBatchCommand``: counts and status, never ``raw_text``) +to tell apart: + +- ``not_collected`` – a side never ran a command (or the slice) for the sheet +- ``suspect_parse`` – the device printed lines but the parser produced no rows +- ``no_data`` – both sides ran fine and are really empty +- ``ok`` – ordinary comparison +""" + +from __future__ import annotations + +from typing import Any, Mapping, Sequence + +from sqlalchemy.orm import Session + +from ..models import BizStateBatchCommand +from .compare_rules import scalar_text + +# More than this many CLI lines with zero parsed rows is not "an empty table". +SUSPECT_RAW_LINES = 5 +DATA_STATES = ("ok", "no_data", "suspect_parse", "not_collected") +UNJUDGED_STATES = ("no_data", "not_collected") + + +def _has_receipts(db: Session, batch_id: str) -> bool: + return bool(batch_id) and db.query(BizStateBatchCommand.id).filter( + BizStateBatchCommand.batch_id == batch_id).limit(1).first() is not None + + +def side_receipts(db: Session, batch_id: str, metric_id: str) -> list[dict[str, Any]]: + if not batch_id or not metric_id: + return [] + rows = ( + db.query( + BizStateBatchCommand.profile_id, BizStateBatchCommand.parse_status, + BizStateBatchCommand.row_count, BizStateBatchCommand.raw_line_count, + BizStateBatchCommand.declared_total, + ) + .filter(BizStateBatchCommand.batch_id == batch_id, BizStateBatchCommand.metric_id == metric_id) + .all() + ) + return [ + { + "profile_id": str(r[0] or ""), "parse_status": str(r[1] or ""), + "row_count": int(r[2] or 0), "raw_line_count": int(r[3] or 0), + "declared_total": int(r[4] or 0), + } + for r in rows + ] + + +def collect_quality(receipts: Sequence[Mapping[str, Any]]) -> dict[str, Any]: + statuses: dict[str, int] = {} + for r in receipts: + key = str(r.get("parse_status") or "unknown") + statuses[key] = statuses.get(key, 0) + 1 + return { + "commands": len(receipts), + "parse_status": statuses, + "row_count": sum(int(r.get("row_count") or 0) for r in receipts), + "raw_line_count": sum(int(r.get("raw_line_count") or 0) for r in receipts), + "declared_total": sum(int(r.get("declared_total") or 0) for r in receipts), + "profile_ids": sorted({str(r.get("profile_id") or "") for r in receipts if r.get("profile_id")}), + } + + +def sheet_slice(sheet: Mapping[str, Any]) -> dict[str, str]: + """``{field: value}`` the sheet pins with top-level ``eq`` row filters.""" + out: dict[str, str] = {} + for f in sheet.get("row_filters") or []: + if not isinstance(f, Mapping) or "any" in f or "all" in f: + continue + if str(f.get("op") or "eq").strip().lower() not in ("eq", "==", "ci_eq"): + continue + field = str(f.get("field") or "").strip() + value = scalar_text(f.get("value")).strip().lower() + if field and value: + out[field] = value + return out + + +def covers_slice( + receipts: Sequence[Mapping[str, Any]], wanted: Mapping[str, str], *, profile_slice: Any = None, +) -> bool: + """True when at least one successful command could have produced ``wanted``. + + A command without a declared slice (or unknown profile) covers everything. + """ + if profile_slice is None: + from .profiles import get_profile + + def profile_slice(pid: str) -> dict[str, str]: # noqa: F811 + prof = get_profile(pid) if pid else None + return dict(prof.slice) if prof is not None and prof.slice else {} + + for r in receipts: + if str(r.get("parse_status") or "") in ("failed", "skipped_custom"): + continue + declared = profile_slice(str(r.get("profile_id") or "")) + if all( + f not in declared or str(declared[f]).strip().lower() == v + for f, v in wanted.items() + ): + return True + return False + + +def classify_sheet( + *, + before: Sequence[Mapping[str, Any]] | None, + after: Sequence[Mapping[str, Any]] | None, + before_rows: int, + after_rows: int, + wanted_slice: Mapping[str, str] | None = None, +) -> tuple[str, list[str]]: + """``(data_state, reasons)`` for one sheet; reasons name the offending side. + + ``None`` receipts mean the batch carries no receipts at all (imported or + legacy data): that side cannot be judged and is trusted as-is. + """ + wanted = dict(wanted_slice or {}) + reasons: list[str] = [] + for side, receipts in (("before", before), ("after", after)): + if receipts is None: + continue + if not receipts: + reasons.append(f"{side}:no_command") + elif wanted and not covers_slice(receipts, wanted): + reasons.append(f"{side}:slice_not_collected") + if reasons: + return "not_collected", reasons + for side, receipts in (("before", before), ("after", after)): + if receipts is None: + continue + q = collect_quality(receipts) + if q["row_count"] == 0 and q["raw_line_count"] > SUSPECT_RAW_LINES: + reasons.append(f"{side}:parsed_0_of_{q['raw_line_count']}_lines") + elif q["commands"] and q["parse_status"].get("failed", 0) == q["commands"]: + reasons.append(f"{side}:all_commands_failed") + if reasons: + return "suspect_parse", reasons + if before_rows == 0 and after_rows == 0: + return "no_data", ["both_sides_empty"] + return "ok", [] + + +def assess_sheet( + db: Session, sheet: Mapping[str, Any], *, before_batch_id: str, after_batch_id: str, + before_rows: int, after_rows: int, +) -> dict[str, Any]: + """Quality block stored on a sheet summary.""" + metric_id = str(sheet.get("metric_id") or "") + before = side_receipts(db, before_batch_id, metric_id) if _has_receipts(db, before_batch_id) else None + after = side_receipts(db, after_batch_id, metric_id) if _has_receipts(db, after_batch_id) else None + state, reasons = classify_sheet( + before=before, after=after, before_rows=before_rows, after_rows=after_rows, + wanted_slice=sheet_slice(sheet), + ) + return { + "data_state": state, + "data_state_reasons": reasons, + "collect_quality": { + "before": collect_quality(before or []), "after": collect_quality(after or []), + "unknown_sides": [n for n, r in (("before", before), ("after", after)) if r is None], + }, + } + + +def command_coverage( + db: Session, *, before_batch_id: str, after_batch_id: str, +) -> dict[str, Any]: + """Collected "metric{slice}" labels per side and their difference. + + Profile ids differ per vendor (``zte.x`` vs ``huawei.x``), so the comparison + is on what a command *produces*: its metric and slice. + """ + from .profiles import get_profile + + def labels(batch_id: str) -> set[str]: + if not batch_id: + return set() + rows = db.query(BizStateBatchCommand.metric_id, BizStateBatchCommand.profile_id).filter( + BizStateBatchCommand.batch_id == batch_id, BizStateBatchCommand.metric_id != "", + ).distinct().all() + out: set[str] = set() + for metric_id, profile_id in rows: + prof = get_profile(str(profile_id or "")) if profile_id else None + sl = dict(prof.slice) if prof is not None and prof.slice else {} + suffix = ",".join(f"{k}={v}" for k, v in sorted(sl.items())) + out.add(f"{metric_id}{{{suffix}}}" if suffix else str(metric_id)) + return out + + b, a = labels(before_batch_id), labels(after_batch_id) + return {"before": sorted(b), "after": sorted(a), + "only_before": sorted(b - a), "only_after": sorted(a - b)} diff --git a/netx_api/biz_state/compare_rules.py b/netx_api/biz_state/compare_rules.py index 269a32a..928fcdb 100644 --- a/netx_api/biz_state/compare_rules.py +++ b/netx_api/biz_state/compare_rules.py @@ -137,6 +137,8 @@ def field_rule_map(rules: Sequence[Mapping[str, Any]] | None) -> dict[str, dict[ name = str(raw.get("field") or "").strip() if not name: continue + if str(raw.get("side") or "").strip(): + continue # side-scoped rules feed the identity pipeline, not the compare out[name] = dict(raw) return out @@ -210,6 +212,50 @@ def _parse_float(text: str) -> float | None: return None +DIRECTIONAL_MODES = ("not_worse", "numeric_min", "numeric_max") + + +def _rank_index(value: str, rank: Sequence[Any]) -> int | None: + wanted = value.strip().lower() + for i, item in enumerate(rank or []): + if scalar_text(item).strip().lower() == wanted: + return i + return None + + +def _tolerance(rule: Mapping[str, Any]) -> float: + try: + return float(rule.get("tolerance") or 0) + except (TypeError, ValueError): + return 0.0 + + +def _directional_verdict(bv: str, av: str, rule: Mapping[str, Any]) -> tuple[bool, str]: + """``(equal_or_better, reason)`` for not_worse / numeric_min / numeric_max.""" + mode = str(rule.get("compare") or "").strip().lower() + if mode == "not_worse": + rank = rule.get("rank") or [] + bi, ai = _rank_index(bv, rank), _rank_index(av, rank) + if bi is None or ai is None: + return bv == av, "neq" + return ai <= bi, (f"worse: {bv}→{av}" if ai > bi else "") + bn, an = _parse_float(bv), _parse_float(av) + if bn is None or an is None: + return bv == av, "neq" + tol = _tolerance(rule) + if mode == "numeric_min": + floor = bn * (1 - tol / 100.0) + if an >= floor: + return True, "" + drop = (bn - an) / abs(bn) * 100.0 if bn else 0.0 + return False, f"drop {drop:.0f}% > {tol:g}%" + ceil = bn * (1 + tol / 100.0) + if an <= ceil: + return True, "" + rise = (an - bn) / abs(bn) * 100.0 if bn else 0.0 + return False, f"rise {rise:.0f}% > {tol:g}%" + + def explain_diff( before: Any, after: Any, @@ -222,6 +268,8 @@ def explain_diff( bv = normalize_value(before, norm) av = normalize_value(after, norm) mode = str(rule.get("compare") or "eq").strip().lower() or "eq" + if mode in DIRECTIONAL_MODES: + return _directional_verdict(bv, av, rule)[1] if mode in ("percent", "pct", "rel"): bn = _parse_float(bv) an = _parse_float(av) @@ -262,6 +310,8 @@ def values_equal( mode = str(rule.get("compare") or "eq").strip().lower() or "eq" if mode in ("ignore", "skip", "off"): return True + if mode in DIRECTIONAL_MODES: + return _directional_verdict(bv, av, rule)[0] if mode in ("numeric", "number", "int", "float", "percent", "pct", "rel"): bn = _parse_float(bv) an = _parse_float(av) diff --git a/netx_api/biz_state/compare_service.py b/netx_api/biz_state/compare_service.py index 286c29b..d54cb35 100644 --- a/netx_api/biz_state/compare_service.py +++ b/netx_api/biz_state/compare_service.py @@ -22,6 +22,8 @@ from ..models import ( BizCompareJob, BizCompareRun, BizCompareTemplate, + BizMigrationProject, + BizMonitorTemplate, BizPortMapping, BizPortMappingRow, BizStateBatch, @@ -34,6 +36,7 @@ from .compare_engine import apply_port_map, compare_rows, mapping_stats, row_key from .compare_rules import ( ROW_FILTER_PRESETS, apply_row_filters, + scalar_text, arp_dynamic_row_filters, effective_compare_fields, effective_display_fields, @@ -43,12 +46,19 @@ from .iface_normalize import ( apply_iface_normalize_rows, default_zte_iface_normalize_rules, normalize_iface_rules, + normalize_port_map, ) +from .compare_quality import UNJUDGED_STATES, assess_sheet +from .identity_pipeline import Pipeline, build_pipeline, legacy_pipeline from .profiles import metric_field_map from .compare_validation import validate_template_body _log = logging.getLogger("netx.biz_state.compare") +# v1: port-map keys/values must be written post-normalization. +# v2: port-map keys/values are normalized with the template iface rules (A1). +CONFIG_SNAPSHOT_VERSION = 2 + # Per-job mutex so auto + manual run_compare do not dual-write the same job. _auto_compare_locks: dict[str, threading.Lock] = {} _auto_compare_locks_guard = threading.Lock() @@ -426,11 +436,25 @@ def _metric_rows_by_ids(db: Session, ids: list[str]) -> dict[str, dict[str, Any] return out +def _attach_raw_key_before(items: list[dict[str, Any]], key_fields: list[str]) -> list[dict[str, Any]]: + """``raw_key_before``: the before-side key as printed, only where it differs from the match key.""" + for it in items: + before = it.get("before") + if not before or it.get("raw_key_before"): + continue + shown = it.get("key") or {} + raw = {f: scalar_text(before.get(f)).strip() for f in key_fields} + if any(raw[f] != scalar_text(shown.get(f)).strip() for f in key_fields if f in shown): + it["raw_key_before"] = raw + return items + + def _hydrate_diff_rows( db: Session, items: list[dict[str, Any]], *, run: BizCompareRun | None = None, sheet: dict[str, Any] | None = None, ) -> list[dict[str, Any]]: """Fill empty before/after from metric tables when row_ids are present.""" + key_fields = list((sheet or {}).get("key_fields") or []) need: list[str] = [] for it in items: if not it.get("before") and it.get("before_row_id"): @@ -438,27 +462,19 @@ def _hydrate_diff_rows( if not it.get("after") and it.get("after_row_id"): need.append(str(it["after_row_id"])) if not need: - return items + return _attach_raw_key_before(items, key_fields) by_id = _metric_rows_by_ids(db, need) - pmap, norm_rules = _run_transform_config(db, run) if run else ({}, []) - iface_fields = list((sheet or {}).get("iface_fields") or []) + pipeline = _run_pipeline(db, run, sheet or {}) for it in items: brid = str(it.get("before_row_id") or "") arid = str(it.get("after_row_id") or "") if not it.get("before") and brid and brid in by_id: it["before"] = by_id[brid] if not it.get("after") and arid and arid in by_id: - it["after"] = apply_iface_normalize_rows( - [by_id[arid]], iface_fields=iface_fields, rules=norm_rules, - )[0] + it["after"] = pipeline.normalize(by_id[arid], "after") if not it.get("mapped_before") and it.get("before"): - normalized = apply_iface_normalize_rows( - [it["before"]], iface_fields=iface_fields, rules=norm_rules, - )[0] - it["mapped_before"] = apply_port_map( - normalized, iface_fields=iface_fields, port_map=pmap, - ) - return items + it["mapped_before"] = pipeline.transform(it["before"], "before") + return _attach_raw_key_before(items, key_fields) def _diff_row_out(r: BizCompareDiff) -> dict[str, Any]: @@ -653,18 +669,41 @@ def _normalize_row_filters(raw: Any) -> list[dict[str, Any]]: return out +# Optional sheet-level switches that must survive normalization. +_SHEET_OPTION_VALUES: dict[str, tuple[str, ...]] = { + "unmapped_port_policy": ("by_name", "ignore_port"), + "duplicate_pairing": ("sorted", "ordered"), +} + + +def _sheet_options(raw: dict[str, Any]) -> dict[str, Any]: + out: dict[str, Any] = {} + for key, allowed in _SHEET_OPTION_VALUES.items(): + val = str(raw.get(key) or "").strip().lower() + if val in allowed: + out[key] = val + if raw.get("builtin_vendor_canonical") is False: + out["builtin_vendor_canonical"] = False + if raw.get("scope_by_port_map") is not None: + out["scope_by_port_map"] = bool(raw.get("scope_by_port_map")) + return out + + def _normalize_field_rules(raw: Any) -> list[dict[str, Any]]: if not isinstance(raw, list): return [] out: list[dict[str, Any]] = [] - seen: set[str] = set() + seen: set[tuple[str, str]] = set() for item in raw: if not isinstance(item, dict): continue name = str(item.get("field") or "").strip() - if not name or name in seen: + side = str(item.get("side") or "").strip().lower() + side = side if side in ("before", "after") else "" + # A field may carry one rule per side (e.g. a different value_map each). + if not name or (name, side) in seen: continue - seen.add(name) + seen.add((name, side)) rule: dict[str, Any] = {"field": name} compare = str(item.get("compare") or "").strip().lower() if compare: @@ -680,6 +719,16 @@ def _normalize_field_rules(raw: Any) -> list[dict[str, Any]]: rule["tolerance"] = float(item.get("tolerance")) except (TypeError, ValueError): pass + vmap = item.get("value_map") + if isinstance(vmap, dict) and vmap: + rule["value_map"] = {str(k): str(v) for k, v in vmap.items()} + if side: + rule["side"] = side + if isinstance(item.get("apply_to_key"), bool): + rule["apply_to_key"] = item["apply_to_key"] + rank = item.get("rank") + if isinstance(rank, list) and rank: + rule["rank"] = [str(x) for x in rank] # Drop empty rules (only field name) if len(rule) > 1: out.append(rule) @@ -709,6 +758,7 @@ def _sheet_def( row_filters: list[dict[str, Any]] | None = None, field_rules: list[dict[str, Any]] | None = None, ignore_port_changes: bool | None = None, + options: dict[str, Any] | None = None, ) -> dict[str, Any]: mid = str(metric_id or "").strip() sid = str(sheet_id or "").strip() or mid @@ -749,6 +799,7 @@ def _sheet_def( } if ignore_port_changes is not None: sheet["ignore_port_changes"] = bool(ignore_port_changes) + sheet.update(options or {}) return sheet @@ -946,6 +997,49 @@ def _builtin_status_needs_packaged_upgrade(existing: list[dict[str, Any]]) -> bo return len(packaged_keys - have_keys) >= 3 +# A7: (metric_id, kind, field, old packaged value, new value). A stored sheet is +# only touched when its value still equals the *old* packaged one, so anything +# an operator edited survives the upgrade. +_INTERFACE_DETAIL_OLD_COMPARE = [ + "admin", "input_bps", "output_bps", "port_status", "in_util", "out_util", + "ip_mtu", "mtu", "mpls_mtu", "ipv6_mtu", "port_media", "negotiation", +] +_BUILTIN_STATUS_VALUE_UPGRADES: tuple[tuple[str, str, str, Any, Any], ...] = ( + ("interface_detail", "rule", "output_bps", + {"compare": "numeric", "tolerance": 5}, {"compare": "percent", "tolerance": 5}), + ("interface_detail", "list", "compare_fields", _INTERFACE_DETAIL_OLD_COMPARE, + [f for f in _INTERFACE_DETAIL_OLD_COMPARE if f not in ("input_bps", "output_bps", "in_util", "out_util")]), + ("optical_brief", "rule", "rx_power", + {"compare": "percent", "tolerance": 5}, {"compare": "numeric", "tolerance": 2}), + ("l2vpn_mac", "list", "iface_fields", [], ["ac_port"]), + # ARP: filter on the canonical ``entry_type`` (every vendor parser emits it); the + # age-timer clause stays only as a fallback for rows parsed before entry_type existed. + ("arp", "list", "row_filters", [{"op": "age_timer", "field": "age", "value": ""}], arp_dynamic_row_filters()), +) + + +def _upgrade_builtin_status_values(sheet: dict[str, Any]) -> tuple[dict[str, Any], bool]: + """Apply ``_BUILTIN_STATUS_VALUE_UPGRADES`` to one stored sheet.""" + mid = str(sheet.get("metric_id") or "") + out = deepcopy(sheet) + changed = False + for metric_id, kind, field, old, new in _BUILTIN_STATUS_VALUE_UPGRADES: + if mid != metric_id: + continue + if kind == "list": + if list(out.get(field) or []) == old: + out[field] = list(new) + changed = True + continue + for rule in out.get("field_rules") or []: + if isinstance(rule, dict) and rule.get("field") == field and all( + rule.get(k) == v for k, v in old.items() + ): + rule.update(new) + changed = True + return out, changed + + def _default_zte_config_sheets() -> list[dict[str, Any]]: """Config-intent metrics for cutover / intent-vs-intent compare.""" return [ @@ -1009,6 +1103,7 @@ def _normalize_sheet(raw: Any) -> dict[str, Any] | None: row_filters=_normalize_row_filters(raw.get("row_filters")), field_rules=rules, ignore_port_changes=ignore_ports, + options=_sheet_options(raw), ) @@ -1252,6 +1347,8 @@ def ensure_default_zte_status_template(db: Session) -> BizCompareTemplate: if not cur.get("field_rules") and src and src.get("field_rules"): cur["field_rules"] = list(src["field_rules"]) changed = True + cur, value_changed = _upgrade_builtin_status_values(cur) + changed = changed or value_changed upgraded.append(_normalize_sheet(cur) or cur) have_mids = {str(s.get("metric_id") or "") for s in upgraded} for s in sheets: @@ -1389,7 +1486,16 @@ def list_templates(db: Session) -> list[dict[str, Any]]: upgrade_builtin_split_sheets(db) rows = db.query(BizCompareTemplate).order_by(BizCompareTemplate.name.asc()).all() usage = dict(db.query(BizCompareJob.template_id, func.count(BizCompareJob.id)).group_by(BizCompareJob.template_id).all()) - return [{**_template_out(t), "job_count": usage.get(t.id, 0)} for t in rows] + monitor_usage = dict( + db.query(BizMonitorTemplate.compare_template_id, func.count(BizMonitorTemplate.id)) + .filter(BizMonitorTemplate.compare_template_id != "") + .group_by(BizMonitorTemplate.compare_template_id).all() + ) + return [ + {**_template_out(t), "job_count": usage.get(t.id, 0), + "monitor_template_count": monitor_usage.get(t.id, 0)} + for t in rows + ] def list_metric_schemas() -> list[dict[str, Any]]: @@ -1522,8 +1628,19 @@ def delete_template(db: Session, template_id: str) -> None: t = db.get(BizCompareTemplate, template_id) if not t: raise HTTPException(status_code=404, detail="template_not_found") - if db.query(BizCompareJob.id).filter(BizCompareJob.template_id == template_id).first(): - raise HTTPException(status_code=409, detail="template_in_use") + jobs = [n or i for i, n in db.query(BizCompareJob.id, BizCompareJob.name) + .filter(BizCompareJob.template_id == template_id).all()] + monitors = [n or i for i, n in db.query(BizMonitorTemplate.id, BizMonitorTemplate.name) + .filter(BizMonitorTemplate.compare_template_id == template_id).all()] + if jobs or monitors: + raise HTTPException(status_code=409, detail={ + "code": "template_in_use", + "message": "template_in_use: " + "; ".join(filter(None, [ + f"jobs={', '.join(jobs)}" if jobs else "", + f"monitor_templates={', '.join(monitors)}" if monitors else "", + ])), + "jobs": jobs, "monitor_templates": monitors, + }) db.delete(t) db.commit() @@ -1544,9 +1661,30 @@ def _mapping_out(db: Session, m: BizPortMapping) -> dict[str, Any]: } +def _mapping_references(db: Session, mapping_id: str) -> tuple[list[str], list[str]]: + """(compare job names, cutover project names) that reference this mapping.""" + jobs = [n or i for i, n in db.query(BizCompareJob.id, BizCompareJob.name) + .filter(BizCompareJob.mapping_id == mapping_id).all()] + projects = [n or i for i, n in db.query(BizMigrationProject.id, BizMigrationProject.name) + .filter(BizMigrationProject.mapping_id == mapping_id).all()] + return jobs, projects + + def list_mappings(db: Session) -> list[dict[str, Any]]: rows = db.query(BizPortMapping).order_by(BizPortMapping.name.asc()).all() - return [_mapping_out(db, m) for m in rows] + job_counts = dict( + db.query(BizCompareJob.mapping_id, func.count(BizCompareJob.id)) + .filter(BizCompareJob.mapping_id != "").group_by(BizCompareJob.mapping_id).all() + ) + project_counts = dict( + db.query(BizMigrationProject.mapping_id, func.count(BizMigrationProject.id)) + .filter(BizMigrationProject.mapping_id != "").group_by(BizMigrationProject.mapping_id).all() + ) + return [ + {**_mapping_out(db, m), "job_count": job_counts.get(m.id, 0), + "project_count": project_counts.get(m.id, 0)} + for m in rows + ] def create_mapping(db: Session, body: dict[str, Any]) -> dict[str, Any]: @@ -1604,6 +1742,16 @@ def delete_mapping(db: Session, mapping_id: str) -> None: m = db.get(BizPortMapping, mapping_id) if not m: raise HTTPException(status_code=404, detail="mapping_not_found") + jobs, projects = _mapping_references(db, mapping_id) + if jobs or projects: + raise HTTPException(status_code=409, detail={ + "code": "mapping_in_use", + "message": "mapping_in_use: " + "; ".join(filter(None, [ + f"jobs={', '.join(jobs)}" if jobs else "", + f"projects={', '.join(projects)}" if projects else "", + ])), + "jobs": jobs, "projects": projects, + }) db.query(BizPortMappingRow).filter(BizPortMappingRow.mapping_id == mapping_id).delete() db.delete(m) db.commit() @@ -1822,11 +1970,11 @@ def validate_mapping( sheets = template_metrics(tpl) if not sheets: raise HTTPException(status_code=400, detail="template_has_no_metrics") - pmap = _port_map_dict(db, mapping_id) # Validate against first sheet that has iface fields (or first sheet) target = next((s for s in sheets if s.get("iface_fields")), sheets[0]) iface_fields = list(target.get("iface_fields") or []) norm = template_iface_normalize(tpl) + pmap, _ = normalize_port_map(_port_map_dict(db, mapping_id), norm) before = apply_iface_normalize_rows( _load_metric_rows(db, batch_id=before_batch_id, metric_id=target["metric_id"]), iface_fields=iface_fields, @@ -1845,6 +1993,66 @@ def validate_mapping( ) +def _mapping_rows_to_dict(rows: list[Any]) -> tuple[dict[str, str], list[dict[str, str]]]: + """Editor rows → raw map; a repeated before port is reported, first one wins.""" + out: dict[str, str] = {} + dups: list[dict[str, str]] = [] + for r in rows or []: + raw = r if isinstance(r, dict) else dict(r) + before = str(raw.get("before_if") or "").strip() + after = str(raw.get("after_if") or "").strip() + if not before or not after: + continue + if before in out: + dups.append({"before_if": before, "kept_to": out[before], "dropped_to": after}) + continue + out[before] = after + return out, dups + + +def preflight_compare(db: Session, body: dict[str, Any]) -> dict[str, Any]: + """Mapping coverage + data-quality report for a template and two batches.""" + from .compare_preflight import preflight + + ensure_default_templates(db) + template_id = str(body.get("template_id") or "") + tpl = db.get(BizCompareTemplate, template_id) if template_id else ensure_default_cutover_template(db) + if not tpl: + raise HTTPException(status_code=404, detail="template_not_found") + before_id = str(body.get("before_batch_id") or "") + after_id = str(body.get("after_batch_id") or "") + if not db.get(BizStateBatch, before_id) or not db.get(BizStateBatch, after_id): + raise HTTPException(status_code=404, detail="batch_not_found") + sheets = _filter_enabled_sheets(template_metrics(tpl), _str_list(body.get("enabled_sheet_ids"))) + if not sheets: + raise HTTPException(status_code=400, detail="no_enabled_sheets") + dups: list[dict[str, str]] = [] + if body.get("mapping_rows") is not None: + raw_map, dups = _mapping_rows_to_dict(list(body.get("mapping_rows") or [])) + else: + raw_map = _port_map_dict(db, str(body.get("mapping_id") or "")) + cache: dict[tuple[str, str], list[dict[str, Any]]] = {} + + def load(side: str, sheet: dict[str, Any]) -> list[dict[str, Any]]: + batch_id = before_id if side == "before" else after_id + key = (batch_id, str(sheet["metric_id"])) + if key not in cache: + cache[key] = _load_metric_rows(db, batch_id=batch_id, metric_id=key[1]) + return cache[key] + + report = preflight( + db, sheets=sheets, iface_rules=template_iface_normalize(tpl), raw_port_map=raw_map, + before_batch_id=before_id, after_batch_id=after_id, + before_vendor=_batch_vendor_key(db, before_id), after_vendor=_batch_vendor_key(db, after_id), + load_rows=load, + ) + if dups: + report["mapping"]["duplicate_before"] = dups + report["risk_reasons"] = sorted({*report["risk_reasons"], "duplicate_before"}) + report["verdict"] = "risk" + return report + + def _job_out(j: BizCompareJob) -> dict[str, Any]: return { "id": j.id, @@ -2040,14 +2248,23 @@ def _sheet_result_envelope( } +def _sheet_quality( + db: Session, sheet: dict[str, Any], before_batch_id: str, after_batch_id: str, + summary: dict[str, Any], +) -> dict[str, Any]: + return assess_sheet( + db, sheet, before_batch_id=before_batch_id, after_batch_id=after_batch_id, + before_rows=int(summary.get("before_count") or 0), after_rows=int(summary.get("after_count") or 0), + ) + + def _run_sheet( db: Session, *, sheet: dict[str, Any], before_batch_id: str, after_batch_id: str, - port_map: dict[str, str], - iface_normalize_rules: list[dict[str, str]] | None = None, + pipeline: Pipeline, store_unchanged: str = "auto", on_load_progress: Callable[..., None] | None = None, ) -> dict[str, Any]: @@ -2083,12 +2300,7 @@ def _run_sheet( # PostgreSQL path: pushdown-safe sheets join in-DB (BGP-scale). from .compare_sql import SqlCompareSkip, run_sql_sheet_compare, sql_compare_skip_reason - skip_reason = sql_compare_skip_reason( - db, - sheet, - port_map=port_map, - iface_normalize_rules=iface_normalize_rules, - ) + skip_reason = sql_compare_skip_reason(db, sheet, pipeline=pipeline) if not skip_reason: try: result = run_sql_sheet_compare( @@ -2098,6 +2310,7 @@ def _run_sheet( after_batch_id=after_batch_id, store_unchanged=store_unchanged, on_progress=_emit_load, + duplicate_pairing=pipeline.duplicate_pairing, ) summary = dict(result["summary"]) # Engine already sets raw counts / policy; keep keys stable @@ -2107,6 +2320,7 @@ def _run_sheet( before_n=int(summary.get("before_count") or 0), after_n=int(summary.get("after_count") or 0), ) + summary.update(_sheet_quality(db, sheet, before_batch_id, after_batch_id, summary)) return _sheet_result_envelope( sheet, key_fields=key_fields, @@ -2161,39 +2375,38 @@ def _run_sheet( def _after_chunk(n: int) -> None: _emit_load("after", n, engine="python", note=skip_reason or "python", phase="loading") + # Only filters that read untransformed fields may run in SQL on the raw rows; + # the pipeline applies every filter (post-normalize for v2) in ``prepare``. before_raw = _load_metric_rows( db, batch_id=before_batch_id, metric_id=mid, on_chunk=_before_chunk, - row_filters=row_filters, + row_filters=pipeline.pushdown_filters("before"), ) after_raw = _load_metric_rows( db, batch_id=after_batch_id, metric_id=mid, on_chunk=_after_chunk, - row_filters=row_filters, + row_filters=pipeline.pushdown_filters("after"), ) - # Filters already applied in load when pushdown-safe; keep apply for safety - before_rows = apply_row_filters(before_raw, row_filters) if row_filters else before_raw - after_rows = apply_row_filters(after_raw, row_filters) if row_filters else after_raw + before_pairs = pipeline.prepare(before_raw, "before") + after_pairs = pipeline.prepare(after_raw, "after") policy = resolve_unchanged_policy( - store_unchanged, before_n=len(before_rows), after_n=len(after_rows) + store_unchanged, before_n=len(before_pairs), after_n=len(after_pairs) ) result = compare_rows( - before_rows=before_rows, - after_rows=after_rows, + before_rows=[], after_rows=[], key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields, - port_map=port_map, field_rules=field_rules, - iface_normalize_rules=iface_normalize_rules, - ignore_port_changes=ignore_ports, include_unchanged=bool(policy["include"]), unchanged_limit=policy.get("limit"), compact_unchanged=bool(policy.get("compact")), + pipeline=pipeline, + prepared=(before_pairs, after_pairs), ) summary = dict(result["summary"]) summary["before_raw_count"] = len(before_raw) @@ -2201,6 +2414,14 @@ def _run_sheet( summary["row_filters"] = len(row_filters) summary["unchanged_policy"] = policy summary.setdefault("engine", "python") + if pipeline.port_map and pipeline.iface_fields: + from .compare_preflight import sheet_mapping_coverage + + coverage = sheet_mapping_coverage(pipeline, before_pairs, after_pairs) + coverage.pop("_after_ifaces", None) + coverage.pop("_used_keys", None) + summary["mapping_coverage"] = coverage + summary.update(_sheet_quality(db, sheet, before_batch_id, after_batch_id, summary)) return _sheet_result_envelope( sheet, key_fields=key_fields, @@ -2227,6 +2448,8 @@ def _validate_compare_job( tpl = db.get(BizCompareTemplate, j.template_id) if not tpl: raise HTTPException(status_code=404, detail="template_not_found") + if j.mapping_id and not db.get(BizPortMapping, j.mapping_id): + raise HTTPException(status_code=400, detail="mapping_missing") before_batch_id = str(j.before_batch_id or "") after_batch_id = str(force_after_batch_id or "").strip() or _resolve_after_batch(db, j) if not before_batch_id or not after_batch_id: @@ -2272,15 +2495,31 @@ def _create_running_run( before_batch_id: str, after_batch_id: str, sheets_cfg: list[dict[str, Any]], + purpose: str = "manual", ) -> BizCompareRun: first_metric = str(sheets_cfg[0].get("metric_id") or "") pending_sheets = [_pending_sheet_meta(s) for s in sheets_cfg] mapping = db.get(BizPortMapping, job.mapping_id) if job.mapping_id else None + vendors = { + "before": _batch_vendor_key(db, before_batch_id), + "after": _batch_vendor_key(db, after_batch_id), + } + raw_port_map = _port_map_dict(db, str(job.mapping_id or "")) + iface_rules = template_iface_normalize(tpl) + identity = { + sheet_key(sh): build_pipeline( + sh, iface_rules=iface_rules, port_map=raw_port_map, + before_vendor=vendors["before"], after_vendor=vendors["after"], + ).describe() + for sh in sheets_cfg + } config_snapshot = deepcopy({ - "version": 1, + "version": CONFIG_SNAPSHOT_VERSION, "template": _template_out(tpl), "sheets": sheets_cfg, - "port_map": _port_map_dict(db, str(job.mapping_id or "")), + "port_map": raw_port_map, + "vendors": vendors, + "identity": identity, "store_unchanged": normalize_store_unchanged(getattr(job, "store_unchanged", None)), "job_name": job.name or "", "before_task_id": job.before_task_id or "", @@ -2297,6 +2536,7 @@ def _create_running_run( metric_id=first_metric, status="running", summary_json={ + "purpose": purpose, "config_snapshot": config_snapshot, "progress": { "phase": "queued", @@ -2329,7 +2569,7 @@ def _create_running_run( def _run_config_snapshot(run: BizCompareRun) -> dict[str, Any] | None: snapshot = (run.summary_json or {}).get("config_snapshot") - if isinstance(snapshot, dict) and snapshot.get("version") == 1: + if isinstance(snapshot, dict) and snapshot.get("version") in (1, 2): return snapshot return None @@ -2339,14 +2579,66 @@ def _run_transform_config( ) -> tuple[dict[str, str], list[dict[str, str]]]: snapshot = _run_config_snapshot(run) if snapshot is not None: - return dict(snapshot.get("port_map") or {}), normalize_iface_rules( + rules = normalize_iface_rules( (snapshot.get("template") or {}).get("iface_normalize_rules") or [], ) + pmap = dict(snapshot.get("port_map") or {}) + if int(snapshot.get("version") or 1) >= 2: + # v2 runs: map keys/values are written as the device prints them. + pmap, _ = normalize_port_map(pmap, rules) + return pmap, rules # Legacy runs did not record these settings; retain their fallback behavior. tpl = db.get(BizCompareTemplate, run.template_id) if run.template_id else None return _port_map_dict(db, str(run.mapping_id or "")), template_iface_normalize(tpl) +def _batch_vendor_key(db: Session, batch_id: str) -> str: + """Vendor key (``huawei``/``zte``/…) for a batch, '' when unknown.""" + from ..topology_lldp import resolve_vendor_key + + batch = db.get(BizStateBatch, batch_id) if batch_id else None + if batch is None: + return "" + task = db.get(BizStateTask, batch.task_id) if batch.task_id else None + key = resolve_vendor_key( + batch.vendor or (task.vendor if task else ""), (task.device_type if task else "") or "", + ) + return "" if key == "generic" else key + + +def _run_pipeline(db: Session, run: BizCompareRun | None, sheet: dict[str, Any]) -> Pipeline: + """The identity pipeline a run used (or would use) for one sheet. + + Snapshot v2 runs get the full pipeline; v1 and snapshot-less runs get the + legacy one so stored results replay exactly as they were produced. + """ + sid = sheet_key(sheet) + snapshot = _run_config_snapshot(run) if run is not None else None + if snapshot is None: + pmap, rules = _run_transform_config(db, run) if run is not None else ({}, []) + return legacy_pipeline( + key_fields=sheet.get("key_fields") or [], iface_fields=sheet.get("iface_fields") or [], + compare_fields=sheet.get("compare_fields") or [], field_rules=sheet.get("field_rules"), + port_map=pmap, iface_normalize_rules=rules, row_filters=sheet.get("row_filters"), + ignore_port_changes=sheet.get("ignore_port_changes"), + ) + authoritative = next( + (sh for sh in snapshot.get("sheets") or [] if isinstance(sh, dict) and sheet_key(sh) == sid), None, + ) + cfg = authoritative or sheet + version = int(snapshot.get("version") or 1) + vendors = snapshot.get("vendors") or {} + return build_pipeline( + cfg, + iface_rules=(snapshot.get("template") or {}).get("iface_normalize_rules") or [], + port_map=dict(snapshot.get("port_map") or {}), + before_vendor=str(vendors.get("before") or ""), + after_vendor=str(vendors.get("after") or ""), + legacy=version < 2, + normalize_map=version >= 2, + ) + + def _set_run_progress( db: Session, run: BizCompareRun, @@ -2458,7 +2750,6 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]: store_mode = normalize_store_unchanged( snapshot.get("store_unchanged") if snapshot is not None else getattr(j, "store_unchanged", None), ) - pmap, norm_rules = _run_transform_config(db, run) unchanged_listed_total = 0 unchanged_truncated_any = False unchanged_compact_any = False @@ -2563,8 +2854,7 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]: sheet=sheet, before_batch_id=before_batch_id, after_batch_id=after_batch_id, - port_map=pmap, - iface_normalize_rules=norm_rules, + pipeline=_run_pipeline(db, run, sheet), store_unchanged=store_mode, on_load_progress=_on_load, ) @@ -2755,6 +3045,7 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]: if snapshot is not None: summary_payload["config_snapshot"] = snapshot run = db.get(BizCompareRun, run_id) or run + summary_payload["purpose"] = (run.summary_json or {}).get("purpose") or "manual" if str(run.status or "") == "cancelled": return get_run(db, run.id) run.status = "success" @@ -2882,7 +3173,9 @@ def cancel_compare_run(db: Session, run_id: str) -> dict[str, Any]: return get_run(db, run_id) -def run_compare(db: Session, job_id: str, *, force_after_batch_id: str = "") -> dict[str, Any]: +def run_compare( + db: Session, job_id: str, *, force_after_batch_id: str = "", purpose: str = "manual", +) -> dict[str, Any]: """Synchronous compare (auto-compare / tests). Blocks the caller until done.""" if _job_has_active_run(db, job_id): raise HTTPException(status_code=409, detail="compare_already_running") @@ -2898,6 +3191,7 @@ def run_compare(db: Session, job_id: str, *, force_after_batch_id: str = "") -> before_batch_id=ctx["before_batch_id"], after_batch_id=ctx["after_batch_id"], sheets_cfg=ctx["sheets_cfg"], + purpose=purpose, ) return _execute_compare_into_run(db, run.id) finally: @@ -2972,7 +3266,9 @@ def _sheet_csv_lines(sheet: dict[str, Any], diffs: Iterable[dict[str, Any]]) -> ) # Non-key columns already ordered Key→Compare→Display by effective_display_fields extra = [f for f in display if f not in key_set] - headers = ["kind", *keys] + # Interface keys also get the name as the device printed it (before side). + raw_cols = [k for k in keys if k in set(sheet.get("iface_fields") or [])] + headers = ["kind", *keys, *[f"{k}__raw_before" for k in raw_cols]] for f in extra: if f in compare_set: headers.append(f"{f}__pre") @@ -2988,6 +3284,9 @@ def _sheet_csv_lines(sheet: dict[str, Any], diffs: Iterable[dict[str, Any]]) -> row = [kind] for k in keys: row.append(key.get(k, pre.get(k, post.get(k, "")))) + raw_before = dict(d.get("raw_key_before") or {}) + for k in raw_cols: + row.append(raw_before.get(k, "")) for f in extra: if f in compare_set: if kind == "added": @@ -3012,6 +3311,18 @@ def _sheet_csv(sheet: dict[str, Any]) -> str: return "\ufeff" + "\n".join(_sheet_csv_lines(sheet, sheet.get("diffs") or [])) + "\n" +def _task_ok(sheets: list[dict[str, Any]], fail_count: int, any_pending: bool) -> bool | None: + """True/False once judged; None when data quality leaves the verdict open.""" + if any_pending: + return False + states = [str((sh.get("summary") or {}).get("data_state") or "ok") for sh in sheets] + if any(st in ("suspect_parse", "not_collected") for st in states): + return None + if states and all(st == "no_data" for st in states): + return None + return fail_count == 0 + + def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> dict[str, Any]: added = int(summary.get("added") or 0) removed = int(summary.get("removed") or 0) @@ -3020,16 +3331,28 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di before_count = int(summary.get("before_count") or 0) after_count = int(summary.get("after_count") or 0) total = added + removed + changed + unchanged + + def _data_state(sh: dict[str, Any]) -> str: + return str((sh.get("summary") or {}).get("data_state") or "ok") + + # Sheets whose side was never collected cannot be judged: their "missing" + # rows are a collection gap, not a business loss, so they stay out of the totals. + dropped = [sh for sh in sheets if _data_state(sh) == "not_collected" + and str(sh.get("status") or "done") not in ("pending", "running", "queued")] + drop_fail = sum(int((sh.get("summary") or {}).get("removed") or 0) + + int((sh.get("summary") or {}).get("changed") or 0) for sh in dropped) + drop_ok = sum(int((sh.get("summary") or {}).get("unchanged") or 0) for sh in dropped) # Compare verdict: fail = missing + mismatch; success = match; added is special - fail_count = removed + changed - success_count = unchanged + fail_count = removed + changed - drop_fail + success_count = unchanged - drop_ok judged = fail_count + success_count matched = changed + unchanged diff_count = fail_count + # Nothing compared at all is "not judged" (None), never a 100% pass. pass_rate = ( - compute_pass_rate(success=success_count, fail=fail_count, empty_as=100.0 if total == 0 else 0.0) + compute_pass_rate(success=success_count, fail=fail_count) if judged - else (100.0 if total == 0 else 0.0) + else (None if total == 0 else 0.0) ) diff_rate = round((fail_count / judged) * 100, 1) if judged else 0.0 @@ -3045,8 +3368,12 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di sj = sf + su status = str(sh.get("status") or "done") pending = status in ("pending", "running", "queued") + data_state = str(ss.get("data_state") or "ok") sheet_cards.append( { + "data_state": data_state, + "data_state_reasons": list(ss.get("data_state_reasons") or []), + "collect_quality": ss.get("collect_quality") or {}, "sheet_id": sheet_key(sh), "title": sheet_title(sh), "metric_id": sh.get("metric_id") or "", @@ -3063,12 +3390,8 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di "diff_count": sf, # Pending sheets must not look like "100% pass" "pass_rate": None - if pending - else ( - compute_pass_rate(success=su, fail=sf, empty_as=100.0 if st == 0 else 0.0) - if sj - else (100.0 if st == 0 else 0.0) - ), + if pending or data_state in UNJUDGED_STATES + else (compute_pass_rate(success=su, fail=sf) if sj else (None if st == 0 else 0.0)), } ) @@ -3105,7 +3428,9 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di "diff_count": diff_count, "pass_rate": None if any_pending else pass_rate, "diff_rate": diff_rate, - "ok": False if any_pending else fail_count == 0, + "ok": _task_ok(sheets, fail_count, any_pending), + "unjudged_sheets": sum(1 for sh in sheets if _data_state(sh) in UNJUDGED_STATES), + "suspect_sheets": sum(1 for sh in sheets if _data_state(sh) == "suspect_parse"), "sheet_cards": sheet_cards, "top_changed_fields": top_fields, "duration_ms": int(summary.get("duration_ms") or 0), @@ -3449,21 +3774,25 @@ def _load_metric_rows_for_search( def _load_search_key_groups( db: Session, *, batch_id: str, metric_id: str, keys: set[tuple[str, ...]], - match_keys: list[str], iface_fields: list[str], row_filters: list[dict[str, Any]], - norm_rules: list[dict[str, str]], port_map: dict[str, str], + match_keys: list[str], pipeline: Pipeline, side: str, ) -> tuple[list[dict[str, Any]], set[tuple[str, ...]]]: """Complete candidate key groups so a one-sided search cannot change verdicts. - Interface transforms run in Python. Other identity fields narrow the SQL - query. If the load budget is exceeded, omit entire groups rather than compare - partial groups and manufacture additions/deletions. + Identity rules (normalize / value map / port map) run in the pipeline, in + Python. Key fields they never touch narrow the SQL query. If the load budget + is exceeded, omit entire groups rather than compare partial groups and + manufacture additions/deletions. """ from ..models import BizStateMetricRow model = BizStateLldpNeighbor if metric_id == "lldp_neighbor" else BizStateMetricRow - invariant = [f for f in match_keys if f not in iface_fields] + # Narrowing compares raw SQL text, so only fields no rule rewrites may be used, + # and only when key tuples line up 1:1 with ``match_keys`` (no mapped/unmapped marker). + invariant = [] if pipeline.ignores_ports_by_policy() else [ + f for f in match_keys if f not in pipeline._transformed # noqa: SLF001 + ] positions = [match_keys.index(f) for f in invariant] - coarse = sorted({tuple(k[i] for i in positions) for k in keys}) + coarse = sorted({tuple(k[i] for i in positions) for k in keys}) if invariant else [()] out: list[dict[str, Any]] = [] complete: set[tuple[str, ...]] = set() # Bound memory even when a candidate identifies a very large duplicate group. @@ -3471,7 +3800,7 @@ def _load_search_key_groups( for start in range(0, len(coarse), 100): group = coarse[start:start + 100] group_set = set(group) - target = {k for k in keys if tuple(k[i] for i in positions) in group_set} + target = {k for k in keys if tuple(k[i] for i in positions) in group_set} if invariant else set(keys) q = db.query(model).filter(model.batch_id == batch_id) if metric_id == "lldp_neighbor": q = q.order_by(model.id.asc()) @@ -3497,13 +3826,10 @@ def _load_search_key_groups( ) row = {**data, "_netx": {"row_id": r.id}} db.expunge(r) - if row_filters and not all(row_matches_filter(row, f) for f in row_filters): + norm = pipeline.normalize(row, side) + if not pipeline.accepts(row, norm, side): continue - normalized = apply_iface_normalize_rows( - [row], iface_fields=iface_fields, rules=norm_rules, - )[0] - mapped = apply_port_map(normalized, iface_fields=iface_fields, port_map=port_map) - if row_key(mapped, match_keys) in target: + if pipeline.key_of(norm, side, match_keys)[1] in target: out.append(row) return out, complete @@ -3566,13 +3892,15 @@ def _live_search_sheet_diffs( ignore_ports = sheet.get("ignore_port_changes") if ignore_ports is not None: ignore_ports = bool(ignore_ports) - pmap, norm_rules = _run_transform_config(db, run) + pipeline = _run_pipeline(db, run, sheet) + if pipeline.legacy and not pipeline.row_filters and row_filters: + pipeline.row_filters = [dict(f) for f in row_filters if isinstance(f, dict)] before_rows, trunc_b = _load_metric_rows_for_search( db, batch_id=str(run.before_batch_id or ""), metric_id=mid_src, - row_filters=row_filters, + row_filters=pipeline.pushdown_filters("before"), key_fields=key_fields, kw=free_kw, field_q=merged_q, @@ -3581,49 +3909,46 @@ def _live_search_sheet_diffs( db, batch_id=str(run.after_batch_id or ""), metric_id=mid_src, - row_filters=row_filters, + row_filters=pipeline.pushdown_filters("after"), key_fields=key_fields, kw=free_kw, field_q=merged_q, ) match_keys = list((sheet.get("summary") or {}).get("match_key_fields") or key_fields) - if ignore_ports is True and not pmap: + if ignore_ports is True and not pipeline.port_map: match_keys = [f for f in key_fields if f not in iface_fields] or key_fields keys: set[tuple[str, ...]] = set() - for rows, mapping in ((before_rows, pmap), (after_rows, {})): - for row in apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules): - keys.add(row_key(apply_port_map(row, iface_fields=iface_fields, port_map=mapping), match_keys)) + for rows, side in ((before_rows, "before"), (after_rows, "after")): + for raw, norm in pipeline.prepare(rows, side): + keys.add(pipeline.key_of(norm, side, match_keys)[1]) before_rows, complete_b = _load_search_key_groups( db, batch_id=str(run.before_batch_id or ""), metric_id=mid_src, keys=keys, - match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters, - norm_rules=norm_rules, port_map=pmap, + match_keys=match_keys, pipeline=pipeline, side="before", ) after_rows, complete_a = _load_search_key_groups( db, batch_id=str(run.after_batch_id or ""), metric_id=mid_src, keys=keys, - match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters, - norm_rules=norm_rules, port_map={}, + match_keys=match_keys, pipeline=pipeline, side="after", ) complete = complete_b & complete_a - def keep_complete(rows: list[dict[str, Any]], mapping: dict[str, str]) -> list[dict[str, Any]]: - normalized = apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules) - return [orig for orig, norm in zip(rows, normalized) if row_key( - apply_port_map(norm, iface_fields=iface_fields, port_map=mapping), match_keys, - ) in complete] - before_rows = keep_complete(before_rows, pmap) - after_rows = keep_complete(after_rows, {}) + + def keep_complete(rows: list[dict[str, Any]], side: str) -> list[dict[str, Any]]: + return [raw for raw, norm in pipeline.prepare(rows, side) + if pipeline.key_of(norm, side, match_keys)[1] in complete] + + before_rows = keep_complete(before_rows, "before") + after_rows = keep_complete(after_rows, "after") result = compare_rows( before_rows=before_rows, after_rows=after_rows, key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields, - port_map=pmap, field_rules=field_rules, - iface_normalize_rules=norm_rules, - ignore_port_changes=(match_keys != key_fields) if not pmap else False, include_unchanged=True, unchanged_limit=None, compact_unchanged=False, + pipeline=pipeline, + match_keys_override=match_keys, ) kind_n = (kind or "diff").strip().lower() filtered = [ @@ -3636,7 +3961,7 @@ def _live_search_sheet_diffs( filtered = filtered[:_LIVE_SEARCH_RESULT_CAP] total = len(filtered) start = (page - 1) * page_size - page_items = filtered[start : start + page_size] + page_items = _attach_raw_key_before(filtered[start : start + page_size], key_fields) return { "total": total, "page": page, @@ -3844,7 +4169,7 @@ def export_run_zip(db: Session, run_id: str) -> bytes: csv_file.write(b"\xef\xbb\xbf") for line in _sheet_csv_lines(sheet, _iter_sheet_diffs(db, run_id, sid)): csv_file.write((line + "\n").encode("utf-8")) - sum_lines = ["metric_id,mode,before,after,added,removed,changed,unchanged,diff_count,pass_rate"] + sum_lines = ["metric_id,mode,before,after,added,removed,changed,unchanged,diff_count,pass_rate,data_state"] for card in list(s.get("sheet_cards") or []): sum_lines.append( ",".join( @@ -3860,6 +4185,7 @@ def export_run_zip(db: Session, run_id: str) -> bytes: card.get("unchanged"), card.get("diff_count"), card.get("pass_rate"), + card.get("data_state") or "ok", ) ) ) @@ -3927,7 +4253,7 @@ def try_auto_compare_for_task(db: Session, task_id: str, batch_id: str) -> int: if not j.before_batch_id: continue try: - run_compare(db, j.id, force_after_batch_id=batch_id) + run_compare(db, j.id, force_after_batch_id=batch_id, purpose="auto") n += 1 except HTTPException as exc: if int(getattr(exc, "status_code", 0) or 0) == 409: diff --git a/netx_api/biz_state/compare_sql.py b/netx_api/biz_state/compare_sql.py index 672a15a..d417475 100644 --- a/netx_api/biz_state/compare_sql.py +++ b/netx_api/biz_state/compare_sql.py @@ -14,6 +14,7 @@ from uuid import uuid4 from sqlalchemy import text from sqlalchemy.orm import Session +from .compare_engine import key_tuple from .compare_rules import effective_compare_fields, field_rule_map, NUMBER_PATTERN, scalar_text _log = logging.getLogger("netx.biz_state.compare_sql") @@ -154,16 +155,23 @@ def sql_compare_skip_reason( *, port_map: Mapping[str, str] | None = None, iface_normalize_rules: Sequence[Mapping[str, str]] | None = None, + pipeline: Any = None, ) -> str: """Empty string when SQL is allowed; otherwise a short reason for progress/logs. Simple ``row_filters`` (eq/in/contains/…) used for BGP sheet splits are fine — they do **not** force Python by themselves. + + A port map only blocks SQL when the sheet actually reads an interface field + in its key or compare columns; a BGP route table with no interface fields is + unaffected by it. ``pipeline`` (v2 identity rules) adds its own blockers. """ if not _dialect_is_postgres(db): return "not_postgres" - if port_map: - return "port_map" + if pipeline is not None: + port_map = pipeline.port_map + if iface_normalize_rules is None and (pipeline.before.has_iface_rules or pipeline.after.has_iface_rules): + iface_normalize_rules = [{"from": "*", "to": "*"}] key_fields = [str(k).strip() for k in (sheet.get("key_fields") or []) if str(k).strip()] if not key_fields: return "no_key_fields" @@ -171,12 +179,24 @@ def sql_compare_skip_reason( return "unsafe_key_field" iface_fields = [str(f).strip() for f in (sheet.get("iface_fields") or []) if str(f).strip()] iface_set = set(iface_fields) + if port_map and iface_set: + early_cmp = effective_compare_fields( + list(sheet.get("compare_fields") or []), list(sheet.get("field_rules") or []), + ) + if iface_set & (set(key_fields) | {str(f).strip() for f in early_cmp}): + return "port_map" ignore_ports = sheet.get("ignore_port_changes") if ignore_ports is True: return "ignore_port_changes" if ignore_ports is None and iface_set and any(k in iface_set for k in key_fields): return "auto_ignore_ports" field_rules = list(sheet.get("field_rules") or []) + if any( + isinstance(r, dict) and str(r.get("compare") or "").strip().lower() in + ("not_worse", "numeric_min", "numeric_max") + for r in field_rules + ): + return "directional_compare" if not _field_rules_sql_compatible(field_rules): return "field_rules" compare_fields = effective_compare_fields( @@ -186,8 +206,14 @@ def sql_compare_skip_reason( if any(not _FIELD_RE.match(str(f).strip()) for f in compare_fields if str(f).strip()): return "unsafe_compare_field" used = set(key_fields) | {str(f).strip() for f in compare_fields if str(f).strip()} + if port_map and (used & iface_set): + return "port_map" if iface_normalize_rules and (used & iface_set): return "iface_normalize" + if pipeline is not None: + blocked = pipeline.sql_blockers(used) + if blocked: + return blocked if not _filters_sql_compatible(list(sheet.get("row_filters") or [])): return "row_filters" if not str(sheet.get("metric_id") or "").strip(): @@ -371,7 +397,7 @@ def _changed_predicate( def _key_obj_from_data(data: dict[str, Any] | None, key_fields: list[str]) -> dict[str, Any]: row = data if isinstance(data, dict) else {} - return {f: row.get(f, "") for f in key_fields} + return dict(zip(key_fields, key_tuple(row, key_fields))) def _changes_from_rows( @@ -399,6 +425,22 @@ def _changes_from_rows( return out +def dup_pairing_order_sql(compare_fields: Sequence[str], rules: Mapping[str, Mapping[str, Any]]) -> str: + """``ORDER BY`` prefix that pairs same-key rows by value (mirrors ``Pipeline.pair_sort_key``). + + Byte-order collation matches Python's code-point string order, so both + engines choose the same pairs. Returns '' (or a trailing-comma list). + """ + parts = [] + for f in compare_fields: + name = str(f).strip() + if not name: + continue + mode = str((rules.get(name) or {}).get("normalize") or "strip").strip().lower() or "strip" + parts.append(f'({_norm_expr("data_json", name, mode)}) COLLATE "C" ASC') + return ", ".join(parts) + ", " if parts else "" + + def run_sql_sheet_compare( db: Session, *, @@ -407,6 +449,7 @@ def run_sql_sheet_compare( after_batch_id: str, store_unchanged: str = "auto", on_progress: Callable[..., None] | None = None, + duplicate_pairing: str = "ordered", ) -> dict[str, Any]: """Compare one sheet via PostgreSQL TEMP tables + FULL OUTER JOIN. @@ -443,6 +486,7 @@ def run_sql_sheet_compare( filter_sql, filter_params = compile_row_filters_sql(row_filters) rk = _rk_sql(key_fields) + dup_order = dup_pairing_order_sql(compare_fields, rules) if duplicate_pairing == "sorted" else "" tag = uuid4().hex[:8] tb = f"_netx_cmp_b_{tag}" ta = f"_netx_cmp_a_{tag}" @@ -500,7 +544,7 @@ def run_sql_sheet_compare( data_json AS data, row_number() OVER ( PARTITION BY ({rk}) - ORDER BY seq ASC, id ASC + ORDER BY {dup_order}seq ASC, id ASC ) AS dup_rn FROM biz_state_metric_row WHERE batch_id = :bid diff --git a/netx_api/biz_state/compare_validation.py b/netx_api/biz_state/compare_validation.py index 442b10f..3fba6c7 100644 --- a/netx_api/biz_state/compare_validation.py +++ b/netx_api/biz_state/compare_validation.py @@ -8,7 +8,10 @@ from typing import Any from fastapi import HTTPException FILTER_OPS = {"eq", "==", "ci_eq", "ne", "!=", "in", "not_in", "nin", "contains", "empty", "not_empty", "nonempty", "regex", "age_timer"} -COMPARE_MODES = {"", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel"} +COMPARE_MODES = {"", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel", + "not_worse", "numeric_min", "numeric_max"} +SIDES = {"", "both", "before", "after"} +MAX_VALUE_MAP = 500 NORMALIZE_MODES = {"", "none", "strip", "lower", "upper", "mac", "empty_as_blank"} @@ -34,6 +37,8 @@ def validate_filters(value: Any, path: str, depth: int = 0) -> None: here = f"{path}[{i}]" if not isinstance(node, dict) or not node: invalid(here, "filter_required") + if depth == 0 and str(node.get("side") or "").strip().lower() not in SIDES: + invalid(here, "unknown_filter_side") groups = [key for key in ("any", "all") if key in node] if groups: if len(groups) != 1 or any(node.get(key) is not None for key in ("field", "op", "value")): @@ -73,6 +78,12 @@ def validate_sheet(sheet: Any, path: str, *, partial: bool = False) -> None: for key in ("key_fields", "iface_fields", "compare_fields", "display_fields", "ignore_fields"): if key in sheet or (key == "key_fields" and not partial): _strings(sheet.get(key), f"{path}.{key}", required=key == "key_fields") + for opt, allowed in (("unmapped_port_policy", {"by_name", "ignore_port"}), + ("duplicate_pairing", {"sorted", "ordered"})): + if str(sheet.get(opt) or "").strip() and str(sheet[opt]).strip().lower() not in allowed: + invalid(path, f"unknown_{opt}") + if sheet.get("builtin_vendor_canonical") is not None and not isinstance(sheet["builtin_vendor_canonical"], bool): + invalid(path, "boolean_builtin_vendor_canonical_required") if "ignore_port_changes" in sheet and sheet["ignore_port_changes"] is not None and not isinstance(sheet["ignore_port_changes"], bool): invalid(path, "boolean_port_policy_required") if "row_filters" in sheet: @@ -87,9 +98,30 @@ def validate_sheet(sheet: Any, path: str, *, partial: bool = False) -> None: if not isinstance(rule, dict) or not isinstance(rule.get("field"), str) or not rule["field"].strip(): invalid(here, "rule_field_required") field = rule["field"].strip() - if field in seen: + rule_side = str(rule.get("side") or "").strip().lower() + if rule_side not in SIDES: + invalid(here, "unknown_rule_side") + marker = (field, "" if rule_side == "both" else rule_side) + if marker in seen: invalid(here, "duplicate_field_rule") - seen.add(field) + seen.add(marker) + vmap = rule.get("value_map") + if vmap is not None: + if not isinstance(vmap, dict) or any( + not isinstance(k, str) or not isinstance(v, str) for k, v in vmap.items() + ): + invalid(here, "value_map_string_pairs_required") + if len(vmap) > MAX_VALUE_MAP: + invalid(here, "value_map_too_large") + if rule.get("apply_to_key") is not None and not isinstance(rule["apply_to_key"], bool): + invalid(here, "boolean_apply_to_key_required") + mode = str(rule.get("compare") or "").strip().lower() + if mode == "not_worse": + rank = rule.get("rank") + if not isinstance(rank, list) or len(rank) < 2 or any(not isinstance(x, str) or not x.strip() for x in rank): + invalid(here, "rank_list_required") + elif rule.get("rank") is not None: + invalid(here, "rank_only_for_not_worse") if str(rule.get("compare") or "").strip().lower() not in COMPARE_MODES: invalid(here, "unknown_compare_mode") if str(rule.get("normalize") or "").strip().lower() not in NORMALIZE_MODES: @@ -124,7 +156,13 @@ def validate_template_body(body: dict[str, Any], *, partial: bool = False) -> No path = f"iface_normalize_rules[{i}]" if not isinstance(rule, dict) or any(not isinstance(rule.get(k), str) or not rule[k].strip() for k in ("from", "to")): invalid(path, "alias_pair_required") - key = rule["from"].strip().lower() + rside = str(rule.get("side") or "").strip().lower() + if rside not in SIDES: + invalid(path, "unknown_alias_side") + if rule.get("vendor") is not None and not isinstance(rule["vendor"], str): + invalid(path, "alias_vendor_string_required") + key = (rule["from"].strip().lower(), "" if rside == "both" else rside, + str(rule.get("vendor") or "").strip().lower()) if key in seen: invalid(path, "duplicate_alias") seen.add(key) diff --git a/netx_api/biz_state/data/default_zte_status_template.json b/netx_api/biz_state/data/default_zte_status_template.json index d643cbc..7549a38 100644 --- a/netx_api/biz_state/data/default_zte_status_template.json +++ b/netx_api/biz_state/data/default_zte_status_template.json @@ -131,9 +131,25 @@ ], "row_filters": [ { - "op": "age_timer", - "field": "age", - "value": "" + "any": [ + { + "field": "entry_type", + "op": "eq", + "value": "dynamic" + }, + { + "all": [ + { + "field": "entry_type", + "op": "empty" + }, + { + "field": "age", + "op": "age_timer" + } + ] + } + ] } ], "field_rules": [ @@ -415,11 +431,7 @@ ], "compare_fields": [ "admin", - "input_bps", - "output_bps", "port_status", - "in_util", - "out_util", "ip_mtu", "mtu", "mpls_mtu", @@ -454,7 +466,7 @@ }, { "field": "output_bps", - "compare": "numeric", + "compare": "percent", "tolerance": 5 }, { @@ -593,8 +605,8 @@ "field_rules": [ { "field": "rx_power", - "compare": "percent", - "tolerance": 5 + "compare": "numeric", + "tolerance": 2 } ] }, @@ -668,7 +680,9 @@ "vpn_sid", "neighbor_sid" ], - "iface_fields": [], + "iface_fields": [ + "ac_port" + ], "compare_fields": [], "display_fields": [ "mac", diff --git a/netx_api/biz_state/identity_pipeline.py b/netx_api/biz_state/identity_pipeline.py new file mode 100644 index 0000000..d3f1b26 --- /dev/null +++ b/netx_api/biz_state/identity_pipeline.py @@ -0,0 +1,454 @@ +"""One identity pipeline for every compare path. + +A row's *match key* must come out identical whether it is computed while a run +executes, while its results are searched / hydrated / exported, while a mapping +is validated, or while a cutover monitor evaluates the same sheet. All of those +paths therefore build a :class:`Pipeline` and call the same methods instead of +re-implementing "normalize → map → key". + +Stages for one row on one side (``before`` or ``after``):: + + raw row ──normalize──▶ iface aliases (+vendor canonical) ▶ key normalize ▶ value_map + ──filter────▶ ``row_filters`` (v2: on the normalized row; legacy: on the raw row) + ──map───────▶ port map (before side only) + ──key───────▶ ``key_tuple`` over the match key fields + +``legacy=True`` reproduces the pre-pipeline behaviour (snapshot v1 runs): the +template alias rules are the only normalization, keys are never normalized, +there is no value map and the filters see the raw row. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Mapping, Sequence + +from .compare_rules import ( + normalize_value, + row_matches_filter, + scalar_text, +) +from .iface_normalize import ( + normalize_iface_name, + normalize_iface_rules, + normalize_port_map, + resolve_mapped_iface, + rules_for_side, + vendor_canonical_rules, +) + +SIDES = ("before", "after") +DIRECTIONAL_MODES = {"not_worse", "numeric_min", "numeric_max"} +UNMAPPED_POLICIES = ("by_name", "ignore_port") +DUPLICATE_PAIRING = ("sorted", "ordered") + + +def filter_fields(filt: Mapping[str, Any] | None) -> set[str]: + """Every field name a (possibly nested) row filter reads.""" + out: set[str] = set() + if not isinstance(filt, Mapping): + return out + for group in ("any", "all"): + if group in filt: + for kid in filt.get(group) or []: + out |= filter_fields(kid) + return out + name = str(filt.get("field") or "").strip() + if name: + out.add(name) + return out + + +def _vendor_key(vendor: str = "", device_type: str = "") -> str: + from ..topology_lldp import resolve_vendor_key + + key = resolve_vendor_key(vendor, device_type) + return "" if key == "generic" else key + + +@dataclass(eq=False) +class SideSpec: + side: str + vendor: str = "" + canonical: list[dict[str, str]] = field(default_factory=list) + alias_rules: list[dict[str, str]] = field(default_factory=list) + # field -> {lowercased source value: canonical value} + value_maps: dict[str, dict[str, str]] = field(default_factory=dict) + # key field -> normalize mode (only fields whose rule asks for one) + key_norms: dict[str, str] = field(default_factory=dict) + strip_inner_space: bool = True + + def iface(self, name: Any) -> str: + text = str(name or "").strip() + if not text: + return "" + if self.strip_inner_space: + text = "".join(text.split()) + if self.canonical: + text = normalize_iface_name(text, self.canonical) + if self.alias_rules: + text = normalize_iface_name(text, self.alias_rules) + return text + + @property + def has_iface_rules(self) -> bool: + return bool(self.canonical or self.alias_rules) + + def describe(self) -> dict[str, Any]: + """JSON-able record of the rules in force (stored in the run snapshot).""" + return { + "vendor": self.vendor, + "canonical": list(self.canonical), + "alias_rules": list(self.alias_rules), + "value_maps": {k: dict(v) for k, v in self.value_maps.items()}, + "key_norms": dict(self.key_norms), + } + + +@dataclass(eq=False) +class Pipeline: + key_fields: list[str] + iface_fields: list[str] + compare_fields: list[str] + field_rules: list[dict[str, Any]] + row_filters: list[dict[str, Any]] + port_map: dict[str, str] + before: SideSpec + after: SideSpec + legacy: bool = False + ignore_port_changes: bool | None = None + unmapped_port_policy: str = "by_name" + duplicate_pairing: str = "ordered" + _iface_set: set[str] = field(default_factory=set, repr=False) + _transformed: set[str] = field(default_factory=set, repr=False) + _targets: set[str] = field(default_factory=set, repr=False) + + def __post_init__(self) -> None: + self._iface_set = set(self.iface_fields) + self._transformed = set(self._iface_set) + for spec in (self.before, self.after): + self._transformed |= set(spec.value_maps) | set(spec.key_norms) + self._targets = set(self.port_map.values()) + + # -- side helpers ---------------------------------------------------- + def spec(self, side: str) -> SideSpec: + return self.before if side == "before" else self.after + + # -- stage 1: normalize ---------------------------------------------- + def normalize(self, row: Mapping[str, Any], side: str) -> dict[str, Any]: + out = dict(row) + spec = self.spec(side) + for f in self.iface_fields: + val = scalar_text(out.get(f)).strip() + if val: + out[f] = spec.iface(val) + if self.legacy: + return out + for f, mode in spec.key_norms.items(): + if f in out: + out[f] = normalize_value(out[f], mode) + for f, vmap in spec.value_maps.items(): + if f not in out: + continue + raw = scalar_text(out[f]).strip() + hit = vmap.get(raw.lower()) + if hit is not None: + out[f] = hit + return out + + def normalize_name(self, name: Any, side: str) -> str: + """Interface-name normalization only (port-map keys / values, selectors).""" + return self.spec(side).iface(name) + + # -- stage 2: filter ------------------------------------------------- + def filters_for(self, side: str) -> list[dict[str, Any]]: + out = [] + for f in self.row_filters: + fs = str(f.get("side") or "").strip().lower() + if not fs or fs == "both" or fs == side: + out.append(f) + return out + + def keep(self, row: Mapping[str, Any], side: str) -> bool: + """Evaluate row filters. v2 expects the *normalized* row, legacy the raw one.""" + return all(row_matches_filter(row, f) for f in self.filters_for(side)) + + def accepts(self, raw: Mapping[str, Any], norm: Mapping[str, Any], side: str) -> bool: + """Row-filter verdict: legacy reads the raw row, v2 the normalized one.""" + return self.keep(raw if self.legacy else norm, side) + + def pushdown_filters(self, side: str) -> list[dict[str, Any]]: + """Filters that read only untransformed fields, so they may run in SQL on raw rows.""" + if self.legacy: + return list(self.row_filters) + return [ + f for f in self.row_filters + if not str(f.get("side") or "").strip() + and not (filter_fields(f) & self._transformed) + ] + + def prepare(self, rows: Sequence[Mapping[str, Any]], side: str) -> list[tuple[dict[str, Any], dict[str, Any]]]: + """``[(original row, normalized row)]`` after filtering, in input order.""" + out: list[tuple[dict[str, Any], dict[str, Any]]] = [] + has_filters = bool(self.filters_for(side)) + for raw in rows: + if not isinstance(raw, dict): + continue + norm = self.normalize(raw, side) + if has_filters and not self.accepts(raw, norm, side): + continue + out.append((raw, norm)) + return out + + # -- stage 3: port map ----------------------------------------------- + def apply_port_map(self, norm_row: Mapping[str, Any]) -> dict[str, Any]: + out = dict(norm_row) + if not self.port_map: + return out + for f in self.iface_fields: + val = scalar_text(out.get(f)).strip() + if val: + out[f] = resolve_mapped_iface(val, self.port_map) + return out + + def is_mapped(self, norm_row: Mapping[str, Any], side: str) -> bool: + """True when every interface value of the row is covered by the port map.""" + if not self.iface_fields: + return True + values = [scalar_text(norm_row.get(f)).strip() for f in self.iface_fields] + values = [v for v in values if v] + if not values: + return False + for v in values: + if side == "before": + if resolve_mapped_iface(v, self.port_map) == v and v not in self.port_map: + return False + else: + if not self._after_value_is_target(v): + return False + return True + + def _after_value_is_target(self, value: str) -> bool: + if value in self._targets: + return True + parts = value.split(".") + return any(".".join(parts[:i]) in self._targets for i in range(len(parts) - 1, 0, -1)) + + def transform(self, row: Mapping[str, Any], side: str) -> dict[str, Any]: + norm = self.normalize(row, side) + return self.apply_port_map(norm) if side == "before" else norm + + # -- stage 4: key ---------------------------------------------------- + def effective_match_keys(self, *, unique_without_iface: bool | None = None) -> list[str]: + """Match-key fields once the port policy is resolved. + + With a port map the full key is used (the map aligns the names). Without + one, ``ignore_port_changes`` may drop the interface fields; ``None`` means + "auto", resolved by the caller via ``unique_without_iface``. + """ + keys = list(self.key_fields) + if self.port_map or not self._iface_set: + return keys + candidate = [k for k in keys if k not in self._iface_set] + if not candidate: + return keys + policy = self.ignore_port_changes + if policy is True: + return candidate + if policy is False: + return keys + return candidate if unique_without_iface else keys + + def ignores_ports_by_policy(self) -> bool: + return bool(self.port_map) and self.unmapped_port_policy == "ignore_port" and bool(self._iface_set) + + def key_of( + self, norm_row: Mapping[str, Any], side: str, match_keys: Sequence[str], + ) -> tuple[dict[str, Any], tuple[str, ...]]: + """``(row after port map, match key)`` for a *normalized* row.""" + row = self.apply_port_map(norm_row) if side == "before" else dict(norm_row) + if self.ignores_ports_by_policy(): + if not self.is_mapped(norm_row, side): + reduced = [k for k in match_keys if k not in self._iface_set] or list(match_keys) + return row, ("U",) + tuple(scalar_text(row.get(k)).strip() for k in reduced) + return row, ("M",) + tuple(scalar_text(row.get(k)).strip() for k in match_keys) + return row, tuple(scalar_text(row.get(k)).strip() for k in match_keys) + + def match_key( + self, row: Mapping[str, Any], side: str, match_keys: Sequence[str] | None = None, + ) -> tuple[str, ...]: + """Match key of a *raw* row (normalize → port map → key).""" + keys = list(match_keys) if match_keys is not None else self.effective_match_keys() + return self.key_of(self.normalize(row, side), side, keys)[1] + + # -- duplicate pairing ----------------------------------------------- + def pair_sort_key(self, norm_row: Mapping[str, Any]) -> tuple[str, ...]: + rules = {str(r.get("field")): r for r in self.field_rules if isinstance(r, dict)} + out = [] + for f in self.compare_fields: + mode = str((rules.get(f) or {}).get("normalize") or "strip") + out.append(normalize_value(norm_row.get(f), mode)) + return tuple(out) + + # -- SQL eligibility ------------------------------------------------- + def sql_blockers(self, used_fields: set[str]) -> str: + """Reason the SQL engine must not run this sheet ('' when it may).""" + if self.legacy: + return "" + for spec in (self.before, self.after): + if set(spec.value_maps) & used_fields: + return "value_map" + if set(spec.key_norms) & set(self.key_fields): + return "key_normalize" + if used_fields & self._iface_set and (self.before.has_iface_rules or self.after.has_iface_rules): + return "iface_normalize" + if any(str(f.get("side") or "").strip() for f in self.row_filters): + return "row_filters_side" + if any(filter_fields(f) & self._transformed for f in self.row_filters): + return "row_filters_post_normalize" + if any( + str(r.get("compare") or "").strip().lower() in DIRECTIONAL_MODES + for r in self.field_rules if isinstance(r, dict) + ): + return "directional_compare" + return "" + + def describe(self) -> dict[str, Any]: + return { + "legacy": self.legacy, + "before": self.before.describe(), + "after": self.after.describe(), + "unmapped_port_policy": self.unmapped_port_policy, + "duplicate_pairing": self.duplicate_pairing, + } + + +def _field_rule_specs( + field_rules: Sequence[Mapping[str, Any]], key_fields: Sequence[str], side: str, +) -> tuple[dict[str, dict[str, str]], dict[str, str]]: + value_maps: dict[str, dict[str, str]] = {} + key_norms: dict[str, str] = {} + key_set = set(key_fields) + for rule in field_rules or []: + if not isinstance(rule, Mapping): + continue + name = str(rule.get("field") or "").strip() + if not name: + continue + rs = str(rule.get("side") or "").strip().lower() + if rs and rs != "both" and rs != side: + continue + apply_to_key = rule.get("apply_to_key") + apply_to_key = (name in key_set) if apply_to_key is None else bool(apply_to_key) + is_key = name in key_set + if is_key and not apply_to_key: + continue + mode = str(rule.get("normalize") or "").strip().lower() + vmap = rule.get("value_map") + if is_key and mode and mode not in ("none", "strip"): + key_norms[name] = mode + if isinstance(vmap, Mapping) and vmap: + pre = mode if (mode and mode not in ("none", "strip")) else "strip" + built: dict[str, str] = {} + for src, dst in vmap.items(): + built[normalize_value(src, pre).lower()] = scalar_text(dst).strip() + value_maps[name] = built + return value_maps, key_norms + + +def build_pipeline( + sheet: Mapping[str, Any], + *, + iface_rules: Sequence[Mapping[str, Any]] | None, + port_map: Mapping[str, str] | None, + before_vendor: str = "", + after_vendor: str = "", + legacy: bool = False, + normalize_map: bool = True, +) -> Pipeline: + """Build the pipeline for one sheet. + + ``port_map`` is the map as the user wrote it. Unless ``normalize_map`` is + false (legacy runs, whose maps were written post-normalization) its keys are + normalized with the *before* side rules and its values with the *after* side + rules. + """ + key_fields = [str(k).strip() for k in (sheet.get("key_fields") or []) if str(k).strip()] + iface_fields = [str(k).strip() for k in (sheet.get("iface_fields") or []) if str(k).strip()] + field_rules = [dict(r) for r in (sheet.get("field_rules") or []) if isinstance(r, Mapping)] + from .compare_rules import effective_compare_fields + + compare_fields = effective_compare_fields(list(sheet.get("compare_fields") or []), field_rules) + canonical_on = sheet.get("builtin_vendor_canonical", True) is not False and not legacy + all_rules = normalize_iface_rules(list(iface_rules or [])) + + specs: dict[str, SideSpec] = {} + for side, vendor_raw in (("before", before_vendor), ("after", after_vendor)): + vendor = _vendor_key(vendor_raw) if vendor_raw else "" + if legacy: + alias = normalize_iface_rules([{"from": r["from"], "to": r["to"]} for r in all_rules]) + value_maps, key_norms = {}, {} + else: + alias = rules_for_side(all_rules, side, vendor) + value_maps, key_norms = _field_rule_specs(field_rules, key_fields, side) + specs[side] = SideSpec( + side=side, vendor=vendor, + canonical=vendor_canonical_rules(vendor) if canonical_on else [], + alias_rules=alias, value_maps=value_maps, key_norms=key_norms, + strip_inner_space=not legacy, + ) + + pmap = dict(port_map or {}) + if pmap and normalize_map and not legacy: + pmap, _ = normalize_port_map( + pmap, None, + key_normalizer=specs["before"].iface, value_normalizer=specs["after"].iface, + ) + ignore_ports = sheet.get("ignore_port_changes") + policy = str(sheet.get("unmapped_port_policy") or "by_name").strip().lower() + pairing = str(sheet.get("duplicate_pairing") or ("ordered" if legacy else "sorted")).strip().lower() + return Pipeline( + key_fields=key_fields, + iface_fields=iface_fields, + compare_fields=compare_fields, + field_rules=field_rules, + row_filters=[dict(f) for f in (sheet.get("row_filters") or []) if isinstance(f, Mapping)], + port_map=pmap, + before=specs["before"], + after=specs["after"], + legacy=legacy, + ignore_port_changes=None if ignore_ports is None else bool(ignore_ports), + unmapped_port_policy=policy if policy in UNMAPPED_POLICIES else "by_name", + duplicate_pairing="ordered" if legacy else (pairing if pairing in DUPLICATE_PAIRING else "sorted"), + ) + + +def legacy_pipeline( + *, + key_fields: Sequence[str], + iface_fields: Sequence[str], + compare_fields: Sequence[str], + field_rules: Sequence[Mapping[str, Any]] | None = None, + port_map: Mapping[str, str] | None = None, + iface_normalize_rules: Sequence[Mapping[str, Any]] | None = None, + ignore_port_changes: bool | None = None, + row_filters: Sequence[Mapping[str, Any]] | None = None, + duplicate_pairing: str = "ordered", +) -> Pipeline: + """Pipeline with the pre-v2 semantics, for snapshot-v1 runs and bare callers.""" + alias = normalize_iface_rules( + [{"from": r.get("from"), "to": r.get("to")} for r in (iface_normalize_rules or [])] + ) + return Pipeline( + key_fields=[str(k) for k in key_fields], + iface_fields=[str(f) for f in iface_fields if str(f).strip()], + compare_fields=[str(f) for f in compare_fields], + field_rules=[dict(r) for r in (field_rules or []) if isinstance(r, Mapping)], + row_filters=[dict(f) for f in (row_filters or []) if isinstance(f, Mapping)], + port_map=dict(port_map or {}), + before=SideSpec(side="before", alias_rules=list(alias), strip_inner_space=False), + after=SideSpec(side="after", alias_rules=list(alias), strip_inner_space=False), + legacy=True, + ignore_port_changes=ignore_port_changes, + duplicate_pairing=duplicate_pairing if duplicate_pairing in DUPLICATE_PAIRING else "ordered", + ) diff --git a/netx_api/biz_state/iface_normalize.py b/netx_api/biz_state/iface_normalize.py index 5fdb793..7018d63 100644 --- a/netx_api/biz_state/iface_normalize.py +++ b/netx_api/biz_state/iface_normalize.py @@ -39,7 +39,14 @@ def normalize_iface_rules(raw: Any) -> list[dict[str, str]]: fr = str(item.get("from") or item.get("src") or "").strip() to = str(item.get("to") or item.get("dst") or "").strip() if fr and to: - out.append({"from": fr, "to": to}) + rule = {"from": fr, "to": to} + side = str(item.get("side") or "").strip().lower() + if side in ("before", "after"): + rule["side"] = side + vendor = str(item.get("vendor") or "").strip().lower() + if vendor: + rule["vendor"] = vendor + out.append(rule) # Longest prefix first so XXVGE wins over XGE / GE. out.sort(key=lambda r: len(r["from"]), reverse=True) return out @@ -107,6 +114,47 @@ def apply_iface_normalize_rows( ] +def normalize_port_map( + port_map: Mapping[str, str] | None, + rules: Sequence[Mapping[str, str]] | None, + value_rules: Sequence[Mapping[str, str]] | None = None, + *, + key_normalizer: Any = None, + value_normalizer: Any = None, +) -> tuple[dict[str, str], list[dict[str, str]]]: + """Normalize port-map keys and values with the same rules as iface fields. + + ``value_rules`` (default: ``rules``) normalizes the after-side values, for + per-side rules. ``key_normalizer`` / ``value_normalizer`` are callables that + replace the rule-based normalization (used by the identity pipeline). + + Returns ``(normalized_map, conflicts)``. A conflict is two raw keys that + normalize to the same key but point at different (normalized) targets; the + first one wins in ``normalized_map``. + """ + out: dict[str, str] = {} + origin: dict[str, str] = {} + conflicts: list[dict[str, str]] = [] + for raw_k, raw_v in (port_map or {}).items(): + k = key_normalizer(raw_k) if key_normalizer else normalize_iface_name(raw_k, rules) + v = ( + value_normalizer(raw_v) if value_normalizer + else normalize_iface_name(raw_v, rules if value_rules is None else value_rules) + ) + if not k or not v: + continue + if k in out: + if out[k] != v: + conflicts.append({ + "before_if": k, "kept_from": origin[k], "kept_to": out[k], + "dropped_from": str(raw_k), "dropped_to": v, + }) + continue + out[k] = v + origin[k] = str(raw_k) + return out, conflicts + + def resolve_mapped_iface(name: str, port_map: Mapping[str, str] | None) -> str: """Exact map hit, else longest mapped parent + remaining suffix (QinQ-safe). @@ -142,3 +190,53 @@ def default_zte_iface_normalize_rules() -> list[dict[str, str]]: {"from": "SG", "to": "smartgroup"}, ] ) + + +# --------------------------------------------------------------------------- +# Built-in per-vendor canonical names (same-vendor synonyms only; cross-vendor +# numbering is the port map's job). Applied before template alias rules. +# --------------------------------------------------------------------------- + +VENDOR_CANONICAL: dict[str, list[dict[str, str]]] = { + "huawei": [ + {"from": "XGigabitEthernet", "to": "XGE"}, {"from": "GigabitEthernet", "to": "GE"}, + {"from": "100GE", "to": "100GE"}, {"from": "40GE", "to": "40GE"}, {"from": "25GE", "to": "25GE"}, + {"from": "10GE", "to": "10GE"}, {"from": "Eth-Trunk", "to": "Eth-Trunk"}, + ], + "cisco": [ + {"from": "GigabitEthernet", "to": "Gi"}, {"from": "TenGigabitEthernet", "to": "Te"}, + {"from": "TenGigE", "to": "Te"}, {"from": "HundredGigE", "to": "Hu"}, {"from": "FortyGigE", "to": "Fo"}, + {"from": "TwentyFiveGigE", "to": "Twe"}, {"from": "FastEthernet", "to": "Fa"}, + {"from": "Bundle-Ether", "to": "BE"}, {"from": "Loopback", "to": "Lo"}, + ], + "h3c": [ + {"from": "Ten-GigabitEthernet", "to": "XGE"}, {"from": "GigabitEthernet", "to": "GE"}, + {"from": "FortyGigE", "to": "FGE"}, {"from": "HundredGigE", "to": "HGE"}, + {"from": "Bridge-Aggregation", "to": "BAGG"}, + ], + "zte": [ + {"from": "xxvgei", "to": "xxvgei"}, {"from": "xgei", "to": "xgei"}, {"from": "cgei", "to": "cgei"}, + {"from": "gei", "to": "gei"}, {"from": "smartgroup", "to": "smartgroup"}, + ], +} + + +def vendor_canonical_rules(vendor: str) -> list[dict[str, str]]: + return normalize_iface_rules(VENDOR_CANONICAL.get(str(vendor or "").strip().lower(), [])) + + +def rules_for_side( + rules: Sequence[Mapping[str, str]] | None, side: str, vendor: str = "", +) -> list[dict[str, str]]: + """Template alias rules that apply to ``side`` (``before``/``after``) and ``vendor``.""" + vendor_n = str(vendor or "").strip().lower() + out: list[dict[str, str]] = [] + for r in rules or []: + rs = str(r.get("side") or "").strip().lower() + if rs and rs != side: + continue + rv = str(r.get("vendor") or "").strip().lower() + if rv and rv != vendor_n: + continue + out.append({"from": str(r.get("from") or ""), "to": str(r.get("to") or "")}) + return normalize_iface_rules(out) diff --git a/netx_api/biz_state/profiles.py b/netx_api/biz_state/profiles.py index 996d46b..70fcf88 100644 --- a/netx_api/biz_state/profiles.py +++ b/netx_api/biz_state/profiles.py @@ -85,6 +85,9 @@ class ParseProfile: enrich_joins: list[EnrichJoin] = field(default_factory=list) # light: default shared SSH lane; heavy: dedicated long-timeout connection collect_lane: str = "light" + # Which slice of the metric this command produces, e.g. {"afi": "ipv4"}. + # Empty = the command covers the whole metric (or its slicing is unknown). + slice: dict[str, str] = field(default_factory=dict) _LLDP_FIELDS: list[FieldDef] = [ @@ -1651,6 +1654,11 @@ _HEAVY_LANE_METRICS = frozenset( ) +def metric_lane(metric_id: str) -> str: + """``heavy`` for huge CLI dumps (own SSH connection), else ``light``.""" + return "heavy" if str(metric_id or "").strip() in _HEAVY_LANE_METRICS else "light" + + def _apply_collect_lanes(profiles: list[ParseProfile]) -> list[ParseProfile]: for p in profiles: if str(p.metric_id or "").strip() in _HEAVY_LANE_METRICS: @@ -1752,12 +1760,41 @@ def _validate_profiles(profiles: list[ParseProfile]) -> None: ) +# profile_id -> slice. Compare sheets filter on these fields (``afi``/``af``/``direction``), +# so a missing command is detected as "slice not collected" instead of "business lost". +_PROFILE_SLICES: dict[str, dict[str, str]] = { + **{f"zte.bgp_{afi}_summary": {"afi": afi} for afi in ("vpnv4", "ipv4", "vpnv6", "ipv6", "evpn", "vpls")}, + "zte.bgp_vpnv4_vrf_summary": {"afi": "vpnv4"}, + "zte.bgp_vpnv6_vrf_summary": {"afi": "vpnv6"}, + "zte.vrrp_ipv4": {"af": "ipv4"}, + "zte.vrrp_ipv6": {"af": "ipv6"}, + **{ + f"zte.bgp_{afi}_neighbor_{direction}": {"afi": afi, "direction": direction} + for afi in ("vpnv4", "vpnv6", "ipv4", "ipv6") + for direction in ("in", "out") + }, + **{ + f"zte.bgp_{afi}_vrf_neighbor_{direction}": {"afi": afi, "direction": direction} + for afi in ("vpnv4", "vpnv6") + for direction in ("in", "out") + }, +} + + +def _apply_profile_slices(profiles: list[ParseProfile]) -> list[ParseProfile]: + for p in profiles: + declared = _PROFILE_SLICES.get(p.profile_id) + if declared and not p.slice: + p.slice = dict(declared) + return profiles + + def _finalize_profiles(profiles: list[ParseProfile]) -> list[ParseProfile]: """Normalize discover→aux, validate, apply collect lanes.""" for p in profiles: _normalize_discover_aux(p) _validate_profiles(profiles) - return _apply_collect_lanes(profiles) + return _apply_profile_slices(_apply_collect_lanes(profiles)) def all_profiles() -> list[ParseProfile]: diff --git a/netx_api/biz_state/retention.py b/netx_api/biz_state/retention.py index fde4107..bd3ccf8 100644 --- a/netx_api/biz_state/retention.py +++ b/netx_api/biz_state/retention.py @@ -6,13 +6,17 @@ from collections import defaultdict from datetime import datetime, timedelta from typing import Any -from sqlalchemy import or_, select +from sqlalchemy import func, or_, select from sqlalchemy.orm import Session from ..models import ( + BizCompareDiff, BizCompareJob, BizCompareRun, + BizMigrationBatch, + BizMigrationDiff, BizMigrationProject, + BizMigrationRedTicket, BizMigrationRun, BizStateBatch, BizStateBatchCommand, @@ -76,9 +80,103 @@ def protected_batch_map( add(before_id, before_reason) add(after_id, after_reason) + # Every batch a *kept* acceptance/manual run (or the newest auto run of a wave) + # compared, not only the first binding. Older auto runs are bounded by + # ``purge_auto_runs`` and keep protecting just their primary pair above. + for run_batches in _kept_run_metric_batches(db): + for bid in run_batches: + add(bid, "migration_run_metric") + return dict(out) +def _kept_run_metric_batches(db: Session, *, limit: int = 500) -> list[set[str]]: + newest_auto = ( + db.query(func.max(BizMigrationRun.created_at).label("at"), BizMigrationRun.batch_id.label("bid")) + .filter(BizMigrationRun.purpose == "auto").group_by(BizMigrationRun.batch_id).subquery() + ) + latest_auto_ids = [ + r[0] for r in db.query(BizMigrationRun.id).join( + newest_auto, + (BizMigrationRun.batch_id == newest_auto.c.bid) & (BizMigrationRun.created_at == newest_auto.c.at), + ).limit(limit).all() + ] + kept_ids = [ + r[0] for r in db.query(BizMigrationRun.id).filter(BizMigrationRun.purpose != "auto") + .order_by(BizMigrationRun.created_at.desc()).limit(limit).all() + ] + latest_auto_ids + out: list[set[str]] = [] + for start in range(0, len(kept_ids), 100): + for (summary,) in db.query(BizMigrationRun.summary_json).filter( + BizMigrationRun.id.in_(kept_ids[start:start + 100]) + ): + batches: set[str] = set() + for item in ((summary or {}).get("metric_batches") or {}).values(): + if isinstance(item, dict): + batches.update(str(item.get(k) or "") for k in ("old_batch_id", "new_batch_id")) + batches.discard("") + if batches: + out.append(batches) + return out + + +def purge_auto_runs(db: Session, *, migration_keep: int = 300, compare_keep: int = 50) -> dict[str, int]: + """Bound auto-generated run history. + + Cutover monitoring writes a run (+ one diff row per object) every minute and + auto-compare writes one per new sample. Acceptance / manual runs and runs a + red ticket points at are kept forever; only the newest ``keep`` auto runs of + each wave / job survive. + """ + dropped = {"migration_runs": 0, "compare_runs": 0} + ticket_runs = {r[0] for r in db.query(BizMigrationRedTicket.run_id).filter( + BizMigrationRedTicket.run_id != "").distinct()} + accepted = {r[0] for r in db.query(BizMigrationBatch.accept_run_id).filter( + BizMigrationBatch.accept_run_id != "")} + keep_forever = ticket_runs | accepted + + def surplus(model, group_col, extra_filter, keep: int) -> list[str]: + over = [g for g, n in db.query(group_col, func.count(model.id)).filter(*extra_filter) + .group_by(group_col).all() if n > keep] + ids: list[str] = [] + for g in over: + rows = db.query(model.id).filter(group_col == g, *extra_filter).order_by( + model.created_at.desc(), model.id.desc()).offset(keep).all() + ids.extend(r[0] for r in rows if r[0] not in keep_forever) + return ids + + mig_ids = surplus(BizMigrationRun, BizMigrationRun.batch_id, [BizMigrationRun.purpose == "auto"], + max(1, int(migration_keep))) + for start in range(0, len(mig_ids), 500): + chunk = mig_ids[start:start + 500] + db.query(BizMigrationDiff).filter(BizMigrationDiff.run_id.in_(chunk)).delete(synchronize_session=False) + db.query(BizMigrationRun).filter(BizMigrationRun.id.in_(chunk)).delete(synchronize_session=False) + dropped["migration_runs"] = len(mig_ids) + + # Auto-mode jobs only. A run's origin lives in summary_json["purpose"]; unmarked runs + # are manual and stay. Only the surplus beyond ``compare_keep`` is inspected. + cmp_ids: list[str] = [] + keep_n = max(1, int(compare_keep)) + for (job_id,) in db.query(BizCompareJob.id).filter(BizCompareJob.mode == "auto").all(): + surplus_ids = [r[0] for r in db.query(BizCompareRun.id).filter( + BizCompareRun.job_id == job_id, BizCompareRun.status.notin_(("queued", "running")), + ).order_by(BizCompareRun.created_at.desc(), BizCompareRun.id.desc()).offset(keep_n).all()] + for start in range(0, len(surplus_ids), 200): + for rid, summary in db.query(BizCompareRun.id, BizCompareRun.summary_json).filter( + BizCompareRun.id.in_(surplus_ids[start:start + 200]) + ): + if (summary or {}).get("purpose") == "auto": + cmp_ids.append(rid) + for start in range(0, len(cmp_ids), 500): + chunk = cmp_ids[start:start + 500] + db.query(BizCompareDiff).filter(BizCompareDiff.run_id.in_(chunk)).delete(synchronize_session=False) + db.query(BizCompareRun).filter(BizCompareRun.id.in_(chunk)).delete(synchronize_session=False) + dropped["compare_runs"] = len(cmp_ids) + if mig_ids or cmp_ids: + db.commit() + return dropped + + def protected_batch_ids(db: Session, *, task_id: str = "") -> set[str]: return set(protected_batch_map(db, task_id=task_id).keys()) diff --git a/netx_api/biz_state_router.py b/netx_api/biz_state_router.py index fa1e936..0714e1c 100644 --- a/netx_api/biz_state_router.py +++ b/netx_api/biz_state_router.py @@ -549,6 +549,12 @@ class FieldRuleIn(BaseModel): normalize: str = "" ignore: bool = False tolerance: float | None = None + # Cross-vendor identity rules (see identity_pipeline). + value_map: dict[str, str] | None = None + side: str = "" + apply_to_key: bool | None = None + # Order for compare=not_worse: earlier entries are better. + rank: list[str] | None = None class RowFilterIn(BaseModel): @@ -575,6 +581,9 @@ class TemplateMetricIn(BaseModel): row_filters: list[dict[str, Any]] = Field(default_factory=list) field_rules: list[FieldRuleIn] = Field(default_factory=list) ignore_port_changes: bool | None = None + unmapped_port_policy: str = "" + duplicate_pairing: str = "" + builtin_vendor_canonical: bool | None = None class TemplateIn(BaseModel): @@ -623,6 +632,16 @@ class MappingIn(BaseModel): rows: list[MappingRowIn] = Field(default_factory=list) +class PreflightIn(BaseModel): + template_id: str = "" + enabled_sheet_ids: list[str] = Field(default_factory=list) + mapping_id: str = "" + # Unsaved editor rows win over ``mapping_id`` when given. + mapping_rows: list[MappingRowIn] | None = None + before_batch_id: str + after_batch_id: str + + class ValidateMappingIn(BaseModel): mapping_id: str before_batch_id: str @@ -711,6 +730,11 @@ def api_validate_mapping(body: ValidateMappingIn, db: Session = Depends(get_db)) ) +@router.post("/compare/preflight") +def api_compare_preflight(body: PreflightIn, db: Session = Depends(get_db)) -> dict[str, Any]: + return cmp_svc.preflight_compare(db, body.model_dump()) + + @router.get("/compare/jobs") def api_list_jobs(db: Session = Depends(get_db)) -> dict[str, Any]: return {"items": cmp_svc.list_jobs(db)} diff --git a/netx_api/biz_state_scheduler.py b/netx_api/biz_state_scheduler.py index 49dcd07..77b6338 100644 --- a/netx_api/biz_state_scheduler.py +++ b/netx_api/biz_state_scheduler.py @@ -47,6 +47,15 @@ def _maybe_purge_retention() -> None: from .biz_state.retention import purge_all_tasks info = purge_all_tasks(db) + from .biz_state.retention import purge_auto_runs + + runs = purge_auto_runs( + db, + migration_keep=int(getattr(settings, "biz_migration_auto_run_keep", 300) or 300), + compare_keep=int(getattr(settings, "biz_compare_auto_run_keep", 50) or 50), + ) + if any(runs.values()): + _log.info("biz_state periodic run purge %s", runs) if info.get("dropped"): _log.info( "biz_state periodic retention dropped=%s tasks=%s", diff --git a/netx_api/config.py b/netx_api/config.py index e12b5fd..2ebe55f 100644 --- a/netx_api/config.py +++ b/netx_api/config.py @@ -137,6 +137,13 @@ class Settings(BaseSettings): biz_state_dedicated_workers: bool = True # Global ceiling for simultaneous running batches (across all workers). biz_state_max_concurrent_tasks: int = 16 + # Auto-generated run history kept per cutover wave / auto compare job (acceptance + manual runs are never purged). + biz_migration_auto_run_keep: int = 300 + biz_compare_auto_run_keep: int = 50 + # Rows of immutable baseline batches cached across cutover evaluations (0 disables). + biz_migration_baseline_cache_rows: int = 500000 + # Concurrent SSH collections per network element (0 = unlimited). + biz_state_max_sessions_per_ne: int = 2 biz_state_worker_collect_threads: int = 8 biz_state_persist_workers: int = 4 # TextFSM / normalize workers (off SSH collect threads). Default 8. diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..6b2d017 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,12 @@ +"""Shared test hygiene.""" +import pytest + + +@pytest.fixture(autouse=True) +def _fresh_baseline_row_cache(): + """Test databases reuse batch ids; the process-wide baseline cache must not leak between tests.""" + from netx_api.biz_migration.service import BASELINE_ROWS + + BASELINE_ROWS.invalidate() + yield + BASELINE_ROWS.invalidate() diff --git a/tests/test_biz_compare_templates.py b/tests/test_biz_compare_templates.py index a1f76f0..83dd0f1 100644 --- a/tests/test_biz_compare_templates.py +++ b/tests/test_biz_compare_templates.py @@ -143,7 +143,7 @@ def test_template_usage_counts_are_batched(db, monkeypatch): event.listen(db.get_bind(), "before_cursor_execute", lambda c, cur, statement, p, ctx, many: sql.append(statement)) counts = {t["id"]: t["job_count"] for t in svc.list_templates(db)} assert counts == {used["id"]: 2, unused["id"]: 0} - assert len(sql) == 2 # one template query and one grouped reference query + assert len(sql) == 3 # template query + one grouped query each for jobs and monitor templates @pytest.mark.parametrize("status", ["queued", "running", "failed", "cancelled", "partial"]) @@ -188,3 +188,93 @@ def test_legacy_invalid_template_cannot_execute_as_partial_or_fail_open(db, bad) assert exc.value.detail["error"] == "invalid_template" assert "metrics[1]" in exc.value.detail["path"] assert db.query(BizCompareRun).count() == 0 + + +def _builtin_sheet(row, metric_id): + return next(s for s in svc.template_metrics(row) if s["metric_id"] == metric_id) + + +def _rule(sheet, field): + return next(r for r in sheet["field_rules"] if r["field"] == field) + + +def test_packaged_status_template_uses_corrected_defaults(db): + row = svc.ensure_default_zte_status_template(db) + detail = _builtin_sheet(row, "interface_detail") + assert _rule(detail, "output_bps")["compare"] == "percent" + for f in ("input_bps", "output_bps", "in_util", "out_util"): + assert f not in detail["compare_fields"] and f in detail["display_fields"] + rx = _rule(_builtin_sheet(row, "optical_brief"), "rx_power") + assert (rx["compare"], rx["tolerance"]) == ("numeric", 2) + assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["ac_port"] + + +def _store_old_packaged_values(db, row): + sheets = svc.template_metrics(row) + for s in sheets: + if s["metric_id"] == "interface_detail": + _rule(s, "output_bps").update(compare="numeric", tolerance=5) + s["compare_fields"] = list(svc._INTERFACE_DETAIL_OLD_COMPARE) + elif s["metric_id"] == "optical_brief": + _rule(s, "rx_power").update(compare="percent", tolerance=5) + elif s["metric_id"] == "l2vpn_mac": + s["iface_fields"] = [] + svc._apply_sheets_to_row(row, sheets) + db.commit() + + +def test_old_packaged_values_are_upgraded_in_place(db): + row = svc.ensure_default_zte_status_template(db) + _store_old_packaged_values(db, row) + row = svc.ensure_default_zte_status_template(db) + detail = _builtin_sheet(row, "interface_detail") + assert _rule(detail, "output_bps")["compare"] == "percent" + assert "output_bps" not in detail["compare_fields"] + assert _rule(_builtin_sheet(row, "optical_brief"), "rx_power")["compare"] == "numeric" + assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["ac_port"] + + +def test_operator_customised_values_survive_upgrade(db): + row = svc.ensure_default_zte_status_template(db) + _store_old_packaged_values(db, row) + sheets = svc.template_metrics(row) + for s in sheets: + if s["metric_id"] == "optical_brief": + _rule(s, "rx_power").update(compare="percent", tolerance=1) # operator tightened it + elif s["metric_id"] == "interface_detail": + s["compare_fields"] = ["admin", "output_bps"] # operator trimmed the list + elif s["metric_id"] == "l2vpn_mac": + s["iface_fields"] = ["neighbor"] + svc._apply_sheets_to_row(row, sheets) + db.commit() + row = svc.ensure_default_zte_status_template(db) + rx = _rule(_builtin_sheet(row, "optical_brief"), "rx_power") + assert (rx["compare"], rx["tolerance"]) == ("percent", 1) + assert _builtin_sheet(row, "interface_detail")["compare_fields"] == ["admin", "output_bps"] + assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["neighbor"] + + +def test_packaged_arp_filter_uses_canonical_entry_type_and_old_filter_is_upgraded(db): + from netx_api.biz_state.compare_rules import arp_dynamic_row_filters + + row = svc.ensure_default_zte_status_template(db) + assert _builtin_sheet(row, "arp")["row_filters"] == arp_dynamic_row_filters() + sheets = svc.template_metrics(row) + for s in sheets: + if s["metric_id"] == "arp": + s["row_filters"] = [{"op": "age_timer", "field": "age", "value": ""}] + svc._apply_sheets_to_row(row, sheets) + db.commit() + assert _builtin_sheet(svc.ensure_default_zte_status_template(db), "arp")["row_filters"] == arp_dynamic_row_filters() + + +def test_customised_arp_filter_is_left_alone(db): + row = svc.ensure_default_zte_status_template(db) + sheets = svc.template_metrics(row) + mine = [{"op": "eq", "field": "entry_type", "value": "static"}] + for s in sheets: + if s["metric_id"] == "arp": + s["row_filters"] = mine + svc._apply_sheets_to_row(row, sheets) + db.commit() + assert _builtin_sheet(svc.ensure_default_zte_status_template(db), "arp")["row_filters"] == mine diff --git a/tests/test_biz_migration_async.py b/tests/test_biz_migration_async.py new file mode 100644 index 0000000..7c60cf1 --- /dev/null +++ b/tests/test_biz_migration_async.py @@ -0,0 +1,284 @@ +"""H2/H3/H5: background evaluation, manual route tables, duplicate-key scope.""" +from __future__ import annotations + +import unittest +from unittest import mock + +from fastapi import HTTPException +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_migration import service as mig +from netx_api.models import BizMigrationBatch, BizMigrationRun, BizStateBatchCommand, BizStateMetricRow +import tests.test_biz_migration_evidence as evidence + + +class _Inline: + """threading.Thread stand-in: remember the target, run it on demand.""" + + started: list = [] + + def __init__(self, target=None, **_): + self.target = target + + def start(self): + _Inline.started.append(self.target) + + +class _Fixture(unittest.TestCase): + def setUp(self) -> None: + self.ctx = evidence.EvidencePersistTests("test_evaluate_persists_display_ab_and_evidence_with_command") + self.ctx.setUp() + self.db = self.ctx.db + self.factory = sessionmaker(bind=self.db.get_bind(), expire_on_commit=False) + self.session_patch = mock.patch("netx_api.db.SessionLocal", self.factory) + self.session_patch.start() + _Inline.started = [] + proj = self.ctx._create_project() + c = self.ctx + for bid, tid in (("bl_old", c.old_p.id), ("bl_new", c.new_p.id), ("cur_old", proj["old_hf_task_id"]), + ("cur_new", proj["new_hf_task_id"])): + c._mk_batch(bid, tid) + c._seed_if_row(batch_id="bl_old", task_id=c.old_p.id, ne_id="ne-old", command_id="c1", + raw_command="show interface brief", interface="GE-old") + c._seed_if_row(batch_id="cur_new", task_id=proj["new_hf_task_id"], ne_id="ne-new", command_id="c2", + raw_command="show interface brief", interface="xgei-new") + mig.patch_project(self.db, proj["id"], {"old_baseline_batch_id": "bl_old", "new_baseline_batch_id": "bl_new"}) + self.batch = mig.create_batch(self.db, proj["id"], {"batch_label": "w", "expect_set": {"ports": ["gei-old"]}}) + mig.patch_batch(self.db, self.batch["id"], {"status": "active"}) + self.project_id = proj["id"] + + def tearDown(self) -> None: + self.session_patch.stop() + self.ctx.tearDown() + + def run_background(self): + for target in list(_Inline.started): + target() + _Inline.started.clear() + + +class AsyncEvaluateTests(_Fixture): + def test_evaluate_returns_a_running_run_immediately_then_completes(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + started = mig.enqueue_evaluate(self.db, self.batch["id"]) + self.assertEqual(started["status"], "running") + self.assertEqual(self.db.get(BizMigrationRun, started["id"]).status, "running") + self.assertEqual(len(_Inline.started), 1) + self.run_background() + self.db.expire_all() + run = self.db.get(BizMigrationRun, started["id"]) + self.assertEqual(run.status, "success") + self.assertEqual(run.purpose, "manual") + self.assertIn("sheet_cards", run.summary_json) + self.assertEqual(self.db.query(BizMigrationRun).count(), 1) # the claim row was filled, not duplicated + + def test_second_request_while_one_runs_is_409(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + mig.enqueue_evaluate(self.db, self.batch["id"]) + with self.assertRaises(HTTPException) as exc: + mig.enqueue_evaluate(self.db, self.batch["id"]) + with self.assertRaises(HTTPException) as exc_finish: + mig.enqueue_finish(self.db, self.batch["id"]) + self.assertEqual(exc.exception.status_code, 409) + self.assertEqual(exc.exception.detail, "evaluation_in_progress") + self.assertEqual(exc_finish.exception.status_code, 409) + + def test_failure_is_recorded_on_the_run_and_releases_the_project(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + started = mig.enqueue_evaluate(self.db, self.batch["id"]) + with mock.patch.object(mig, "run_evaluate", side_effect=HTTPException(status_code=400, detail="boom")): + self.run_background() + self.db.expire_all() + run = self.db.get(BizMigrationRun, started["id"]) + self.assertEqual((run.status, run.message), ("failed", "boom")) + with mock.patch.object(mig.threading, "Thread", _Inline): + mig.enqueue_evaluate(self.db, self.batch["id"]) # no longer blocked + + def test_restart_cancels_orphaned_runs(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + started = mig.enqueue_evaluate(self.db, self.batch["id"]) + self.assertEqual(mig.recover_interrupted_evaluations_on_startup(self.db), 1) + self.db.expire_all() + run = self.db.get(BizMigrationRun, started["id"]) + self.assertEqual((run.status, run.message), ("cancelled", "interrupted_by_restart")) + # a cancelled claim must not be resurrected by a late worker + with self.assertRaises(HTTPException) as exc: + mig.run_evaluate(self.db, batch_id=self.batch["id"], _run_id=started["id"]) + self.assertEqual(exc.exception.status_code, 409) + + def test_background_evaluation_locks_the_project_only_for_the_final_write(self): + from sqlalchemy.orm import Query + + calls: list[str] = [] + original = Query.with_for_update + + def spy(self_, *a, **k): + calls.append("lock") + return original(self_, *a, **k) + + with mock.patch.object(mig.threading, "Thread", _Inline): + mig.enqueue_evaluate(self.db, self.batch["id"]) # claim = one short lock + with mock.patch.object(Query, "with_for_update", spy): + self.run_background() + self.assertEqual(len(calls), 1, "computation must be unlocked; only the write takes the project lock") + calls.clear() + with mock.patch.object(Query, "with_for_update", spy): + self.db.query(BizMigrationRun).delete() + self.db.commit() + mig.run_evaluate(self.db, batch_id=self.batch["id"]) # synchronous callers keep the old behaviour + self.assertGreaterEqual(len(calls), 1) + + def test_finish_runs_in_the_background_and_closes_the_wave_only_on_success(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + started = mig.enqueue_finish(self.db, self.batch["id"]) + self.assertEqual((started["status"], started["run"]["purpose"]), ("running", "acceptance")) + self.assertEqual(self.db.get(BizMigrationBatch, self.batch["id"]).status, "active") + self.run_background() + self.db.expire_all() + batch = self.db.get(BizMigrationBatch, self.batch["id"]) + self.assertEqual(batch.status, "review") + self.assertEqual(batch.accept_run_id, started["run_id"]) + self.assertEqual(self.db.get(BizMigrationRun, started["run_id"]).status, "success") + + def test_finish_failure_keeps_the_wave_operable(self): + with mock.patch.object(mig.threading, "Thread", _Inline): + started = mig.enqueue_finish(self.db, self.batch["id"]) + with mock.patch.object(mig, "run_evaluate", side_effect=HTTPException(status_code=400, detail="x")): + self.run_background() + self.db.expire_all() + self.assertEqual(self.db.get(BizMigrationBatch, self.batch["id"]).status, "active") + self.assertEqual(self.db.get(BizMigrationRun, started["run_id"]).status, "failed") + + def test_auto_monitor_stays_out_of_a_running_evaluation(self): + from netx_api.biz_migration import auto_monitor + + with mock.patch.object(mig.threading, "Thread", _Inline): + mig.enqueue_evaluate(self.db, self.batch["id"]) + with mock.patch.object(mig, "run_evaluate") as run_evaluate: + auto_monitor.try_auto_monitor_for_task(self.db, self.ctx.old_p.id) + run_evaluate.assert_not_called() + + +class ManualRouteAndDuplicateScopeTests(_Fixture): + """Share the fixture, add route sheets / duplicate rows to the compare template.""" + + def _add_sheet(self, sheet): + self.ctx.ct.metrics_json = [*self.ctx.ct.metrics_json, sheet] + self.ctx.mt.collect_metric_ids_json = [] + self.db.commit() + + def test_uncollected_route_table_is_manual_pending_and_does_not_fail_acceptance(self): + self._add_sheet({"sheet_id": "bgp_route", "title": "routes", "metric_id": "bgp_route", + "key_fields": ["network"], "iface_fields": [], "compare_fields": [], "row_filters": [], + "field_rules": []}) + out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance") + card = next(c for c in out["summary"]["sheet_cards"] if c["metric_id"] == "bgp_route") + self.assertTrue(card["manual_pending"]) + self.assertFalse(card.get("current_missing")) + self.assertIn("routes", out["summary"]["manual_pending_metrics"]) + self.assertNotIn("routes", out["summary"]["current_missing_metrics"]) + + def test_duplicate_keys_outside_the_wave_are_warnings_not_failures(self): + c = self.ctx + for i, cmd in enumerate(("d1", "d2")): + self.db.add(BizStateMetricRow(id=f"dup-{i}", batch_id="bl_old", metric_id="interface_brief", seq=10 + i, + data_json={"interface": "GE-other", "admin": "up", "phy": "up", "prot": "up"})) + self.db.commit() + out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance") + summary = out["summary"] + self.assertEqual(summary["duplicate_keys"], 0) + self.assertEqual(summary["duplicate_warnings"][0]["count"] >= 1, True) + + def test_duplicate_keys_inside_the_wave_still_fail(self): + for i in range(2): + self.db.add(BizStateMetricRow(id=f"dup-{i}", batch_id="bl_old", metric_id="interface_brief", seq=10 + i, + data_json={"interface": "GE-old", "admin": "up", "phy": "up", "prot": "up"})) + self.db.commit() + out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance") + self.assertGreaterEqual(out["summary"]["duplicate_keys"], 1) + self.assertFalse(out["summary"]["coverage_complete"]) + + +class BaselineCacheTests(_Fixture): + def test_second_evaluation_does_not_read_baseline_rows_again(self): + mig.BASELINE_ROWS.invalidate() + real = mig._load_metric_rows + loaded: list[str] = [] + + def spy(db, *, batch_id, metric_id, **kw): + loaded.append(batch_id) + return real(db, batch_id=batch_id, metric_id=metric_id, **kw) + + with mock.patch.object(mig, "_load_metric_rows", spy): + mig.run_evaluate(self.db, batch_id=self.batch["id"]) + first = list(loaded) + loaded.clear() + mig.run_evaluate(self.db, batch_id=self.batch["id"]) + self.assertIn("bl_old", first) + self.assertNotIn("bl_old", loaded) + self.assertNotIn("bl_new", loaded) + self.assertIn("cur_new", loaded) # current samples are always re-read + + def test_lru_is_bounded_by_total_rows_and_can_be_invalidated(self): + cache = mig.BaselineRowCache(max_rows=5) + rows = lambda n: [{"i": i} for i in range(n)] # noqa: E731 + cache.get_or_load("a", "m", lambda: rows(3)) + cache.get_or_load("b", "m", lambda: rows(3)) # evicts a (3 + 3 > 5) + calls: list[str] = [] + cache.get_or_load("a", "m", lambda: calls.append("a") or rows(3)) + self.assertEqual(calls, ["a"]) + cache.get_or_load("big", "m", lambda: rows(9)) # larger than the cap: never cached + cache.get_or_load("big", "m", lambda: calls.append("big") or rows(9)) + self.assertEqual(calls, ["a", "big"]) + cache.invalidate("a") + cache.get_or_load("a", "m", lambda: calls.append("a2") or rows(1)) + self.assertEqual(calls[-1], "a2") + + +class RedTicketResolvableTests(_Fixture): + def _diff(self, run_id, key, color, verdict): + from netx_api.models import BizMigrationDiff + + self.db.add(BizMigrationDiff(id=f"d-{run_id}-{key}", run_id=run_id, metric_id="interface_brief", color=color, + verdict=verdict, key_json={"match_old_key": key, "sheet_id": "interface_brief", + "key_str": key})) + + def test_ticket_for_a_key_that_turned_green_is_flagged_not_closed(self): + from netx_api.models import BizMigrationRedTicket + + pid = self.project_id + for key in ("gei-1", "gei-2"): + self.db.add(BizMigrationRedTicket( + id=f"t-{key}", project_id=pid, batch_id="old", run_id="r0", metric_id="interface_brief", + match_key_str=key, key_str=key, status="open", detail_json={"sheet_id": "interface_brief", "match_old_key": key})) + self.db.add(BizMigrationRun(id="r1", project_id=pid, batch_id=self.batch["id"], purpose="acceptance")) + self._diff("r1", "gei-1", "green", "migrated") + self._diff("r1", "gei-2", "red", "anomaly") + self.db.commit() + mig._persist_red_tickets_from_run(self.db, project_id=pid, batch_id=self.batch["id"], run_id="r1") + self.db.commit() + from netx_api.models import BizMigrationRedTicket as T + + flagged = {t.id: mig._red_ticket_to_dict(t) for t in self.db.query(T)} + self.assertTrue(flagged["t-gei-1"]["resolvable"]) + self.assertFalse(flagged["t-gei-2"]["resolvable"]) + self.assertEqual(flagged["t-gei-1"]["status"], "open") + + out = mig.resolve_resolvable_red_tickets(self.db, pid) + self.assertEqual(out["ids"], ["t-gei-1"]) + self.assertEqual(self.db.get(T, "t-gei-1").status, "resolved") + self.assertEqual(self.db.get(T, "t-gei-2").status, "open") + + +class UncoveredSliceTests(_Fixture): + def test_a_slice_one_side_never_collected_is_not_collected_not_lost(self): + sheet = {"sheet_id": "bgp_peer.ipv4", "metric_id": "bgp_peer", "key_fields": ["neighbor"], + "iface_fields": [], "compare_fields": ["state"], + "row_filters": [{"op": "eq", "field": "afi", "value": "ipv4"}], "field_rules": []} + for batch, profile in (("cur_old", "zte.bgp_ipv4_summary"), ("cur_new", "zte.bgp_vpnv4_summary")): + self.db.add(BizStateBatchCommand(id=f"slice-{batch}", batch_id=batch, metric_id="bgp_peer", + profile_id=profile, parse_status="ok", row_count=0, raw_line_count=0)) + self.db.commit() + self.assertEqual(mig._uncovered_slice_sides(self.db, sheet, "bgp_peer", "cur_old", "cur_new"), ["new"]) + self.assertEqual(mig._uncovered_slice_sides(self.db, {**sheet, "row_filters": []}, "bgp_peer", + "cur_old", "cur_new"), []) diff --git a/tests/test_biz_migration_hf_decouple.py b/tests/test_biz_migration_hf_decouple.py index cf9a575..8e34c77 100644 --- a/tests/test_biz_migration_hf_decouple.py +++ b/tests/test_biz_migration_hf_decouple.py @@ -144,16 +144,34 @@ class HfDecoupleFlowTests(unittest.TestCase): self.assertNotEqual(out["old_hf_task_id"], "old_p") self.assertNotEqual(out["new_hf_task_id"], "new_p") - def test_same_interval_metrics_are_separate_paired_tasks(self) -> None: + def test_same_interval_light_metrics_share_one_paired_task(self) -> None: out = self._create_project(collect_metric_ids=["interface_brief", "arp"]) old, new = out["old_hf_bindings"], out["new_hf_bindings"] - self.assertEqual(len(old), 2) - self.assertEqual({b["interval_sec"] for b in old}, {60}) - for o, n in zip(old, new): - self.assertEqual(len(o["metric_ids"]), 1) - self.assertTrue(o["collect_group_id"]) - self.assertEqual(o["collect_group_id"], n["collect_group_id"]) - self.assertEqual(o["metric_ids"], n["metric_ids"]) + self.assertEqual((len(old), len(new)), (1, 1)) + self.assertEqual(old[0]["metric_ids"], ["interface_brief", "arp"]) + self.assertEqual(old[0]["interval_sec"], 60) + self.assertTrue(old[0]["collect_group_id"]) + self.assertEqual(old[0]["collect_group_id"], new[0]["collect_group_id"]) + from netx_api.models import BizStateTaskItem + + items = self.db.query(BizStateTaskItem).filter(BizStateTaskItem.task_id == old[0]["task_id"]).count() + self.assertEqual(items, 2) # one SSH login runs both commands + + def test_different_interval_or_lane_or_manual_route_stay_apart(self) -> None: + out = self._create_project(collect_metric_ids=["interface_brief", "arp", "interface_detail", "ip_route"], + metric_interval_sec={"arp": 300}) + by_metrics = {tuple(b["metric_ids"]): b for b in out["old_hf_bindings"]} + self.assertEqual(set(by_metrics), {("interface_brief",), ("arp",), ("interface_detail",), ("ip_route",)}) + self.assertEqual(by_metrics[("arp",)]["interval_sec"], 300) + self.assertTrue(by_metrics[("ip_route",)]["manual_only"]) + self.assertEqual(len({b["collect_group_id"] for b in out["old_hf_bindings"]}), 4) + + def test_pair_lookup_works_for_a_metric_inside_a_merged_task(self) -> None: + out = self._create_project(collect_metric_ids=["interface_brief", "arp"]) + proj = self.db.get(BizMigrationProject, out["id"]) + self.assertEqual(mig._hf_task_id_for_metric(proj, "old", "arp"), out["old_hf_bindings"][0]["task_id"]) + self.assertEqual(mig._hf_task_id_for_metric(proj, "new", "interface_brief"), + out["new_hf_bindings"][0]["task_id"]) def test_dynamic_routes_manual_only_and_removal_cancels_queued_pairs(self) -> None: from netx_api.models import BizStateTaskItem, BizStateTaskItemBinding @@ -563,3 +581,77 @@ class HfDecoupleFlowTests(unittest.TestCase): if __name__ == "__main__": unittest.main() + + +class SliceUnionAndSameDeviceTests(HfDecoupleFlowTests): + """V2 (both devices collect the same slices) and H7 (same device on both sides).""" + + def _portrait_items(self, task_id, *profile_ids): + from netx_api.models import BizStateTaskItem + + for i, pid in enumerate(profile_ids): + self.db.add(BizStateTaskItem(id=f"{task_id}-{i}", task_id=task_id, source_profile_id=pid, + kind="catalog", enabled=True, sort_order=i)) + self.db.commit() + + def _hf_profiles(self, task_id): + from netx_api.models import BizStateTaskItem + + return sorted(r.source_profile_id for r in + self.db.query(BizStateTaskItem).filter(BizStateTaskItem.task_id == task_id)) + + def test_union_gives_the_device_that_lacks_a_slice_this_vendors_profile(self) -> None: + old = [{"source_profile_id": "zte.bgp_ipv4_summary", "bindings": []}, + {"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}] + new = [{"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}] + old2, new2 = mig._union_slice_items("bgp_peer", old, new, "zte", "zte") + self.assertEqual({i["source_profile_id"] for i in new2}, {"zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary"}) + self.assertEqual(old2, old) + + def test_missing_vendor_profile_is_a_clear_error_not_a_silent_gap(self) -> None: + old = [{"source_profile_id": "zte.bgp_ipv4_summary", "bindings": []}] + new = [{"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}] + with self.assertRaises(Exception) as ctx: + mig._union_slice_items("bgp_peer", old, new, "zte", "huawei") + self.assertEqual(ctx.exception.detail, "no_profile_for_slice:bgp_peer:afi=ipv4:huawei") + + def test_wholemetric_profiles_need_no_union(self) -> None: + item = [{"source_profile_id": "zte.interface_brief", "bindings": []}] + self.assertEqual(mig._union_slice_items("interface_brief", item, [], "zte", "zte"), (item, [])) + + def test_project_hf_tasks_collect_the_union_of_both_portraits(self) -> None: + self._portrait_items("old_p", "zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary") + self._portrait_items("new_p", "zte.bgp_vpnv4_summary") + out = self._create_project(collect_metric_ids=["bgp_peer"]) + want = ["zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary"] + self.assertEqual(self._hf_profiles(out["old_hf_bindings"][0]["task_id"]), want) + self.assertEqual(self._hf_profiles(out["new_hf_bindings"][0]["task_id"]), want) + + def test_same_device_project_uses_one_task_for_both_sides(self) -> None: + self.new_portrait.ne_id = self.old_portrait.ne_id + self.db.commit() + out = self._create_project(collect_metric_ids=["interface_brief", "arp"]) + old, new = out["old_hf_bindings"], out["new_hf_bindings"] + self.assertEqual(len(old), 1) + self.assertEqual(old[0]["task_id"], new[0]["task_id"]) + self.assertEqual(old[0]["collect_group_id"], new[0]["collect_group_id"]) + self.assertEqual(self.db.query(BizStateTask).filter(BizStateTask.purpose == mig.PURPOSE_CUTOVER_HF).count(), 1) + + def test_same_device_sample_is_that_devices_latest_finished_round(self) -> None: + self.new_portrait.ne_id = self.old_portrait.ne_id + self.db.commit() + out = self._create_project(collect_metric_ids=["interface_brief"]) + proj = self.db.get(BizMigrationProject, out["id"]) + tid = out["old_hf_bindings"][0]["task_id"] + self.assertEqual(mig._current_pair_for_metric(self.db, proj, "interface_brief")[2]["sample_issues"], + ["collect_pair_missing"]) + batch = self._mk_batch("solo-1", tid) + old, new, meta = mig._current_pair_for_metric(self.db, proj, "interface_brief") + self.assertIs(old, new) + self.assertEqual(old.id, batch.id) + self.assertEqual(meta["old_collect_status"], "success") + + def test_two_devices_are_still_rejected_when_they_are_in_fact_one(self) -> None: + # different NE ids but resolving to one task id must keep failing loudly + out = self._create_project(collect_metric_ids=["interface_brief"]) + self.assertNotEqual(out["old_hf_bindings"][0]["task_id"], out["new_hf_bindings"][0]["task_id"]) diff --git a/tests/test_biz_migration_integrity.py b/tests/test_biz_migration_integrity.py index 6f2070d..079b8e8 100644 --- a/tests/test_biz_migration_integrity.py +++ b/tests/test_biz_migration_integrity.py @@ -406,3 +406,21 @@ def test_diff_filters_apply_before_pagination_and_escape_wildcards(db): def test_invalid_correction_rules_fail_before_evaluation(condition): with pytest.raises(HTTPException): validate_monitor_rules({}, [{"metric_id": "arp", "success": [{"old_groups": [[condition]], "new": ["added"]}]}]) + + +@pytest.mark.parametrize("raw", [None, True, " v "]) +def test_odd_key_values_still_match_expected_key(raw): + """A6: diff-derived key strings must use the same form as the expect set.""" + from netx_api.biz_migration.monitor_templates import preset_override_for_metric + from netx_api.biz_state.compare_rules import scalar_text + + old = {"ip": "a", "vrf": raw, "mac": "m", "port": "old"} + new = {"ip": "a", "vrf": raw, "mac": "m", "port": "new"} + out = evaluate( + key_fields=["ip", "vrf"], compare_fields=["mac"], iface_fields=[], + old_baseline_rows=[old], old_current_rows=[], new_baseline_rows=[], new_current_rows=[new], + port_map={}, window_active=False, acceptance=True, + sheet_override=preset_override_for_metric("arp"), + expect=parse_expect_set({"items": [{"metric_id": "arp", "key": ["a", scalar_text(raw).strip()]}]}), + ) + assert [r["verdict"] for r in out["rows"]] == ["migrated"] diff --git a/tests/test_biz_migration_presets.py b/tests/test_biz_migration_presets.py new file mode 100644 index 0000000..15bfec3 --- /dev/null +++ b/tests/test_biz_migration_presets.py @@ -0,0 +1,156 @@ +"""Default monitor presets: port success condition and legacy upgrade (plan A4).""" +from copy import deepcopy + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_migration import monitor_templates as mt +from netx_api.biz_migration.evaluate import dual_verdict_ex +from netx_api.db import Base +from netx_api.models import BizCompareTemplate, BizMonitorTemplate + +NEW_UP = {"admin": "up", "phy": "up"} + +OLD_SIDES = { + "fibre_pulled_no_shutdown": ("changed", {"admin": "up", "phy": "down"}), + "shutdown_link_still_up": ("changed", {"admin": "down", "phy": "up"}), + "both_down": ("changed", {"admin": "down", "phy": "down"}), + "port_deleted": ("removed", {}), +} + + +def verdict(old_kind, old_row, new_row, *, acceptance, ov=None): + ov = ov or mt.preset_override_for_metric("interface_brief") + return dual_verdict_ex( + old_kind=old_kind, new_kind="changed", in_expect=True, + window_active=not acceptance, acceptance=acceptance, + old_status="none", new_status="up", + success_patterns=ov["success"], anomaly_patterns=ov["anomaly"], + old_row=old_row, new_row=new_row, + ) + + +@pytest.mark.parametrize("acceptance", [False, True], ids=["window", "acceptance"]) +@pytest.mark.parametrize("case", OLD_SIDES) +def test_old_port_out_of_service_with_new_up_is_migrated(case, acceptance): + kind, row = OLD_SIDES[case] + v, color, _hit = verdict(kind, row, NEW_UP, acceptance=acceptance) + assert (v, color) == ("migrated", "green") + + +@pytest.mark.parametrize("acceptance", [False, True], ids=["window", "acceptance"]) +def test_new_port_phy_down_is_never_green(acceptance): + v, color, _ = verdict("changed", {"admin": "down", "phy": "down"}, + {"admin": "up", "phy": "down"}, acceptance=acceptance) + assert color != "green" + + +def test_old_port_still_in_service_is_not_migrated(): + v, color, _ = verdict("unchanged", {"admin": "up", "phy": "up"}, NEW_UP, acceptance=True) + assert color != "green" + + +def test_legacy_form_still_requires_both_down(): + legacy = mt.preset_override_for_metric("interface_brief", legacy_port_and=True) + v, color, _ = verdict("changed", {"admin": "up", "phy": "down"}, NEW_UP, + acceptance=True, ov=legacy) + assert color != "green" # the pre-A4 bug this task fixes + + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + yield session + session.close() + engine.dispose() + + +def _template(db, overrides): + db.add(BizMonitorTemplate(id="m", name="m", compare_template_id="", sheet_overrides_json=overrides)) + db.commit() + + +def test_upgrade_replaces_exact_legacy_preset_only(db): + legacy = mt.preset_override_for_metric("interface_brief", legacy_port_and=True) + arp = mt.preset_override_for_metric("arp") + _template(db, [legacy, arp]) + assert mt.upgrade_legacy_port_overrides(db) == 1 + stored = db.get(BizMonitorTemplate, "m").sheet_overrides_json + assert stored[0] == mt.preset_override_for_metric("interface_brief") + assert stored[1] == arp + assert mt.upgrade_legacy_port_overrides(db) == 0 # idempotent + + +def test_customised_override_is_kept_and_warned(db): + custom = mt.preset_override_for_metric("interface_brief", legacy_port_and=True) + custom["down_values"] = ["down", "lowerlayerdown"] + _template(db, [deepcopy(custom)]) + assert mt.upgrade_legacy_port_overrides(db) == 0 + assert db.get(BizMonitorTemplate, "m").sheet_overrides_json == [custom] + db.add(BizCompareTemplate(id="ct", name="ct", metrics_json=[ + {"sheet_id": "interface_brief", "metric_id": "interface_brief", + "key_fields": ["interface"], "compare_fields": ["admin"]}])) + db.commit() + warnings = mt.validate_sheet_overrides(db, compare_template_id="ct", sheet_overrides=[custom]) + assert {"sheet_id": "", "metric_id": "interface_brief", + "reason": "port_success_legacy_and_semantics"} in warnings + assert not any(w["reason"] == "port_success_legacy_and_semantics" for w in + mt.validate_sheet_overrides(db, compare_template_id="ct", sheet_overrides=[ + mt.preset_override_for_metric("interface_brief")])) + + +def test_key_tuple_is_the_single_canonical_key_form(): + from netx_api.biz_state.compare_engine import key_tuple + + assert key_tuple({"a": None, "b": True, "c": " x "}, ["a", "b", "c", "missing"]) == ("", "true", "x", "") + + +# ---- H8: scope_by_port_map is declared, not guessed from the metric name ---------------------- + +def _scoped(override): + from netx_api.biz_migration.evaluate import evaluate_metric_dual, parse_expect_set + + return evaluate_metric_dual( + metric_id="custom_t", key_fields=["ifname"], iface_fields=["ifname"], compare_fields=[], + old_baseline_rows=[{"ifname": "a"}], old_current_rows=[], new_baseline_rows=[], + new_current_rows=[{"ifname": "b"}], port_map={"a": "b"}, expect=parse_expect_set({}), + window_active=True, sheet_override=override, allow_mapping_scope=True, + ) + + +def test_a_port_map_scopes_an_interface_sheet_unless_the_override_says_otherwise(): + assert _scoped({})["progress_total"] == 1 + assert _scoped({"scope_by_port_map": True})["progress_total"] == 1 + out = _scoped({"scope_by_port_map": False}) + assert out["progress_total"] == 0 and out["effective_scope_by_port_map"] is False + + +def test_presets_turn_scoping_off_for_bgp_and_pseudowire_only(): + assert mt.preset_override_for_metric("bgp_peer")["scope_by_port_map"] is False + assert mt.preset_override_for_metric("l2vpn_pw")["scope_by_port_map"] is False + assert "scope_by_port_map" not in mt.preset_override_for_metric("interface_brief") + assert "scope_by_port_map" not in mt.preset_override_for_metric("arp") + + +def test_stored_overrides_get_the_old_name_based_default_written_down(db): + _template(db, [{"metric_id": "bgp_peer", "sheet_id": "bgp_peer.vpnv4"}, {"metric_id": "arp"}, + {"metric_id": "bgp_peer", "sheet_id": "x", "scope_by_port_map": True}]) + assert mt.fill_scope_by_port_map(db) == 1 + stored = db.get(BizMonitorTemplate, "m").sheet_overrides_json + assert stored[0]["scope_by_port_map"] is False + assert "scope_by_port_map" not in stored[1] + assert stored[2]["scope_by_port_map"] is True # an explicit choice is kept + assert mt.fill_scope_by_port_map(db) == 0 + + +def test_scope_flag_must_be_boolean(): + from fastapi import HTTPException + + from netx_api.biz_migration.validation import validate_monitor_rules + + with pytest.raises(HTTPException): + validate_monitor_rules({}, [{"metric_id": "arp", "scope_by_port_map": "no"}]) + validate_monitor_rules({}, [{"metric_id": "arp", "scope_by_port_map": False}]) diff --git a/tests/test_biz_migration_retention.py b/tests/test_biz_migration_retention.py new file mode 100644 index 0000000..f021f6f --- /dev/null +++ b/tests/test_biz_migration_retention.py @@ -0,0 +1,96 @@ +"""H1: bounded auto-run history and run-aware batch protection.""" +from datetime import timedelta + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_state.retention import protected_batch_map, purge_auto_runs +from netx_api.db import Base +from netx_api.models import ( + BizCompareDiff, BizCompareJob, BizCompareRun, BizMigrationBatch, BizMigrationDiff, BizMigrationRedTicket, + BizMigrationRun, BizStateBatch, BizStateTask, +) +from netx_api.timeutil import utcnow_naive + + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + yield session + session.close() + engine.dispose() + + +def _runs(db, n, *, purpose="auto", batch="wave", prefix="r"): + base = utcnow_naive() - timedelta(hours=10) + for i in range(n): + db.add(BizMigrationRun(id=f"{prefix}{i}", project_id="p", batch_id=batch, purpose=purpose, + old_batch_id=f"o-{prefix}{i}", new_batch_id=f"n-{prefix}{i}", + created_at=base + timedelta(minutes=i), summary_json={})) + db.add(BizMigrationDiff(id=f"d-{prefix}{i}", run_id=f"{prefix}{i}", metric_id="m")) + db.commit() + + +def test_only_the_newest_auto_runs_survive_and_acceptance_runs_stay(db): + _runs(db, 400) + _runs(db, 1, purpose="acceptance", prefix="acc") + out = purge_auto_runs(db, migration_keep=300) + assert out["migration_runs"] == 100 + assert db.query(BizMigrationRun).count() == 301 + assert db.get(BizMigrationRun, "acc0") is not None + assert db.get(BizMigrationRun, "r399") is not None and db.get(BizMigrationRun, "r99") is None + assert db.query(BizMigrationDiff).count() == 301 # diffs follow their run + + +def test_runs_pinned_by_red_tickets_or_acceptance_are_never_dropped(db): + _runs(db, 12) + db.add(BizMigrationRedTicket(id="t", project_id="p", batch_id="wave", run_id="r0")) + db.add(BizMigrationBatch(id="wave", project_id="p", accept_run_id="r1")) + db.commit() + purge_auto_runs(db, migration_keep=5) + assert db.get(BizMigrationRun, "r0") is not None and db.get(BizMigrationRun, "r1") is not None + assert db.get(BizMigrationRun, "r2") is None and db.query(BizMigrationRun).count() == 7 + + +def test_keep_is_per_wave(db): + _runs(db, 6, batch="w1", prefix="a") + _runs(db, 6, batch="w2", prefix="b") + assert purge_auto_runs(db, migration_keep=4)["migration_runs"] == 4 + assert db.query(BizMigrationRun).filter_by(batch_id="w1").count() == 4 + + +def test_compare_keeps_manual_runs_and_trims_auto_runs_of_auto_jobs(db): + db.add(BizCompareJob(id="job", name="j", mode="auto")) + base = utcnow_naive() - timedelta(hours=5) + for i in range(8): + db.add(BizCompareRun(id=f"c{i}", job_id="job", status="success", created_at=base + timedelta(minutes=i), + summary_json={"purpose": "auto" if i != 0 else "manual"})) + db.add(BizCompareDiff(id=f"cd{i}", run_id=f"c{i}", metric_id="m")) + db.commit() + out = purge_auto_runs(db, compare_keep=3) + assert out["compare_runs"] == 4 # c1..c4; c0 is manual and survives, c5..c7 are the newest + assert {r.id for r in db.query(BizCompareRun)} == {"c0", "c5", "c6", "c7"} + assert db.query(BizCompareDiff).count() == 4 + + +def _batch(db, bid, task="t"): + db.add(BizStateBatch(id=bid, task_id=task, status="success")) + + +def test_protection_follows_surviving_runs_and_all_metric_batches(db): + db.add(BizStateTask(id="t", source="managed", ne_id="n")) + for bid in ("o-r0", "n-r0", "hf-a", "hf-b", "o-r1", "n-r1", "o-acc0", "n-acc0"): + _batch(db, bid) + db.commit() + _runs(db, 2) + _runs(db, 1, purpose="acceptance", prefix="acc") + acc = db.get(BizMigrationRun, "acc0") + acc.summary_json = {"metric_batches": {"arp": {"old_batch_id": "hf-a", "new_batch_id": "hf-b"}}} + db.commit() + reasons = protected_batch_map(db) + assert "hf-a" in reasons and "hf-b" in reasons # beyond the first binding + purge_auto_runs(db, migration_keep=1) # r0 goes away + assert "o-r0" not in protected_batch_map(db) and "o-r1" in protected_batch_map(db) diff --git a/tests/test_biz_state_claim.py b/tests/test_biz_state_claim.py index 12beb4f..8224ac5 100644 --- a/tests/test_biz_state_claim.py +++ b/tests/test_biz_state_claim.py @@ -95,7 +95,8 @@ class BizStateClaimTests(unittest.TestCase): self.assertEqual(r2.get("reason"), "already_collecting") def test_claim_ne_mutex_skips_same_ne(self) -> None: - with patch.object(claim_mod, "max_concurrent_tasks", return_value=10): + # Session cap 1 reproduces the classic "one collection per NE" mutex. + with patch.object(claim_mod, "max_concurrent_tasks", return_value=10), patch.object(claim_mod, "max_sessions_per_ne", return_value=1): ra = claim_mod.enqueue_collect("ta", manual=True) rb = claim_mod.enqueue_collect("tb", manual=True) rc = claim_mod.enqueue_collect("tc", manual=True) @@ -111,6 +112,47 @@ class BizStateClaimTests(unittest.TestCase): self.assertEqual(self.db.get(BizStateBatch, rb["batch_id"]).status, "queued") self.assertEqual(self.db.get(BizStateBatch, rc["batch_id"]).status, "running") + def test_a_ne_runs_up_to_its_session_cap_and_no_more(self) -> None: + self.db.add_all([BizStateTask(id=f"x{i}", source="managed", ne_id="ne-a", status="running") for i in range(3)]) + for i in range(3): + self.db.add(BizStateTaskItem(id=uuid4().hex, task_id=f"x{i}", source_profile_id="zte.arp", + kind="catalog", enabled=True)) + self.db.commit() + with patch.object(claim_mod, "max_concurrent_tasks", return_value=10), patch.object(claim_mod, "max_sessions_per_ne", return_value=2): + queued = [claim_mod.enqueue_collect(f"x{i}", manual=True)["batch_id"] for i in range(3)] + first = claim_mod.claim_queued_batches(10) + again = claim_mod.claim_queued_batches(10) + self.assertEqual(len(first), 2) # three batches queued for ne-a, only two sessions allowed + self.assertEqual(again, []) + self.db.expire_all() + states = sorted(self.db.get(BizStateBatch, b).status for b in queued) + self.assertEqual(states, ["queued", "running", "running"]) + + def test_default_session_cap_is_two(self) -> None: + self.assertEqual(claim_mod.max_sessions_per_ne(), 2) + + def test_single_task_group_is_a_valid_round_for_a_same_device_cutover(self) -> None: + task = self.db.get(BizStateTask, "ta") + task.collect_group_id = "solo" + self.db.commit() + with patch.object(claim_mod, "max_concurrent_tasks", return_value=10): + r = claim_mod.enqueue_collect("ta", manual=True) + self.assertTrue(r["queued"], r) + claimed = claim_mod.claim_queued_batches(10) + self.assertEqual([c["task_id"] for c in claimed], ["ta"]) + + def test_a_pair_needs_both_sides_before_it_is_claimed(self) -> None: + for tid, group in (("ta", "pair"), ("tc", "pair")): + self.db.get(BizStateTask, tid).collect_group_id = group + self.db.commit() + with patch.object(claim_mod, "max_concurrent_tasks", return_value=10): + r = claim_mod.enqueue_collect("ta", manual=True) + self.assertTrue(r["queued"], r) + self.db.expire_all() + self.db.query(BizStateBatch).filter(BizStateBatch.task_id == "tc").delete() + self.db.commit() + self.assertEqual(claim_mod.claim_queued_batches(10), []) + def test_claim_respects_slot_ceiling(self) -> None: with patch.object(claim_mod, "max_concurrent_tasks", return_value=1): claim_mod.enqueue_collect("ta", manual=True) diff --git a/tests/test_biz_state_compare_snapshot.py b/tests/test_biz_state_compare_snapshot.py index 3736fa8..a531e0e 100644 --- a/tests/test_biz_state_compare_snapshot.py +++ b/tests/test_biz_state_compare_snapshot.py @@ -90,6 +90,33 @@ class CompareSnapshotTests(unittest.TestCase): self.assertEqual(detail["sheets"][0]["compare_fields"], ["state"]) self.assertEqual(self.run.summary_json["config_snapshot"], original) + def _use_raw_port_map(self, version: int): + # Map written exactly as the device prints it (not the normalized form). + self.pair.before_if, self.pair.after_if = "GE-1", "GE-2" + self.db.commit() + self.run = _create_running_run( + self.db, job=self.job, tpl=self.tpl, before_batch_id="before", + after_batch_id="after", sheets_cfg=template_metrics(self.tpl), + ) + snap = deepcopy(self.run.summary_json) + snap["config_snapshot"]["version"] = version + self.run.summary_json = snap + self.db.commit() + + def test_v2_run_normalizes_raw_port_map_names(self): + self._use_raw_port_map(2) + detail = self.execute() + self.assertEqual(detail["summary"]["unchanged"], 1) + self.assertEqual(detail["summary"]["added"], 0) + self.assertEqual(detail["summary"]["removed"], 0) + + def test_v1_run_keeps_legacy_port_map_matching(self): + self._use_raw_port_map(1) + detail = self.execute() + self.assertEqual(detail["summary"]["unchanged"], 0) + self.assertEqual(detail["summary"]["added"], 1) + self.assertEqual(detail["summary"]["removed"], 1) + def test_compact_hydration_restores_original_normalized_values(self): self.execute() self.edit_current_settings() @@ -98,6 +125,15 @@ class CompareSnapshotTests(unittest.TestCase): self.assertEqual(diff["mapped_before"]["interface"], "gei-2.100") self.assertEqual(diff["after"]["interface"], "gei-2.100") + def test_diff_rows_expose_the_before_side_name_as_printed(self): + self.execute() + diff = list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged")["items"][0] + self.assertEqual(diff["key"]["interface"], "gei-2.100") + self.assertEqual(diff["raw_key_before"], {"id": "1", "interface": "GE-1.100"}) + with zipfile.ZipFile(io.BytesIO(export_run_zip(self.db, self.run.id))) as archive: + rows = list(csv.DictReader(io.StringIO(archive.read("tables/sheet.csv").decode("utf-8-sig")))) + self.assertEqual(rows[0]["interface__raw_before"], "GE-1.100") + def test_live_search_does_not_inherit_new_filters_or_transforms(self): with patch("netx_api.biz_state.compare_service._UNCHANGED_SAMPLE_MAX", 0): self.execute() @@ -132,7 +168,7 @@ class CompareSnapshotTests(unittest.TestCase): self.edit_current_settings() detail = get_run(self.db, self.run.id) self.assertEqual(detail["job_name"], "Original job") - self.assertEqual(detail["config_snapshot_version"], 1) + self.assertEqual(detail["config_snapshot_version"], 2) self.assertEqual(detail["template"]["iface_normalize_rules"], [{"from": "GE", "to": "gei"}]) self.assertNotIn("config_snapshot", detail["summary"]) detail["template"]["iface_normalize_rules"].clear() diff --git a/tests/test_biz_state_paired_sampling.py b/tests/test_biz_state_paired_sampling.py index 1594824..22f6923 100644 --- a/tests/test_biz_state_paired_sampling.py +++ b/tests/test_biz_state_paired_sampling.py @@ -65,7 +65,9 @@ def test_pair_claim_requires_two_slots(db): assert len(claim.claim_queued_batches(2)) == 2 -def test_busy_device_keeps_both_queued_and_no_ssh_overlap(db): +def test_busy_device_keeps_both_queued_and_no_ssh_overlap(db, monkeypatch): + # These cases are about one collection per device; the cap defaults to 2. + monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1) db.add(BizStateBatch(id="portrait", task_id="other", ne_id="old", status="running")) db.commit() claim.enqueue_collect("old") @@ -125,7 +127,9 @@ def test_inline_pair_uses_two_workers_without_waiting_for_one_side(db, monkeypat assert set(synchronized) == {"old", "new"} -def test_faster_metrics_claim_before_manual_route_round(db): +def test_faster_metrics_claim_before_manual_route_round(db, monkeypatch): + # These cases are about one collection per device; the cap defaults to 2. + monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1) for tid in ("ro", "rn"): db.add(BizStateTask(id=tid, ne_id="old" if tid == "ro" else "new", status="paused", collect_group_id="routes", collect_manual_only=True)) @@ -137,7 +141,9 @@ def test_faster_metrics_claim_before_manual_route_round(db): assert {job["task_id"] for job in jobs} == {"old", "new"} -def test_manual_route_round_cannot_starve_behind_continuous_fast_metrics(db): +def test_manual_route_round_cannot_starve_behind_continuous_fast_metrics(db, monkeypatch): + # These cases are about one collection per device; the cap defaults to 2. + monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1) for tid in ("ro", "rn"): db.add(BizStateTask(id=tid, ne_id="old" if tid == "ro" else "new", status="paused", collect_group_id="routes", collect_manual_only=True)) diff --git a/tests/test_compare_preflight.py b/tests/test_compare_preflight.py new file mode 100644 index 0000000..408b380 --- /dev/null +++ b/tests/test_compare_preflight.py @@ -0,0 +1,167 @@ +"""Mapping pre-flight (B5), N:1 reporting (B7) and command-slice coverage (V2).""" +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_migration import service as mig +from netx_api.biz_state import compare_service as svc +from netx_api.biz_state.compare_quality import classify_sheet, covers_slice, sheet_slice +from netx_api.db import Base +from netx_api.models import ( + BizCompareTemplate, BizMigrationProject, BizMonitorTemplate, BizPortMapping, BizPortMappingRow, + BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask, +) + + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + yield session + session.close() + engine.dispose() + + +SHEET = {"sheet_id": "ifb", "metric_id": "iface_t", "key_fields": ["interface"], "iface_fields": ["interface"], + "compare_fields": ["state"], "field_rules": [], "row_filters": []} + + +def _batches(db, before_ifaces, after_ifaces, *, receipts=True): + db.add_all([BizStateTask(id="t1", source="managed", ne_id="a", vendor="ZTE"), + BizStateTask(id="t2", source="managed", ne_id="b", vendor="ZTE"), + BizStateBatch(id="b1", task_id="t1", status="success"), + BizStateBatch(id="b2", task_id="t2", status="success"), + BizCompareTemplate(id="tpl", name="t", metrics_json=[SHEET], + iface_normalize_json=[{"from": "GE", "to": "gei"}])]) + for batch, names in (("b1", before_ifaces), ("b2", after_ifaces)): + for i, name in enumerate(names): + db.add(BizStateMetricRow(id=f"{batch}-{i}", batch_id=batch, metric_id="iface_t", seq=i, + data_json={"interface": name, "state": "up"})) + if receipts: + db.add(BizStateBatchCommand(id=f"c-{batch}", batch_id=batch, metric_id="iface_t", + profile_id="zte.x", parse_status="ok", row_count=len(names), + raw_line_count=len(names) + 3)) + db.commit() + + +def _preflight(db, **extra): + return svc.preflight_compare(db, {"template_id": "tpl", "before_batch_id": "b1", "after_batch_id": "b2", **extra}) + + +def test_exact_parent_same_name_and_suspect_are_classified(db): + _batches(db, ["GE0/1/0/1", "GE0/1/0/2.100", "GE0/1/0/3", "GE0/1/0/9"], + ["gei0/2/0/1", "gei0/2/0/2.100", "gei0/1/0/3"]) + report = _preflight(db, mapping_rows=[{"before_if": "GE0/1/0/1", "after_if": "GE0/2/0/1"}, + {"before_if": "GE0/1/0/2", "after_if": "GE0/2/0/2"}]) + (sheet,) = report["sheets"] + assert sheet["mapped_exact"] == ["gei0/1/0/1"] + assert sheet["mapped_by_parent"] == ["gei0/1/0/2.100"] + assert sheet["identity_same_name"] == ["gei0/1/0/3"] + assert sheet["suspect_unmapped"] == ["gei0/1/0/9"] + assert report["coverage"] == {"mapped": 2, "suspect_unmapped": 1, "identity_same_name": 1, "rate": 66.7} + assert report["verdict"] == "risk" and "suspect_unmapped" in report["risk_reasons"] + + +def test_unsaved_rows_are_enough_and_clean_map_is_ready(db): + _batches(db, ["GE0/1"], ["gei0/2"]) + report = _preflight(db, mapping_rows=[{"before_if": "GE0/1", "after_if": "GE0/2"}]) + assert report["verdict"] == "ready" and report["coverage"]["rate"] == 100.0 + assert report["mapping"]["unused_map_keys"] == [] + + +def test_unused_keys_missing_targets_and_n_to_1_are_listed(db): + _batches(db, ["GE0/1", "GE0/2"], ["gei0/9"]) + report = _preflight(db, mapping_rows=[ + {"before_if": "GE0/1", "after_if": "GE0/9"}, {"before_if": "GE0/2", "after_if": "GE0/9"}, + {"before_if": "GE0/77", "after_if": "GE0/88"}]) + m = report["mapping"] + assert m["n_to_1_targets"] == [{"target": "gei0/9", "sources": ["gei0/1", "gei0/2"]}] + assert m["unused_map_keys"] == ["gei0/77"] + assert m["map_targets_missing_on_after"] == ["gei0/88"] + assert report["verdict"] == "risk" + + +def test_duplicate_before_rows_are_reported(db): + _batches(db, ["GE0/1"], ["gei0/2"]) + report = _preflight(db, mapping_rows=[{"before_if": "GE0/1", "after_if": "GE0/2"}, + {"before_if": "GE0/1", "after_if": "GE0/3"}]) + assert report["mapping"]["duplicate_before"][0]["dropped_to"] == "GE0/3" + assert report["verdict"] == "risk" + + +def test_saved_mapping_id_is_used_when_no_rows_given(db): + _batches(db, ["GE0/1"], ["gei0/2"]) + db.add_all([BizPortMapping(id="m", name="m"), + BizPortMappingRow(mapping_id="m", before_if="GE0/1", after_if="GE0/2")]) + db.commit() + assert _preflight(db, mapping_id="m")["coverage"]["mapped"] == 1 + + +def test_missing_batch_is_404(db): + _batches(db, ["GE0/1"], ["gei0/2"]) + with pytest.raises(HTTPException) as exc: + svc.preflight_compare(db, {"template_id": "tpl", "before_batch_id": "nope", "after_batch_id": "b2"}) + assert exc.value.status_code == 404 + + +def test_n_to_1_is_rejected_by_monitoring_with_the_colliding_rows(): + with pytest.raises(HTTPException) as exc: + mig._reject_ambiguous_mapping({"a": "x", "b": "x", "c": "y"}) + assert exc.value.status_code == 400 + assert exc.value.detail["code"] == "mapping_target_ambiguous" + assert exc.value.detail["n_to_1"] == [{"target": "x", "sources": ["a", "b"]}] + assert "a, b -> x" in exc.value.detail["message"] + mig._reject_ambiguous_mapping({"a": "x", "b": "y"}) + + +def test_project_preflight_uses_the_portrait_baselines(db): + _batches(db, ["GE0/1", "GE0/5"], ["gei0/2"]) + db.add_all([BizPortMapping(id="m", name="m"), + BizPortMappingRow(mapping_id="m", before_if="GE0/1", after_if="GE0/2"), + BizMonitorTemplate(id="mt", name="mt", compare_template_id="tpl"), + BizMigrationProject(id="p", name="p", mapping_id="m", monitor_template_id="mt", old_task_id="t1", + new_task_id="t2", old_baseline_batch_id="b1", new_baseline_batch_id="b2")]) + db.commit() + report = mig.preflight_project(db, "p") + assert report["sheets"][0]["suspect_unmapped"] == ["gei0/5"] + assert report["verdict"] == "risk" + + +# ---- V2: slices --------------------------------------------------------------------------- + +def receipts(*profile_ids, status="ok"): + return [{"profile_id": p, "parse_status": status, "row_count": 1, "raw_line_count": 4} for p in profile_ids] + + +def test_sheet_slice_reads_only_top_level_eq_filters(): + s = {"row_filters": [{"op": "eq", "field": "afi", "value": "IPv4"}, + {"op": "contains", "field": "x", "value": "y"}, + {"any": [{"op": "eq", "field": "z", "value": "1"}]}]} + assert sheet_slice(s) == {"afi": "ipv4"} + + +def test_missing_afi_command_means_not_collected_not_lost(): + only_vpnv4 = receipts("zte.bgp_vpnv4_summary") + both = receipts("zte.bgp_vpnv4_summary", "zte.bgp_ipv4_summary") + state, reasons = classify_sheet(before=both, after=only_vpnv4, before_rows=3, after_rows=0, + wanted_slice={"afi": "ipv4"}) + assert state == "not_collected" and reasons == ["after:slice_not_collected"] + ok, _ = classify_sheet(before=both, after=both, before_rows=3, after_rows=3, wanted_slice={"afi": "ipv4"}) + assert ok == "ok" + + +def test_command_without_declared_slice_covers_everything(): + assert covers_slice(receipts("custom.unknown"), {"afi": "ipv4"}) + assert not covers_slice(receipts("zte.bgp_vpnv6_summary"), {"afi": "ipv4"}) + assert not covers_slice(receipts("zte.bgp_ipv4_summary", status="failed"), {"afi": "ipv4"}) + + +def test_command_coverage_compares_what_commands_produce(db): + _batches(db, ["GE0/1"], ["gei0/2"]) + db.add(BizStateBatchCommand(id="x", batch_id="b1", metric_id="bgp_peer", profile_id="zte.bgp_ipv4_summary", + parse_status="ok")) + db.commit() + cov = _preflight(db, mapping_rows=[])["command_coverage"] + assert cov["only_before"] == ["bgp_peer{afi=ipv4}"] diff --git a/tests/test_compare_quality.py b/tests/test_compare_quality.py new file mode 100644 index 0000000..c6079a4 --- /dev/null +++ b/tests/test_compare_quality.py @@ -0,0 +1,163 @@ +"""Sheet data quality (V1) and directional compare modes (V3).""" +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_state import compare_service as svc +from netx_api.biz_state.compare_rules import explain_diff, values_equal +from netx_api.biz_state.identity_pipeline import build_pipeline +from netx_api.db import Base +from netx_api.models import BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask + +SHEET = {"sheet_id": "peers", "metric_id": "bgp_peer", "key_fields": ["neighbor"], "iface_fields": [], + "compare_fields": ["state"], "field_rules": [], "row_filters": []} + + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + session.add_all([BizStateTask(id="t", source="managed", ne_id="n"), + BizStateBatch(id="b1", task_id="t", status="success"), + BizStateBatch(id="b2", task_id="t", status="success")]) + session.commit() + yield session + session.close() + engine.dispose() + + +def collect(db, batch, rows, *, lines=None, status="ok", command=True): + if command: + db.add(BizStateBatchCommand(id=f"c-{batch}", batch_id=batch, metric_id="bgp_peer", profile_id="zte.x", + parse_status=status, row_count=len(rows), + raw_line_count=len(rows) + 2 if lines is None else lines)) + for i, peer in enumerate(rows): + db.add(BizStateMetricRow(id=f"{batch}-{i}", batch_id=batch, metric_id="bgp_peer", seq=i, + data_json={"neighbor": peer, "state": "up"})) + db.commit() + + +def run(db): + return svc._run_sheet(db, sheet=SHEET, before_batch_id="b1", after_batch_id="b2", + pipeline=build_pipeline(SHEET, iface_rules=[], port_map={}))["summary"] + + +def enrich(summary): + sheet = {"sheet_id": "peers", "metric_id": "bgp_peer", "status": "done", "summary": summary} + return svc._enrich_summary({k: summary.get(k, 0) for k in + ("added", "removed", "changed", "unchanged", "before_count", "after_count")}, [sheet]) + + +def test_empty_against_empty_is_no_data_not_a_hundred_percent(db): + collect(db, "b1", [], lines=0) + collect(db, "b2", [], lines=0) + summary = run(db) + assert summary["data_state"] == "no_data" + out = enrich(summary) + assert out["pass_rate"] is None and out["sheet_cards"][0]["pass_rate"] is None + assert out["ok"] is None + + +def test_unparsed_output_is_suspect_parse_and_leaves_the_verdict_open(db): + collect(db, "b1", [f"p{i}" for i in range(10)]) + collect(db, "b2", [], lines=40) # 40 CLI lines, 0 rows parsed + summary = run(db) + assert summary["data_state"] == "suspect_parse" + assert summary["data_state_reasons"] == ["after:parsed_0_of_40_lines"] + out = enrich(summary) + assert out["sheet_cards"][0]["removed"] == 10 # still shown + assert out["ok"] is None and out["suspect_sheets"] == 1 + + +def test_side_without_any_command_is_not_collected_and_leaves_the_totals(db): + collect(db, "b1", ["a", "b"]) + db.add(BizStateBatchCommand(id="other", batch_id="b2", metric_id="other_metric", profile_id="zte.y", + parse_status="ok", row_count=1, raw_line_count=1)) + db.commit() + summary = run(db) + assert summary["data_state"] == "not_collected" + out = enrich(summary) + assert out["sheet_cards"][0]["pass_rate"] is None + assert out["fail_count"] == 0 and out["ok"] is None and out["unjudged_sheets"] == 1 + + +def test_batches_without_receipts_are_trusted(db): + collect(db, "b1", ["a"], command=False) + collect(db, "b2", ["a"], command=False) + summary = run(db) + assert summary["data_state"] == "ok" + assert enrich(summary)["pass_rate"] == 100.0 + + +def test_normal_comparison_keeps_the_verdict(db): + collect(db, "b1", ["a", "b"]) + collect(db, "b2", ["a"]) + out = enrich(run(db)) + assert out["ok"] is False and out["fail_count"] == 1 and out["pass_rate"] == 50.0 + + +# ---- V3 ----------------------------------------------------------------------------------- + +NOT_WORSE = {"compare": "not_worse", "rank": ["up", "down"]} + + +def test_not_worse_accepts_recovery_and_rejects_degradation(): + assert values_equal("down", "up", rule=NOT_WORSE) + assert values_equal("up", "up", rule=NOT_WORSE) + assert not values_equal("up", "down", rule=NOT_WORSE) + assert explain_diff("up", "down", rule=NOT_WORSE) == "worse: up→down" + + +def test_not_worse_falls_back_to_equality_for_unranked_values(): + assert not values_equal("up", "flapping", rule=NOT_WORSE) + assert values_equal("flapping", "flapping", rule=NOT_WORSE) + + +def test_numeric_min_allows_growth_and_bounded_loss(): + rule = {"compare": "numeric_min", "tolerance": 5} + assert values_equal("1000", "1200", rule=rule) + assert values_equal("1000", "960", rule=rule) + assert not values_equal("1000", "900", rule=rule) + assert explain_diff("1000", "900", rule=rule) == "drop 10% > 5%" + + +def test_numeric_max_is_the_mirror_image(): + rule = {"compare": "numeric_max", "tolerance": 10} + assert values_equal("100", "60", rule=rule) + assert values_equal("100", "110", rule=rule) + assert not values_equal("100", "150", rule=rule) + assert explain_diff("100", "150", rule=rule) == "rise 50% > 10%" + + +def test_directional_modes_pass_validation_and_keep_their_rank(): + from netx_api.biz_state.compare_validation import validate_template_body + + sheet = {"metric_id": "m", "key_fields": ["k"], "compare_fields": ["s"], + "field_rules": [{"field": "s", "compare": "not_worse", "rank": ["up", "down"]}]} + validate_template_body({"metrics": [sheet]}) + normalized = svc._normalize_sheet(sheet) + assert normalized["field_rules"][0]["rank"] == ["up", "down"] + with pytest.raises(Exception): + validate_template_body({"metrics": [{**sheet, "field_rules": [{"field": "s", "compare": "not_worse"}]}]}) + + +def test_directional_compare_runs_end_to_end_and_blocks_sql(db): + from tests.test_identity_pipeline import pg_db + from netx_api.biz_state.compare_engine import compare_rows + from netx_api.biz_state.compare_sql import sql_compare_skip_reason + + sheet = {**SHEET, "field_rules": [{"field": "state", **NOT_WORSE}]} + out = compare_rows(before_rows=[{"neighbor": "a", "state": "down"}, {"neighbor": "b", "state": "up"}], + after_rows=[{"neighbor": "a", "state": "up"}, {"neighbor": "b", "state": "down"}], + key_fields=["neighbor"], iface_fields=[], compare_fields=["state"], + field_rules=sheet["field_rules"]) + assert (out["summary"]["unchanged"], out["summary"]["changed"]) == (1, 1) + assert sql_compare_skip_reason(pg_db(), sheet) == "directional_compare" + + +def test_sorted_pairing_is_the_default_for_new_runs_only(): + assert build_pipeline(SHEET, iface_rules=[], port_map={}).duplicate_pairing == "sorted" + assert build_pipeline({**SHEET, "duplicate_pairing": "ordered"}, iface_rules=[], + port_map={}).duplicate_pairing == "ordered" + assert build_pipeline(SHEET, iface_rules=[], port_map={}, legacy=True).duplicate_pairing == "ordered" diff --git a/tests/test_cutover_references.py b/tests/test_cutover_references.py new file mode 100644 index 0000000..0f97c36 --- /dev/null +++ b/tests/test_cutover_references.py @@ -0,0 +1,204 @@ +"""Reference protection and input validation for compare/cutover (plan A2, A5).""" +import pytest +from fastapi import HTTPException +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_migration import service as mig +from netx_api.biz_state import compare_service as svc +from netx_api.db import Base +from netx_api.models import ( + BizCompareJob, BizCompareTemplate, BizMigrationProject, BizMonitorTemplate, + BizPortMapping, BizPortMappingRow, BizStateBatch, BizStateTask, +) + + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + yield session + session.close() + engine.dispose() + + +def _template(db, tid="tpl"): + db.add(BizCompareTemplate(id=tid, name=f"name-{tid}", metrics_json=[])) + db.commit() + + +def _mapping(db, mid="map"): + db.add(BizPortMapping(id=mid, name=f"map-{mid}")) + db.add(BizPortMappingRow(mapping_id=mid, before_if="a", after_if="b")) + db.commit() + + +def test_mapping_used_by_job_cannot_be_deleted(db): + _mapping(db) + db.add(BizCompareJob(id="j", name="job-1", template_id="tpl", mapping_id="map")) + db.commit() + with pytest.raises(HTTPException) as exc: + svc.delete_mapping(db, "map") + assert exc.value.status_code == 409 + assert exc.value.detail["code"] == "mapping_in_use" + assert exc.value.detail["jobs"] == ["job-1"] + assert db.get(BizPortMapping, "map") is not None + + +def test_mapping_used_by_project_cannot_be_deleted(db): + _mapping(db) + db.add(BizMigrationProject(id="p", name="proj-1", mapping_id="map")) + db.commit() + with pytest.raises(HTTPException) as exc: + svc.delete_mapping(db, "map") + assert exc.value.status_code == 409 + assert exc.value.detail["projects"] == ["proj-1"] + assert "proj-1" in exc.value.detail["message"] + + +def test_unreferenced_mapping_is_deleted_with_rows(db): + _mapping(db) + svc.delete_mapping(db, "map") + assert db.get(BizPortMapping, "map") is None + assert db.query(BizPortMappingRow).count() == 0 + + +def test_list_mappings_reports_reference_counts(db): + _mapping(db) + db.add_all([BizCompareJob(id="j1", template_id="tpl", mapping_id="map"), + BizCompareJob(id="j2", template_id="tpl", mapping_id="map"), + BizMigrationProject(id="p", name="p", mapping_id="map")]) + db.commit() + (item,) = svc.list_mappings(db) + assert (item["job_count"], item["project_count"]) == (2, 1) + + +def test_template_used_by_monitor_template_cannot_be_deleted(db): + _template(db) + db.add(BizMonitorTemplate(id="mt", name="monitor-1", compare_template_id="tpl")) + db.commit() + with pytest.raises(HTTPException) as exc: + svc.delete_template(db, "tpl") + assert exc.value.status_code == 409 + assert exc.value.detail["code"] == "template_in_use" + assert exc.value.detail["monitor_templates"] == ["monitor-1"] + assert exc.value.detail["jobs"] == [] + + +def test_template_used_by_job_reports_job_names(db): + _template(db) + db.add(BizCompareJob(id="j", name="job-1", template_id="tpl")) + db.commit() + with pytest.raises(HTTPException) as exc: + svc.delete_template(db, "tpl") + assert exc.value.detail["jobs"] == ["job-1"] + + +def test_list_templates_counts_monitor_templates(db, monkeypatch): + monkeypatch.setattr(svc, "ensure_default_templates", lambda db: None) + monkeypatch.setattr(svc, "upgrade_builtin_split_sheets", lambda db: None) + _template(db) + db.add_all([BizMonitorTemplate(id="m1", compare_template_id="tpl"), + BizMonitorTemplate(id="m2", compare_template_id="tpl")]) + db.commit() + (item,) = svc.list_templates(db) + assert item["monitor_template_count"] == 2 + + +def test_job_with_deleted_mapping_is_rejected_not_silently_unmapped(db): + _template(db) + db.add(BizCompareJob(id="j", template_id="tpl", mapping_id="gone", + before_batch_id="b", after_batch_id="a")) + db.commit() + with pytest.raises(HTTPException) as exc: + svc._validate_compare_job(db, "j") + assert exc.value.status_code == 400 + assert exc.value.detail == "mapping_missing" + + +def _project_tasks(db): + db.add_all([BizStateTask(id="old_p", source="managed", ne_id="ne-old", status="running"), + BizStateTask(id="new_p", source="managed", ne_id="ne-new", status="running"), + BizStateTask(id="other", source="managed", ne_id="ne-x", status="running")]) + db.add_all([BizStateBatch(id="old_ok", task_id="old_p", status="success"), + BizStateBatch(id="new_ok", task_id="new_p", status="partial"), + BizStateBatch(id="foreign", task_id="other", status="success"), + BizStateBatch(id="old_failed", task_id="old_p", status="failed")]) + db.commit() + + +def _body(**extra): + return {"name": "n", "old_task_id": "old_p", "new_task_id": "new_p", **extra} + + +@pytest.mark.parametrize("extra,code", [ + ({"old_baseline_batch_id": "foreign"}, "old_baseline_invalid"), + ({"new_baseline_batch_id": "foreign"}, "new_baseline_invalid"), + ({"old_baseline_batch_id": "old_failed"}, "old_baseline_invalid"), + ({"new_baseline_batch_id": "old_ok"}, "new_baseline_invalid"), +]) +def test_create_project_rejects_baseline_from_wrong_task(db, extra, code): + _project_tasks(db) + with pytest.raises(HTTPException) as exc: + mig.create_project(db, _body(**extra)) + assert exc.value.status_code == 400 + assert exc.value.detail == code + assert db.query(BizMigrationProject).count() == 0 + + +def test_baseline_ownership_accepts_success_and_partial(db): + _project_tasks(db) + mig._validate_baseline_ownership(db, "old", "old_ok", "old_p") + mig._validate_baseline_ownership(db, "new", "new_ok", "new_p") + mig._validate_baseline_ownership(db, "old", "", "old_p") + + +# ---- H9: reads do not write; project lists do not fan out -------------------------------------- + +def _project(db, n, **extra): + db.add(BizStateTask(id=f"to{n}", source="managed", ne_id=f"o{n}", status="running")) + db.add(BizStateTask(id=f"tn{n}", source="managed", ne_id=f"n{n}", status="running")) + db.add(BizStateBatch(id=f"bo{n}", task_id=f"to{n}", status="success")) + db.add(BizStateBatch(id=f"bn{n}", task_id=f"tn{n}", status="success")) + db.add(BizMigrationProject(id=f"p{n}", name=f"p{n}", old_task_id=f"to{n}", new_task_id=f"tn{n}", + old_baseline_batch_id=f"bo{n}", new_baseline_batch_id=f"bn{n}", **extra)) + + +def _statements(db, work): + from sqlalchemy import event + + seen: list[str] = [] + listener = lambda conn, cur, stmt, params, ctx, many: seen.append(stmt) # noqa: E731 + event.listen(db.get_bind(), "before_cursor_execute", listener) + try: + work() + finally: + event.remove(db.get_bind(), "before_cursor_execute", listener) + return seen + + +def test_get_project_with_unbound_monitor_template_does_not_write(db): + db.add_all([BizCompareTemplate(id="ct", name="ZTE status default", metrics_json=[]), + BizMonitorTemplate(id="mt", name="默认割接监控", compare_template_id="ct")]) + _project(db, 1) + db.commit() + seen = _statements(db, lambda: mig.get_project(db, "p1")) + assert not [s for s in seen if s.lstrip().upper().startswith(("INSERT", "UPDATE", "DELETE"))] + assert db.get(BizMigrationProject, "p1").monitor_template_id == "" + + +def test_project_list_query_count_does_not_grow_with_projects(db): + db.add_all([BizCompareTemplate(id="ct", name="c", metrics_json=[]), + BizMonitorTemplate(id="mt", name="默认割接监控", compare_template_id="ct")]) + for n in range(3): + _project(db, n, monitor_template_id="mt") + db.commit() + few = len(_statements(db, lambda: mig.list_projects(db))) + db.expire_all() + for n in range(3, 15): + _project(db, n, monitor_template_id="mt") + db.commit() + db.expire_all() + many = len(_statements(db, lambda: mig.list_projects(db))) + assert many <= few + 2, (few, many) diff --git a/tests/test_identity_pipeline.py b/tests/test_identity_pipeline.py new file mode 100644 index 0000000..a3ee5aa --- /dev/null +++ b/tests/test_identity_pipeline.py @@ -0,0 +1,341 @@ +"""Identity pipeline: one match key everywhere, per-side rules, value maps, post-normalize filters.""" +from __future__ import annotations + +from unittest.mock import MagicMock + +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +from netx_api.biz_migration.evaluate import evaluate_metric_dual, parse_expect_set +from netx_api.biz_state import compare_service as svc +from netx_api.biz_state.compare_engine import compare_rows +from netx_api.biz_state.compare_sql import dup_pairing_order_sql, sql_compare_skip_reason +from netx_api.biz_state.identity_pipeline import build_pipeline, legacy_pipeline +from netx_api.db import Base +from netx_api.models import ( + BizCompareRun, BizCompareTemplate, BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask, +) + +ZTE_RULES = [{"from": "GE", "to": "gei"}, {"from": "XGE", "to": "xgei"}] + + +def sheet(**over): + base = { + "sheet_id": "s", "metric_id": "iface_t", "key_fields": ["interface"], "iface_fields": ["interface"], + "compare_fields": ["state"], "field_rules": [], "row_filters": [], + } + return {**base, **over} + + +def pg_db(): + db = MagicMock() + db.get_bind.return_value = MagicMock(dialect=MagicMock()) + db.get_bind.return_value.dialect.name = "postgresql" + return db + + +# ---- B2: per-side rules + vendor canonical ------------------------------------------------- + +def test_huawei_long_and_short_names_are_the_same_interface(): + p = build_pipeline(sheet(), iface_rules=ZTE_RULES, port_map={}, before_vendor="huawei", after_vendor="huawei") + a = p.match_key({"interface": "GigabitEthernet0/0/1"}, "before") + b = p.match_key({"interface": "GE0/0/1"}, "before") + assert a == b == ("gei0/0/1",) + + +def test_inner_spaces_are_removed(): + p = build_pipeline(sheet(), iface_rules=[], port_map={}, before_vendor="cisco") + assert p.match_key({"interface": "Gigabit Ethernet 0/0/1"}, "before") == p.match_key( + {"interface": "GigabitEthernet0/0/1"}, "before") + + +def test_cisco_and_h3c_canonical_forms(): + p = build_pipeline(sheet(), iface_rules=[], port_map={}, before_vendor="cisco", after_vendor="h3c") + assert p.normalize({"interface": "TenGigE0/0/0/1"}, "before")["interface"] == "Te0/0/0/1" + assert p.normalize({"interface": "Ten-GigabitEthernet1/0/1"}, "after")["interface"] == "XGE1/0/1" + + +def test_builtin_canonical_can_be_switched_off_per_sheet(): + p = build_pipeline(sheet(builtin_vendor_canonical=False), iface_rules=[], port_map={}, before_vendor="huawei") + assert p.normalize({"interface": "GigabitEthernet0/0/1"}, "before")["interface"] == "GigabitEthernet0/0/1" + + +def test_side_scoped_rule_only_touches_its_side(): + rules = [{"from": "GE", "to": "gei", "side": "before"}] + p = build_pipeline(sheet(), iface_rules=rules, port_map={}, before_vendor="huawei", after_vendor="zte") + assert p.normalize({"interface": "GE0/1"}, "before")["interface"] == "gei0/1" + assert p.normalize({"interface": "GE0/1"}, "after")["interface"] == "GE0/1" + + +def test_vendor_scoped_rule_follows_the_batch_vendor(): + rules = [{"from": "Eth-Trunk", "to": "ae", "vendor": "huawei"}] + p = build_pipeline(sheet(), iface_rules=rules, port_map={}, before_vendor="huawei", after_vendor="zte") + assert p.normalize({"interface": "Eth-Trunk1"}, "before")["interface"] == "ae1" + assert p.normalize({"interface": "Eth-Trunk1"}, "after")["interface"] == "Eth-Trunk1" + + +def test_port_map_keys_use_before_rules_and_values_use_after_rules(): + rules = [{"from": "GE", "to": "gei", "side": "after"}] + p = build_pipeline(sheet(), iface_rules=rules, port_map={"GE0/1": "GE0/2"}, + before_vendor="huawei", after_vendor="zte") + # before-side name stays "GE0/1" (rule is after-only); after-side value is normalized. + assert p.port_map == {"GigabitEthernet0/1".replace("GigabitEthernet", "GE"): "gei0/2"} + row = p.transform({"interface": "GigabitEthernet0/1"}, "before") + assert row["interface"] == "gei0/2" + + +def test_zte_to_huawei_numbering_change_with_alias_rules(): + p = build_pipeline( + sheet(), iface_rules=ZTE_RULES, port_map={"GE0/1/0/1": "GigabitEthernet0/2/0/1"}, + before_vendor="zte", after_vendor="huawei", + ) + out = compare_rows( + before_rows=[{"interface": "gei0/1/0/1", "state": "up"}], + after_rows=[{"interface": "GE0/2/0/1", "state": "up"}], + key_fields=["interface"], iface_fields=["interface"], compare_fields=["state"], pipeline=p, + ) + assert out["summary"]["unchanged"] == 1 + + +# ---- B3: value maps ---------------------------------------------------------------------- + +def test_value_map_aligns_vendor_state_names(): + rules = [{"field": "state", "value_map": {"estab": "established", "*down": "down"}}] + p = build_pipeline(sheet(iface_fields=[], key_fields=["peer"], field_rules=rules), iface_rules=[], + port_map={}) + out = compare_rows( + before_rows=[{"peer": "1.1.1.1", "state": "Estab"}, {"peer": "2.2.2.2", "state": "*down"}], + after_rows=[{"peer": "1.1.1.1", "state": "established"}, {"peer": "2.2.2.2", "state": "down"}], + key_fields=["peer"], iface_fields=[], compare_fields=["state"], pipeline=p, + ) + assert out["summary"]["unchanged"] == 2 and out["summary"]["changed"] == 0 + + +def test_value_map_can_be_scoped_to_one_side(): + rules = [{"field": "state", "side": "before", "value_map": {"estab": "established"}}] + p = build_pipeline(sheet(iface_fields=[], key_fields=["peer"], field_rules=rules), iface_rules=[], port_map={}) + assert p.normalize({"peer": "a", "state": "Estab"}, "before")["state"] == "established" + assert p.normalize({"peer": "a", "state": "Estab"}, "after")["state"] == "Estab" + + +def test_mac_key_with_different_formats_pairs(): + rules = [{"field": "mac", "normalize": "mac"}] + p = build_pipeline(sheet(metric_id="mac_t", iface_fields=[], key_fields=["mac"], compare_fields=["port"], + field_rules=rules), iface_rules=[], port_map={}) + out = compare_rows( + before_rows=[{"mac": "00e0-fc12-3456", "port": "1"}], after_rows=[{"mac": "00e0.fc12.3456", "port": "1"}], + key_fields=["mac"], iface_fields=[], compare_fields=["port"], pipeline=p, + ) + assert out["summary"]["unchanged"] == 1 and out["summary"]["added"] == out["summary"]["removed"] == 0 + + +def test_legacy_pipeline_never_normalizes_keys(): + rules = [{"field": "mac", "normalize": "mac"}] + p = legacy_pipeline(key_fields=["mac"], iface_fields=[], compare_fields=["port"], field_rules=rules) + out = compare_rows( + before_rows=[{"mac": "00e0-fc12-3456", "port": "1"}], after_rows=[{"mac": "00e0.fc12.3456", "port": "1"}], + key_fields=["mac"], iface_fields=[], compare_fields=["port"], field_rules=rules, pipeline=p, + ) + assert out["summary"]["added"] == out["summary"]["removed"] == 1 + + +def test_value_map_and_key_normalize_block_sql(): + s = sheet(metric_id="bgp_route", iface_fields=[], key_fields=["mac"], + field_rules=[{"field": "state", "value_map": {"a": "b"}}], compare_fields=["state"]) + p = build_pipeline(s, iface_rules=[], port_map={}) + assert sql_compare_skip_reason(pg_db(), s, pipeline=p) == "value_map" + s2 = sheet(metric_id="m", iface_fields=[], key_fields=["mac"], compare_fields=[], + field_rules=[{"field": "mac", "normalize": "lower"}]) + assert sql_compare_skip_reason(pg_db(), s2, pipeline=build_pipeline(s2, iface_rules=[], port_map={})) \ + == "key_normalize" + + +# ---- B4: filters run after normalization ----------------------------------------------------- + +CONTAINS_GEI = [{"field": "interface", "op": "contains", "value": "gei"}] + + +def test_v2_filter_sees_normalized_names_but_legacy_sees_raw(): + s = sheet(row_filters=CONTAINS_GEI) + v2 = build_pipeline(s, iface_rules=ZTE_RULES, port_map={}, before_vendor="huawei") + assert [raw["interface"] for raw, _ in v2.prepare([{"interface": "GE0/0/1"}], "before")] == ["GE0/0/1"] + old = legacy_pipeline(key_fields=["interface"], iface_fields=["interface"], compare_fields=[], + iface_normalize_rules=ZTE_RULES, row_filters=CONTAINS_GEI) + assert old.prepare([{"interface": "GE0/0/1"}], "before") == [] # the cross-vendor bug, kept for v1 + + +def test_filter_can_target_one_side(): + s = sheet(iface_fields=[], row_filters=[{"field": "type", "op": "eq", "value": "dynamic", "side": "before"}]) + p = build_pipeline(s, iface_rules=[], port_map={}) + rows = [{"interface": "a", "type": "static"}] + assert p.prepare(rows, "before") == [] + assert len(p.prepare(rows, "after")) == 1 + + +def test_pushdown_only_for_filters_that_read_untransformed_fields(): + s = sheet(row_filters=[CONTAINS_GEI[0], {"field": "afi", "op": "eq", "value": "ipv4"}]) + p = build_pipeline(s, iface_rules=ZTE_RULES, port_map={}) + assert p.pushdown_filters("before") == [{"field": "afi", "op": "eq", "value": "ipv4"}] + assert sql_compare_skip_reason(pg_db(), sheet(metric_id="m", iface_fields=["interface"], key_fields=["id"], + compare_fields=[], row_filters=CONTAINS_GEI, + ignore_port_changes=False), + pipeline=build_pipeline(sheet(row_filters=CONTAINS_GEI), iface_rules=[], + port_map={})) == "row_filters_post_normalize" + + +# ---- B6: a port map no longer pushes interface-free sheets to Python ---------------------------- + +def test_port_map_only_blocks_sql_for_sheets_that_use_interfaces(): + route = {"metric_id": "bgp_route", "key_fields": ["network"], "iface_fields": [], + "compare_fields": ["path"], "field_rules": [], "row_filters": []} + assert sql_compare_skip_reason(pg_db(), route, port_map={"a": "b"}) == "" + brief = {**route, "metric_id": "interface_brief", "key_fields": ["interface"], "iface_fields": ["interface"]} + assert sql_compare_skip_reason(pg_db(), brief, port_map={"a": "b"}) == "port_map" + # a map on a sheet whose interface column is only displayed does not matter either + shown = {**route, "iface_fields": ["interface"]} + assert sql_compare_skip_reason(pg_db(), shown, port_map={"a": "b"}) == "" + + +# ---- V4: duplicate keys pair by value ---------------------------------------------------------- + +def test_same_key_rows_pair_by_value_not_output_order(): + s = sheet(iface_fields=[], key_fields=["prefix"], compare_fields=["hop"]) + rows_b = [{"prefix": "p", "hop": "a"}, {"prefix": "p", "hop": "b"}] + rows_a = [{"prefix": "p", "hop": "b"}, {"prefix": "p", "hop": "a"}] + kw = dict(before_rows=rows_b, after_rows=rows_a, key_fields=["prefix"], iface_fields=[], compare_fields=["hop"]) + new = compare_rows(**kw, pipeline=build_pipeline(s, iface_rules=[], port_map={})) + assert new["summary"]["unchanged"] == 2 and new["summary"]["changed"] == 0 + old = compare_rows(**kw) # ordered, legacy + assert old["summary"]["changed"] == 2 + + +def test_sql_dup_order_mirrors_python_sort_key(): + sql = dup_pairing_order_sql(["hop", "metric"], {"hop": {"normalize": "lower"}}) + assert 'lower(trim(both from coalesce(data_json->>\'hop\', \'\'))) COLLATE "C" ASC' in sql.replace("(", "", 1) \ + or 'COLLATE "C"' in sql + assert sql.endswith(", ") + assert dup_pairing_order_sql([], {}) == "" + + +# ---- B7: partial maps --------------------------------------------------------------------------- + +def test_ignore_port_policy_keeps_unmapped_ports_matching_by_content(): + s = sheet(iface_fields=["interface"], key_fields=["interface", "ip"], compare_fields=["mac"], + unmapped_port_policy="ignore_port") + p = build_pipeline(s, iface_rules=[], port_map={"gei-1": "gei-9"}) + out = compare_rows( + before_rows=[{"interface": "gei-1", "ip": "10.0.0.1", "mac": "m1"}, + {"interface": "gei-2", "ip": "10.0.0.2", "mac": "m2"}], + after_rows=[{"interface": "gei-9", "ip": "10.0.0.1", "mac": "m1"}, + {"interface": "gei-7", "ip": "10.0.0.2", "mac": "m2"}], + key_fields=["interface", "ip"], iface_fields=["interface"], compare_fields=["mac"], pipeline=p, + ) + assert out["summary"]["unchanged"] == 2 + by_name = build_pipeline({**s, "unmapped_port_policy": "by_name"}, iface_rules=[], port_map={"gei-1": "gei-9"}) + out2 = compare_rows( + before_rows=[{"interface": "gei-2", "ip": "10.0.0.2", "mac": "m2"}], + after_rows=[{"interface": "gei-7", "ip": "10.0.0.2", "mac": "m2"}], + key_fields=["interface", "ip"], iface_fields=["interface"], compare_fields=["mac"], pipeline=by_name, + ) + assert out2["summary"]["added"] == out2["summary"]["removed"] == 1 + + +# ---- B1: every path derives the same match key -------------------------------------------------- + +@pytest.fixture() +def db(): + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + session = sessionmaker(bind=engine, expire_on_commit=False)() + yield session + session.close() + engine.dispose() + + +def _seed_two_batches(db): + db.add_all([ + BizStateTask(id="t-old", source="managed", ne_id="o", vendor="Huawei", device_type="huawei"), + BizStateTask(id="t-new", source="managed", ne_id="n", vendor="ZTE", device_type="zte_zxros"), + BizStateBatch(id="b-old", task_id="t-old", status="success", vendor="Huawei"), + BizStateBatch(id="b-new", task_id="t-new", status="success", vendor="ZTE"), + ]) + db.add_all([ + BizStateMetricRow(id="r-old", batch_id="b-old", metric_id="iface_t", seq=0, + data_json={"interface": "GigabitEthernet0/1/0/1", "state": "up"}), + BizStateMetricRow(id="r-new", batch_id="b-new", metric_id="iface_t", seq=0, + data_json={"interface": "gei-0/2/0/1", "state": "up"}), + ]) + db.commit() + + +def test_all_paths_agree_on_the_match_key(db): + _seed_two_batches(db) + raw_map = {"GigabitEthernet0/1/0/1": "gei-0/2/0/1"} + rules = [{"from": "GE", "to": "gei"}] + s = sheet(sheet_id="s") + tpl = BizCompareTemplate(id="tpl", name="t", metrics_json=[s], iface_normalize_json=rules) + db.add(tpl) + db.commit() + from netx_api.biz_state.compare_service import _create_running_run, template_metrics + from netx_api.models import BizCompareJob, BizPortMapping, BizPortMappingRow + + db.add_all([BizPortMapping(id="m", name="m"), + BizPortMappingRow(mapping_id="m", before_if="GigabitEthernet0/1/0/1", after_if="gei-0/2/0/1"), + BizCompareJob(id="j", name="j", template_id="tpl", mapping_id="m")]) + db.commit() + job = db.get(BizCompareJob, "j") + run = _create_running_run(db, job=job, tpl=tpl, before_batch_id="b-old", after_batch_id="b-new", + sheets_cfg=template_metrics(tpl)) + assert run.summary_json["config_snapshot"]["vendors"] == {"before": "huawei", "after": "zte"} + pipe = svc._run_pipeline(db, run, s) + expected = ("gei-0/2/0/1",) + + # 1. execution + result = compare_rows(before_rows=[{"interface": "GigabitEthernet0/1/0/1", "state": "up"}], + after_rows=[{"interface": "gei-0/2/0/1", "state": "up"}], + key_fields=["interface"], iface_fields=["interface"], compare_fields=["state"], + pipeline=pipe) + assert result["summary"]["unchanged"] == 1 + # 2. pipeline directly + assert pipe.match_key({"interface": "GigabitEthernet0/1/0/1"}, "before") == expected + assert pipe.match_key({"interface": "gei-0/2/0/1"}, "after") == expected + # 3. search key groups + rows, complete = svc._load_search_key_groups( + db, batch_id="b-old", metric_id="iface_t", keys={expected}, match_keys=["interface"], + pipeline=pipe, side="before") + assert [r["interface"] for r in rows] == ["GigabitEthernet0/1/0/1"] and complete == {expected} + # 4. hydrate + item = {"before": {"interface": "GigabitEthernet0/1/0/1"}, "after": {}, "mapped_before": {}, + "after_row_id": "r-new", "before_row_id": ""} + hydrated = svc._hydrate_diff_rows(db, [item], run=run, sheet=s)[0] + assert hydrated["mapped_before"]["interface"] == hydrated["after"]["interface"] == "gei-0/2/0/1" + # 5. cutover evaluation (old = before, new = after) + dual = evaluate_metric_dual( + metric_id="iface_t", sheet_id="s", key_fields=["interface"], iface_fields=["interface"], + compare_fields=["state"], old_baseline_rows=[{"interface": "GigabitEthernet0/1/0/1", "state": "up"}], + old_current_rows=[], new_baseline_rows=[{"interface": "gei-0/2/0/1", "state": "down"}], + new_current_rows=[{"interface": "gei-0/2/0/1", "state": "up"}], port_map=pipe.port_map, + expect=parse_expect_set({"items": [{"metric_id": "iface_t", "key": "GE0/1/0/1"}]}), + window_active=False, acceptance=True, pipeline=pipe, + ) + (row,) = dual["rows"] + assert (row["match_old_key"], row["match_new_key"]) == ("gei0/1/0/1", "gei-0/2/0/1") + assert row["old_key"] == "GigabitEthernet0/1/0/1" and row["verdict"] == "migrated" + # 6. baseline expect picker + assert pipe.transform({"interface": "GigabitEthernet0/1/0/1"}, "before")["interface"] == "gei-0/2/0/1" + + +# ---- v1 replay must not change ----------------------------------------------------------------- + +def test_v1_snapshot_run_replays_with_legacy_semantics(db): + _seed_two_batches(db) + s = sheet(sheet_id="s", row_filters=CONTAINS_GEI) + run = BizCompareRun(id="r1", job_id="j", template_id="", before_batch_id="b-old", after_batch_id="b-new", + summary_json={"config_snapshot": { + "version": 1, "template": {"iface_normalize_rules": [{"from": "GE", "to": "gei"}]}, + "sheets": [s], "port_map": {"gei-0/1/0/1": "gei-0/2/0/1"}}}) + pipe = svc._run_pipeline(db, run, s) + assert pipe.legacy and pipe.duplicate_pairing == "ordered" + assert pipe.port_map == {"gei-0/1/0/1": "gei-0/2/0/1"} + assert pipe.prepare([{"interface": "GE0/1"}], "before") == [] diff --git a/tests/test_iface_normalize.py b/tests/test_iface_normalize.py index 94bbcbb..f266fa7 100644 --- a/tests/test_iface_normalize.py +++ b/tests/test_iface_normalize.py @@ -9,6 +9,7 @@ from netx_api.biz_state.iface_normalize import ( default_zte_iface_normalize_rules, normalize_iface_name, normalize_iface_rules, + normalize_port_map, resolve_mapped_iface, ) @@ -71,6 +72,32 @@ class PortMapSubifTests(unittest.TestCase): self.assertEqual(row["interface"], "xgei-0/1/0/1.55") +class NormalizePortMapTests(unittest.TestCase): + def test_keys_and_values_use_template_rules(self) -> None: + out, conflicts = normalize_port_map( + {"GE0/1/0/1": "GE0/2/0/1"}, + normalize_iface_rules([{"from": "GE", "to": "gei"}]), + ) + self.assertEqual(out, {"gei0/1/0/1": "gei0/2/0/1"}) + self.assertEqual(conflicts, []) + + def test_conflicting_keys_are_reported_first_wins(self) -> None: + out, conflicts = normalize_port_map( + {"GE0/1": "gei0/2", "gei0/1": "gei0/3"}, + normalize_iface_rules([{"from": "GE", "to": "gei"}]), + ) + self.assertEqual(out, {"gei0/1": "gei0/2"}) + self.assertEqual(len(conflicts), 1) + self.assertEqual(conflicts[0]["dropped_to"], "gei0/3") + + def test_same_target_duplicate_is_not_a_conflict(self) -> None: + _, conflicts = normalize_port_map( + {"GE0/1": "gei0/2", "gei0/1": "gei0/2"}, + normalize_iface_rules([{"from": "GE", "to": "gei"}]), + ) + self.assertEqual(conflicts, []) + + class ComparePipelineTests(unittest.TestCase): def test_normalize_then_map_matches(self) -> None: before = [{"interface": "GE-0/0/0/1.100", "admin": "up"}] diff --git a/web/src/features/cutover/BizComparePage.tsx b/web/src/features/cutover/BizComparePage.tsx index 8163a98..011e2a8 100644 --- a/web/src/features/cutover/BizComparePage.tsx +++ b/web/src/features/cutover/BizComparePage.tsx @@ -1,8 +1,15 @@ import { Button, Input, Modal } from "@heroui/react"; import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from "react"; import type { KeyboardEvent as ReactKeyboardEvent } from "react"; -import { parseTemplateImport, parseIfaceNormText, toOrGroups, fromOrGroups, validateTemplateDraft } from "./lib/compareTemplateModel"; -import type { MetricSheet, FieldRule, RowFilter } from "./lib/compareTemplateModel"; +import { + parseTemplateImport, parseIfaceNormText, formatIfaceNormText, parseValueMapText, formatValueMapText, + sheetPayload, sheetOptions, toOrGroups, fromOrGroups, validateTemplateDraft, +} from "./lib/compareTemplateModel"; +import type { MetricSheet, FieldRule, RowFilter, IfaceRule, Side } from "./lib/compareTemplateModel"; +import { MappingTable } from "./components/MappingTable"; +import { PreflightReport, dataStateLabel } from "./components/PreflightReport"; +import type { PreflightReportData } from "./components/PreflightReport"; +import { parseMapText, mapTextDirty } from "./lib/mappingText"; import { ListPager } from "../../components/ListPager"; import { AppModalShell } from "../../components/ui/AppModalShell"; import { FieldSelect } from "../../components/ui/FieldSelect"; @@ -29,7 +36,7 @@ import { bizCompareUpdateJob, bizCompareUpdateMapping, bizCompareUpdateTemplate, - bizCompareValidateMapping, + bizCompareMappingPreflight, bizStateListBatches, bizStateListTasks, formatErr, @@ -72,12 +79,19 @@ type Template = { key_fields: string[]; iface_fields: string[]; compare_fields: string[]; - iface_normalize_rules?: { from: string; to: string }[]; + iface_normalize_rules?: IfaceRule[]; note: string; job_count?: number; + monitor_template_count?: number; }; -type Mapping = { id: string; name: string; rows: { before_if: string; after_if: string }[] }; +type Mapping = { + id: string; + name: string; + rows: { before_if: string; after_if: string }[]; + job_count?: number; + project_count?: number; +}; type StoreUnchanged = "auto" | "always" | "never" | "sample" | "keys"; type Job = { @@ -102,6 +116,8 @@ type DiffRow = { before: Record | null; after: Record | null; mapped_before?: Record | null; + /** Before-side key exactly as the device printed it; only present when it differs from the match key. */ + raw_key_before?: Record | null; changes: Record; }; @@ -525,6 +541,7 @@ function cloneSheet(s: MetricSheet): MetricSheet { row_filters: JSON.parse(JSON.stringify(s.row_filters || [])), field_rules: JSON.parse(JSON.stringify(s.field_rules || [])), ignore_port_changes: s.ignore_port_changes, + ...sheetOptions(s as unknown as Record), }; } @@ -532,21 +549,9 @@ const TEMPLATE_EXPORT_FORMAT = "netx.biz_compare_template"; const TEMPLATE_EXPORT_VERSION = 1; function templateExportPayload(tpl: Template) { - const metrics = templateSheets(tpl).map((s) => ({ - sheet_id: s.sheet_id || s.metric_id, - title: s.title || s.sheet_id || s.metric_id, - metric_id: s.metric_id, - key_fields: [...(s.key_fields || [])], - iface_fields: [...(s.iface_fields || [])], - compare_fields: [...(s.compare_fields || [])], - display_fields: - s.display_fields && s.display_fields.length - ? [...s.display_fields] - : [...(s.key_fields || []), ...(s.compare_fields || [])], - row_filters: JSON.parse(JSON.stringify(s.row_filters || [])), - field_rules: JSON.parse(JSON.stringify(s.field_rules || [])), - ignore_port_changes: s.ignore_port_changes, - })); + const metrics = templateSheets(tpl).map((s) => + JSON.parse(JSON.stringify(sheetPayload({ ...s, display_fields: s.display_fields?.length ? s.display_fields : undefined }))), + ); return { format: TEMPLATE_EXPORT_FORMAT, version: TEMPLATE_EXPORT_VERSION, @@ -558,8 +563,8 @@ function templateExportPayload(tpl: Template) { }; } -function ifaceNormRulesToText(rules?: { from: string; to: string }[]): string { - return (rules || []).map((r) => `${r.from},${r.to}`).join("\n"); +function ifaceNormRulesToText(rules?: IfaceRule[]): string { + return formatIfaceNormText(rules); } function downloadJsonFile(filename: string, data: unknown) { @@ -572,13 +577,16 @@ function downloadJsonFile(filename: string, data: unknown) { URL.revokeObjectURL(url); } +const isSided = (r: FieldRule) => r.side === "before" || r.side === "after"; + +/** The rule that drives the comparison of a field (side-scoped value maps are separate rules). */ function ruleForField(sheet: MetricSheet, field: string): FieldRule { - return (sheet.field_rules || []).find((r) => r.field === field) || { field }; + return (sheet.field_rules || []).find((r) => r.field === field && !isSided(r)) || { field }; } function upsertFieldRule(sheet: MetricSheet, field: string, patch: Partial): FieldRule[] { const rules = [...(sheet.field_rules || [])]; - const idx = rules.findIndex((r) => r.field === field); + const idx = rules.findIndex((r) => r.field === field && !isSided(r)); const next: FieldRule = { ...(idx >= 0 ? rules[idx] : { field }), ...patch, field }; const norm = (next.normalize || "").toLowerCase(); const cmp = (next.compare || "").toLowerCase(); @@ -586,15 +594,33 @@ function upsertFieldRule(sheet: MetricSheet, field: string, patch: Partial r.field !== field); + return rules.filter((_r, i) => i !== idx); } if (idx >= 0) rules[idx] = next; else rules.push(next); return rules; } +/** Value map of a field for one scope: "both" lives on the main rule, before/after on their own rules. */ +function valueMapFor(sheet: MetricSheet, field: string, scope: Side): Record { + const rule = (sheet.field_rules || []).find((r) => r.field === field && (scope === "before" || scope === "after" ? r.side === scope : !isSided(r))); + return rule?.value_map || {}; +} + +function setValueMapFor(sheet: MetricSheet, field: string, scope: Side, map: Record): FieldRule[] { + if (scope !== "before" && scope !== "after") { + return upsertFieldRule(sheet, field, { value_map: Object.keys(map).length ? map : undefined }); + } + const rules = (sheet.field_rules || []).filter((r) => !(r.field === field && r.side === scope)); + if (Object.keys(map).length) rules.push({ field, side: scope, value_map: map }); + return rules; +} + function emptyLeaf(field = ""): RowFilter { return { field, op: "eq", value: "" }; } @@ -611,6 +637,69 @@ const FILTER_OPS = [ "age_timer", ] as const; +/** Value-map editor for one field: text is parsed on blur so half-typed lines never clobber the rule. */ +function ValueMapCell({ + field, value, scope, onScope, onCommit, rank, onRank, showRank, t, +}: { + field: string; + value: Record; + scope: Side; + onScope: (s: Side) => void; + onCommit: (map: Record) => void; + rank: string[]; + onRank: (rank: string[]) => void; + showRank: boolean; + t: (k: string, vars?: Record) => string; +}) { + const external = formatValueMapText(value); + const [text, setText] = useState(external); + const [error, setError] = useState(""); + useEffect(() => { + setText(external); + setError(""); + }, [external, scope]); + return ( +
+ {showRank ? ( + onRank(e.target.value.split(",").map((x) => x.trim()).filter(Boolean))} + /> + ) : null} + +