mirror of
https://github.com/hansjone/netx.git
synced 2026-10-14 15:43:15 +08:00
Optimize cutover compare/monitor per the 2026-10-12 plan.
Correctness (A): normalize port-map names with the template rules (snapshot v2, v1 replays unchanged); protect mappings/templates that are referenced; stop dropping edited mapping text; accept pulled-but-not-shutdown ports as migrated; validate baseline ownership on project create; one canonical key form; fix built-in template tolerances and upgrade old packaged values in place. Cross-vendor identity (B): single identity pipeline for every path, per-side and per-vendor interface rules, value maps, filters after normalization, mapping pre-flight for compare and cutover projects, SQL eligibility fixes. Data quality (V): no_data / suspect_parse / not_collected sheet states, command slice coverage, directional compare modes, value-sorted duplicate pairing, before-side raw names in results and CSV. Robustness (H): bounded auto-run history, background evaluate/finish with polling, manual route tables out of the acceptance gate, baseline row cache, scoped duplicate-key failures, merged HF tasks with a per-NE session cap, same-device cutover, explicit scope_by_port_map, read-only GETs, resolvable red tickets. Frontend: pre-flight report, mapping table view, template editor support for the new rules, data-state banners. Docs: canonical field contract. Not included: Huawei status parsers (no real samples), SQL-path route compare. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
fff7d63b9b
commit
315539e0fd
59 changed files with 6483 additions and 487 deletions
68
docs/biz-state-canonical-fields.md
Normal file
68
docs/biz-state-canonical-fields.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# 割接对比：规范值契约
|
||||
|
||||
适用范围：业务对比（`BizCompare*`）和割接监控（`BizMigration*`）。两种模式用同一条身份流水线（`netx_api/biz_state/identity_pipeline.py`），跨厂商对比靠它把两侧设备的原始输出对齐到同一套“规范值”。
|
||||
|
||||
## 1. 流水线
|
||||
|
||||
同一行在某一侧（`before` / `after`）依次经过：
|
||||
|
||||
1. **归一化**：接口字段去内部空格 → 厂商内规范名（`VENDOR_CANONICAL`，如华为 `GigabitEthernet`→`GE`）→ 模板别名规则（`iface_normalize_rules`，如 `GE`→`gei`）；键字段按 `field_rules[].normalize`（`lower`/`upper`/`mac`…）；再按 `field_rules[].value_map` 改写状态值。
|
||||
2. **行过滤**：`row_filters` 在归一化之后求值，所以按规范值书写；带 `side` 的过滤只作用于该侧。
|
||||
3. **端口映射**：只作用于 `before` 侧，键用 before 侧规则归一化、值用 after 侧规则归一化。
|
||||
4. **取键**：`scalar_text(value).strip()` 逐字段取值，得到 match key。
|
||||
|
||||
快照版本 `config_snapshot.version >= 2` 的 run 走完整流水线；`version == 1` 的历史 run 保持旧语义（过滤看原始行、键不归一化、同键按采集顺序配对），回填、检索、导出结果不变。
|
||||
|
||||
## 2. 规范值
|
||||
|
||||
比对层只认下面这些取值。设备原始值不同的，用 `value_map` 或 parser 输出规范值解决，**引擎里不写厂商特例**。
|
||||
|
||||
| metric | 字段 | 规范取值 |
|
||||
|---|---|---|
|
||||
| `interface_brief` | `admin` / `phy` / `prot` | `up` / `down`（华为 `*down` 记 `admin=down`） |
|
||||
| `bgp_peer` | `state` | `established` / `idle` / `active` / `connect` / `opensent` / `openconfirm` |
|
||||
| `bgp_peer` | `afi` | `ipv4` / `ipv6` / `vpnv4` / `vpnv6` / `evpn` / `vpls` |
|
||||
| `arp` | `entry_type` | `dynamic` / `static` / `interface` |
|
||||
| `isis_adjacency` / `ospf_neighbor` | `state` | `up` / `full` / `2way` / `down` / `init` / `idle` |
|
||||
| `vrrp` | `af` | `ipv4` / `ipv6` |
|
||||
|
||||
### 中兴（已实现）现状
|
||||
|
||||
中兴 parser 输出设备打印的原值，没有改写：
|
||||
|
||||
- `interface_brief.admin/phy/prot`：`up` / `down`，已是规范值。
|
||||
- `arp.entry_type`：由 `age` 推断，`HH:MM:SS` 为 `dynamic`，否则 `static`。
|
||||
- `bgp_peer.state`：设备原值，如 `Established`（首字母大写）。与中兴自身对比时一致；与其他厂商对比时在模板里加值映射：
|
||||
|
||||
```json
|
||||
{"field": "state", "value_map": {"established": "established", "estab": "established", "*down": "down"}}
|
||||
```
|
||||
|
||||
`value_map` 大小写不敏感匹配，输出为右侧的规范值；两侧都经过同一张表。
|
||||
|
||||
> 没有改动中兴 parser 的输出值：已采集批次里的值是原值，直接改成小写会让“旧批次 vs 新批次”出现假差异。需要统一时在模板里加 `value_map`。
|
||||
|
||||
### 其他厂商（待办）
|
||||
|
||||
华为 VRP 的状态 parser（`interface_brief`、`arp`、`bgp_peer`、`isis_adjacency`、`ospf_neighbor`）**尚未实现**：仓库里没有真实输出样本，按规则不凭空编写 TextFSM。收到样本后按 `skills/biz-state/netx-biz-state/SKILL.md` 第 2 节逐项完成，输出与中兴相同的字段名，状态值输出上表的规范值，并给 profile 补 `slice`。
|
||||
|
||||
目前只有 LLDP 在跨厂商场景下有真实 parser。
|
||||
|
||||
## 3. 命令切片（`ParseProfile.slice`）
|
||||
|
||||
一个采集命令产出 metric 的哪一部分，由 profile 的 `slice` 声明，例如 `zte.bgp_ipv4_summary` → `{"afi": "ipv4"}`、`zte.vrrp_ipv6` → `{"af": "ipv6"}`。比对表单（sheet）用顶层 `eq` 行过滤声明自己要的切片。
|
||||
|
||||
- 一侧没有任何命令覆盖该切片 → 该 sheet 状态 `not_collected`，不判业务缺失、不计入通过率。
|
||||
- 割接监控创建 HF 采集任务时，取两侧画像命令切片的并集；某侧找不到对应切片的 profile 时返回 `no_profile_for_slice:<metric>:<slice>:<vendor>`。
|
||||
- 没声明 `slice` 的命令视为覆盖整个 metric。
|
||||
|
||||
## 4. sheet 数据状态
|
||||
|
||||
| 状态 | 条件 | 通过率 |
|
||||
|---|---|---|
|
||||
| `ok` | 两侧采集正常 | 正常计算 |
|
||||
| `no_data` | 两侧过滤后都是 0 行，命令都成功 | `null`，不计入总通过率 |
|
||||
| `suspect_parse` | 某侧 `raw_line_count > 5` 但 `row_count == 0`，或全部命令失败 | 失败行照常显示，任务级 `ok = null` |
|
||||
| `not_collected` | 某侧没有任何命令（或切片）产出该 metric | `null`，不计入总通过率 |
|
||||
|
||||
没有任何命令回执的批次（导入或历史数据）无法判断，按正常处理。
|
||||
477
docs/reviews/cutover-optimization-plan-2026-10-12.md
Normal file
477
docs/reviews/cutover-optimization-plan-2026-10-12.md
Normal file
|
|
@ -0,0 +1,477 @@
|
|||
# 割接对比（业务对比 / 割接监控）问题分析与优化计划
|
||||
|
||||
日期：2026-10-12。范围：网络管理 → 割接对比下的 5 个子模块：业务监控（采集）、业务对比、割接监控、对比模板、监控模板。
|
||||
用途：交给 Cursor 按任务逐项实施；Claude 负责验收。
|
||||
|
||||
本文基于代码阅读和本地脚本验证，没有连接真实设备，也没有在真实 PostgreSQL 上执行。标了 **[已验证]** 的问题已经用脚本复现（脚本放在会话 scratchpad，结论见各条目）。基线：开始修改前，相关后端回归 **279 项通过**，`tsc -b` 通过，`cutoverDataCache` / `compareTemplateModel` Node 测试通过。
|
||||
|
||||
---
|
||||
|
||||
## 0. 结论摘要
|
||||
|
||||
1. **两种模式共用一套“归一化 + 映射”概念，但各个代码路径的实现不一致。** 割接监控会先对端口映射的键和值做接口归一化，业务对比不会（**[已验证]**：按设备原始名称填写映射，业务对比全部变成“删除 + 新增”）。行过滤用原始值，映射作用在归一化之后。键字段不受 `field_rules` 归一化影响。跨厂商割接最容易在这里产生误报。
|
||||
2. **目前只有两种映射：接口类型别名（模板级，两侧同一套规则）和端口映射（before→after，一对一）。** 没有分侧（按厂商）归一化，没有字段值映射（状态枚举、对端 IP、VRF 改名），也无法提前检查映射覆盖率。映射漏配会直接表现为业务“缺失”，用户很难判断是映射问题还是业务问题。
|
||||
3. **除 LLDP 外，状态采集解析只实现了中兴。** 华为、思科等的 parser 都是会抛出 `NotImplementedError` 的空壳，且没有注册。内置对比模板（`ZTE status default`）里的过滤和容差也都按中兴设定。所以现在的“跨厂商映射”只对 LLDP 真正有效，其余检查项在跨厂商场景下要么采集不到，要么全量误报。
|
||||
4. **判定语义有几处会把正常割接判成失败：**
|
||||
- 端口预设要求旧端口 `admin=down 且 phy=down`。现场常见的做法是拔纤但不 shutdown，这种情况验收时会判为“未完成/红”（**[已验证]**）。
|
||||
- 两侧都是 0 行时通过率是 100%（空对空误判为通过）。
|
||||
- 解析不到数据会被当成业务缺失。
|
||||
- 新旧两侧的采集命令集不一致时，会被当成业务丢失。
|
||||
- 内置模板对 `output_bps` 用绝对值容差 5 bps，对 `rx_power`（dBm）用百分比容差。
|
||||
5. **割接监控的运行数据无限增长。** 每个 `BizMigrationRun` 或 `BizCompareRun` 引用的采集批次都会被永久保护，run 和 diff 本身也没有保留策略。60 秒一轮的高频监控跑几天，数据库和磁盘会持续膨胀。
|
||||
6. **性能：** 业务对比只要配置了端口映射，所有检查项（包括不含接口字段的 BGP 路由百万行表）都会退回 Python 引擎。割接验收是同步 HTTP 请求，会把路由表全量载入内存。
|
||||
7. **引用保护有漏洞：** 被监控模板引用的对比模板、被任务或割接项目引用的端口映射都能被删除。删除后，业务对比会静默地按“无映射”执行；割接监控会直接报错停摆。
|
||||
|
||||
---
|
||||
|
||||
## 1. 现状框架梳理（给 Cursor 的背景）
|
||||
|
||||
### 1.1 数据流
|
||||
|
||||
```
|
||||
业务监控任务(biz_state_task, 每台设备)
|
||||
└─ 采集批次 batch ──> 每条命令 batch_command(parse_status/row_count/raw_line_count)
|
||||
└─ 解析(ParseProfile → parser → FieldDef) ──> biz_state_metric_row(data_json) / biz_state_lldp_neighbor
|
||||
│
|
||||
对比模板(biz_compare_template.metrics_json = 多个 sheet)
|
||||
sheet: metric_id, key_fields, iface_fields, compare_fields, display_fields,
|
||||
row_filters, field_rules(normalize/compare/tolerance), ignore_port_changes
|
||||
模板级: iface_normalize_json（接口类型别名，两侧同一套）
|
||||
│
|
||||
┌───────────── 业务对比（BizCompareJob/Run/Diff）─────────────┐
|
||||
│ before 批次 vs after 批次（两台设备或同一设备前后） │
|
||||
│ 行过滤(原始值) → 接口归一化(两侧) → 端口映射(仅 before) │
|
||||
│ → 按 match_key 分组 → 同键按采集顺序 zip 配对 │
|
||||
│ → added/removed/changed/unchanged → 通过率 │
|
||||
│ 引擎：PG + 可下推 → SQL FULL OUTER JOIN；否则 Python │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
┌───────────── 割接监控（BizMigrationProject/Batch/Run/Diff）──┐
|
||||
│ 旧设备: 画像基线 vs 高频当前 → old_kind │
|
||||
│ 新设备: 画像基线 vs 高频当前 → new_kind │
|
||||
│ 预期集(expect_set: 本批要迁移的端口/对象) + 监控模板(成功/异常规则)│
|
||||
│ → dual_verdict: migrated/migrating/anomaly/lost/unfinished… │
|
||||
│ → 本批完成(finish_batch)=终验 → 红单(带红继续) │
|
||||
│ 高频采集：每个 metric 一对(新/旧)任务，同轮次配对采样 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 1.2 现有“映射/归一化”层与使用点（不一致的根源）
|
||||
|
||||
| 层 | 归属 | 作用对象 | 业务对比 | 业务对比·实时检索/回填/导出 | 割接监控 |
|
||||
|---|---|---|---|---|---|
|
||||
| 接口类型别名 `iface_normalize_json` | 对比模板 | `iface_fields`，两侧同规则，只替换前缀 | 有 | 有（`_run_transform_config`） | 有 |
|
||||
| 端口映射 `biz_port_mapping` | 任务 / 项目 | `iface_fields`，before→after，精确匹配 + 父接口继承 | **键/值不归一化** | 同左 | **键/值先归一化**（`service.py:1019`） |
|
||||
| 行过滤 `row_filters` | sheet | **原始值**（归一化之前） | 有（PG 可下推） | 有 | 有（Python） |
|
||||
| 字段规则 `field_rules.normalize` | sheet | **仅 compare 字段**，键字段不处理 | 有 | 有 | 仅完整性校验 |
|
||||
| 监控模板 `sheet_overrides` | 监控模板 | 状态语义（up/down 值、成功/异常模式） | — | — | 有 |
|
||||
|
||||
关键文件：
|
||||
- 引擎：`netx_api/biz_state/compare_engine.py`、`compare_rules.py`、`iface_normalize.py`、`compare_sql.py`
|
||||
- 业务对比服务：`netx_api/biz_state/compare_service.py`（约 4000 行）
|
||||
- 割接监控：`netx_api/biz_migration/evaluate.py`、`service.py`、`monitor_templates.py`、`auto_monitor.py`
|
||||
- 采集：`netx_api/biz_state/collect_runner.py`、`claim.py`、`profiles.py`、`parsers/*`
|
||||
- 保留：`netx_api/biz_state/retention.py`
|
||||
- 前端：`web/src/features/cutover/BizComparePage.tsx`、`BizMigrationPage.tsx`、`BizMonitorTemplatesPage.tsx`、`BizStatePage.tsx`
|
||||
- 内置模板：`netx_api/biz_state/data/default_zte_status_template.json`
|
||||
|
||||
---
|
||||
|
||||
## 2. 问题清单
|
||||
|
||||
严重度：**P0** 会产生错误结论或数据损坏；**P1** 高概率误报或明显性能/可用性问题；**P2** 边界问题或体验问题；**P3** 清理项。
|
||||
|
||||
### 2.1 业务对比（BizCompare）
|
||||
|
||||
| ID | 严重度 | 问题 | 位置 | 影响 |
|
||||
|---|---|---|---|---|
|
||||
| B-01 | P0 **[已验证]** | 端口映射的键/值不按模板接口归一化规则转换，必须由用户手工填写“归一化后”的名称；割接监控却会自动归一化。两种模式行为不一致。 | `compare_engine.py:224-260`（`apply_port_map` 用原始 `pmap`），`compare_service.py:2461`、`_run_transform_config:2337` | 模板有 `GE→gei` 规则时，映射 `GE0/1/0/1→gei0/2/0/1` 完全不命中，结果 added=1 / removed=1、`map_ok=False` |
|
||||
| B-02 | P1 | 任务里只要有端口映射，所有 sheet 都会跳过 SQL 引擎，包括没有接口字段的 BGP/路由表 | `compare_sql.py:165`（`if port_map: return "port_map"`） | 百万行 vpnv4 路由退回 Python，内存和耗时都大幅上升；割接任务几乎都会配映射 |
|
||||
| B-03 | P1 | 端口映射被删除后，业务对比静默地按“无映射”执行（还会触发自动忽略端口的启发式），删除时也不检查引用 | `delete_mapping:1603`，`_port_map_dict:1612`，`_validate_compare_job:2220` | 结果语义悄悄改变 |
|
||||
| B-04 | P1 | 对比模板删除只检查 `BizCompareJob` 的引用，不检查 `BizMonitorTemplate.compare_template_id` | `delete_template:1521`，`list_templates:1391` 的 `job_count` | 割接窗口中删掉模板后，`run_evaluate` 报 `compare_template_missing`，监控停摆 |
|
||||
| B-05 | P1 | 创建向导或任务配置里编辑了映射文本但没点“保存映射”，创建/保存/执行任务时映射文本被丢弃，没有任何提示 | `BizComparePage.tsx:1814 jobConfigBody`、`1895 createJob`、`1997 saveJobConfig`、`2029 runNow` | 用户以为配了映射，实际没生效，产生大量假“缺失/新增” |
|
||||
| B-06 | P2 | 映射文本解析只认 `, | Tab` 作分隔符；空格分隔或格式不对的行被静默丢弃，没有解析计数或错误提示 | `BizComparePage.tsx:1720 parseMapRows` | 映射少行而不自知 |
|
||||
| B-07 | P1 | 映射校验只针对“第一个含接口字段的 sheet”，结果以原始 JSON 展示；只报告“映射键未命中”，不报告“before 侧端口没有映射且在 after 侧不存在”（漏配） | `validate_mapping:1810`，`mapping_stats`（`compare_engine.py:100`），`BizComparePage.tsx:2272` | 漏配映射最终表现为业务缺失 |
|
||||
| B-08 | P1 | `row_filters` 作用在原始行（归一化、映射之前）；跨厂商时，同一条过滤在两侧含义不同 | `_load_metric_rows:1619`（SQL/Python 都在加载时过滤），`_run_sheet:2179` | 例如 `interface contains gei` 会把华为 `GE0/0/1` 整侧滤掉；ARP 的 `age_timer`（HH:MM:SS）只适用于中兴 |
|
||||
| B-09 | P2 | 键字段不应用 `field_rules.normalize`（如 MAC 格式），跨厂商的 MAC/大小写差异导致配对失败 | `compare_engine.py:96 row_key`，`compare_sql.py _rk_sql` | 键里含 MAC 的 sheet（如 `l2vpn_mac`）全量误报 |
|
||||
| B-10 | P1 | 两侧都是 0 行时通过率为 100%；解析不到数据（命令执行了但解析 0 行）被当成业务“缺失” | `_enrich_summary:3015`，`compute_pass_rate:210` | 空对空误判为通过；模板不匹配被当成业务丢失 |
|
||||
| B-11 | P1 | 新旧两侧采集命令集不一致（例如旧设备采了 bgp ipv4 summary，新设备没采），对比结果是“全部缺失” | 没有任何检查 | 采集配置问题被误判为业务问题 |
|
||||
| B-12 | P2 | 同键多行按采集顺序 zip 配对，CLI 输出顺序变化会产生假“不一致” | `compare_engine.py:331+`、`compare_sql.py` 中 `dup_rn ORDER BY seq` | ECMP/多路径场景误报 |
|
||||
| B-13 | P2 | 只要提供了映射，`ignore_port_changes` 就被忽略；部分映射时，未映射端口既不按名称匹配也不忽略端口 | `compare_engine.py:224` | 部分映射时策略不可控 |
|
||||
| B-14 | P2 | 结果表的键列只显示映射后的名称，看不到 before 侧原始端口名 | `BizComparePage.tsx:4161`（`d.key[k] ?? pre[k]`，`pre` 取 `mapped_before`） | 跨厂商时现场人员无法对照原端口 |
|
||||
| B-15 | P2 | 判定没有方向：状态 down→up、前缀数增加也算失败；流量计数器被当成严格比较项 | `compare_rules.values_equal` | 割接后业务恢复或流量迁入被判为失败 |
|
||||
| B-16 | P3 | `batch_metric_collect_ok` 是死代码；`_load_metric_rows` 中 LLDP 分支重复过滤；PG 下推时 `before_raw_count` 实际上是过滤后的数量 | `compare_service.py:78`、`1675-1688`、`2199` | 清理 |
|
||||
| B-17 | P3 | 防止同一任务并发对比的锁是进程内的；`_job_has_active_run` 检查和插入不是原子操作。biz_state worker 多副本时，自动对比有极小概率并发 | `compare_service.py:57`、`2887` | 低概率 |
|
||||
|
||||
### 2.2 割接监控（BizMigration）
|
||||
|
||||
| ID | 严重度 | 问题 | 位置 | 影响 |
|
||||
|---|---|---|---|---|
|
||||
| M-01 | P0 **[已验证]** | 端口预设的成功条件要求旧侧 `removed` 或 `admin=down 且 phy=down`。旧端口拔纤但没有 shutdown（admin up / phy down）、新端口已 up 时：窗口内是 `migrating/黄`，**验收时是 `unfinished/红`** | `monitor_templates.py:255-275`（`_success_stateful` 的 `old_down_groups` 用的是 AND） | 正常割接被判为验收失败 |
|
||||
| M-02 | P1 | 每个 run 引用的 HF 批次都会被永久保护，run 和 diff 本身也没有保留策略 | `retention.py:65-75` | 高频监控期间数据库无限增长 |
|
||||
| M-03 | P1 | 验收（`finish_batch`）和手动评估是同步 HTTP 请求；路由类 sheet（`bgp_route`/`ip_route`）会把 4 个批次的全量行载入 Python | `biz_migration_router.py` 的 evaluate/finish，`service.py:1148-1163` | 超时、内存暴涨；而且这期间一直持有项目行锁（`with_for_update`，`service.py:940`） |
|
||||
| M-04 | P1 | 默认监控模板会采集对比模板里的全部 metric，包括只能手动采集的路由表。没有手动采集时，该 sheet 是 `current_missing`，验收**必然**失败（`current_missing` / `coverage_incomplete`） | `resolve_collect_metric_ids:525`、`finish_batch:2358-2371`、`MANUAL_ROUTE_METRICS` | 默认配置跑不通验收 |
|
||||
| M-05 | P1 | 只要任意 sheet 有重复键，验收就判失败（`duplicate_keys`，同时 `coverage_complete=False`），不管这个 sheet 是否在本批范围内 | `service.py:1299`、`2366` | 例如 ISIS L1/L2 两行同键，就永远过不了验收 |
|
||||
| M-06 | P1 | 新旧两侧画像任务的命令集可能不同，HF 任务沿用各自画像的命令项，旧侧有、新侧没采的切片会被判为 `lost` | `_portrait_metric_items:1741`、`ensure_highfreq:1919` | 采集配置问题被当成业务丢失 |
|
||||
| M-07 | P1 | 创建项目时不校验基线批次是否属于该侧画像任务（patch 时会校验） | `create_project:686-691`（对比 `patch_project:744-750`） | 可能拿错设备的基线；直到评估时才报 `baseline_task_mismatch` |
|
||||
| M-08 | P2 | 不支持同设备割接（板卡更换、同框端口迁移）：`collect_pair_same_device` 直接报错 | `service.py:2016` | 常见场景无法使用 |
|
||||
| M-09 | P2 | 每个 metric、每台设备一个 HF 任务；15 个检查项就是每分钟每台设备约 15 次 SSH 登录；没有单设备并发上限 | `ensure_highfreq:1943`，`claim.py` | VTY 耗尽、AAA 日志刷屏，进而采集失败，看板大量 `current_missing` |
|
||||
| M-10 | P2 | 每轮自动评估都重新载入基线全量行（基线不会变） | `run_evaluate.metric_rows:1059` | 白白消耗 CPU/IO |
|
||||
| M-11 | P2 | 双侧评估里，诊断索引的键串用 `str(v)` 构造，而其他地方用 `scalar_text().strip()`：`None` 变成 `"None"`，`True` 变成 `"True"`，空白不裁剪，预期键匹配不上 | `evaluate.py:759`、`compare_engine.py:273 _key_obj` | **[已验证]**：键字段为 `None` 时，存在且未变化的对象被判为 `unfinished` |
|
||||
| M-12 | P2 | 用 metric 名称包含 `bgp` / `pw` 来决定是否用端口映射推导范围（字符串启发式） | `evaluate.py:804` | 新增 metric 时行为不可预期，也违反“引擎不写 metric 特例”的原则 |
|
||||
| M-13 | P2 | 映射允许 N:1 吗？业务对比允许（结果是重复键），割接监控直接报 `mapping_target_ambiguous`；1:N（拆分 breakout）两边都不支持（`before_if` 唯一） | `models/biz_state.py BizPortMappingRow`，`service.py:991/1021` | 端口汇聚、拆分割接无法表达，报错也没说明是哪几行 |
|
||||
| M-14 | P3 | `project_to_dict`（GET/列表）里会写库（`resolve_project_monitor_template` 持久化默认模板）；`list_projects` 存在 N+1 | `service.py:486-503`、`543` | 只读接口有副作用 |
|
||||
| M-15 | P3 | 一个键在后续验收中已经变绿，对应红单仍然保持 open，只能人工关闭 | `_persist_red_tickets_from_run:2326` | 红单堆积 |
|
||||
|
||||
### 2.3 内置模板与跨厂商能力
|
||||
|
||||
| ID | 严重度 | 问题 | 位置 |
|
||||
|---|---|---|---|
|
||||
| T-01 | P1 | `interface_detail.output_bps` 用 `numeric` 绝对容差 5（bps），而 `input_bps` 用 5%：相当于对出方向流量做严格比较 | `default_zte_status_template.json` |
|
||||
| T-02 | P1 | `optical_brief.rx_power` 用 `percent 5`：dBm 是对数单位，-0.5 和 -0.6 已经差 20%。应改为绝对值 dB 容差 | 同上 |
|
||||
| T-03 | P2 | 流量计数器（bps/util）默认参与比较；割接后流量迁移是预期行为，会产生大量噪声 | 同上 |
|
||||
| T-04 | P2 | `l2vpn_mac` 键里的 `ac_port` 是接口，但没放进 `iface_fields`，端口映射不生效 | 同上 |
|
||||
| T-05 | P1 | 华为、思科、华三、Juniper、诺基亚、爱立信的状态 parser 都是空壳（没有注册）；只有 LLDP 跨厂商可用 | `parsers/huawei/__init__.py` 等 |
|
||||
| T-06 | P2 | 各厂商的状态值不统一（`Established`/`ESTAB`、`UP`/`up`/`*down`、`Full/DR`），而对比层只有 `lower`/`upper`/`mac` 几种归一化，没有枚举别名映射 | `compare_rules.normalize_value` |
|
||||
|
||||
---
|
||||
|
||||
## 3. 优化方案（分阶段任务，按顺序执行）
|
||||
|
||||
### 通用执行约束（Cursor 每个任务都要遵守）
|
||||
|
||||
1. **引擎里不写 metric 特例**：所有 metric 相关的行为放进模板、sheet 或 profile 声明（遵循 `skills/biz-state/netx-biz-state/SKILL.md` 第 4、5 节）。
|
||||
2. **Python 和 SQL 必须一致**：凡是影响配对或判定的改动，要么两个引擎都实现，要么在 `sql_compare_skip_reason` 里让 SQL 自动降级到 Python，并补“同输入两引擎结果一致”的对照测试（SQL 侧至少断言生成的 SQL 和降级原因）。
|
||||
3. **历史 run 不变**：新增的执行配置写进 `config_snapshot`，并把 `version` 升到 2。`version==1` 和旧记录的回填、检索、导出保持原行为，不能因为新逻辑改变历史结果。
|
||||
4. **数据库结构**：尽量用 JSON 字段扩展，不加列。确实需要加列或表时，同时补 `alembic/versions/` 和 `netx_api/schema_patches.py`（参照现有做法），并写回归测试。
|
||||
5. **每个任务**：补单元测试；跑第 5 节的回归命令；前端改动跑 `tsc -b` 和 `vite build`，涉及交互的补 `web/tests/*.browser.mjs` 场景（全部 `/v1` 用固定响应）。
|
||||
6. 一个任务一个提交，提交信息写清楚任务 ID（例如 `fix(cutover): B-01 normalize port-map keys`）。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 A：正确性快修（P0/P1，改动小，先做）
|
||||
|
||||
#### A1 端口映射键/值统一归一化（B-01）
|
||||
- 新增函数 `normalize_port_map(port_map, rules) -> (normalized_map, conflicts)`，放在 `iface_normalize.py`：键和值都用 `normalize_iface_name` 处理。归一化后键冲突（两个原始键归一后相同但目标不同）要作为 conflicts 返回。
|
||||
- 业务对比：在 `_run_transform_config`（`compare_service.py:2337`）统一处理，覆盖执行、实时检索、回填、导出所有路径。**只对 `config_snapshot.version>=2` 的 run 生效**；`version==1` 和旧记录保持原样。
|
||||
- 割接监控：`run_evaluate` 和 `list_baseline_expect_objects` 里的内联归一化改为调用这个函数，消除重复实现。
|
||||
- `mapping_stats` 的 `hint` 文案去掉“必须填写归一化后名称”。
|
||||
- **验收测试**：模板规则 `GE→gei`、映射写原始名 `GE0/1/0/1→gei0/2/0/1`，新 run 的结果是 `unchanged=1`；同样的数据在 `version=1` 的 run 上回填，结果不变。
|
||||
|
||||
#### A2 引用保护和缺失检测（B-03、B-04）
|
||||
- `delete_mapping`：被 `BizCompareJob.mapping_id` 或 `BizMigrationProject.mapping_id` 引用时返回 409 `mapping_in_use`，detail 里带引用列表（任务名/项目名）。
|
||||
- `delete_template`：同时检查 `BizMonitorTemplate.compare_template_id`，返回 409 `template_in_use`（detail 区分 job/monitor）。`list_templates` 增加 `monitor_template_count`。
|
||||
- `_validate_compare_job`：`job.mapping_id` 非空但映射不存在时返回 400 `mapping_missing`。
|
||||
- 前端：删除失败时显示引用对象；模板列表显示两类引用数。
|
||||
- **验收测试**：上述 3 种 409/400 的单测；被引用的映射无法删除。
|
||||
|
||||
#### A3 映射文本不会丢失（B-05、B-06）
|
||||
- `parseMapRows` 改为返回 `{rows, invalidLines[]}`：两段文本即使用空格分隔也接受（正好两个 token 时）；重复的 before 行要提示。
|
||||
- 映射文本相对于已保存版本“有修改”时：创建任务、保存配置、立即执行都要先拦截，提供“保存映射并继续 / 放弃修改”两个选项（或者自动先保存映射再保存任务，二选一，推荐自动保存并提示）。
|
||||
- 编辑区下方实时显示“已解析 N 对 / 无效 M 行（行号）”。
|
||||
- 保存一个被其他任务或项目共用的映射时，提示“该映射被 X 个任务 / Y 个割接项目使用”（后端 `list_mappings` 返回引用计数）。
|
||||
- **验收（浏览器脚本）**：在向导里输入映射后直接点“创建”，请求体里的 `mapping_id` 非空，或者被拦截并出现提示；空格分隔的行能被解析；无效行有提示。
|
||||
|
||||
#### A4 端口预设成功条件修正（M-01）
|
||||
- `preset_override_for_metric("interface_brief")` 中旧侧的条件改为三个 OR 组：`[removed]`、`[phy in down]`、`[admin in down]`；新侧保持 `admin up 且 phy up`。
|
||||
- 已存在的监控模板：在 `ensure_default_monitor_templates` 中增加一个幂等升级，**只替换结构上和旧预设完全一致的 override**（逐字段对比），用户自定义过的不动，并在 `override_warnings` 里提示“端口成功条件为旧版 AND 语义”。
|
||||
- **验收测试**：新增 4 个用例——旧侧 admin up/phy down、admin down/phy up、两者都 down、端口已删除；在窗口内和验收时，新侧 up 都应判为 `migrated/green`。另外，新侧 phy down 时仍判为红。
|
||||
|
||||
#### A5 创建项目时校验基线归属（M-07）
|
||||
- `create_project` 复用 `patch_project` 的校验：基线批次的 `task_id` 必须等于对应侧的画像任务，状态必须是 `success` 或 `partial`。抽成公共函数 `_validate_baseline_ownership`。
|
||||
- **验收测试**：拿其他设备的批次作为基线创建项目，返回 400 `old_baseline_invalid` / `new_baseline_invalid`。
|
||||
|
||||
#### A6 键串规范化统一（M-11）
|
||||
- 在 `compare_engine.py` 新增 `key_tuple(row, fields)`，统一用 `scalar_text(...).strip()`。`_key_obj` 的输出键值和 `build_diff_index_from_compare` 都改用它（`evaluate.py:759` 不再用 `str(v)`）。
|
||||
- **验收测试**：键字段为 `None`、`True`、带首尾空格时，预期键能命中，判定为 `ok`/`migrated` 而不是 `unfinished`。
|
||||
|
||||
#### A7 内置模板参数修正（T-01、T-02、T-04）
|
||||
- `default_zte_status_template.json`：
|
||||
- `output_bps` 改为 `percent 5`（和 `input_bps` 对称）。
|
||||
- `rx_power` 改为 `numeric`，`tolerance: 2`（单位 dB）。
|
||||
- `l2vpn_mac.iface_fields` 加上 `ac_port`。
|
||||
- `interface_detail` 的 `input_bps/output_bps/in_util/out_util` 移出 `compare_fields`，只保留在 `display_fields`（见 V3 的方向性规则；在 V3 完成之前先不比较）。
|
||||
- `ensure_default_zte_status_template` 增量升级：**只修改值仍等于旧打包值的字段**，用户改过的保留。
|
||||
- **验收测试**：旧打包值的模板升级后变成新值；用户自定义过 `rx_power` 规则的模板升级后保持不变。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 B：统一的“身份归一化流水线”（跨厂商核心）
|
||||
|
||||
目标：业务对比执行、实时检索、回填、导出、映射校验、割接评估、预期集选择器这 7 条路径，**对同一行算出相同的 match key**，并支持按侧（按厂商）的规则和字段值映射。
|
||||
|
||||
#### B1 抽出 `IdentityPipeline`（解决“各路径各写一遍”）
|
||||
- 新文件 `netx_api/biz_state/identity_pipeline.py`：
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class SideSpec:
|
||||
iface_rules: list[dict] # 该侧接口归一化（已合并 both + side）
|
||||
value_maps: dict[str, dict] # field -> {from_lower: to}
|
||||
@dataclass(frozen=True)
|
||||
class Pipeline:
|
||||
before: SideSpec; after: SideSpec
|
||||
port_map: dict[str, str] # 已归一化（A1）
|
||||
key_fields, iface_fields, compare_fields, field_rules, row_filters, match_keys
|
||||
def transform(self, row, side) -> dict # 归一化 → 值映射 →（before 侧）端口映射
|
||||
def match_key(self, row, side) -> tuple
|
||||
def keep(self, row, side) -> bool # 行过滤（在 transform 之后求值，见 B4）
|
||||
```
|
||||
- 由 `build_pipeline(sheet, snapshot_or_template, port_map, before_vendor, after_vendor)` 统一构造；`compare_rows`、`_live_search_sheet_diffs`、`_load_search_key_groups`、`_hydrate_diff_rows`、`validate_mapping`、`evaluate_metric_dual`、`list_baseline_expect_objects` 全部改用它。
|
||||
- **验收测试**：参数化对照测试，用同一份 fixture 断言 7 条路径对同一行给出的 match key 相同（重点覆盖：映射、父接口继承、QinQ、值映射、归一化规则）。
|
||||
|
||||
#### B2 分侧接口归一化 + 厂商内置规范化
|
||||
- `iface_normalize_rules` 每条增加可选字段 `side: "both"|"before"|"after"`（默认 `both`）和 `vendor`（可选，按批次的厂商自动生效）。前端模板编辑器支持选择。
|
||||
- 内置“厂商内全称/缩写”规范化表（`iface_normalize.py` 中的 `VENDOR_CANONICAL`）：
|
||||
- 华为：`GigabitEthernet`↔`GE`、`XGigabitEthernet`↔`XGE`、`100GE`、`Eth-Trunk`
|
||||
- 思科：`GigabitEthernet`↔`Gi`、`TenGigE`↔`Te`、`HundredGigE`↔`Hu`、`Bundle-Ether`↔`BE`
|
||||
- 华三：`GigabitEthernet`↔`GE`、`Ten-GigabitEthernet`↔`XGE`、`Bridge-Aggregation`↔`BAGG`
|
||||
- 中兴：大小写
|
||||
- 统一去掉名称中的空格。
|
||||
|
||||
只做厂商内同义统一，**不做跨厂商编号转换**（那是端口映射的职责）。模板 sheet 可以用 `builtin_vendor_canonical: false` 关闭。
|
||||
- 批次的厂商取 `BizStateBatch.vendor`，取不到再用任务的 vendor，经 `resolve_vendor_key` 解析。
|
||||
- `config_snapshot` v2 记录两侧最终生效的规则。
|
||||
- **验收测试**：华为 `GigabitEthernet0/0/1` 与 `GE0/0/1` 互认；before 华为、after 中兴，只对 before 生效的规则不影响 after。
|
||||
|
||||
#### B3 字段值映射（T-06、B-09）
|
||||
- `field_rules[]` 新增可选字段：
|
||||
- `value_map: {"estab": "established", "*down": "down", ...}`（大小写不敏感匹配，输出规范值）
|
||||
- `side: both|before|after`
|
||||
- `apply_to_key: true|false`（默认：字段在 key_fields 中时为 true）
|
||||
- `normalize` 的处理顺序：`normalize`（strip/lower/mac…）→ `value_map`。**键字段也要应用**（解决 MAC 作为键时跨厂商对不上的问题）。
|
||||
- SQL：`value_map` 作用在键或比较字段上时，`sql_compare_skip_reason` 返回 `value_map`，降级到 Python（以后可以用 `CASE` 下推，本期不做）。
|
||||
- `compare_validation.py` 校验 `value_map` 必须是 `str→str` 映射，并且每个 sheet 最多 500 项。
|
||||
- 模板编辑器：字段规则行增加“值映射”编辑（每行一对，可以粘贴）。
|
||||
- **验收测试**：`state: Estab` 对 `established`，配置值映射后为 unchanged；MAC 作为键字段时，`00e0-fc12-3456` 与 `00e0.fc12.3456` 能够配对；Python 与 SQL 降级原因断言。
|
||||
|
||||
#### B4 行过滤在归一化之后求值（B-08）
|
||||
- 新 run（snapshot v2）：先 `transform` 再 `row_filters`。
|
||||
- PG 下推只在“过滤涉及的字段都不在 `iface_fields` 中、也没有值映射”时启用；否则 Python 过滤。
|
||||
- 支持过滤条件带 `side`（只作用于某一侧），满足“旧侧只看 dynamic ARP”这类需求。
|
||||
- 模板编辑器提示“过滤在归一化之后执行，请按规范值书写”。
|
||||
- **验收测试**：规则 `GE→gei` 加过滤 `interface contains gei`，before 侧华为 `GE0/0/1` 不再被滤掉；`version=1` 的 run 回放结果不变。
|
||||
|
||||
#### B5 端口映射覆盖率预检（B-07、B-11、M-06 的基础）
|
||||
- 新接口：`POST /v1/biz-state/compare/preflight`，请求体 `{template_id, enabled_sheet_ids, mapping_id | mapping_rows, before_batch_id, after_batch_id}`。割接监控复用同一个服务函数：`/v1/biz-migration/projects/{id}/preflight`，传入基线和最新 HF 批次。
|
||||
- 按 sheet 返回：
|
||||
- `before_count / after_count`（过滤后）、`parse` 状态（见 V1）
|
||||
- `mapped_exact[]`、`mapped_by_parent[]`、`identity_same_name[]`（未映射但两侧同名）
|
||||
- **`suspect_unmapped[]`**：before 侧接口没有映射，after 侧也不存在 → 疑似漏配
|
||||
- `unused_map_keys[]`、`map_targets_missing_on_after[]`、`n_to_1_targets[]`
|
||||
- `command_coverage`：两侧执行过的 profile_id 集合和差集（见 V2）
|
||||
- 汇总：映射覆盖率 = 已映射 / (已映射 + 疑似漏配)，以及“可执行 / 有风险”的结论。
|
||||
- 业务对比 run 的 summary 也写入同结构的 `mapping_coverage`（按 sheet），结果页可以查看。
|
||||
- 前端：映射块里的“校验”按钮改为显示预检表格（取代原始 JSON）；疑似漏配可以一键加入映射编辑区（目标留空，待填）。
|
||||
- **验收测试**：构造 3 个端口，分别是精确映射、父接口继承、漏配，预检分类正确；`mapping_rows` 未保存时也能预检。
|
||||
|
||||
#### B6 SQL 引擎条件放宽（B-02）
|
||||
- `sql_compare_skip_reason`：`port_map` 只在 `iface_fields ∩ (key_fields ∪ compare_fields)` 非空时才阻止 SQL。
|
||||
- **验收测试**：有端口映射的任务，`bgp_route` sheet（`iface_fields=[]`）走 SQL（断言 skip reason 为空），`interface_brief` 仍走 Python。
|
||||
|
||||
#### B7 映射语义：N:1 与部分映射（B-13、M-13）
|
||||
- 决策见第 6 节，默认方案：
|
||||
- **N:1**：允许，但预检和 summary 中必须列出；业务对比按重复键规则处理（V4）；割接监控继续拒绝，但错误信息里列出冲突行。
|
||||
- **1:N（breakout）**：本期不支持，在映射编辑器里检测到重复 before 时明确报错，不再只报 `duplicate_before_if`。
|
||||
- **部分映射**：新增 sheet 选项 `unmapped_port_policy: "by_name"|"ignore_port"`（默认 `by_name`，即当前行为），让 `ignore_port_changes` 与映射可以组合使用。
|
||||
- **验收测试**：N:1 在预检中出现在 `n_to_1_targets`；割接监控的报错包含冲突对。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 C：判定语义与数据可信度
|
||||
|
||||
#### V1 sheet 级数据可信度（B-10）
|
||||
- `_run_sheet` 和割接评估为每侧补充 `collect_quality`：该 metric 的命令数、`parse_status` 计数、`row_count`、`raw_line_count`、`declared_total`（只读这些列，不读 `raw_text`）。
|
||||
- 新增 sheet 状态：
|
||||
- `no_data`：两侧过滤后都是 0 行，且命令都执行成功 → `pass_rate=null`，不计入总通过率，卡片显示“无数据”。
|
||||
- `suspect_parse`：某侧 `raw_line_count > 5` 但 `row_count == 0` → 卡片显示“疑似解析失败”，失败行数照常显示，但给出明确提示。
|
||||
- `not_collected`：某侧没有任何命令产生该 metric → 不做判定，`pass_rate=null`。
|
||||
- 任务级 `ok`：任一 sheet 处于 `not_collected` / `suspect_parse` 时为 `null`（未判定），不再是 `true`。
|
||||
- 前端卡片和 CSV `_summary.csv` 增加状态列。
|
||||
- **验收测试**：0 对 0 → `no_data`；before 1000 行、after 有原文但 0 行 → `suspect_parse`；after 侧无命令 → `not_collected`。
|
||||
|
||||
#### V2 采集命令集对齐（B-11、M-06）
|
||||
- `ParseProfile` 增加可选字段 `slice: dict[str, str]`（例如 `zte.bgp_ipv4_summary` 的 slice 是 `{"afi": "ipv4"}`），声明这个 profile 产出哪个切片。先给中兴的 `bgp_peer` / `bgp_route` / `vrrp` / `isis` profile 补齐。
|
||||
- 判断一个 sheet 是否“被某侧覆盖”：该侧存在 `metric_id` 相同的成功命令，且（sheet 的 row_filters 里有 `eq` 条件时）存在 slice 与之相符的 profile。
|
||||
- 业务对比的预检（B5）和执行时都做这个检查：未覆盖的 sheet 标记 `not_collected`（V1）。
|
||||
- 割接监控：`ensure_highfreq` 生成 HF 任务项时，取**两侧画像命令项的并集**（按 profile 的 slice 转换成该侧厂商可用的 profile；找不到时返回明确错误 `no_profile_for_slice:<metric>:<slice>:<vendor>`）。评估时，未覆盖的 sheet 标记为 `not_collected`，不判红。
|
||||
- **验收测试**：旧画像有 ipv4 + vpnv4，新画像只有 vpnv4 → HF 任务两侧都有 ipv4 + vpnv4；如果强制只有一侧有，评估结果是 `not_collected`，而不是 `lost`。
|
||||
|
||||
#### V3 方向性判定（B-15、T-03）
|
||||
- `field_rules.compare` 新增：
|
||||
- `not_worse`：配合 `rank: ["up", "down"]`（排在前面表示更好）；after 不差于 before 即通过。
|
||||
- `numeric_min`：配合 `tolerance`（百分比），`after >= before*(1-tol%)` 即通过（适用于前缀数、流量“不应明显下降”）。
|
||||
- `numeric_max`：对称方向。
|
||||
- `explain_diff` 输出方向原因，例如 `worse: up→down`、`drop 37% > 10%`。
|
||||
- SQL：先降级（skip reason 为 `directional_compare`），后续再下推。
|
||||
- 内置模板：`interface_brief.prot`、`bgp_peer.state` 默认仍为 `eq`（保守）；在模板编辑器里提供这些新模式供选择；流量计数器若要比较，推荐用 `numeric_min`。
|
||||
- **验收测试**：`down→up` 用 `not_worse` 判为通过，`up→down` 判为失败；前缀数 1000→1200 用 `numeric_min 5%` 判为通过，1000→900 判为失败。
|
||||
|
||||
#### V4 重复键配对：先排序再配对（B-12）
|
||||
- Python：同一个 match key 下，两侧各自按“归一化后的 compare 字段元组, 原始 seq”排序，然后 zip。
|
||||
- SQL：`dup_rn` 改为 `row_number() OVER (PARTITION BY rk ORDER BY <compare 字段归一化表达式...>, seq, id)`。
|
||||
- 用 sheet 选项 `duplicate_pairing: "sorted"|"ordered"` 控制，默认 `sorted`，只对 snapshot v2 生效；v1 回放保持 `ordered`。
|
||||
- **验收测试**：同键两行的值 `[a, b]` 对 `[b, a]` → 2 个 unchanged（原来是 2 个 changed）；Python 和 SQL 生成 SQL 的断言。
|
||||
|
||||
#### V5 原始身份展示（B-14）
|
||||
- diff 增加 `raw_key_before`（before 原始键值，未归一化/映射）。实现方式：在 Python 路径生成；对精简行，在回填时由 `IdentityPipeline` 反推；SQL 路径没有映射，原始键等于 key。
|
||||
- 结果表：当原始值与映射后的值不同时，键单元格显示 `cgei-0/2/0/1 ← GE0/0/1`；CSV 增加 `<field>__raw_before` 列（只针对接口键）。
|
||||
- **验收（浏览器脚本）**：映射行同时显示两侧端口名。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 D：割接监控的健壮性与性能
|
||||
|
||||
#### H1 run、diff 保留策略 + 批次保护收敛（M-02）
|
||||
- 新配置项：`biz_migration_auto_run_keep`（默认每个 batch 保留 300 个 auto run）、`biz_compare_auto_run_keep`（默认每个 job 保留 50 个 auto run）。
|
||||
- 清理规则：
|
||||
- 永久保留：`purpose in (acceptance, manual)` 的 run，以及被红单 `run_id` 引用的 run。
|
||||
- 其余 auto run：超过数量上限的删除（连同其 diff）。
|
||||
- 保护范围：`retention.protected_batch_map` 只保护“仍然存在的 run”所引用的批次，并**加上 `summary.metric_batches` 里的所有批次**（目前只保护首个 binding 的批次）。
|
||||
- 清理挂在 `biz_state_scheduler._maybe_purge_retention` 上，每小时执行一次。
|
||||
- **验收测试**：插入 400 个 auto run 和 1 个验收 run，清理后剩 300 + 1；被删除 run 所引用的 HF 批次不再受保护，能按 `retention_days` 被清理；验收 run 引用的所有批次（包括 `metric_batches`）都受保护。
|
||||
|
||||
#### H2 评估和验收改为后台执行（M-03）
|
||||
- 仿照业务对比的 `enqueue_compare`：`POST /batches/{id}/evaluate` 和 `/finish` 返回 `{run_id, status: "running"}`，在后台线程里执行，`BizMigrationRun.status` 依次经过 `running/success/failed`，`summary.progress` 记录进度。
|
||||
- 项目行锁只在“写 run 和 diff”这个短事务里持有，不再覆盖整个计算过程。
|
||||
- 重启恢复：参照 `recover_interrupted_compares_on_startup`，把遗留的 `running` 标记为 `cancelled`。
|
||||
- 前端：轮询 run 的状态；验收按钮在执行期间不可重复点击。
|
||||
- **验收测试**：评估请求立即返回；并发两次评估，第二次返回 409；模拟重启后状态被正确恢复。
|
||||
|
||||
#### H3 路由类 sheet 的处理（M-04）
|
||||
- 默认：`MANUAL_ROUTE_METRICS` 对应的 sheet **不参与验收门槛**。没有在本批窗口内手动采集时，显示为 `manual_pending`（灰色），不计入 `current_missing` 或 `coverage_incomplete`；已采集时参与判定。
|
||||
- 路由 sheet 的双侧比对改为复用业务对比的 SQL 路径（旧基线 vs 旧当前、新基线 vs 新当前，各执行一次 `run_sql_sheet_compare`，只取计数和失败行），避免在 Python 里载入百万行。
|
||||
- **验收测试**：默认监控模板、没有采集路由时，验收不因路由失败；采集了路由时，判定会出现在结果里。
|
||||
|
||||
#### H4 基线行缓存（M-10）
|
||||
- 进程内 LRU：键为 `(batch_id, metric_id)`，只缓存基线批次（基线不可变），总行数上限可配置（默认 50 万行）。基线变更或项目删除时失效。
|
||||
- **验收测试**：连续两次 auto 评估，第二次不再查询基线行（mock 计数）。
|
||||
|
||||
#### H5 重复键的处理口径（M-05）
|
||||
- `duplicate_keys` 只在“本批范围内的键（expect）有重复”时作为验收失败原因；其他 sheet 的重复键作为警告，写进 `accept_summary.warnings`。
|
||||
- 卡片上展示具体是哪些重复键（已有 `duplicate_key_list`），并提示“完善 key_fields（例如 ISIS 加上 `lev`）或在 sheet 上启用 `duplicate_pairing: sorted`”。
|
||||
- **验收测试**：范围外的 sheet 有重复键时验收可以通过；范围内有重复时验收失败。
|
||||
|
||||
#### H6 HF 采集合并与单设备并发上限（M-09）
|
||||
- 决策见第 6 节，默认：**同间隔、同 lane（light/heavy）的 metric 合并为一对任务**（一次 SSH 采多条命令）；heavy 和 manual 的路由表保持独立。
|
||||
- 新配置项 `biz_state_max_sessions_per_ne`（默认 2）：`claim_queued_batches` 认领时，同一个 `ne_id` 正在运行的批次达到上限就跳过（成对认领时要两侧都满足）。
|
||||
- `_hf_bindings` 的结构保持兼容（`metric_ids` 本来就是列表）。
|
||||
- **验收测试**：15 个 light metric、同一间隔 → 每侧 1 个 HF 任务；单设备并发上限生效（3 个同 NE 批次只认领 2 个）。
|
||||
|
||||
#### H7 同设备割接模式（M-08）
|
||||
- 当 `old_ne == new_ne` 时允许创建项目：每个 metric 只建一个 HF 任务，新旧两侧共用同一批次（`collect_group_id` 只含一个任务，轮次天然对齐），映射表达同框端口迁移。
|
||||
- `evaluate_metric_dual` 不需要改（两侧是同一份行数据，靠映射区分新旧端口）。
|
||||
- `enqueue_collect_group` 允许单任务组（`mode=single_ne`）。
|
||||
- **验收测试**：同设备项目可以创建；端口 A 迁移到 B（A 变为 down、B 变为 up）判为 `migrated`。
|
||||
|
||||
#### H8 去掉 metric 名称启发式（M-12）
|
||||
- `sheet_overrides` 增加 `scope_by_port_map: true|false`；不填时按当前规则推导，并把推导结果写进 override 的 `effective_*` 字段供界面展示。`_map_defines_iface_expect` 不再读取 metric 名称。
|
||||
- **验收测试**：一个名称不含 bgp 的 sheet 设置 `scope_by_port_map=false` 后，不会被映射推导出范围。
|
||||
|
||||
#### H9 其他（M-14、M-15）
|
||||
- GET 接口不写库：`project_to_dict` 中的默认模板绑定改为只读解析，持久化只在 create/patch 时进行。
|
||||
- 红单：在后续验收中已转绿的键标记为 `resolvable=true`（不自动关闭），前端提供“一键关闭可关闭红单”。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 E：跨厂商解析（需要你提供真实样本）
|
||||
|
||||
#### P1 华为 VRP 首批状态解析（T-05）
|
||||
- **前提**：每个命令提供至少 2 份真实输出样本，放到 `tests/fixtures/huawei/`，并做脱敏（不同版本、带子接口和 VRF 的各一份）。**没有样本时 Cursor 不得凭空编写 TextFSM。**
|
||||
- 顺序：`interface_brief`（`display interface brief`）、`arp`（`display arp all`）、`bgp_peer`（`display bgp peer` / `display bgp vpnv4 all peer`）、`isis_adjacency`（`display isis peer`）、`ospf_neighbor`（`display ospf peer brief`）。
|
||||
- 每个 metric 输出**与中兴相同的字段名**（FieldDef 共用），状态值输出**规范枚举**：
|
||||
- 接口 `admin/phy/prot ∈ {up, down}`；华为的 `*down` 输出为 `admin=down`
|
||||
- BGP `state ∈ {established, idle, active, connect, opensent, openconfirm}`
|
||||
- ARP `entry_type ∈ {dynamic, static, interface}`
|
||||
|
||||
这样对比层的过滤和预设不需要按厂商分叉。
|
||||
- 按 `skills/biz-state/netx-biz-state/SKILL.md` 第 2 节逐项完成：TextFSM、`prefer_fsm`、注册、profile（附 `slice`）、单测。
|
||||
- 新增“规范值契约”文档：`docs/biz-state-canonical-fields.md`，按 metric 列出字段、取值范围，以及各厂商原始值到规范值的对照。中兴 parser 也要对照这份契约补齐规范化（例如 `entry_type`）。
|
||||
- ARP 过滤：内置模板从 `age_timer` 改为 `entry_type eq dynamic`（依赖各厂商 parser 输出规范的 `entry_type`）；中兴 parser 在 `entry_type` 为空时，根据 age 推断出 `dynamic`。
|
||||
- **验收**：fixture 单测；中兴→华为的跨厂商对比用例（固定行数据）在配置映射和值映射后通过率为 100%。
|
||||
|
||||
---
|
||||
|
||||
### 阶段 F：前端体验（和上面各任务一起做）
|
||||
|
||||
1. 映射编辑器升级为表格模式（保留文本模式）：两列可编辑，支持粘贴 CSV/Excel，逐行显示校验状态（重复、无效、归一化后的值）。“从基线生成”：左列 before 端口、右列 after 端口，可按“相同槽位/端口号”自动推荐。
|
||||
2. 预检结果表（B5）：疑似漏配、未使用的键、N:1、命令覆盖差集，每项都可以展开查看明细。
|
||||
3. sheet 卡片显示 `no_data` / `suspect_parse` / `not_collected` / `manual_pending` 状态和原因（V1、H3）。
|
||||
4. 结果表的键列显示原始端口名（V5）。
|
||||
5. 模板编辑器：接口归一化规则增加 side/vendor 选择；字段规则增加值映射和方向性比较；过滤条件可选 side；行过滤处提示“按归一化后的值书写”。
|
||||
6. 割接看板：重复键警告、未覆盖切片、预设语义说明（端口成功条件）。
|
||||
|
||||
---
|
||||
|
||||
## 4. 建议执行顺序
|
||||
|
||||
| 顺序 | 任务 | 说明 |
|
||||
|---|---|---|
|
||||
| 1 | A1–A7 | 正确性快修，相互独立，可以并行 |
|
||||
| 2 | B1 → B2 → B3 → B4 | 流水线是 B2–B4 的基础，必须先做 |
|
||||
| 3 | B5、B6、B7 | 预检依赖流水线 |
|
||||
| 4 | V1 → V2 → V3 → V4 → V5 | V2 依赖 V1 的状态定义 |
|
||||
| 5 | H1 → H2 → H3 → H4 → H5 → H6 → H7 → H8 → H9 | H1/H2 优先（生产风险最大） |
|
||||
| 6 | 阶段 E | 等你提供样本 |
|
||||
| 7 | 阶段 F | 随各任务交付；最后做一次整体走查 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 验收方式（Claude 验收时执行）
|
||||
|
||||
### 5.1 回归命令
|
||||
```powershell
|
||||
# 项目根目录：割接对比相关后端回归（基线 279 项通过）
|
||||
.venv\Scripts\python.exe -m pytest tests/test_biz_state_compare.py tests/test_biz_state_compare_sql.py tests/test_biz_state_compare_search.py tests/test_biz_state_compare_snapshot.py tests/test_biz_state_compare_export.py tests/test_biz_state_compare_recovery.py tests/test_biz_compare_templates.py tests/test_biz_migration_evaluate.py tests/test_biz_migration_evidence.py tests/test_biz_migration_hf_decouple.py tests/test_biz_migration_integrity.py tests/test_iface_normalize.py tests/test_biz_state_monitor_service.py tests/test_biz_state_paired_sampling.py tests/test_biz_state_retention.py -q
|
||||
# 全量
|
||||
.venv\Scripts\python.exe -m pytest -q
|
||||
|
||||
# web 目录
|
||||
node node_modules/typescript/bin/tsc -b
|
||||
node node_modules/vite/bin/vite.js build
|
||||
node --experimental-strip-types --test tests/cutoverDataCache.test.mjs tests/compareTemplateModel.test.mjs
|
||||
# 浏览器（先启动 vite :5179）
|
||||
node tests/bizCompare.browser.mjs
|
||||
node tests/bizCompareTemplates.browser.mjs
|
||||
node tests/bizMigration.browser.mjs
|
||||
node tests/bizState.browser.mjs
|
||||
```
|
||||
|
||||
### 5.2 必须新增的测试文件（便于验收定位）
|
||||
- `tests/test_identity_pipeline.py`：B1 的 7 路径对照、B2、B3、B4
|
||||
- `tests/test_compare_preflight.py`：B5、B7、V2
|
||||
- `tests/test_compare_quality.py`：V1、V3、V4
|
||||
- `tests/test_biz_migration_presets.py`：A4 的 4 个端口场景、H5、H8
|
||||
- `tests/test_biz_migration_retention.py`：H1
|
||||
- `tests/test_biz_migration_async.py`：H2、H3
|
||||
- `tests/test_cutover_references.py`：A2、A5
|
||||
- 浏览器：`bizCompare.browser.mjs` 增加 A3、B5、V5 场景；`bizMigration.browser.mjs` 增加 H2、V1 卡片状态场景
|
||||
|
||||
### 5.3 我会逐条核对的场景
|
||||
1. 原始名称映射（A1）：`GE0/1/0/1→gei0/2/0/1` 加规则 `GE→gei`，结果为 unchanged。
|
||||
2. 拔纤但未 shutdown（A4）：验收判 `migrated`。
|
||||
3. 删除被引用的映射或模板（A2）：返回 409。
|
||||
4. 向导中映射未保存（A3）：不会被静默丢弃。
|
||||
5. 跨厂商过滤（B4）：华为侧不会被 `contains gei` 整侧滤掉。
|
||||
6. MAC 作为键（B3）：不同格式能配对。
|
||||
7. 空对空（V1）：显示 `no_data`，不是 100%。
|
||||
8. 命令集不一致（V2）：显示 `not_collected`，不是 `lost`。
|
||||
9. 同键乱序（V4）：不再出现假“不一致”。
|
||||
10. 有映射时 BGP 路由走 SQL（B6）。
|
||||
11. auto run 被清理、被引用批次的保护随之释放（H1）。
|
||||
12. 验收异步执行、不阻塞请求（H2）；默认模板不因路由未采集而验收失败（H3）。
|
||||
13. 历史 run（snapshot v1）的检索、回填、导出结果与改动前一致。
|
||||
|
||||
---
|
||||
|
||||
## 6. 需要你确认的决策（Cursor 按“默认”执行，除非你另行指定）
|
||||
|
||||
| # | 决策 | 默认 |
|
||||
|---|---|---|
|
||||
| D1 | 业务对比是否允许 N:1 映射 | 允许，但在预检和结果中明确列出；割接监控继续拒绝并说明冲突行 |
|
||||
| D2 | 1:N（端口拆分 breakout）本期是否支持 | 不支持，明确报错 |
|
||||
| D3 | 空对空 / 疑似解析失败时，任务级“是否通过” | `null`（未判定），不计入通过率，不当作失败 |
|
||||
| D4 | 路由表是否参与割接验收门槛 | 未采集时不参与（`manual_pending`），采集后参与 |
|
||||
| D5 | HF 采集是否按“同间隔 + 同 lane”合并任务 | 合并；每台设备并发 SSH 上限默认 2 |
|
||||
| D6 | 是否支持同设备割接 | 支持（H7） |
|
||||
| D7 | auto run 保留数量 | 割接每批 300、业务对比每个任务 50；验收和手动 run 永久保留 |
|
||||
| D8 | 流量计数器默认是否参与比较 | 不参与；需要时用 `numeric_min` |
|
||||
| D9 | 华为样本由谁提供、首批覆盖哪些命令 | 由你提供；首批为接口、ARP、BGP、ISIS、OSPF |
|
||||
|
|
@ -170,6 +170,14 @@ def run_api_startup() -> None:
|
|||
)
|
||||
except Exception:
|
||||
_log.exception("startup: biz compare recovery failed")
|
||||
try:
|
||||
from .biz_migration.service import recover_interrupted_evaluations_on_startup
|
||||
|
||||
n_eval = recover_interrupted_evaluations_on_startup(db)
|
||||
if n_eval:
|
||||
_log.info("startup: cancelled %s interrupted cutover evaluation(s)", n_eval)
|
||||
except Exception:
|
||||
_log.exception("startup: cutover evaluation recovery failed")
|
||||
try:
|
||||
from .port_traffic_migrate import backfill_port_traffic_series
|
||||
|
||||
|
|
|
|||
|
|
@ -44,6 +44,11 @@ def try_auto_monitor_for_task(db: Session, task_id: str) -> int:
|
|||
if not batch or proj.status == "done":
|
||||
db.rollback()
|
||||
continue
|
||||
if db.query(BizMigrationRun.id).filter(
|
||||
BizMigrationRun.project_id == proj.id, BizMigrationRun.status == "running"
|
||||
).first():
|
||||
db.rollback() # a manual / acceptance evaluation owns the project right now
|
||||
continue
|
||||
mt = db.get(BizMonitorTemplate, proj.monitor_template_id) if proj.monitor_template_id else None
|
||||
if not mt:
|
||||
db.rollback()
|
||||
|
|
@ -60,7 +65,8 @@ def try_auto_monitor_for_task(db: Session, task_id: str) -> int:
|
|||
batch.expect_set_json, service._port_map_dict(db, proj.mapping_id),
|
||||
sheets, overrides, defaults, norms, service.resolve_collect_metric_ids(db, proj)]
|
||||
fingerprint = hashlib.sha256(json.dumps(payload, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
|
||||
last = db.query(BizMigrationRun).filter(BizMigrationRun.batch_id == batch.id).order_by(
|
||||
last = db.query(BizMigrationRun).filter(
|
||||
BizMigrationRun.batch_id == batch.id, BizMigrationRun.status == "success").order_by(
|
||||
BizMigrationRun.created_at.desc()).first()
|
||||
if last and (last.summary_json or {}).get("auto_fingerprint") == fingerprint:
|
||||
db.rollback()
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ from __future__ import annotations
|
|||
|
||||
from typing import Any
|
||||
|
||||
from ..biz_state.compare_engine import apply_port_map, compare_rows
|
||||
from ..biz_state.compare_engine import apply_port_map, compare_rows, key_tuple
|
||||
from ..biz_state.compare_rules import (
|
||||
effective_compare_fields, field_rule_map, scalar_text, values_equal,
|
||||
)
|
||||
|
|
@ -756,10 +756,10 @@ def build_diff_index_from_compare(result: dict[str, Any]) -> dict[str, dict[str,
|
|||
continue
|
||||
key = d.get("key")
|
||||
if isinstance(key, dict):
|
||||
key_list = [str(v) for v in key.values()]
|
||||
key_list = [scalar_text(v).strip() for v in key.values()]
|
||||
ks = _key_str(key_list)
|
||||
elif isinstance(key, (list, tuple)):
|
||||
key_list = [str(x) for x in key]
|
||||
key_list = [scalar_text(x).strip() for x in key]
|
||||
ks = _key_str(key_list)
|
||||
else:
|
||||
ks = str(key or "")
|
||||
|
|
@ -801,17 +801,18 @@ def expect_keys_for_metric(
|
|||
return keys
|
||||
|
||||
|
||||
def _map_defines_iface_expect(metric_id: str, iface_fields: list[str], port_map: dict[str, str]) -> bool:
|
||||
"""Iface sheets: a non-empty port map is the expect scope.
|
||||
def _map_defines_iface_expect(
|
||||
iface_fields: list[str], port_map: dict[str, str], scope_by_port_map: bool | None = None,
|
||||
) -> bool:
|
||||
"""Whether a non-empty port map is this sheet's expect scope.
|
||||
|
||||
BGP / PW keep the batch expect picker even if a column is marked iface.
|
||||
The sheet override decides (``scope_by_port_map``); the built-in presets set it to
|
||||
false for BGP / pseudowire sheets, which keep the batch expect picker. Unset means
|
||||
"an interface sheet with a map is scoped by the map".
|
||||
"""
|
||||
if not port_map or not iface_fields:
|
||||
return False
|
||||
mid = str(metric_id or "").strip().lower()
|
||||
if "bgp" in mid or "pw" in mid or "pseudowire" in mid:
|
||||
return False
|
||||
return True
|
||||
return True if scope_by_port_map is None else bool(scope_by_port_map)
|
||||
|
||||
|
||||
def _iface_values(key_str: str, *, key_fields: list[str], iface_fields: list[str]) -> list[str]:
|
||||
|
|
@ -1022,11 +1023,14 @@ def row_match_key(
|
|||
key_fields: list[str],
|
||||
iface_fields: list[str],
|
||||
rules: list[dict[str, str]] | None,
|
||||
normalizer: Any = None,
|
||||
) -> str:
|
||||
from ..biz_state.iface_normalize import apply_iface_normalize
|
||||
|
||||
data = strip_netx(row)
|
||||
if iface_fields and rules:
|
||||
if normalizer is not None:
|
||||
data = normalizer(dict(data))
|
||||
elif iface_fields and rules:
|
||||
data = apply_iface_normalize(dict(data), iface_fields=iface_fields, rules=rules)
|
||||
return KEY_SEP.join(scalar_text(data.get(k)).strip() for k in key_fields)
|
||||
|
||||
|
|
@ -1037,11 +1041,14 @@ def index_raw_by_match_key(
|
|||
key_fields: list[str],
|
||||
iface_fields: list[str],
|
||||
rules: list[dict[str, str]] | None,
|
||||
normalizer: Any = None,
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""First raw row per internal match key (normalized iface, raw elsewhere)."""
|
||||
idx: dict[str, dict[str, Any]] = {}
|
||||
for raw in raw_rows:
|
||||
ks = row_match_key(raw, key_fields=key_fields, iface_fields=iface_fields, rules=rules)
|
||||
ks = row_match_key(
|
||||
raw, key_fields=key_fields, iface_fields=iface_fields, rules=rules, normalizer=normalizer,
|
||||
)
|
||||
if ks and ks not in idx:
|
||||
idx[ks] = raw
|
||||
return idx
|
||||
|
|
@ -1062,6 +1069,7 @@ def iface_lineage(
|
|||
rules: list[dict[str, str]] | None,
|
||||
port_map: dict[str, str] | None,
|
||||
apply_map: bool,
|
||||
namer: Any = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Per iface field: raw → normalized → (optional) mapped."""
|
||||
from ..biz_state.iface_normalize import normalize_iface_name, resolve_mapped_iface
|
||||
|
|
@ -1073,7 +1081,7 @@ def iface_lineage(
|
|||
out: list[dict[str, Any]] = []
|
||||
for f in iface_fields:
|
||||
raw_v = scalar_text(data.get(f)).strip()
|
||||
norm_v = normalize_iface_name(raw_v, rules) if rules else raw_v
|
||||
norm_v = namer(raw_v) if namer is not None else (normalize_iface_name(raw_v, rules) if rules else raw_v)
|
||||
mapped = False
|
||||
map_to = ""
|
||||
if apply_map and pmap and norm_v:
|
||||
|
|
@ -1113,6 +1121,7 @@ def evaluate_metric_dual(
|
|||
iface_normalize_rules: list[dict[str, str]] | None = None,
|
||||
previous_expect: dict[str, set[str]] | None = None,
|
||||
allow_mapping_scope: bool = True,
|
||||
pipeline: Any = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Run old vs old-baseline, new vs new-baseline (or mapped old baseline), dual merge.
|
||||
|
||||
|
|
@ -1120,10 +1129,12 @@ def evaluate_metric_dual(
|
|||
Board / AI fields ``old_key`` / ``new_key`` and ``old`` / ``new`` stay **raw**
|
||||
device values (A/B), never the post-map BB form.
|
||||
"""
|
||||
from ..biz_state.iface_normalize import (
|
||||
apply_iface_normalize_rows,
|
||||
normalize_iface_rules,
|
||||
normalize_iface_name,
|
||||
from ..biz_state.identity_pipeline import legacy_pipeline
|
||||
|
||||
# Old device = "before" side, new device = "after" side of the identity pipeline.
|
||||
pipe = pipeline or legacy_pipeline(
|
||||
key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields,
|
||||
field_rules=field_rules, port_map=port_map, iface_normalize_rules=iface_normalize_rules,
|
||||
)
|
||||
|
||||
sid = str(sheet_id or "").strip() or str(metric_id or "").strip()
|
||||
|
|
@ -1159,48 +1170,44 @@ def evaluate_metric_dual(
|
|||
[dict(r) for r in new_baseline_rows] if new_baseline_rows is not None else None
|
||||
)
|
||||
|
||||
norm_rules = normalize_iface_rules(iface_normalize_rules)
|
||||
def normalized_key(key: str) -> str:
|
||||
parts = key.split(KEY_SEP) if KEY_SEP in key else ([key] if len(key_fields) == 1 else key.split("|"))
|
||||
if len(parts) != len(key_fields):
|
||||
return key
|
||||
return KEY_SEP.join(normalize_iface_name(p, norm_rules) if f in iface_fields else p
|
||||
return KEY_SEP.join(pipe.normalize_name(p, "before") if f in iface_fields else p
|
||||
for p, f in zip(parts, key_fields))
|
||||
|
||||
def norm_before(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return pipe.normalize(row, "before")
|
||||
|
||||
def norm_after(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return pipe.normalize(row, "after")
|
||||
|
||||
expect_keys = {normalized_key(key) for key in expect_keys}
|
||||
previous_keys = {normalized_key(key) for key in previous_keys}
|
||||
old_baseline_rows = apply_iface_normalize_rows(
|
||||
[strip_netx(r) for r in raw_old_base], iface_fields=iface_fields, rules=norm_rules
|
||||
)
|
||||
old_current_rows = apply_iface_normalize_rows(
|
||||
[strip_netx(r) for r in raw_old_cur], iface_fields=iface_fields, rules=norm_rules
|
||||
)
|
||||
new_current_rows = apply_iface_normalize_rows(
|
||||
[strip_netx(r) for r in raw_new_cur], iface_fields=iface_fields, rules=norm_rules
|
||||
)
|
||||
old_baseline_rows = [norm_before(strip_netx(r)) for r in raw_old_base]
|
||||
old_current_rows = [norm_before(strip_netx(r)) for r in raw_old_cur]
|
||||
new_current_rows = [norm_after(strip_netx(r)) for r in raw_new_cur]
|
||||
if raw_new_base is not None:
|
||||
new_baseline_rows = apply_iface_normalize_rows(
|
||||
[strip_netx(r) for r in raw_new_base],
|
||||
iface_fields=iface_fields,
|
||||
rules=norm_rules,
|
||||
)
|
||||
new_baseline_rows = [norm_after(strip_netx(r)) for r in raw_new_base]
|
||||
else:
|
||||
new_baseline_rows = None
|
||||
|
||||
old_raw_cur_idx = index_raw_by_match_key(
|
||||
raw_old_cur, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules
|
||||
raw_old_cur, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_before
|
||||
)
|
||||
old_raw_base_idx = index_raw_by_match_key(
|
||||
raw_old_base, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules
|
||||
raw_old_base, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_before
|
||||
)
|
||||
new_raw_cur_idx = index_raw_by_match_key(
|
||||
raw_new_cur, key_fields=key_fields, iface_fields=iface_fields, rules=norm_rules
|
||||
raw_new_cur, key_fields=key_fields, iface_fields=iface_fields, rules=None, normalizer=norm_after
|
||||
)
|
||||
new_raw_base_idx = index_raw_by_match_key(
|
||||
raw_new_base or [],
|
||||
key_fields=key_fields,
|
||||
iface_fields=iface_fields,
|
||||
rules=norm_rules,
|
||||
rules=None,
|
||||
normalizer=norm_after,
|
||||
)
|
||||
|
||||
old_cmp = compare_rows(
|
||||
|
|
@ -1213,6 +1220,7 @@ def evaluate_metric_dual(
|
|||
field_rules=field_rules,
|
||||
iface_normalize_rules=None, # already applied
|
||||
include_unchanged=True,
|
||||
duplicate_pairing=pipe.duplicate_pairing,
|
||||
)
|
||||
old_idx = build_diff_index_from_compare(old_cmp)
|
||||
|
||||
|
|
@ -1244,15 +1252,16 @@ def evaluate_metric_dual(
|
|||
port_map=None,
|
||||
field_rules=field_rules,
|
||||
include_unchanged=True,
|
||||
duplicate_pairing=pipe.duplicate_pairing,
|
||||
)
|
||||
new_idx = build_diff_index_from_compare(new_cmp)
|
||||
|
||||
rev_map = {v: k for k, v in port_map.items()}
|
||||
map_scoped = _map_defines_iface_expect(metric_id, iface_fields, port_map)
|
||||
map_scoped = _map_defines_iface_expect(iface_fields, port_map, ov.get("scope_by_port_map"))
|
||||
# Mapping describes identity; an explicit batch selection defines scope.
|
||||
# Port selection also covers business rows attached to those interfaces.
|
||||
for scope, keys in ((expect, expect_keys), (previous_expect or {}, previous_keys)):
|
||||
ports = {normalize_iface_name(p, norm_rules) for p in scope.get("_ports") or set()}
|
||||
ports = {pipe.normalize_name(p, "before") for p in scope.get("_ports") or set()}
|
||||
if ports and not keys and iface_fields:
|
||||
for ks, diff in old_idx.items():
|
||||
row = diff.get("before") or diff.get("after") or {}
|
||||
|
|
@ -1278,6 +1287,21 @@ def evaluate_metric_dual(
|
|||
scoped.add(canon)
|
||||
expect_keys = scoped
|
||||
|
||||
# Duplicate keys only block when they sit inside this wave's scope.
|
||||
from collections import Counter
|
||||
|
||||
def _dup_keys(rows: list[dict[str, Any]]) -> set[str]:
|
||||
return {KEY_SEP.join(k) for k, n in Counter(key_tuple(r, key_fields) for r in rows).items() if n > 1}
|
||||
|
||||
in_scope = expect_keys | previous_keys
|
||||
new_dups = _dup_keys(new_current_rows) | _dup_keys(new_baseline_rows or [])
|
||||
flagged = (
|
||||
_dup_keys(old_baseline_rows) | _dup_keys(old_current_rows) | new_dups
|
||||
| {_reverse_remap_key_str(k, key_fields=key_fields, iface_fields=iface_fields, rev_map=rev_map)
|
||||
for k in new_dups}
|
||||
)
|
||||
duplicate_in_expect = sorted(flagged & in_scope)[:64]
|
||||
|
||||
canon_keys: list[str] = []
|
||||
seen: set[str] = set()
|
||||
|
||||
|
|
@ -1447,9 +1471,10 @@ def evaluate_metric_dual(
|
|||
"iface": iface_lineage(
|
||||
raw_old,
|
||||
iface_fields=iface_fields,
|
||||
rules=norm_rules,
|
||||
rules=None,
|
||||
port_map=port_map,
|
||||
apply_map=True,
|
||||
namer=lambda v: pipe.normalize_name(v, "before"),
|
||||
),
|
||||
},
|
||||
"new": {
|
||||
|
|
@ -1457,9 +1482,10 @@ def evaluate_metric_dual(
|
|||
"iface": iface_lineage(
|
||||
raw_new,
|
||||
iface_fields=iface_fields,
|
||||
rules=norm_rules,
|
||||
rules=None,
|
||||
port_map=None,
|
||||
apply_map=False,
|
||||
namer=lambda v: pipe.normalize_name(v, "after"),
|
||||
),
|
||||
},
|
||||
"port_map": {
|
||||
|
|
@ -1522,6 +1548,7 @@ def evaluate_metric_dual(
|
|||
|
||||
return {
|
||||
"metric_id": metric_id,
|
||||
"effective_scope_by_port_map": map_scoped,
|
||||
"old_summary": old_cmp.get("summary") or {},
|
||||
"new_summary": new_cmp.get("summary") or {},
|
||||
"progress_ok": progress_ok,
|
||||
|
|
@ -1536,6 +1563,7 @@ def evaluate_metric_dual(
|
|||
+ int((new_cmp.get("summary") or {}).get("duplicate_keys_before") or 0),
|
||||
"duplicate_keys_after": int((old_cmp.get("summary") or {}).get("duplicate_keys_after") or 0)
|
||||
+ int((new_cmp.get("summary") or {}).get("duplicate_keys_after") or 0),
|
||||
"duplicate_in_expect": duplicate_in_expect,
|
||||
"duplicate_key_list": list(
|
||||
dict.fromkeys(
|
||||
list((old_cmp.get("summary") or {}).get("duplicate_key_list") or [])
|
||||
|
|
|
|||
|
|
@ -61,6 +61,18 @@ def _out(row: BizMonitorTemplate, compare_name: str = "", *, db: Session | None
|
|||
return out
|
||||
|
||||
|
||||
def _has_legacy_port_and_group(override: dict[str, Any]) -> bool:
|
||||
"""True when a success rule's old side demands admin down AND phy down."""
|
||||
legacy = _port_old_down_groups(legacy_and=True)[0]
|
||||
for rule in override.get("success") or []:
|
||||
if not isinstance(rule, dict):
|
||||
continue
|
||||
for group in rule.get("old_groups") or []:
|
||||
if group == legacy:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def validate_sheet_overrides(
|
||||
db: Session,
|
||||
*,
|
||||
|
|
@ -94,6 +106,9 @@ def validate_sheet_overrides(
|
|||
for ov in overrides:
|
||||
sid = str(ov.get("sheet_id") or "").strip()
|
||||
mid = str(ov.get("metric_id") or "").strip()
|
||||
if _has_legacy_port_and_group(ov):
|
||||
warnings.append({"sheet_id": sid, "metric_id": mid,
|
||||
"reason": "port_success_legacy_and_semantics"})
|
||||
if sid:
|
||||
if sid not in sheet_ids:
|
||||
warnings.append(
|
||||
|
|
@ -228,9 +243,29 @@ def _success_stateful(
|
|||
}
|
||||
|
||||
|
||||
def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[str, Any]:
|
||||
_PORT_ADMIN_DOWN = {"type": "value", "field": "admin", "op": "in", "value": ["down"]}
|
||||
_PORT_PHY_DOWN = {"type": "value", "field": "phy", "op": "in", "value": ["down"]}
|
||||
|
||||
|
||||
def _port_old_down_groups(*, legacy_and: bool = False) -> list[list[dict[str, Any]]]:
|
||||
"""Old-side "port is out of service" groups (OR of groups, AND within one).
|
||||
|
||||
A fibre pulled without ``shutdown`` leaves admin up / phy down, so either
|
||||
signal alone counts. ``legacy_and`` is the pre-A4 form requiring both.
|
||||
"""
|
||||
if legacy_and:
|
||||
return [[dict(_PORT_ADMIN_DOWN), dict(_PORT_PHY_DOWN)]]
|
||||
return [[dict(_PORT_PHY_DOWN)], [dict(_PORT_ADMIN_DOWN)]]
|
||||
|
||||
|
||||
def preset_override_for_metric(
|
||||
metric_id: str, *, sheet_id: str = "", legacy_port_and: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Default dual-verdict overlay for a compare-template sheet.
|
||||
|
||||
``legacy_port_and`` rebuilds the pre-A4 port preset (old side must be
|
||||
admin down AND phy down) so stored copies can be recognised and upgraded.
|
||||
|
||||
``metric_id`` is the collected source table; ``sheet_id`` identifies a
|
||||
filtered split (e.g. bgp_peer.vpnv4). Presets key off the source metric.
|
||||
"""
|
||||
|
|
@ -244,6 +279,8 @@ def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[st
|
|||
look = mid.split(".", 1)[0] if "." in mid else mid
|
||||
|
||||
def _out(body: dict[str, Any]) -> dict[str, Any]:
|
||||
if _scope_by_map_default_off(look):
|
||||
body["scope_by_port_map"] = False
|
||||
body["metric_id"] = mid or look
|
||||
if sid and sid != (mid or look):
|
||||
body["sheet_id"] = sid
|
||||
|
|
@ -261,12 +298,7 @@ def preset_override_for_metric(metric_id: str, *, sheet_id: str = "") -> dict[st
|
|||
"up_values": ["up"],
|
||||
"success": [
|
||||
_success_stateful(
|
||||
old_down_groups=[
|
||||
[
|
||||
{"type": "value", "field": "admin", "op": "in", "value": ["down"]},
|
||||
{"type": "value", "field": "phy", "op": "in", "value": ["down"]},
|
||||
]
|
||||
],
|
||||
old_down_groups=_port_old_down_groups(legacy_and=legacy_port_and),
|
||||
new_up_conds=up,
|
||||
)
|
||||
],
|
||||
|
|
@ -369,6 +401,8 @@ DEFAULT_MONITOR_TEMPLATE_NAME = "默认割接监控"
|
|||
def ensure_default_monitor_templates(db: Session) -> None:
|
||||
"""Seed once when monitor-template table is empty — one template on the full status compare."""
|
||||
if db.query(BizMonitorTemplate.id).limit(1).first():
|
||||
upgrade_legacy_port_overrides(db)
|
||||
fill_scope_by_port_map(db)
|
||||
return
|
||||
cmp_svc.ensure_default_templates(db)
|
||||
zte = (
|
||||
|
|
@ -403,9 +437,68 @@ def ensure_default_monitor_templates(db: Session) -> None:
|
|||
db.commit()
|
||||
|
||||
|
||||
def default_port_monitor_template_id(db: Session) -> str:
|
||||
"""Ensure seeds exist and return the default monitor template id."""
|
||||
ensure_default_monitor_templates(db)
|
||||
def _scope_by_map_default_off(name: str) -> bool:
|
||||
"""BGP / pseudowire sheets keep the batch expect picker; a port map does not scope them."""
|
||||
low = str(name or "").lower()
|
||||
return "bgp" in low or "pw" in low or "pseudowire" in low
|
||||
|
||||
|
||||
def fill_scope_by_port_map(db: Session) -> int:
|
||||
"""Make the old name-based default explicit on stored overrides (one-off, idempotent)."""
|
||||
changed = 0
|
||||
for row in db.query(BizMonitorTemplate).all():
|
||||
out = []
|
||||
touched = False
|
||||
for ov in list(row.sheet_overrides_json or []):
|
||||
if (isinstance(ov, dict) and "scope_by_port_map" not in ov
|
||||
and _scope_by_map_default_off(f"{ov.get('metric_id') or ''} {ov.get('sheet_id') or ''}")):
|
||||
ov = {**ov, "scope_by_port_map": False}
|
||||
touched = True
|
||||
out.append(ov)
|
||||
if touched:
|
||||
row.sheet_overrides_json = out
|
||||
row.updated_at = _utcnow()
|
||||
changed += 1
|
||||
if changed:
|
||||
db.commit()
|
||||
return changed
|
||||
|
||||
|
||||
def upgrade_legacy_port_overrides(db: Session) -> int:
|
||||
"""Rewrite stored port overrides that are exactly the pre-A4 preset (AND semantics).
|
||||
|
||||
Field-by-field equality only: anything a user customised is left alone and
|
||||
surfaces ``port_success_legacy_and_semantics`` in ``override_warnings``.
|
||||
"""
|
||||
changed = 0
|
||||
for row in db.query(BizMonitorTemplate).all():
|
||||
overrides = list(row.sheet_overrides_json or [])
|
||||
out: list[Any] = []
|
||||
touched = False
|
||||
for ov in overrides:
|
||||
if isinstance(ov, dict):
|
||||
mid = str(ov.get("metric_id") or "").strip()
|
||||
sid = str(ov.get("sheet_id") or "").strip()
|
||||
if (mid or sid).split(".", 1)[0] == PORT_METRIC_ID and ov == preset_override_for_metric(
|
||||
mid, sheet_id=sid, legacy_port_and=True,
|
||||
):
|
||||
out.append(preset_override_for_metric(mid, sheet_id=sid))
|
||||
touched = True
|
||||
continue
|
||||
out.append(ov)
|
||||
if touched:
|
||||
row.sheet_overrides_json = out
|
||||
row.updated_at = _utcnow()
|
||||
changed += 1
|
||||
if changed:
|
||||
db.commit()
|
||||
return changed
|
||||
|
||||
|
||||
def default_port_monitor_template_id(db: Session, *, seed: bool = True) -> str:
|
||||
"""Return the default monitor template id, seeding the built-ins first unless ``seed=False``."""
|
||||
if seed:
|
||||
ensure_default_monitor_templates(db)
|
||||
for name in (
|
||||
DEFAULT_MONITOR_TEMPLATE_NAME,
|
||||
"ZTE 状态割接监控",
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -23,6 +23,8 @@ def validate_monitor_rules(defaults: Any, overrides: Any) -> None:
|
|||
if ident in seen:
|
||||
fail(path + ".duplicate")
|
||||
seen.add(ident)
|
||||
if ov.get("scope_by_port_map") is not None and not isinstance(ov["scope_by_port_map"], bool):
|
||||
fail(path + ".scope_by_port_map")
|
||||
for key in ("status_fields", "up_values", "down_values"):
|
||||
if key in ov and (not isinstance(ov[key], list) or any(not str(v).strip() for v in ov[key])):
|
||||
fail(path + "." + key)
|
||||
|
|
|
|||
|
|
@ -151,6 +151,12 @@ def api_baseline_ports(project_id: str, db: Session = Depends(get_db)):
|
|||
return svc.list_baseline_ports(db, project_id)
|
||||
|
||||
|
||||
@router.get("/projects/{project_id}/preflight")
|
||||
@router.post("/projects/{project_id}/preflight")
|
||||
def api_project_preflight(project_id: str, db: Session = Depends(get_db)):
|
||||
return svc.preflight_project(db, project_id)
|
||||
|
||||
|
||||
@router.get("/projects/{project_id}/baseline-expect")
|
||||
def api_baseline_expect(project_id: str, db: Session = Depends(get_db)):
|
||||
"""Multi-metric baseline keys for expect-set picking (from monitor template sheets)."""
|
||||
|
|
@ -222,8 +228,13 @@ def api_finish_batch(
|
|||
mark_done: bool = False,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""本批完成：终验 + 小结 + 红单（允许带红继续）."""
|
||||
return svc.finish_batch(db, batch_id, mark_done=mark_done)
|
||||
"""本批完成：终验 + 小结 + 红单（允许带红继续）. Runs in the background; poll the run."""
|
||||
return svc.enqueue_finish(db, batch_id, mark_done=mark_done)
|
||||
|
||||
|
||||
@router.post("/projects/{project_id}/red-tickets/resolve-resolvable")
|
||||
def api_resolve_resolvable(project_id: str, db: Session = Depends(get_db)):
|
||||
return svc.resolve_resolvable_red_tickets(db, project_id)
|
||||
|
||||
|
||||
@router.get("/projects/{project_id}/red-tickets")
|
||||
|
|
@ -258,9 +269,10 @@ def api_patch_batch(batch_id: str, body: BatchPatchIn, db: Session = Depends(get
|
|||
@router.post("/batches/{batch_id}/evaluate")
|
||||
def api_evaluate(batch_id: str, body: EvaluateIn | None = None, db: Session = Depends(get_db)):
|
||||
payload = body.model_dump() if body else {}
|
||||
return svc.run_evaluate(
|
||||
# Returns the ``running`` run at once; poll GET /runs/{id} until ``status`` settles.
|
||||
return svc.enqueue_evaluate(
|
||||
db,
|
||||
batch_id=batch_id,
|
||||
batch_id,
|
||||
old_batch_id=str(payload.get("old_batch_id") or ""),
|
||||
new_batch_id=str(payload.get("new_batch_id") or ""),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any
|
||||
|
|
@ -25,6 +27,11 @@ def max_concurrent_tasks() -> int:
|
|||
return max(1, int(getattr(settings, "biz_state_max_concurrent_tasks", 16) or 16))
|
||||
|
||||
|
||||
def max_sessions_per_ne() -> int:
|
||||
"""Concurrent SSH collections allowed per network element (>= 1)."""
|
||||
return max(1, int(getattr(settings, "biz_state_max_sessions_per_ne", 2) or 2))
|
||||
|
||||
|
||||
def enqueue_collect(task_id: str, *, manual: bool = False) -> dict[str, Any]:
|
||||
"""Create a queued batch and mark task collect_running. Does not open SSH.
|
||||
|
||||
|
|
@ -135,15 +142,16 @@ def enqueue_collect_group(group_id: str, *, manual: bool = False,
|
|||
BizStateTask.id).with_for_update().populate_existing().all()
|
||||
result: dict[str, Any] = {"ok": False, "queued": False, "task_id": requested_task_id,
|
||||
"collect_group_id": group_id}
|
||||
if len(tasks) != 2 or len({t.ne_id for t in tasks}) != 2:
|
||||
# One task = a single device used as both sides (same-device cutover); two tasks = a pair.
|
||||
if len(tasks) not in (1, 2) or len({t.ne_id for t in tasks}) != len(tasks):
|
||||
return {**result, "reason": "collect_pair_invalid"}
|
||||
if max_concurrent_tasks() < 2:
|
||||
if max_concurrent_tasks() < len(tasks):
|
||||
return {**result, "reason": "collect_pair_capacity"}
|
||||
from ..cli_budget import clamp_cli_workers
|
||||
|
||||
worker_cap = clamp_cli_workers(int(getattr(settings, "biz_state_worker_collect_threads", None)
|
||||
or getattr(settings, "biz_state_dispatch_workers", 8) or 8))
|
||||
if worker_cap < 2:
|
||||
if worker_cap < len(tasks):
|
||||
return {**result, "reason": "collect_pair_capacity"}
|
||||
if any(t.collect_running for t in tasks):
|
||||
return {**result, "reason": "collect_pair_busy"}
|
||||
|
|
@ -230,13 +238,12 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]:
|
|||
if slots <= 0:
|
||||
return []
|
||||
|
||||
busy_nes = {
|
||||
str(r[0] or "").strip()
|
||||
for r in db.query(BizStateBatch.ne_id)
|
||||
.filter(BizStateBatch.status == "running")
|
||||
.all()
|
||||
if str(r[0] or "").strip()
|
||||
}
|
||||
# Running collections per NE; an NE accepts new ones until it reaches its session cap.
|
||||
per_ne_cap = max_sessions_per_ne()
|
||||
ne_running: dict[str, int] = defaultdict(int)
|
||||
for (running_ne,) in db.query(BizStateBatch.ne_id).filter(BizStateBatch.status == "running").all():
|
||||
if str(running_ne or "").strip():
|
||||
ne_running[str(running_ne).strip()] += 1
|
||||
|
||||
# Manual route rounds yield to fresh monitoring, but must eventually
|
||||
# run even when a busy device always has regular metrics queued.
|
||||
|
|
@ -266,14 +273,19 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]:
|
|||
peers = db.query(BizStateBatch).filter(
|
||||
BizStateBatch.collect_round_id == batch.collect_round_id,
|
||||
).with_for_update(skip_locked=pg).all()
|
||||
if len(peers) != 2 or any(p.status != "queued" for p in peers):
|
||||
expected = db.query(BizStateTask.id).filter(
|
||||
BizStateTask.collect_group_id == batch.collect_group_id).count() if batch.collect_group_id else 2
|
||||
if len(peers) != max(1, expected) or any(p.status != "queued" for p in peers):
|
||||
continue
|
||||
group = peers
|
||||
visited.update(p.id for p in group)
|
||||
if len(group) > slots - len(claimed_rows):
|
||||
continue
|
||||
nes = {str(p.ne_id or "").strip() for p in group}
|
||||
if nes & busy_nes:
|
||||
wanted: dict[str, int] = defaultdict(int)
|
||||
for p in group:
|
||||
if str(p.ne_id or "").strip():
|
||||
wanted[str(p.ne_id).strip()] += 1
|
||||
if any(ne_running[ne] + n > per_ne_cap for ne, n in wanted.items()):
|
||||
continue
|
||||
now = _utcnow()
|
||||
for member in group:
|
||||
|
|
@ -285,7 +297,8 @@ def claim_queued_batches(limit: int | None = None) -> list[dict[str, Any]]:
|
|||
if task:
|
||||
task.last_collect_started_at = now
|
||||
claimed_rows.append(member)
|
||||
busy_nes.update(nes - {""})
|
||||
for ne, n in wanted.items():
|
||||
ne_running[ne] += n
|
||||
|
||||
if not claimed_rows:
|
||||
db.rollback()
|
||||
|
|
|
|||
|
|
@ -93,8 +93,17 @@ def apply_port_map(
|
|||
return out
|
||||
|
||||
|
||||
def key_tuple(row: Mapping[str, Any], fields: Sequence[str]) -> tuple[str, ...]:
|
||||
"""The one canonical match-key form: ``scalar_text(...).strip()`` per field.
|
||||
|
||||
Every place that builds, stores or looks up a key string must go through
|
||||
this (``None`` → ``""``, ``True`` → ``"true"``, surrounding blanks trimmed).
|
||||
"""
|
||||
return tuple(scalar_text(row.get(k)).strip() for k in fields)
|
||||
|
||||
|
||||
def row_key(row: dict[str, Any], key_fields: list[str]) -> tuple[str, ...]:
|
||||
return tuple(scalar_text(row.get(k)).strip() for k in key_fields)
|
||||
return key_tuple(row, key_fields)
|
||||
|
||||
|
||||
def mapping_stats(
|
||||
|
|
@ -109,8 +118,8 @@ def mapping_stats(
|
|||
``hit_before``: map key appears as a normalized before iface, or as the
|
||||
parent of a before subinterface (so main-port-only maps still validate).
|
||||
|
||||
Map keys are matched against **normalized** iface values (same pipeline as
|
||||
compare). Enter mapping keys in normalized form.
|
||||
Map keys are matched against **normalized** iface values; callers pass a
|
||||
port map already normalized with the template iface rules.
|
||||
"""
|
||||
before_ifaces: set[str] = set()
|
||||
after_ifaces: set[str] = set()
|
||||
|
|
@ -156,7 +165,7 @@ def mapping_stats(
|
|||
"miss_after": miss_after,
|
||||
"unused_before_keys": unused,
|
||||
"ok": not miss_before and not miss_after,
|
||||
"hint": "map keys must match normalized iface names (post iface_normalize)",
|
||||
"hint": "map keys/values are normalized with the template iface rules before matching",
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -174,6 +183,10 @@ def compare_rows(
|
|||
include_unchanged: bool = False,
|
||||
unchanged_limit: int | None = None,
|
||||
compact_unchanged: bool = False,
|
||||
pipeline: Any = None,
|
||||
match_keys_override: Sequence[str] | None = None,
|
||||
prepared: tuple[list, list] | None = None,
|
||||
duplicate_pairing: str = "ordered",
|
||||
) -> dict[str, Any]:
|
||||
"""Return summary + diffs list.
|
||||
|
||||
|
|
@ -200,6 +213,11 @@ def compare_rows(
|
|||
- ``True``: force drop iface from match key when a non-empty candidate exists.
|
||||
- ``False``: never drop iface from match key.
|
||||
|
||||
``pipeline`` (``identity_pipeline.Pipeline``) owns normalization, filtering,
|
||||
port mapping and key building; when omitted a legacy pipeline is derived
|
||||
from the loose arguments (callers have already filtered the rows).
|
||||
``match_keys_override`` pins the match key (re-running a stored result).
|
||||
|
||||
``summary.duplicate`` is always 0. ``duplicate_keys_*`` count match keys
|
||||
that appear more than once on a side (diagnostic only).
|
||||
|
||||
|
|
@ -208,32 +226,40 @@ def compare_rows(
|
|||
"""
|
||||
if not key_fields:
|
||||
raise ValueError("key_fields required")
|
||||
pmap = dict(port_map or {})
|
||||
rules = field_rule_map(field_rules)
|
||||
norm_rules = normalize_iface_rules(iface_normalize_rules)
|
||||
iface_list = [str(f) for f in (iface_fields or []) if str(f).strip()]
|
||||
if pipeline is None:
|
||||
from .identity_pipeline import legacy_pipeline
|
||||
|
||||
pipeline = legacy_pipeline(
|
||||
key_fields=key_fields, iface_fields=iface_fields, compare_fields=compare_fields,
|
||||
field_rules=field_rules, port_map=port_map, iface_normalize_rules=iface_normalize_rules,
|
||||
ignore_port_changes=ignore_port_changes, duplicate_pairing=duplicate_pairing,
|
||||
)
|
||||
pmap = dict(pipeline.port_map)
|
||||
rules = field_rule_map(field_rules if field_rules is not None else pipeline.field_rules)
|
||||
iface_list = list(pipeline.iface_fields)
|
||||
iface_set = set(iface_list)
|
||||
|
||||
before_norm = apply_iface_normalize_rows(
|
||||
before_rows, iface_fields=iface_list, rules=norm_rules
|
||||
) if norm_rules and iface_list else before_rows
|
||||
after_norm = apply_iface_normalize_rows(
|
||||
after_rows, iface_fields=iface_list, rules=norm_rules
|
||||
) if norm_rules and iface_list else after_rows
|
||||
if prepared is not None:
|
||||
before_pairs, after_pairs = prepared
|
||||
else:
|
||||
before_pairs = pipeline.prepare(before_rows, "before")
|
||||
after_pairs = pipeline.prepare(after_rows, "after")
|
||||
before_rows = [orig for orig, _ in before_pairs]
|
||||
after_rows = [orig for orig, _ in after_pairs]
|
||||
before_norm = [norm for _, norm in before_pairs]
|
||||
after_norm = [norm for _, norm in after_pairs]
|
||||
|
||||
ignore_ports = False
|
||||
# No map → optionally ignore port renames by dropping iface from match key.
|
||||
if not pmap and iface_set:
|
||||
candidate = [k for k in key_fields if k not in iface_set]
|
||||
if not candidate:
|
||||
match_keys = list(key_fields)
|
||||
elif ignore_port_changes is True:
|
||||
match_keys = candidate
|
||||
ignore_ports = True
|
||||
elif ignore_port_changes is False:
|
||||
match_keys = list(key_fields)
|
||||
else:
|
||||
# Auto heuristic (legacy default)
|
||||
if match_keys_override is not None:
|
||||
match_keys = [str(k) for k in match_keys_override]
|
||||
ignore_ports = match_keys != list(key_fields)
|
||||
else:
|
||||
# No map → optionally ignore port renames by dropping iface from match key.
|
||||
policy = pipeline.ignore_port_changes
|
||||
unique = None
|
||||
if not pmap and iface_set and policy is None:
|
||||
candidate = [k for k in key_fields if k not in iface_set]
|
||||
|
||||
def unique_keys(rows: list[dict[str, Any]]) -> bool:
|
||||
seen: set[tuple[str, ...]] = set()
|
||||
for row in rows:
|
||||
|
|
@ -243,23 +269,20 @@ def compare_rows(
|
|||
seen.add(key)
|
||||
return True
|
||||
|
||||
if unique_keys(before_norm) and unique_keys(after_norm):
|
||||
match_keys = candidate
|
||||
ignore_ports = True
|
||||
else:
|
||||
match_keys = list(key_fields)
|
||||
else:
|
||||
match_keys = list(key_fields)
|
||||
unique = bool(candidate) and unique_keys(before_norm) and unique_keys(after_norm)
|
||||
match_keys = pipeline.effective_match_keys(unique_without_iface=unique)
|
||||
ignore_ports = match_keys != list(key_fields)
|
||||
|
||||
before_groups: dict[tuple[str, ...], list[tuple[dict[str, Any], dict[str, Any]]]] = (
|
||||
defaultdict(list)
|
||||
)
|
||||
after_groups: dict[tuple[str, ...], list[dict[str, Any]]] = defaultdict(list)
|
||||
for orig, norm in zip(before_rows, before_norm):
|
||||
mapped = apply_port_map(norm, iface_fields=iface_list, port_map=pmap) if iface_list else norm
|
||||
before_groups[row_key(mapped, match_keys)].append((orig, mapped))
|
||||
for r in after_norm:
|
||||
after_groups[row_key(r, match_keys)].append(r)
|
||||
mapped, k = pipeline.key_of(norm, "before", match_keys)
|
||||
before_groups[k].append((orig, mapped))
|
||||
for norm in after_norm:
|
||||
after_groups[pipeline.key_of(norm, "after", match_keys)[1]].append(norm)
|
||||
sort_pairs = pipeline.duplicate_pairing == "sorted"
|
||||
|
||||
diffs: list[dict[str, Any]] = []
|
||||
added = removed = changed = unchanged = 0
|
||||
|
|
@ -271,7 +294,7 @@ def compare_rows(
|
|||
sample = _StratifiedSample(limit_n) if include_unchanged and limit_n is not None else None
|
||||
|
||||
def _key_obj(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {f: row.get(f, "") for f in key_fields}
|
||||
return dict(zip(key_fields, key_tuple(row, key_fields)))
|
||||
|
||||
def _row_id(row: dict[str, Any] | None) -> str:
|
||||
if not isinstance(row, dict):
|
||||
|
|
@ -325,6 +348,10 @@ def compare_rows(
|
|||
multi_before_keys.append(k)
|
||||
if len(a_list) > 1:
|
||||
multi_after_keys.append(k)
|
||||
if sort_pairs and (len(b_list) > 1 or len(a_list) > 1):
|
||||
# Same-key rows pair by value, not by CLI output order (ECMP, multipath).
|
||||
b_list = sorted(b_list, key=lambda pair: pipeline.pair_sort_key(pair[1]))
|
||||
a_list = sorted(a_list, key=pipeline.pair_sort_key)
|
||||
n = max(len(b_list), len(a_list))
|
||||
for i in range(n):
|
||||
if i >= len(b_list):
|
||||
|
|
|
|||
194
netx_api/biz_state/compare_preflight.py
Normal file
194
netx_api/biz_state/compare_preflight.py
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
"""Pre-flight check of a port mapping against the two batches it will be applied to.
|
||||
|
||||
The same function backs the compare "validate mapping" button and the cutover
|
||||
project pre-flight, so both modes judge mapping coverage the same way. It answers
|
||||
before anyone runs a comparison:
|
||||
|
||||
- which *before* interfaces are mapped (exact / by parent), unmapped but present on
|
||||
the *after* side by the same name, or **suspect** (not mapped, not on *after*)
|
||||
- which map keys / targets match nothing, which targets receive several sources
|
||||
- whether each sheet's two sides were actually collected (``data_state``)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Callable, Mapping, Sequence
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from .compare_quality import assess_sheet, command_coverage, collect_quality, side_receipts
|
||||
from .identity_pipeline import build_pipeline
|
||||
from .iface_normalize import normalize_port_map, resolve_mapped_iface
|
||||
|
||||
LIST_CAP = 200
|
||||
# Million-row tables are never loaded just to look at interface names.
|
||||
HEAVY_METRICS = frozenset({"bgp_route", "ip_route", "ipv6_route", "l2vpn_mac", "evpn_mac", "interface_detail"})
|
||||
|
||||
RowsLoader = Callable[[str, Mapping[str, Any]], list[dict[str, Any]]]
|
||||
|
||||
|
||||
def _parents(name: str) -> list[str]:
|
||||
parts = name.split(".")
|
||||
return [".".join(parts[:i]) for i in range(len(parts) - 1, 0, -1)]
|
||||
|
||||
|
||||
def _classify(name: str, pmap: Mapping[str, str], after: set[str]) -> str:
|
||||
if name in pmap:
|
||||
return "mapped_exact"
|
||||
if any(p in pmap for p in _parents(name)):
|
||||
return "mapped_by_parent"
|
||||
if name in after:
|
||||
return "identity_same_name"
|
||||
return "suspect_unmapped"
|
||||
|
||||
|
||||
def _present(name: str, names: set[str]) -> bool:
|
||||
"""``name`` exists, or is the parent of an existing sub-interface."""
|
||||
if name in names:
|
||||
return True
|
||||
prefix = name + "."
|
||||
return any(n.startswith(prefix) for n in names)
|
||||
|
||||
|
||||
def _top_level(names: Sequence[str]) -> list[str]:
|
||||
"""Drop sub-interfaces whose parent is already in the list (noise)."""
|
||||
pool = set(names)
|
||||
return sorted(n for n in pool if not any(p in pool for p in _parents(n)))
|
||||
|
||||
|
||||
def sheet_mapping_coverage(
|
||||
pipe: Any,
|
||||
before_pairs: Sequence[tuple[Mapping[str, Any], Mapping[str, Any]]],
|
||||
after_pairs: Sequence[tuple[Mapping[str, Any], Mapping[str, Any]]],
|
||||
) -> dict[str, Any]:
|
||||
"""Classify every *before* interface of a sheet against the map and the *after* side.
|
||||
|
||||
``before_pairs`` / ``after_pairs`` are ``pipeline.prepare`` outputs. The result
|
||||
carries two private sets (``_after_ifaces``, ``_used_keys``) for aggregation.
|
||||
"""
|
||||
b_if = {str(n.get(f) or "").strip() for _, n in before_pairs for f in pipe.iface_fields}
|
||||
a_if = {str(n.get(f) or "").strip() for _, n in after_pairs for f in pipe.iface_fields}
|
||||
b_if.discard("")
|
||||
a_if.discard("")
|
||||
buckets: dict[str, list[str]] = {
|
||||
"mapped_exact": [], "mapped_by_parent": [], "identity_same_name": [], "suspect_unmapped": [],
|
||||
}
|
||||
for name in sorted(b_if):
|
||||
buckets[_classify(name, pipe.port_map, a_if)].append(name)
|
||||
used = {k for name in b_if for k in pipe.port_map if name == k or name.startswith(k + ".")}
|
||||
missing_targets = sorted({
|
||||
resolve_mapped_iface(n, pipe.port_map) for n in buckets["mapped_exact"] + buckets["mapped_by_parent"]
|
||||
if not _present(resolve_mapped_iface(n, pipe.port_map), a_if)
|
||||
})
|
||||
suspects = _top_level(buckets["suspect_unmapped"])
|
||||
return {
|
||||
"counts": {k: len(v) for k, v in buckets.items()},
|
||||
"mapped_exact": buckets["mapped_exact"][:LIST_CAP],
|
||||
"mapped_by_parent": buckets["mapped_by_parent"][:LIST_CAP],
|
||||
"identity_same_name": buckets["identity_same_name"][:LIST_CAP],
|
||||
"suspect_unmapped": suspects[:LIST_CAP],
|
||||
"suspect_unmapped_total": len(suspects),
|
||||
"mapped_target_missing_on_after": missing_targets[:LIST_CAP],
|
||||
"_after_ifaces": a_if,
|
||||
"_used_keys": used,
|
||||
}
|
||||
|
||||
|
||||
def preflight(
|
||||
db: Session,
|
||||
*,
|
||||
sheets: Sequence[Mapping[str, Any]],
|
||||
iface_rules: Sequence[Mapping[str, Any]] | None,
|
||||
raw_port_map: Mapping[str, str] | None,
|
||||
before_batch_id: str,
|
||||
after_batch_id: str,
|
||||
before_vendor: str = "",
|
||||
after_vendor: str = "",
|
||||
load_rows: RowsLoader,
|
||||
) -> dict[str, Any]:
|
||||
"""Build the pre-flight report. ``load_rows(side, sheet)`` returns raw rows."""
|
||||
raw_map = dict(raw_port_map or {})
|
||||
out_sheets: list[dict[str, Any]] = []
|
||||
used_keys: set[str] = set()
|
||||
after_all: set[str] = set()
|
||||
totals = {"mapped": 0, "suspect": 0, "same_name": 0}
|
||||
risk_reasons: list[str] = []
|
||||
full_pmap: dict[str, str] = {}
|
||||
conflicts: list[dict[str, str]] = []
|
||||
|
||||
for sheet in sheets:
|
||||
sid = str(sheet.get("sheet_id") or sheet.get("metric_id") or "")
|
||||
mid = str(sheet.get("metric_id") or "")
|
||||
title = str(sheet.get("title") or sid)
|
||||
pipe = build_pipeline(
|
||||
sheet, iface_rules=iface_rules, port_map=raw_map,
|
||||
before_vendor=before_vendor, after_vendor=after_vendor,
|
||||
)
|
||||
if not full_pmap and pipe.port_map:
|
||||
full_pmap = dict(pipe.port_map)
|
||||
heavy = mid in HEAVY_METRICS
|
||||
card: dict[str, Any] = {"sheet_id": sid, "metric_id": mid, "title": title}
|
||||
if heavy:
|
||||
b_n = collect_quality(side_receipts(db, before_batch_id, mid))["row_count"]
|
||||
a_n = collect_quality(side_receipts(db, after_batch_id, mid))["row_count"]
|
||||
card.update(skipped="heavy_table", before_count=b_n, after_count=a_n)
|
||||
else:
|
||||
b_pairs = pipe.prepare(load_rows("before", sheet), "before")
|
||||
a_pairs = pipe.prepare(load_rows("after", sheet), "after")
|
||||
card.update(before_count=len(b_pairs), after_count=len(a_pairs))
|
||||
if pipe.iface_fields:
|
||||
cov = sheet_mapping_coverage(pipe, b_pairs, a_pairs)
|
||||
after_all |= cov.pop("_after_ifaces")
|
||||
used_keys |= cov.pop("_used_keys")
|
||||
card.update(cov)
|
||||
totals["mapped"] += cov["counts"]["mapped_exact"] + cov["counts"]["mapped_by_parent"]
|
||||
totals["same_name"] += cov["counts"]["identity_same_name"]
|
||||
totals["suspect"] += cov["suspect_unmapped_total"]
|
||||
card.update(assess_sheet(
|
||||
db, sheet, before_batch_id=before_batch_id, after_batch_id=after_batch_id,
|
||||
before_rows=int(card.get("before_count") or 0), after_rows=int(card.get("after_count") or 0),
|
||||
))
|
||||
if card["data_state"] in ("not_collected", "suspect_parse"):
|
||||
risk_reasons.append(f"{sid}:{card['data_state']}")
|
||||
out_sheets.append(card)
|
||||
|
||||
if raw_map and not full_pmap:
|
||||
full_pmap, conflicts = normalize_port_map(raw_map, iface_rules)
|
||||
elif raw_map:
|
||||
_, conflicts = normalize_port_map(raw_map, iface_rules)
|
||||
sources: dict[str, list[str]] = {}
|
||||
for k, v in full_pmap.items():
|
||||
sources.setdefault(v, []).append(k)
|
||||
n_to_1 = [{"target": t, "sources": sorted(src)} for t, src in sorted(sources.items()) if len(src) > 1]
|
||||
unused = sorted(k for k in full_pmap if k not in used_keys)
|
||||
targets_missing = sorted(v for v in set(full_pmap.values()) if after_all and not _present(v, after_all))
|
||||
|
||||
judged = totals["mapped"] + totals["suspect"]
|
||||
coverage = {
|
||||
"mapped": totals["mapped"], "suspect_unmapped": totals["suspect"],
|
||||
"identity_same_name": totals["same_name"],
|
||||
"rate": round(totals["mapped"] / judged * 100, 1) if judged else None,
|
||||
}
|
||||
mapping = {
|
||||
"pairs": len(full_pmap), "conflicts": conflicts[:LIST_CAP],
|
||||
"unused_map_keys": unused[:LIST_CAP], "unused_map_keys_total": len(unused),
|
||||
"map_targets_missing_on_after": targets_missing[:LIST_CAP],
|
||||
"map_targets_missing_on_after_total": len(targets_missing),
|
||||
"n_to_1_targets": n_to_1[:LIST_CAP],
|
||||
}
|
||||
if totals["suspect"]:
|
||||
risk_reasons.append("suspect_unmapped")
|
||||
if unused:
|
||||
risk_reasons.append("unused_map_keys")
|
||||
if targets_missing:
|
||||
risk_reasons.append("map_targets_missing_on_after")
|
||||
if conflicts:
|
||||
risk_reasons.append("map_conflicts")
|
||||
return {
|
||||
"verdict": "risk" if risk_reasons else "ready",
|
||||
"risk_reasons": sorted(set(risk_reasons)),
|
||||
"coverage": coverage,
|
||||
"mapping": mapping,
|
||||
"sheets": out_sheets,
|
||||
"command_coverage": command_coverage(db, before_batch_id=before_batch_id, after_batch_id=after_batch_id),
|
||||
}
|
||||
198
netx_api/biz_state/compare_quality.py
Normal file
198
netx_api/biz_state/compare_quality.py
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
"""Collection quality of a compare side: is a missing row a business loss or a data problem?
|
||||
|
||||
Compare results are only as good as the two batches behind them. This module
|
||||
reads *receipts* (``BizStateBatchCommand``: counts and status, never ``raw_text``)
|
||||
to tell apart:
|
||||
|
||||
- ``not_collected`` – a side never ran a command (or the slice) for the sheet
|
||||
- ``suspect_parse`` – the device printed lines but the parser produced no rows
|
||||
- ``no_data`` – both sides ran fine and are really empty
|
||||
- ``ok`` – ordinary comparison
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models import BizStateBatchCommand
|
||||
from .compare_rules import scalar_text
|
||||
|
||||
# More than this many CLI lines with zero parsed rows is not "an empty table".
|
||||
SUSPECT_RAW_LINES = 5
|
||||
DATA_STATES = ("ok", "no_data", "suspect_parse", "not_collected")
|
||||
UNJUDGED_STATES = ("no_data", "not_collected")
|
||||
|
||||
|
||||
def _has_receipts(db: Session, batch_id: str) -> bool:
|
||||
return bool(batch_id) and db.query(BizStateBatchCommand.id).filter(
|
||||
BizStateBatchCommand.batch_id == batch_id).limit(1).first() is not None
|
||||
|
||||
|
||||
def side_receipts(db: Session, batch_id: str, metric_id: str) -> list[dict[str, Any]]:
|
||||
if not batch_id or not metric_id:
|
||||
return []
|
||||
rows = (
|
||||
db.query(
|
||||
BizStateBatchCommand.profile_id, BizStateBatchCommand.parse_status,
|
||||
BizStateBatchCommand.row_count, BizStateBatchCommand.raw_line_count,
|
||||
BizStateBatchCommand.declared_total,
|
||||
)
|
||||
.filter(BizStateBatchCommand.batch_id == batch_id, BizStateBatchCommand.metric_id == metric_id)
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
{
|
||||
"profile_id": str(r[0] or ""), "parse_status": str(r[1] or ""),
|
||||
"row_count": int(r[2] or 0), "raw_line_count": int(r[3] or 0),
|
||||
"declared_total": int(r[4] or 0),
|
||||
}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
|
||||
def collect_quality(receipts: Sequence[Mapping[str, Any]]) -> dict[str, Any]:
|
||||
statuses: dict[str, int] = {}
|
||||
for r in receipts:
|
||||
key = str(r.get("parse_status") or "unknown")
|
||||
statuses[key] = statuses.get(key, 0) + 1
|
||||
return {
|
||||
"commands": len(receipts),
|
||||
"parse_status": statuses,
|
||||
"row_count": sum(int(r.get("row_count") or 0) for r in receipts),
|
||||
"raw_line_count": sum(int(r.get("raw_line_count") or 0) for r in receipts),
|
||||
"declared_total": sum(int(r.get("declared_total") or 0) for r in receipts),
|
||||
"profile_ids": sorted({str(r.get("profile_id") or "") for r in receipts if r.get("profile_id")}),
|
||||
}
|
||||
|
||||
|
||||
def sheet_slice(sheet: Mapping[str, Any]) -> dict[str, str]:
|
||||
"""``{field: value}`` the sheet pins with top-level ``eq`` row filters."""
|
||||
out: dict[str, str] = {}
|
||||
for f in sheet.get("row_filters") or []:
|
||||
if not isinstance(f, Mapping) or "any" in f or "all" in f:
|
||||
continue
|
||||
if str(f.get("op") or "eq").strip().lower() not in ("eq", "==", "ci_eq"):
|
||||
continue
|
||||
field = str(f.get("field") or "").strip()
|
||||
value = scalar_text(f.get("value")).strip().lower()
|
||||
if field and value:
|
||||
out[field] = value
|
||||
return out
|
||||
|
||||
|
||||
def covers_slice(
|
||||
receipts: Sequence[Mapping[str, Any]], wanted: Mapping[str, str], *, profile_slice: Any = None,
|
||||
) -> bool:
|
||||
"""True when at least one successful command could have produced ``wanted``.
|
||||
|
||||
A command without a declared slice (or unknown profile) covers everything.
|
||||
"""
|
||||
if profile_slice is None:
|
||||
from .profiles import get_profile
|
||||
|
||||
def profile_slice(pid: str) -> dict[str, str]: # noqa: F811
|
||||
prof = get_profile(pid) if pid else None
|
||||
return dict(prof.slice) if prof is not None and prof.slice else {}
|
||||
|
||||
for r in receipts:
|
||||
if str(r.get("parse_status") or "") in ("failed", "skipped_custom"):
|
||||
continue
|
||||
declared = profile_slice(str(r.get("profile_id") or ""))
|
||||
if all(
|
||||
f not in declared or str(declared[f]).strip().lower() == v
|
||||
for f, v in wanted.items()
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def classify_sheet(
|
||||
*,
|
||||
before: Sequence[Mapping[str, Any]] | None,
|
||||
after: Sequence[Mapping[str, Any]] | None,
|
||||
before_rows: int,
|
||||
after_rows: int,
|
||||
wanted_slice: Mapping[str, str] | None = None,
|
||||
) -> tuple[str, list[str]]:
|
||||
"""``(data_state, reasons)`` for one sheet; reasons name the offending side.
|
||||
|
||||
``None`` receipts mean the batch carries no receipts at all (imported or
|
||||
legacy data): that side cannot be judged and is trusted as-is.
|
||||
"""
|
||||
wanted = dict(wanted_slice or {})
|
||||
reasons: list[str] = []
|
||||
for side, receipts in (("before", before), ("after", after)):
|
||||
if receipts is None:
|
||||
continue
|
||||
if not receipts:
|
||||
reasons.append(f"{side}:no_command")
|
||||
elif wanted and not covers_slice(receipts, wanted):
|
||||
reasons.append(f"{side}:slice_not_collected")
|
||||
if reasons:
|
||||
return "not_collected", reasons
|
||||
for side, receipts in (("before", before), ("after", after)):
|
||||
if receipts is None:
|
||||
continue
|
||||
q = collect_quality(receipts)
|
||||
if q["row_count"] == 0 and q["raw_line_count"] > SUSPECT_RAW_LINES:
|
||||
reasons.append(f"{side}:parsed_0_of_{q['raw_line_count']}_lines")
|
||||
elif q["commands"] and q["parse_status"].get("failed", 0) == q["commands"]:
|
||||
reasons.append(f"{side}:all_commands_failed")
|
||||
if reasons:
|
||||
return "suspect_parse", reasons
|
||||
if before_rows == 0 and after_rows == 0:
|
||||
return "no_data", ["both_sides_empty"]
|
||||
return "ok", []
|
||||
|
||||
|
||||
def assess_sheet(
|
||||
db: Session, sheet: Mapping[str, Any], *, before_batch_id: str, after_batch_id: str,
|
||||
before_rows: int, after_rows: int,
|
||||
) -> dict[str, Any]:
|
||||
"""Quality block stored on a sheet summary."""
|
||||
metric_id = str(sheet.get("metric_id") or "")
|
||||
before = side_receipts(db, before_batch_id, metric_id) if _has_receipts(db, before_batch_id) else None
|
||||
after = side_receipts(db, after_batch_id, metric_id) if _has_receipts(db, after_batch_id) else None
|
||||
state, reasons = classify_sheet(
|
||||
before=before, after=after, before_rows=before_rows, after_rows=after_rows,
|
||||
wanted_slice=sheet_slice(sheet),
|
||||
)
|
||||
return {
|
||||
"data_state": state,
|
||||
"data_state_reasons": reasons,
|
||||
"collect_quality": {
|
||||
"before": collect_quality(before or []), "after": collect_quality(after or []),
|
||||
"unknown_sides": [n for n, r in (("before", before), ("after", after)) if r is None],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def command_coverage(
|
||||
db: Session, *, before_batch_id: str, after_batch_id: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Collected "metric{slice}" labels per side and their difference.
|
||||
|
||||
Profile ids differ per vendor (``zte.x`` vs ``huawei.x``), so the comparison
|
||||
is on what a command *produces*: its metric and slice.
|
||||
"""
|
||||
from .profiles import get_profile
|
||||
|
||||
def labels(batch_id: str) -> set[str]:
|
||||
if not batch_id:
|
||||
return set()
|
||||
rows = db.query(BizStateBatchCommand.metric_id, BizStateBatchCommand.profile_id).filter(
|
||||
BizStateBatchCommand.batch_id == batch_id, BizStateBatchCommand.metric_id != "",
|
||||
).distinct().all()
|
||||
out: set[str] = set()
|
||||
for metric_id, profile_id in rows:
|
||||
prof = get_profile(str(profile_id or "")) if profile_id else None
|
||||
sl = dict(prof.slice) if prof is not None and prof.slice else {}
|
||||
suffix = ",".join(f"{k}={v}" for k, v in sorted(sl.items()))
|
||||
out.add(f"{metric_id}{{{suffix}}}" if suffix else str(metric_id))
|
||||
return out
|
||||
|
||||
b, a = labels(before_batch_id), labels(after_batch_id)
|
||||
return {"before": sorted(b), "after": sorted(a),
|
||||
"only_before": sorted(b - a), "only_after": sorted(a - b)}
|
||||
|
|
@ -137,6 +137,8 @@ def field_rule_map(rules: Sequence[Mapping[str, Any]] | None) -> dict[str, dict[
|
|||
name = str(raw.get("field") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
if str(raw.get("side") or "").strip():
|
||||
continue # side-scoped rules feed the identity pipeline, not the compare
|
||||
out[name] = dict(raw)
|
||||
return out
|
||||
|
||||
|
|
@ -210,6 +212,50 @@ def _parse_float(text: str) -> float | None:
|
|||
return None
|
||||
|
||||
|
||||
DIRECTIONAL_MODES = ("not_worse", "numeric_min", "numeric_max")
|
||||
|
||||
|
||||
def _rank_index(value: str, rank: Sequence[Any]) -> int | None:
|
||||
wanted = value.strip().lower()
|
||||
for i, item in enumerate(rank or []):
|
||||
if scalar_text(item).strip().lower() == wanted:
|
||||
return i
|
||||
return None
|
||||
|
||||
|
||||
def _tolerance(rule: Mapping[str, Any]) -> float:
|
||||
try:
|
||||
return float(rule.get("tolerance") or 0)
|
||||
except (TypeError, ValueError):
|
||||
return 0.0
|
||||
|
||||
|
||||
def _directional_verdict(bv: str, av: str, rule: Mapping[str, Any]) -> tuple[bool, str]:
|
||||
"""``(equal_or_better, reason)`` for not_worse / numeric_min / numeric_max."""
|
||||
mode = str(rule.get("compare") or "").strip().lower()
|
||||
if mode == "not_worse":
|
||||
rank = rule.get("rank") or []
|
||||
bi, ai = _rank_index(bv, rank), _rank_index(av, rank)
|
||||
if bi is None or ai is None:
|
||||
return bv == av, "neq"
|
||||
return ai <= bi, (f"worse: {bv}→{av}" if ai > bi else "")
|
||||
bn, an = _parse_float(bv), _parse_float(av)
|
||||
if bn is None or an is None:
|
||||
return bv == av, "neq"
|
||||
tol = _tolerance(rule)
|
||||
if mode == "numeric_min":
|
||||
floor = bn * (1 - tol / 100.0)
|
||||
if an >= floor:
|
||||
return True, ""
|
||||
drop = (bn - an) / abs(bn) * 100.0 if bn else 0.0
|
||||
return False, f"drop {drop:.0f}% > {tol:g}%"
|
||||
ceil = bn * (1 + tol / 100.0)
|
||||
if an <= ceil:
|
||||
return True, ""
|
||||
rise = (an - bn) / abs(bn) * 100.0 if bn else 0.0
|
||||
return False, f"rise {rise:.0f}% > {tol:g}%"
|
||||
|
||||
|
||||
def explain_diff(
|
||||
before: Any,
|
||||
after: Any,
|
||||
|
|
@ -222,6 +268,8 @@ def explain_diff(
|
|||
bv = normalize_value(before, norm)
|
||||
av = normalize_value(after, norm)
|
||||
mode = str(rule.get("compare") or "eq").strip().lower() or "eq"
|
||||
if mode in DIRECTIONAL_MODES:
|
||||
return _directional_verdict(bv, av, rule)[1]
|
||||
if mode in ("percent", "pct", "rel"):
|
||||
bn = _parse_float(bv)
|
||||
an = _parse_float(av)
|
||||
|
|
@ -262,6 +310,8 @@ def values_equal(
|
|||
mode = str(rule.get("compare") or "eq").strip().lower() or "eq"
|
||||
if mode in ("ignore", "skip", "off"):
|
||||
return True
|
||||
if mode in DIRECTIONAL_MODES:
|
||||
return _directional_verdict(bv, av, rule)[0]
|
||||
if mode in ("numeric", "number", "int", "float", "percent", "pct", "rel"):
|
||||
bn = _parse_float(bv)
|
||||
an = _parse_float(av)
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ from ..models import (
|
|||
BizCompareJob,
|
||||
BizCompareRun,
|
||||
BizCompareTemplate,
|
||||
BizMigrationProject,
|
||||
BizMonitorTemplate,
|
||||
BizPortMapping,
|
||||
BizPortMappingRow,
|
||||
BizStateBatch,
|
||||
|
|
@ -34,6 +36,7 @@ from .compare_engine import apply_port_map, compare_rows, mapping_stats, row_key
|
|||
from .compare_rules import (
|
||||
ROW_FILTER_PRESETS,
|
||||
apply_row_filters,
|
||||
scalar_text,
|
||||
arp_dynamic_row_filters,
|
||||
effective_compare_fields,
|
||||
effective_display_fields,
|
||||
|
|
@ -43,12 +46,19 @@ from .iface_normalize import (
|
|||
apply_iface_normalize_rows,
|
||||
default_zte_iface_normalize_rules,
|
||||
normalize_iface_rules,
|
||||
normalize_port_map,
|
||||
)
|
||||
from .compare_quality import UNJUDGED_STATES, assess_sheet
|
||||
from .identity_pipeline import Pipeline, build_pipeline, legacy_pipeline
|
||||
from .profiles import metric_field_map
|
||||
from .compare_validation import validate_template_body
|
||||
|
||||
_log = logging.getLogger("netx.biz_state.compare")
|
||||
|
||||
# v1: port-map keys/values must be written post-normalization.
|
||||
# v2: port-map keys/values are normalized with the template iface rules (A1).
|
||||
CONFIG_SNAPSHOT_VERSION = 2
|
||||
|
||||
# Per-job mutex so auto + manual run_compare do not dual-write the same job.
|
||||
_auto_compare_locks: dict[str, threading.Lock] = {}
|
||||
_auto_compare_locks_guard = threading.Lock()
|
||||
|
|
@ -426,11 +436,25 @@ def _metric_rows_by_ids(db: Session, ids: list[str]) -> dict[str, dict[str, Any]
|
|||
return out
|
||||
|
||||
|
||||
def _attach_raw_key_before(items: list[dict[str, Any]], key_fields: list[str]) -> list[dict[str, Any]]:
|
||||
"""``raw_key_before``: the before-side key as printed, only where it differs from the match key."""
|
||||
for it in items:
|
||||
before = it.get("before")
|
||||
if not before or it.get("raw_key_before"):
|
||||
continue
|
||||
shown = it.get("key") or {}
|
||||
raw = {f: scalar_text(before.get(f)).strip() for f in key_fields}
|
||||
if any(raw[f] != scalar_text(shown.get(f)).strip() for f in key_fields if f in shown):
|
||||
it["raw_key_before"] = raw
|
||||
return items
|
||||
|
||||
|
||||
def _hydrate_diff_rows(
|
||||
db: Session, items: list[dict[str, Any]], *,
|
||||
run: BizCompareRun | None = None, sheet: dict[str, Any] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Fill empty before/after from metric tables when row_ids are present."""
|
||||
key_fields = list((sheet or {}).get("key_fields") or [])
|
||||
need: list[str] = []
|
||||
for it in items:
|
||||
if not it.get("before") and it.get("before_row_id"):
|
||||
|
|
@ -438,27 +462,19 @@ def _hydrate_diff_rows(
|
|||
if not it.get("after") and it.get("after_row_id"):
|
||||
need.append(str(it["after_row_id"]))
|
||||
if not need:
|
||||
return items
|
||||
return _attach_raw_key_before(items, key_fields)
|
||||
by_id = _metric_rows_by_ids(db, need)
|
||||
pmap, norm_rules = _run_transform_config(db, run) if run else ({}, [])
|
||||
iface_fields = list((sheet or {}).get("iface_fields") or [])
|
||||
pipeline = _run_pipeline(db, run, sheet or {})
|
||||
for it in items:
|
||||
brid = str(it.get("before_row_id") or "")
|
||||
arid = str(it.get("after_row_id") or "")
|
||||
if not it.get("before") and brid and brid in by_id:
|
||||
it["before"] = by_id[brid]
|
||||
if not it.get("after") and arid and arid in by_id:
|
||||
it["after"] = apply_iface_normalize_rows(
|
||||
[by_id[arid]], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
it["after"] = pipeline.normalize(by_id[arid], "after")
|
||||
if not it.get("mapped_before") and it.get("before"):
|
||||
normalized = apply_iface_normalize_rows(
|
||||
[it["before"]], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
it["mapped_before"] = apply_port_map(
|
||||
normalized, iface_fields=iface_fields, port_map=pmap,
|
||||
)
|
||||
return items
|
||||
it["mapped_before"] = pipeline.transform(it["before"], "before")
|
||||
return _attach_raw_key_before(items, key_fields)
|
||||
|
||||
|
||||
def _diff_row_out(r: BizCompareDiff) -> dict[str, Any]:
|
||||
|
|
@ -653,18 +669,41 @@ def _normalize_row_filters(raw: Any) -> list[dict[str, Any]]:
|
|||
return out
|
||||
|
||||
|
||||
# Optional sheet-level switches that must survive normalization.
|
||||
_SHEET_OPTION_VALUES: dict[str, tuple[str, ...]] = {
|
||||
"unmapped_port_policy": ("by_name", "ignore_port"),
|
||||
"duplicate_pairing": ("sorted", "ordered"),
|
||||
}
|
||||
|
||||
|
||||
def _sheet_options(raw: dict[str, Any]) -> dict[str, Any]:
|
||||
out: dict[str, Any] = {}
|
||||
for key, allowed in _SHEET_OPTION_VALUES.items():
|
||||
val = str(raw.get(key) or "").strip().lower()
|
||||
if val in allowed:
|
||||
out[key] = val
|
||||
if raw.get("builtin_vendor_canonical") is False:
|
||||
out["builtin_vendor_canonical"] = False
|
||||
if raw.get("scope_by_port_map") is not None:
|
||||
out["scope_by_port_map"] = bool(raw.get("scope_by_port_map"))
|
||||
return out
|
||||
|
||||
|
||||
def _normalize_field_rules(raw: Any) -> list[dict[str, Any]]:
|
||||
if not isinstance(raw, list):
|
||||
return []
|
||||
out: list[dict[str, Any]] = []
|
||||
seen: set[str] = set()
|
||||
seen: set[tuple[str, str]] = set()
|
||||
for item in raw:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
name = str(item.get("field") or "").strip()
|
||||
if not name or name in seen:
|
||||
side = str(item.get("side") or "").strip().lower()
|
||||
side = side if side in ("before", "after") else ""
|
||||
# A field may carry one rule per side (e.g. a different value_map each).
|
||||
if not name or (name, side) in seen:
|
||||
continue
|
||||
seen.add(name)
|
||||
seen.add((name, side))
|
||||
rule: dict[str, Any] = {"field": name}
|
||||
compare = str(item.get("compare") or "").strip().lower()
|
||||
if compare:
|
||||
|
|
@ -680,6 +719,16 @@ def _normalize_field_rules(raw: Any) -> list[dict[str, Any]]:
|
|||
rule["tolerance"] = float(item.get("tolerance"))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
vmap = item.get("value_map")
|
||||
if isinstance(vmap, dict) and vmap:
|
||||
rule["value_map"] = {str(k): str(v) for k, v in vmap.items()}
|
||||
if side:
|
||||
rule["side"] = side
|
||||
if isinstance(item.get("apply_to_key"), bool):
|
||||
rule["apply_to_key"] = item["apply_to_key"]
|
||||
rank = item.get("rank")
|
||||
if isinstance(rank, list) and rank:
|
||||
rule["rank"] = [str(x) for x in rank]
|
||||
# Drop empty rules (only field name)
|
||||
if len(rule) > 1:
|
||||
out.append(rule)
|
||||
|
|
@ -709,6 +758,7 @@ def _sheet_def(
|
|||
row_filters: list[dict[str, Any]] | None = None,
|
||||
field_rules: list[dict[str, Any]] | None = None,
|
||||
ignore_port_changes: bool | None = None,
|
||||
options: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
mid = str(metric_id or "").strip()
|
||||
sid = str(sheet_id or "").strip() or mid
|
||||
|
|
@ -749,6 +799,7 @@ def _sheet_def(
|
|||
}
|
||||
if ignore_port_changes is not None:
|
||||
sheet["ignore_port_changes"] = bool(ignore_port_changes)
|
||||
sheet.update(options or {})
|
||||
return sheet
|
||||
|
||||
|
||||
|
|
@ -946,6 +997,49 @@ def _builtin_status_needs_packaged_upgrade(existing: list[dict[str, Any]]) -> bo
|
|||
return len(packaged_keys - have_keys) >= 3
|
||||
|
||||
|
||||
# A7: (metric_id, kind, field, old packaged value, new value). A stored sheet is
|
||||
# only touched when its value still equals the *old* packaged one, so anything
|
||||
# an operator edited survives the upgrade.
|
||||
_INTERFACE_DETAIL_OLD_COMPARE = [
|
||||
"admin", "input_bps", "output_bps", "port_status", "in_util", "out_util",
|
||||
"ip_mtu", "mtu", "mpls_mtu", "ipv6_mtu", "port_media", "negotiation",
|
||||
]
|
||||
_BUILTIN_STATUS_VALUE_UPGRADES: tuple[tuple[str, str, str, Any, Any], ...] = (
|
||||
("interface_detail", "rule", "output_bps",
|
||||
{"compare": "numeric", "tolerance": 5}, {"compare": "percent", "tolerance": 5}),
|
||||
("interface_detail", "list", "compare_fields", _INTERFACE_DETAIL_OLD_COMPARE,
|
||||
[f for f in _INTERFACE_DETAIL_OLD_COMPARE if f not in ("input_bps", "output_bps", "in_util", "out_util")]),
|
||||
("optical_brief", "rule", "rx_power",
|
||||
{"compare": "percent", "tolerance": 5}, {"compare": "numeric", "tolerance": 2}),
|
||||
("l2vpn_mac", "list", "iface_fields", [], ["ac_port"]),
|
||||
# ARP: filter on the canonical ``entry_type`` (every vendor parser emits it); the
|
||||
# age-timer clause stays only as a fallback for rows parsed before entry_type existed.
|
||||
("arp", "list", "row_filters", [{"op": "age_timer", "field": "age", "value": ""}], arp_dynamic_row_filters()),
|
||||
)
|
||||
|
||||
|
||||
def _upgrade_builtin_status_values(sheet: dict[str, Any]) -> tuple[dict[str, Any], bool]:
|
||||
"""Apply ``_BUILTIN_STATUS_VALUE_UPGRADES`` to one stored sheet."""
|
||||
mid = str(sheet.get("metric_id") or "")
|
||||
out = deepcopy(sheet)
|
||||
changed = False
|
||||
for metric_id, kind, field, old, new in _BUILTIN_STATUS_VALUE_UPGRADES:
|
||||
if mid != metric_id:
|
||||
continue
|
||||
if kind == "list":
|
||||
if list(out.get(field) or []) == old:
|
||||
out[field] = list(new)
|
||||
changed = True
|
||||
continue
|
||||
for rule in out.get("field_rules") or []:
|
||||
if isinstance(rule, dict) and rule.get("field") == field and all(
|
||||
rule.get(k) == v for k, v in old.items()
|
||||
):
|
||||
rule.update(new)
|
||||
changed = True
|
||||
return out, changed
|
||||
|
||||
|
||||
def _default_zte_config_sheets() -> list[dict[str, Any]]:
|
||||
"""Config-intent metrics for cutover / intent-vs-intent compare."""
|
||||
return [
|
||||
|
|
@ -1009,6 +1103,7 @@ def _normalize_sheet(raw: Any) -> dict[str, Any] | None:
|
|||
row_filters=_normalize_row_filters(raw.get("row_filters")),
|
||||
field_rules=rules,
|
||||
ignore_port_changes=ignore_ports,
|
||||
options=_sheet_options(raw),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -1252,6 +1347,8 @@ def ensure_default_zte_status_template(db: Session) -> BizCompareTemplate:
|
|||
if not cur.get("field_rules") and src and src.get("field_rules"):
|
||||
cur["field_rules"] = list(src["field_rules"])
|
||||
changed = True
|
||||
cur, value_changed = _upgrade_builtin_status_values(cur)
|
||||
changed = changed or value_changed
|
||||
upgraded.append(_normalize_sheet(cur) or cur)
|
||||
have_mids = {str(s.get("metric_id") or "") for s in upgraded}
|
||||
for s in sheets:
|
||||
|
|
@ -1389,7 +1486,16 @@ def list_templates(db: Session) -> list[dict[str, Any]]:
|
|||
upgrade_builtin_split_sheets(db)
|
||||
rows = db.query(BizCompareTemplate).order_by(BizCompareTemplate.name.asc()).all()
|
||||
usage = dict(db.query(BizCompareJob.template_id, func.count(BizCompareJob.id)).group_by(BizCompareJob.template_id).all())
|
||||
return [{**_template_out(t), "job_count": usage.get(t.id, 0)} for t in rows]
|
||||
monitor_usage = dict(
|
||||
db.query(BizMonitorTemplate.compare_template_id, func.count(BizMonitorTemplate.id))
|
||||
.filter(BizMonitorTemplate.compare_template_id != "")
|
||||
.group_by(BizMonitorTemplate.compare_template_id).all()
|
||||
)
|
||||
return [
|
||||
{**_template_out(t), "job_count": usage.get(t.id, 0),
|
||||
"monitor_template_count": monitor_usage.get(t.id, 0)}
|
||||
for t in rows
|
||||
]
|
||||
|
||||
|
||||
def list_metric_schemas() -> list[dict[str, Any]]:
|
||||
|
|
@ -1522,8 +1628,19 @@ def delete_template(db: Session, template_id: str) -> None:
|
|||
t = db.get(BizCompareTemplate, template_id)
|
||||
if not t:
|
||||
raise HTTPException(status_code=404, detail="template_not_found")
|
||||
if db.query(BizCompareJob.id).filter(BizCompareJob.template_id == template_id).first():
|
||||
raise HTTPException(status_code=409, detail="template_in_use")
|
||||
jobs = [n or i for i, n in db.query(BizCompareJob.id, BizCompareJob.name)
|
||||
.filter(BizCompareJob.template_id == template_id).all()]
|
||||
monitors = [n or i for i, n in db.query(BizMonitorTemplate.id, BizMonitorTemplate.name)
|
||||
.filter(BizMonitorTemplate.compare_template_id == template_id).all()]
|
||||
if jobs or monitors:
|
||||
raise HTTPException(status_code=409, detail={
|
||||
"code": "template_in_use",
|
||||
"message": "template_in_use: " + "; ".join(filter(None, [
|
||||
f"jobs={', '.join(jobs)}" if jobs else "",
|
||||
f"monitor_templates={', '.join(monitors)}" if monitors else "",
|
||||
])),
|
||||
"jobs": jobs, "monitor_templates": monitors,
|
||||
})
|
||||
db.delete(t)
|
||||
db.commit()
|
||||
|
||||
|
|
@ -1544,9 +1661,30 @@ def _mapping_out(db: Session, m: BizPortMapping) -> dict[str, Any]:
|
|||
}
|
||||
|
||||
|
||||
def _mapping_references(db: Session, mapping_id: str) -> tuple[list[str], list[str]]:
|
||||
"""(compare job names, cutover project names) that reference this mapping."""
|
||||
jobs = [n or i for i, n in db.query(BizCompareJob.id, BizCompareJob.name)
|
||||
.filter(BizCompareJob.mapping_id == mapping_id).all()]
|
||||
projects = [n or i for i, n in db.query(BizMigrationProject.id, BizMigrationProject.name)
|
||||
.filter(BizMigrationProject.mapping_id == mapping_id).all()]
|
||||
return jobs, projects
|
||||
|
||||
|
||||
def list_mappings(db: Session) -> list[dict[str, Any]]:
|
||||
rows = db.query(BizPortMapping).order_by(BizPortMapping.name.asc()).all()
|
||||
return [_mapping_out(db, m) for m in rows]
|
||||
job_counts = dict(
|
||||
db.query(BizCompareJob.mapping_id, func.count(BizCompareJob.id))
|
||||
.filter(BizCompareJob.mapping_id != "").group_by(BizCompareJob.mapping_id).all()
|
||||
)
|
||||
project_counts = dict(
|
||||
db.query(BizMigrationProject.mapping_id, func.count(BizMigrationProject.id))
|
||||
.filter(BizMigrationProject.mapping_id != "").group_by(BizMigrationProject.mapping_id).all()
|
||||
)
|
||||
return [
|
||||
{**_mapping_out(db, m), "job_count": job_counts.get(m.id, 0),
|
||||
"project_count": project_counts.get(m.id, 0)}
|
||||
for m in rows
|
||||
]
|
||||
|
||||
|
||||
def create_mapping(db: Session, body: dict[str, Any]) -> dict[str, Any]:
|
||||
|
|
@ -1604,6 +1742,16 @@ def delete_mapping(db: Session, mapping_id: str) -> None:
|
|||
m = db.get(BizPortMapping, mapping_id)
|
||||
if not m:
|
||||
raise HTTPException(status_code=404, detail="mapping_not_found")
|
||||
jobs, projects = _mapping_references(db, mapping_id)
|
||||
if jobs or projects:
|
||||
raise HTTPException(status_code=409, detail={
|
||||
"code": "mapping_in_use",
|
||||
"message": "mapping_in_use: " + "; ".join(filter(None, [
|
||||
f"jobs={', '.join(jobs)}" if jobs else "",
|
||||
f"projects={', '.join(projects)}" if projects else "",
|
||||
])),
|
||||
"jobs": jobs, "projects": projects,
|
||||
})
|
||||
db.query(BizPortMappingRow).filter(BizPortMappingRow.mapping_id == mapping_id).delete()
|
||||
db.delete(m)
|
||||
db.commit()
|
||||
|
|
@ -1822,11 +1970,11 @@ def validate_mapping(
|
|||
sheets = template_metrics(tpl)
|
||||
if not sheets:
|
||||
raise HTTPException(status_code=400, detail="template_has_no_metrics")
|
||||
pmap = _port_map_dict(db, mapping_id)
|
||||
# Validate against first sheet that has iface fields (or first sheet)
|
||||
target = next((s for s in sheets if s.get("iface_fields")), sheets[0])
|
||||
iface_fields = list(target.get("iface_fields") or [])
|
||||
norm = template_iface_normalize(tpl)
|
||||
pmap, _ = normalize_port_map(_port_map_dict(db, mapping_id), norm)
|
||||
before = apply_iface_normalize_rows(
|
||||
_load_metric_rows(db, batch_id=before_batch_id, metric_id=target["metric_id"]),
|
||||
iface_fields=iface_fields,
|
||||
|
|
@ -1845,6 +1993,66 @@ def validate_mapping(
|
|||
)
|
||||
|
||||
|
||||
def _mapping_rows_to_dict(rows: list[Any]) -> tuple[dict[str, str], list[dict[str, str]]]:
|
||||
"""Editor rows → raw map; a repeated before port is reported, first one wins."""
|
||||
out: dict[str, str] = {}
|
||||
dups: list[dict[str, str]] = []
|
||||
for r in rows or []:
|
||||
raw = r if isinstance(r, dict) else dict(r)
|
||||
before = str(raw.get("before_if") or "").strip()
|
||||
after = str(raw.get("after_if") or "").strip()
|
||||
if not before or not after:
|
||||
continue
|
||||
if before in out:
|
||||
dups.append({"before_if": before, "kept_to": out[before], "dropped_to": after})
|
||||
continue
|
||||
out[before] = after
|
||||
return out, dups
|
||||
|
||||
|
||||
def preflight_compare(db: Session, body: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Mapping coverage + data-quality report for a template and two batches."""
|
||||
from .compare_preflight import preflight
|
||||
|
||||
ensure_default_templates(db)
|
||||
template_id = str(body.get("template_id") or "")
|
||||
tpl = db.get(BizCompareTemplate, template_id) if template_id else ensure_default_cutover_template(db)
|
||||
if not tpl:
|
||||
raise HTTPException(status_code=404, detail="template_not_found")
|
||||
before_id = str(body.get("before_batch_id") or "")
|
||||
after_id = str(body.get("after_batch_id") or "")
|
||||
if not db.get(BizStateBatch, before_id) or not db.get(BizStateBatch, after_id):
|
||||
raise HTTPException(status_code=404, detail="batch_not_found")
|
||||
sheets = _filter_enabled_sheets(template_metrics(tpl), _str_list(body.get("enabled_sheet_ids")))
|
||||
if not sheets:
|
||||
raise HTTPException(status_code=400, detail="no_enabled_sheets")
|
||||
dups: list[dict[str, str]] = []
|
||||
if body.get("mapping_rows") is not None:
|
||||
raw_map, dups = _mapping_rows_to_dict(list(body.get("mapping_rows") or []))
|
||||
else:
|
||||
raw_map = _port_map_dict(db, str(body.get("mapping_id") or ""))
|
||||
cache: dict[tuple[str, str], list[dict[str, Any]]] = {}
|
||||
|
||||
def load(side: str, sheet: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
batch_id = before_id if side == "before" else after_id
|
||||
key = (batch_id, str(sheet["metric_id"]))
|
||||
if key not in cache:
|
||||
cache[key] = _load_metric_rows(db, batch_id=batch_id, metric_id=key[1])
|
||||
return cache[key]
|
||||
|
||||
report = preflight(
|
||||
db, sheets=sheets, iface_rules=template_iface_normalize(tpl), raw_port_map=raw_map,
|
||||
before_batch_id=before_id, after_batch_id=after_id,
|
||||
before_vendor=_batch_vendor_key(db, before_id), after_vendor=_batch_vendor_key(db, after_id),
|
||||
load_rows=load,
|
||||
)
|
||||
if dups:
|
||||
report["mapping"]["duplicate_before"] = dups
|
||||
report["risk_reasons"] = sorted({*report["risk_reasons"], "duplicate_before"})
|
||||
report["verdict"] = "risk"
|
||||
return report
|
||||
|
||||
|
||||
def _job_out(j: BizCompareJob) -> dict[str, Any]:
|
||||
return {
|
||||
"id": j.id,
|
||||
|
|
@ -2040,14 +2248,23 @@ def _sheet_result_envelope(
|
|||
}
|
||||
|
||||
|
||||
def _sheet_quality(
|
||||
db: Session, sheet: dict[str, Any], before_batch_id: str, after_batch_id: str,
|
||||
summary: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
return assess_sheet(
|
||||
db, sheet, before_batch_id=before_batch_id, after_batch_id=after_batch_id,
|
||||
before_rows=int(summary.get("before_count") or 0), after_rows=int(summary.get("after_count") or 0),
|
||||
)
|
||||
|
||||
|
||||
def _run_sheet(
|
||||
db: Session,
|
||||
*,
|
||||
sheet: dict[str, Any],
|
||||
before_batch_id: str,
|
||||
after_batch_id: str,
|
||||
port_map: dict[str, str],
|
||||
iface_normalize_rules: list[dict[str, str]] | None = None,
|
||||
pipeline: Pipeline,
|
||||
store_unchanged: str = "auto",
|
||||
on_load_progress: Callable[..., None] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
|
|
@ -2083,12 +2300,7 @@ def _run_sheet(
|
|||
# PostgreSQL path: pushdown-safe sheets join in-DB (BGP-scale).
|
||||
from .compare_sql import SqlCompareSkip, run_sql_sheet_compare, sql_compare_skip_reason
|
||||
|
||||
skip_reason = sql_compare_skip_reason(
|
||||
db,
|
||||
sheet,
|
||||
port_map=port_map,
|
||||
iface_normalize_rules=iface_normalize_rules,
|
||||
)
|
||||
skip_reason = sql_compare_skip_reason(db, sheet, pipeline=pipeline)
|
||||
if not skip_reason:
|
||||
try:
|
||||
result = run_sql_sheet_compare(
|
||||
|
|
@ -2098,6 +2310,7 @@ def _run_sheet(
|
|||
after_batch_id=after_batch_id,
|
||||
store_unchanged=store_unchanged,
|
||||
on_progress=_emit_load,
|
||||
duplicate_pairing=pipeline.duplicate_pairing,
|
||||
)
|
||||
summary = dict(result["summary"])
|
||||
# Engine already sets raw counts / policy; keep keys stable
|
||||
|
|
@ -2107,6 +2320,7 @@ def _run_sheet(
|
|||
before_n=int(summary.get("before_count") or 0),
|
||||
after_n=int(summary.get("after_count") or 0),
|
||||
)
|
||||
summary.update(_sheet_quality(db, sheet, before_batch_id, after_batch_id, summary))
|
||||
return _sheet_result_envelope(
|
||||
sheet,
|
||||
key_fields=key_fields,
|
||||
|
|
@ -2161,39 +2375,38 @@ def _run_sheet(
|
|||
def _after_chunk(n: int) -> None:
|
||||
_emit_load("after", n, engine="python", note=skip_reason or "python", phase="loading")
|
||||
|
||||
# Only filters that read untransformed fields may run in SQL on the raw rows;
|
||||
# the pipeline applies every filter (post-normalize for v2) in ``prepare``.
|
||||
before_raw = _load_metric_rows(
|
||||
db,
|
||||
batch_id=before_batch_id,
|
||||
metric_id=mid,
|
||||
on_chunk=_before_chunk,
|
||||
row_filters=row_filters,
|
||||
row_filters=pipeline.pushdown_filters("before"),
|
||||
)
|
||||
after_raw = _load_metric_rows(
|
||||
db,
|
||||
batch_id=after_batch_id,
|
||||
metric_id=mid,
|
||||
on_chunk=_after_chunk,
|
||||
row_filters=row_filters,
|
||||
row_filters=pipeline.pushdown_filters("after"),
|
||||
)
|
||||
# Filters already applied in load when pushdown-safe; keep apply for safety
|
||||
before_rows = apply_row_filters(before_raw, row_filters) if row_filters else before_raw
|
||||
after_rows = apply_row_filters(after_raw, row_filters) if row_filters else after_raw
|
||||
before_pairs = pipeline.prepare(before_raw, "before")
|
||||
after_pairs = pipeline.prepare(after_raw, "after")
|
||||
policy = resolve_unchanged_policy(
|
||||
store_unchanged, before_n=len(before_rows), after_n=len(after_rows)
|
||||
store_unchanged, before_n=len(before_pairs), after_n=len(after_pairs)
|
||||
)
|
||||
result = compare_rows(
|
||||
before_rows=before_rows,
|
||||
after_rows=after_rows,
|
||||
before_rows=[], after_rows=[],
|
||||
key_fields=key_fields,
|
||||
iface_fields=iface_fields,
|
||||
compare_fields=compare_fields,
|
||||
port_map=port_map,
|
||||
field_rules=field_rules,
|
||||
iface_normalize_rules=iface_normalize_rules,
|
||||
ignore_port_changes=ignore_ports,
|
||||
include_unchanged=bool(policy["include"]),
|
||||
unchanged_limit=policy.get("limit"),
|
||||
compact_unchanged=bool(policy.get("compact")),
|
||||
pipeline=pipeline,
|
||||
prepared=(before_pairs, after_pairs),
|
||||
)
|
||||
summary = dict(result["summary"])
|
||||
summary["before_raw_count"] = len(before_raw)
|
||||
|
|
@ -2201,6 +2414,14 @@ def _run_sheet(
|
|||
summary["row_filters"] = len(row_filters)
|
||||
summary["unchanged_policy"] = policy
|
||||
summary.setdefault("engine", "python")
|
||||
if pipeline.port_map and pipeline.iface_fields:
|
||||
from .compare_preflight import sheet_mapping_coverage
|
||||
|
||||
coverage = sheet_mapping_coverage(pipeline, before_pairs, after_pairs)
|
||||
coverage.pop("_after_ifaces", None)
|
||||
coverage.pop("_used_keys", None)
|
||||
summary["mapping_coverage"] = coverage
|
||||
summary.update(_sheet_quality(db, sheet, before_batch_id, after_batch_id, summary))
|
||||
return _sheet_result_envelope(
|
||||
sheet,
|
||||
key_fields=key_fields,
|
||||
|
|
@ -2227,6 +2448,8 @@ def _validate_compare_job(
|
|||
tpl = db.get(BizCompareTemplate, j.template_id)
|
||||
if not tpl:
|
||||
raise HTTPException(status_code=404, detail="template_not_found")
|
||||
if j.mapping_id and not db.get(BizPortMapping, j.mapping_id):
|
||||
raise HTTPException(status_code=400, detail="mapping_missing")
|
||||
before_batch_id = str(j.before_batch_id or "")
|
||||
after_batch_id = str(force_after_batch_id or "").strip() or _resolve_after_batch(db, j)
|
||||
if not before_batch_id or not after_batch_id:
|
||||
|
|
@ -2272,15 +2495,31 @@ def _create_running_run(
|
|||
before_batch_id: str,
|
||||
after_batch_id: str,
|
||||
sheets_cfg: list[dict[str, Any]],
|
||||
purpose: str = "manual",
|
||||
) -> BizCompareRun:
|
||||
first_metric = str(sheets_cfg[0].get("metric_id") or "")
|
||||
pending_sheets = [_pending_sheet_meta(s) for s in sheets_cfg]
|
||||
mapping = db.get(BizPortMapping, job.mapping_id) if job.mapping_id else None
|
||||
vendors = {
|
||||
"before": _batch_vendor_key(db, before_batch_id),
|
||||
"after": _batch_vendor_key(db, after_batch_id),
|
||||
}
|
||||
raw_port_map = _port_map_dict(db, str(job.mapping_id or ""))
|
||||
iface_rules = template_iface_normalize(tpl)
|
||||
identity = {
|
||||
sheet_key(sh): build_pipeline(
|
||||
sh, iface_rules=iface_rules, port_map=raw_port_map,
|
||||
before_vendor=vendors["before"], after_vendor=vendors["after"],
|
||||
).describe()
|
||||
for sh in sheets_cfg
|
||||
}
|
||||
config_snapshot = deepcopy({
|
||||
"version": 1,
|
||||
"version": CONFIG_SNAPSHOT_VERSION,
|
||||
"template": _template_out(tpl),
|
||||
"sheets": sheets_cfg,
|
||||
"port_map": _port_map_dict(db, str(job.mapping_id or "")),
|
||||
"port_map": raw_port_map,
|
||||
"vendors": vendors,
|
||||
"identity": identity,
|
||||
"store_unchanged": normalize_store_unchanged(getattr(job, "store_unchanged", None)),
|
||||
"job_name": job.name or "",
|
||||
"before_task_id": job.before_task_id or "",
|
||||
|
|
@ -2297,6 +2536,7 @@ def _create_running_run(
|
|||
metric_id=first_metric,
|
||||
status="running",
|
||||
summary_json={
|
||||
"purpose": purpose,
|
||||
"config_snapshot": config_snapshot,
|
||||
"progress": {
|
||||
"phase": "queued",
|
||||
|
|
@ -2329,7 +2569,7 @@ def _create_running_run(
|
|||
|
||||
def _run_config_snapshot(run: BizCompareRun) -> dict[str, Any] | None:
|
||||
snapshot = (run.summary_json or {}).get("config_snapshot")
|
||||
if isinstance(snapshot, dict) and snapshot.get("version") == 1:
|
||||
if isinstance(snapshot, dict) and snapshot.get("version") in (1, 2):
|
||||
return snapshot
|
||||
return None
|
||||
|
||||
|
|
@ -2339,14 +2579,66 @@ def _run_transform_config(
|
|||
) -> tuple[dict[str, str], list[dict[str, str]]]:
|
||||
snapshot = _run_config_snapshot(run)
|
||||
if snapshot is not None:
|
||||
return dict(snapshot.get("port_map") or {}), normalize_iface_rules(
|
||||
rules = normalize_iface_rules(
|
||||
(snapshot.get("template") or {}).get("iface_normalize_rules") or [],
|
||||
)
|
||||
pmap = dict(snapshot.get("port_map") or {})
|
||||
if int(snapshot.get("version") or 1) >= 2:
|
||||
# v2 runs: map keys/values are written as the device prints them.
|
||||
pmap, _ = normalize_port_map(pmap, rules)
|
||||
return pmap, rules
|
||||
# Legacy runs did not record these settings; retain their fallback behavior.
|
||||
tpl = db.get(BizCompareTemplate, run.template_id) if run.template_id else None
|
||||
return _port_map_dict(db, str(run.mapping_id or "")), template_iface_normalize(tpl)
|
||||
|
||||
|
||||
def _batch_vendor_key(db: Session, batch_id: str) -> str:
|
||||
"""Vendor key (``huawei``/``zte``/…) for a batch, '' when unknown."""
|
||||
from ..topology_lldp import resolve_vendor_key
|
||||
|
||||
batch = db.get(BizStateBatch, batch_id) if batch_id else None
|
||||
if batch is None:
|
||||
return ""
|
||||
task = db.get(BizStateTask, batch.task_id) if batch.task_id else None
|
||||
key = resolve_vendor_key(
|
||||
batch.vendor or (task.vendor if task else ""), (task.device_type if task else "") or "",
|
||||
)
|
||||
return "" if key == "generic" else key
|
||||
|
||||
|
||||
def _run_pipeline(db: Session, run: BizCompareRun | None, sheet: dict[str, Any]) -> Pipeline:
|
||||
"""The identity pipeline a run used (or would use) for one sheet.
|
||||
|
||||
Snapshot v2 runs get the full pipeline; v1 and snapshot-less runs get the
|
||||
legacy one so stored results replay exactly as they were produced.
|
||||
"""
|
||||
sid = sheet_key(sheet)
|
||||
snapshot = _run_config_snapshot(run) if run is not None else None
|
||||
if snapshot is None:
|
||||
pmap, rules = _run_transform_config(db, run) if run is not None else ({}, [])
|
||||
return legacy_pipeline(
|
||||
key_fields=sheet.get("key_fields") or [], iface_fields=sheet.get("iface_fields") or [],
|
||||
compare_fields=sheet.get("compare_fields") or [], field_rules=sheet.get("field_rules"),
|
||||
port_map=pmap, iface_normalize_rules=rules, row_filters=sheet.get("row_filters"),
|
||||
ignore_port_changes=sheet.get("ignore_port_changes"),
|
||||
)
|
||||
authoritative = next(
|
||||
(sh for sh in snapshot.get("sheets") or [] if isinstance(sh, dict) and sheet_key(sh) == sid), None,
|
||||
)
|
||||
cfg = authoritative or sheet
|
||||
version = int(snapshot.get("version") or 1)
|
||||
vendors = snapshot.get("vendors") or {}
|
||||
return build_pipeline(
|
||||
cfg,
|
||||
iface_rules=(snapshot.get("template") or {}).get("iface_normalize_rules") or [],
|
||||
port_map=dict(snapshot.get("port_map") or {}),
|
||||
before_vendor=str(vendors.get("before") or ""),
|
||||
after_vendor=str(vendors.get("after") or ""),
|
||||
legacy=version < 2,
|
||||
normalize_map=version >= 2,
|
||||
)
|
||||
|
||||
|
||||
def _set_run_progress(
|
||||
db: Session,
|
||||
run: BizCompareRun,
|
||||
|
|
@ -2458,7 +2750,6 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
store_mode = normalize_store_unchanged(
|
||||
snapshot.get("store_unchanged") if snapshot is not None else getattr(j, "store_unchanged", None),
|
||||
)
|
||||
pmap, norm_rules = _run_transform_config(db, run)
|
||||
unchanged_listed_total = 0
|
||||
unchanged_truncated_any = False
|
||||
unchanged_compact_any = False
|
||||
|
|
@ -2563,8 +2854,7 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
sheet=sheet,
|
||||
before_batch_id=before_batch_id,
|
||||
after_batch_id=after_batch_id,
|
||||
port_map=pmap,
|
||||
iface_normalize_rules=norm_rules,
|
||||
pipeline=_run_pipeline(db, run, sheet),
|
||||
store_unchanged=store_mode,
|
||||
on_load_progress=_on_load,
|
||||
)
|
||||
|
|
@ -2755,6 +3045,7 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
if snapshot is not None:
|
||||
summary_payload["config_snapshot"] = snapshot
|
||||
run = db.get(BizCompareRun, run_id) or run
|
||||
summary_payload["purpose"] = (run.summary_json or {}).get("purpose") or "manual"
|
||||
if str(run.status or "") == "cancelled":
|
||||
return get_run(db, run.id)
|
||||
run.status = "success"
|
||||
|
|
@ -2882,7 +3173,9 @@ def cancel_compare_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
return get_run(db, run_id)
|
||||
|
||||
|
||||
def run_compare(db: Session, job_id: str, *, force_after_batch_id: str = "") -> dict[str, Any]:
|
||||
def run_compare(
|
||||
db: Session, job_id: str, *, force_after_batch_id: str = "", purpose: str = "manual",
|
||||
) -> dict[str, Any]:
|
||||
"""Synchronous compare (auto-compare / tests). Blocks the caller until done."""
|
||||
if _job_has_active_run(db, job_id):
|
||||
raise HTTPException(status_code=409, detail="compare_already_running")
|
||||
|
|
@ -2898,6 +3191,7 @@ def run_compare(db: Session, job_id: str, *, force_after_batch_id: str = "") ->
|
|||
before_batch_id=ctx["before_batch_id"],
|
||||
after_batch_id=ctx["after_batch_id"],
|
||||
sheets_cfg=ctx["sheets_cfg"],
|
||||
purpose=purpose,
|
||||
)
|
||||
return _execute_compare_into_run(db, run.id)
|
||||
finally:
|
||||
|
|
@ -2972,7 +3266,9 @@ def _sheet_csv_lines(sheet: dict[str, Any], diffs: Iterable[dict[str, Any]]) ->
|
|||
)
|
||||
# Non-key columns already ordered Key→Compare→Display by effective_display_fields
|
||||
extra = [f for f in display if f not in key_set]
|
||||
headers = ["kind", *keys]
|
||||
# Interface keys also get the name as the device printed it (before side).
|
||||
raw_cols = [k for k in keys if k in set(sheet.get("iface_fields") or [])]
|
||||
headers = ["kind", *keys, *[f"{k}__raw_before" for k in raw_cols]]
|
||||
for f in extra:
|
||||
if f in compare_set:
|
||||
headers.append(f"{f}__pre")
|
||||
|
|
@ -2988,6 +3284,9 @@ def _sheet_csv_lines(sheet: dict[str, Any], diffs: Iterable[dict[str, Any]]) ->
|
|||
row = [kind]
|
||||
for k in keys:
|
||||
row.append(key.get(k, pre.get(k, post.get(k, ""))))
|
||||
raw_before = dict(d.get("raw_key_before") or {})
|
||||
for k in raw_cols:
|
||||
row.append(raw_before.get(k, ""))
|
||||
for f in extra:
|
||||
if f in compare_set:
|
||||
if kind == "added":
|
||||
|
|
@ -3012,6 +3311,18 @@ def _sheet_csv(sheet: dict[str, Any]) -> str:
|
|||
return "\ufeff" + "\n".join(_sheet_csv_lines(sheet, sheet.get("diffs") or [])) + "\n"
|
||||
|
||||
|
||||
def _task_ok(sheets: list[dict[str, Any]], fail_count: int, any_pending: bool) -> bool | None:
|
||||
"""True/False once judged; None when data quality leaves the verdict open."""
|
||||
if any_pending:
|
||||
return False
|
||||
states = [str((sh.get("summary") or {}).get("data_state") or "ok") for sh in sheets]
|
||||
if any(st in ("suspect_parse", "not_collected") for st in states):
|
||||
return None
|
||||
if states and all(st == "no_data" for st in states):
|
||||
return None
|
||||
return fail_count == 0
|
||||
|
||||
|
||||
def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
added = int(summary.get("added") or 0)
|
||||
removed = int(summary.get("removed") or 0)
|
||||
|
|
@ -3020,16 +3331,28 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di
|
|||
before_count = int(summary.get("before_count") or 0)
|
||||
after_count = int(summary.get("after_count") or 0)
|
||||
total = added + removed + changed + unchanged
|
||||
|
||||
def _data_state(sh: dict[str, Any]) -> str:
|
||||
return str((sh.get("summary") or {}).get("data_state") or "ok")
|
||||
|
||||
# Sheets whose side was never collected cannot be judged: their "missing"
|
||||
# rows are a collection gap, not a business loss, so they stay out of the totals.
|
||||
dropped = [sh for sh in sheets if _data_state(sh) == "not_collected"
|
||||
and str(sh.get("status") or "done") not in ("pending", "running", "queued")]
|
||||
drop_fail = sum(int((sh.get("summary") or {}).get("removed") or 0)
|
||||
+ int((sh.get("summary") or {}).get("changed") or 0) for sh in dropped)
|
||||
drop_ok = sum(int((sh.get("summary") or {}).get("unchanged") or 0) for sh in dropped)
|
||||
# Compare verdict: fail = missing + mismatch; success = match; added is special
|
||||
fail_count = removed + changed
|
||||
success_count = unchanged
|
||||
fail_count = removed + changed - drop_fail
|
||||
success_count = unchanged - drop_ok
|
||||
judged = fail_count + success_count
|
||||
matched = changed + unchanged
|
||||
diff_count = fail_count
|
||||
# Nothing compared at all is "not judged" (None), never a 100% pass.
|
||||
pass_rate = (
|
||||
compute_pass_rate(success=success_count, fail=fail_count, empty_as=100.0 if total == 0 else 0.0)
|
||||
compute_pass_rate(success=success_count, fail=fail_count)
|
||||
if judged
|
||||
else (100.0 if total == 0 else 0.0)
|
||||
else (None if total == 0 else 0.0)
|
||||
)
|
||||
diff_rate = round((fail_count / judged) * 100, 1) if judged else 0.0
|
||||
|
||||
|
|
@ -3045,8 +3368,12 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di
|
|||
sj = sf + su
|
||||
status = str(sh.get("status") or "done")
|
||||
pending = status in ("pending", "running", "queued")
|
||||
data_state = str(ss.get("data_state") or "ok")
|
||||
sheet_cards.append(
|
||||
{
|
||||
"data_state": data_state,
|
||||
"data_state_reasons": list(ss.get("data_state_reasons") or []),
|
||||
"collect_quality": ss.get("collect_quality") or {},
|
||||
"sheet_id": sheet_key(sh),
|
||||
"title": sheet_title(sh),
|
||||
"metric_id": sh.get("metric_id") or "",
|
||||
|
|
@ -3063,12 +3390,8 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di
|
|||
"diff_count": sf,
|
||||
# Pending sheets must not look like "100% pass"
|
||||
"pass_rate": None
|
||||
if pending
|
||||
else (
|
||||
compute_pass_rate(success=su, fail=sf, empty_as=100.0 if st == 0 else 0.0)
|
||||
if sj
|
||||
else (100.0 if st == 0 else 0.0)
|
||||
),
|
||||
if pending or data_state in UNJUDGED_STATES
|
||||
else (compute_pass_rate(success=su, fail=sf) if sj else (None if st == 0 else 0.0)),
|
||||
}
|
||||
)
|
||||
|
||||
|
|
@ -3105,7 +3428,9 @@ def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> di
|
|||
"diff_count": diff_count,
|
||||
"pass_rate": None if any_pending else pass_rate,
|
||||
"diff_rate": diff_rate,
|
||||
"ok": False if any_pending else fail_count == 0,
|
||||
"ok": _task_ok(sheets, fail_count, any_pending),
|
||||
"unjudged_sheets": sum(1 for sh in sheets if _data_state(sh) in UNJUDGED_STATES),
|
||||
"suspect_sheets": sum(1 for sh in sheets if _data_state(sh) == "suspect_parse"),
|
||||
"sheet_cards": sheet_cards,
|
||||
"top_changed_fields": top_fields,
|
||||
"duration_ms": int(summary.get("duration_ms") or 0),
|
||||
|
|
@ -3449,21 +3774,25 @@ def _load_metric_rows_for_search(
|
|||
|
||||
def _load_search_key_groups(
|
||||
db: Session, *, batch_id: str, metric_id: str, keys: set[tuple[str, ...]],
|
||||
match_keys: list[str], iface_fields: list[str], row_filters: list[dict[str, Any]],
|
||||
norm_rules: list[dict[str, str]], port_map: dict[str, str],
|
||||
match_keys: list[str], pipeline: Pipeline, side: str,
|
||||
) -> tuple[list[dict[str, Any]], set[tuple[str, ...]]]:
|
||||
"""Complete candidate key groups so a one-sided search cannot change verdicts.
|
||||
|
||||
Interface transforms run in Python. Other identity fields narrow the SQL
|
||||
query. If the load budget is exceeded, omit entire groups rather than compare
|
||||
partial groups and manufacture additions/deletions.
|
||||
Identity rules (normalize / value map / port map) run in the pipeline, in
|
||||
Python. Key fields they never touch narrow the SQL query. If the load budget
|
||||
is exceeded, omit entire groups rather than compare partial groups and
|
||||
manufacture additions/deletions.
|
||||
"""
|
||||
from ..models import BizStateMetricRow
|
||||
|
||||
model = BizStateLldpNeighbor if metric_id == "lldp_neighbor" else BizStateMetricRow
|
||||
invariant = [f for f in match_keys if f not in iface_fields]
|
||||
# Narrowing compares raw SQL text, so only fields no rule rewrites may be used,
|
||||
# and only when key tuples line up 1:1 with ``match_keys`` (no mapped/unmapped marker).
|
||||
invariant = [] if pipeline.ignores_ports_by_policy() else [
|
||||
f for f in match_keys if f not in pipeline._transformed # noqa: SLF001
|
||||
]
|
||||
positions = [match_keys.index(f) for f in invariant]
|
||||
coarse = sorted({tuple(k[i] for i in positions) for k in keys})
|
||||
coarse = sorted({tuple(k[i] for i in positions) for k in keys}) if invariant else [()]
|
||||
out: list[dict[str, Any]] = []
|
||||
complete: set[tuple[str, ...]] = set()
|
||||
# Bound memory even when a candidate identifies a very large duplicate group.
|
||||
|
|
@ -3471,7 +3800,7 @@ def _load_search_key_groups(
|
|||
for start in range(0, len(coarse), 100):
|
||||
group = coarse[start:start + 100]
|
||||
group_set = set(group)
|
||||
target = {k for k in keys if tuple(k[i] for i in positions) in group_set}
|
||||
target = {k for k in keys if tuple(k[i] for i in positions) in group_set} if invariant else set(keys)
|
||||
q = db.query(model).filter(model.batch_id == batch_id)
|
||||
if metric_id == "lldp_neighbor":
|
||||
q = q.order_by(model.id.asc())
|
||||
|
|
@ -3497,13 +3826,10 @@ def _load_search_key_groups(
|
|||
)
|
||||
row = {**data, "_netx": {"row_id": r.id}}
|
||||
db.expunge(r)
|
||||
if row_filters and not all(row_matches_filter(row, f) for f in row_filters):
|
||||
norm = pipeline.normalize(row, side)
|
||||
if not pipeline.accepts(row, norm, side):
|
||||
continue
|
||||
normalized = apply_iface_normalize_rows(
|
||||
[row], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
mapped = apply_port_map(normalized, iface_fields=iface_fields, port_map=port_map)
|
||||
if row_key(mapped, match_keys) in target:
|
||||
if pipeline.key_of(norm, side, match_keys)[1] in target:
|
||||
out.append(row)
|
||||
return out, complete
|
||||
|
||||
|
|
@ -3566,13 +3892,15 @@ def _live_search_sheet_diffs(
|
|||
ignore_ports = sheet.get("ignore_port_changes")
|
||||
if ignore_ports is not None:
|
||||
ignore_ports = bool(ignore_ports)
|
||||
pmap, norm_rules = _run_transform_config(db, run)
|
||||
pipeline = _run_pipeline(db, run, sheet)
|
||||
if pipeline.legacy and not pipeline.row_filters and row_filters:
|
||||
pipeline.row_filters = [dict(f) for f in row_filters if isinstance(f, dict)]
|
||||
|
||||
before_rows, trunc_b = _load_metric_rows_for_search(
|
||||
db,
|
||||
batch_id=str(run.before_batch_id or ""),
|
||||
metric_id=mid_src,
|
||||
row_filters=row_filters,
|
||||
row_filters=pipeline.pushdown_filters("before"),
|
||||
key_fields=key_fields,
|
||||
kw=free_kw,
|
||||
field_q=merged_q,
|
||||
|
|
@ -3581,49 +3909,46 @@ def _live_search_sheet_diffs(
|
|||
db,
|
||||
batch_id=str(run.after_batch_id or ""),
|
||||
metric_id=mid_src,
|
||||
row_filters=row_filters,
|
||||
row_filters=pipeline.pushdown_filters("after"),
|
||||
key_fields=key_fields,
|
||||
kw=free_kw,
|
||||
field_q=merged_q,
|
||||
)
|
||||
match_keys = list((sheet.get("summary") or {}).get("match_key_fields") or key_fields)
|
||||
if ignore_ports is True and not pmap:
|
||||
if ignore_ports is True and not pipeline.port_map:
|
||||
match_keys = [f for f in key_fields if f not in iface_fields] or key_fields
|
||||
keys: set[tuple[str, ...]] = set()
|
||||
for rows, mapping in ((before_rows, pmap), (after_rows, {})):
|
||||
for row in apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules):
|
||||
keys.add(row_key(apply_port_map(row, iface_fields=iface_fields, port_map=mapping), match_keys))
|
||||
for rows, side in ((before_rows, "before"), (after_rows, "after")):
|
||||
for raw, norm in pipeline.prepare(rows, side):
|
||||
keys.add(pipeline.key_of(norm, side, match_keys)[1])
|
||||
before_rows, complete_b = _load_search_key_groups(
|
||||
db, batch_id=str(run.before_batch_id or ""), metric_id=mid_src, keys=keys,
|
||||
match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters,
|
||||
norm_rules=norm_rules, port_map=pmap,
|
||||
match_keys=match_keys, pipeline=pipeline, side="before",
|
||||
)
|
||||
after_rows, complete_a = _load_search_key_groups(
|
||||
db, batch_id=str(run.after_batch_id or ""), metric_id=mid_src, keys=keys,
|
||||
match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters,
|
||||
norm_rules=norm_rules, port_map={},
|
||||
match_keys=match_keys, pipeline=pipeline, side="after",
|
||||
)
|
||||
complete = complete_b & complete_a
|
||||
def keep_complete(rows: list[dict[str, Any]], mapping: dict[str, str]) -> list[dict[str, Any]]:
|
||||
normalized = apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules)
|
||||
return [orig for orig, norm in zip(rows, normalized) if row_key(
|
||||
apply_port_map(norm, iface_fields=iface_fields, port_map=mapping), match_keys,
|
||||
) in complete]
|
||||
before_rows = keep_complete(before_rows, pmap)
|
||||
after_rows = keep_complete(after_rows, {})
|
||||
|
||||
def keep_complete(rows: list[dict[str, Any]], side: str) -> list[dict[str, Any]]:
|
||||
return [raw for raw, norm in pipeline.prepare(rows, side)
|
||||
if pipeline.key_of(norm, side, match_keys)[1] in complete]
|
||||
|
||||
before_rows = keep_complete(before_rows, "before")
|
||||
after_rows = keep_complete(after_rows, "after")
|
||||
result = compare_rows(
|
||||
before_rows=before_rows,
|
||||
after_rows=after_rows,
|
||||
key_fields=key_fields,
|
||||
iface_fields=iface_fields,
|
||||
compare_fields=compare_fields,
|
||||
port_map=pmap,
|
||||
field_rules=field_rules,
|
||||
iface_normalize_rules=norm_rules,
|
||||
ignore_port_changes=(match_keys != key_fields) if not pmap else False,
|
||||
include_unchanged=True,
|
||||
unchanged_limit=None,
|
||||
compact_unchanged=False,
|
||||
pipeline=pipeline,
|
||||
match_keys_override=match_keys,
|
||||
)
|
||||
kind_n = (kind or "diff").strip().lower()
|
||||
filtered = [
|
||||
|
|
@ -3636,7 +3961,7 @@ def _live_search_sheet_diffs(
|
|||
filtered = filtered[:_LIVE_SEARCH_RESULT_CAP]
|
||||
total = len(filtered)
|
||||
start = (page - 1) * page_size
|
||||
page_items = filtered[start : start + page_size]
|
||||
page_items = _attach_raw_key_before(filtered[start : start + page_size], key_fields)
|
||||
return {
|
||||
"total": total,
|
||||
"page": page,
|
||||
|
|
@ -3844,7 +4169,7 @@ def export_run_zip(db: Session, run_id: str) -> bytes:
|
|||
csv_file.write(b"\xef\xbb\xbf")
|
||||
for line in _sheet_csv_lines(sheet, _iter_sheet_diffs(db, run_id, sid)):
|
||||
csv_file.write((line + "\n").encode("utf-8"))
|
||||
sum_lines = ["metric_id,mode,before,after,added,removed,changed,unchanged,diff_count,pass_rate"]
|
||||
sum_lines = ["metric_id,mode,before,after,added,removed,changed,unchanged,diff_count,pass_rate,data_state"]
|
||||
for card in list(s.get("sheet_cards") or []):
|
||||
sum_lines.append(
|
||||
",".join(
|
||||
|
|
@ -3860,6 +4185,7 @@ def export_run_zip(db: Session, run_id: str) -> bytes:
|
|||
card.get("unchanged"),
|
||||
card.get("diff_count"),
|
||||
card.get("pass_rate"),
|
||||
card.get("data_state") or "ok",
|
||||
)
|
||||
)
|
||||
)
|
||||
|
|
@ -3927,7 +4253,7 @@ def try_auto_compare_for_task(db: Session, task_id: str, batch_id: str) -> int:
|
|||
if not j.before_batch_id:
|
||||
continue
|
||||
try:
|
||||
run_compare(db, j.id, force_after_batch_id=batch_id)
|
||||
run_compare(db, j.id, force_after_batch_id=batch_id, purpose="auto")
|
||||
n += 1
|
||||
except HTTPException as exc:
|
||||
if int(getattr(exc, "status_code", 0) or 0) == 409:
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ from uuid import uuid4
|
|||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from .compare_engine import key_tuple
|
||||
from .compare_rules import effective_compare_fields, field_rule_map, NUMBER_PATTERN, scalar_text
|
||||
|
||||
_log = logging.getLogger("netx.biz_state.compare_sql")
|
||||
|
|
@ -154,16 +155,23 @@ def sql_compare_skip_reason(
|
|||
*,
|
||||
port_map: Mapping[str, str] | None = None,
|
||||
iface_normalize_rules: Sequence[Mapping[str, str]] | None = None,
|
||||
pipeline: Any = None,
|
||||
) -> str:
|
||||
"""Empty string when SQL is allowed; otherwise a short reason for progress/logs.
|
||||
|
||||
Simple ``row_filters`` (eq/in/contains/…) used for BGP sheet splits are fine —
|
||||
they do **not** force Python by themselves.
|
||||
|
||||
A port map only blocks SQL when the sheet actually reads an interface field
|
||||
in its key or compare columns; a BGP route table with no interface fields is
|
||||
unaffected by it. ``pipeline`` (v2 identity rules) adds its own blockers.
|
||||
"""
|
||||
if not _dialect_is_postgres(db):
|
||||
return "not_postgres"
|
||||
if port_map:
|
||||
return "port_map"
|
||||
if pipeline is not None:
|
||||
port_map = pipeline.port_map
|
||||
if iface_normalize_rules is None and (pipeline.before.has_iface_rules or pipeline.after.has_iface_rules):
|
||||
iface_normalize_rules = [{"from": "*", "to": "*"}]
|
||||
key_fields = [str(k).strip() for k in (sheet.get("key_fields") or []) if str(k).strip()]
|
||||
if not key_fields:
|
||||
return "no_key_fields"
|
||||
|
|
@ -171,12 +179,24 @@ def sql_compare_skip_reason(
|
|||
return "unsafe_key_field"
|
||||
iface_fields = [str(f).strip() for f in (sheet.get("iface_fields") or []) if str(f).strip()]
|
||||
iface_set = set(iface_fields)
|
||||
if port_map and iface_set:
|
||||
early_cmp = effective_compare_fields(
|
||||
list(sheet.get("compare_fields") or []), list(sheet.get("field_rules") or []),
|
||||
)
|
||||
if iface_set & (set(key_fields) | {str(f).strip() for f in early_cmp}):
|
||||
return "port_map"
|
||||
ignore_ports = sheet.get("ignore_port_changes")
|
||||
if ignore_ports is True:
|
||||
return "ignore_port_changes"
|
||||
if ignore_ports is None and iface_set and any(k in iface_set for k in key_fields):
|
||||
return "auto_ignore_ports"
|
||||
field_rules = list(sheet.get("field_rules") or [])
|
||||
if any(
|
||||
isinstance(r, dict) and str(r.get("compare") or "").strip().lower() in
|
||||
("not_worse", "numeric_min", "numeric_max")
|
||||
for r in field_rules
|
||||
):
|
||||
return "directional_compare"
|
||||
if not _field_rules_sql_compatible(field_rules):
|
||||
return "field_rules"
|
||||
compare_fields = effective_compare_fields(
|
||||
|
|
@ -186,8 +206,14 @@ def sql_compare_skip_reason(
|
|||
if any(not _FIELD_RE.match(str(f).strip()) for f in compare_fields if str(f).strip()):
|
||||
return "unsafe_compare_field"
|
||||
used = set(key_fields) | {str(f).strip() for f in compare_fields if str(f).strip()}
|
||||
if port_map and (used & iface_set):
|
||||
return "port_map"
|
||||
if iface_normalize_rules and (used & iface_set):
|
||||
return "iface_normalize"
|
||||
if pipeline is not None:
|
||||
blocked = pipeline.sql_blockers(used)
|
||||
if blocked:
|
||||
return blocked
|
||||
if not _filters_sql_compatible(list(sheet.get("row_filters") or [])):
|
||||
return "row_filters"
|
||||
if not str(sheet.get("metric_id") or "").strip():
|
||||
|
|
@ -371,7 +397,7 @@ def _changed_predicate(
|
|||
|
||||
def _key_obj_from_data(data: dict[str, Any] | None, key_fields: list[str]) -> dict[str, Any]:
|
||||
row = data if isinstance(data, dict) else {}
|
||||
return {f: row.get(f, "") for f in key_fields}
|
||||
return dict(zip(key_fields, key_tuple(row, key_fields)))
|
||||
|
||||
|
||||
def _changes_from_rows(
|
||||
|
|
@ -399,6 +425,22 @@ def _changes_from_rows(
|
|||
return out
|
||||
|
||||
|
||||
def dup_pairing_order_sql(compare_fields: Sequence[str], rules: Mapping[str, Mapping[str, Any]]) -> str:
|
||||
"""``ORDER BY`` prefix that pairs same-key rows by value (mirrors ``Pipeline.pair_sort_key``).
|
||||
|
||||
Byte-order collation matches Python's code-point string order, so both
|
||||
engines choose the same pairs. Returns '' (or a trailing-comma list).
|
||||
"""
|
||||
parts = []
|
||||
for f in compare_fields:
|
||||
name = str(f).strip()
|
||||
if not name:
|
||||
continue
|
||||
mode = str((rules.get(name) or {}).get("normalize") or "strip").strip().lower() or "strip"
|
||||
parts.append(f'({_norm_expr("data_json", name, mode)}) COLLATE "C" ASC')
|
||||
return ", ".join(parts) + ", " if parts else ""
|
||||
|
||||
|
||||
def run_sql_sheet_compare(
|
||||
db: Session,
|
||||
*,
|
||||
|
|
@ -407,6 +449,7 @@ def run_sql_sheet_compare(
|
|||
after_batch_id: str,
|
||||
store_unchanged: str = "auto",
|
||||
on_progress: Callable[..., None] | None = None,
|
||||
duplicate_pairing: str = "ordered",
|
||||
) -> dict[str, Any]:
|
||||
"""Compare one sheet via PostgreSQL TEMP tables + FULL OUTER JOIN.
|
||||
|
||||
|
|
@ -443,6 +486,7 @@ def run_sql_sheet_compare(
|
|||
|
||||
filter_sql, filter_params = compile_row_filters_sql(row_filters)
|
||||
rk = _rk_sql(key_fields)
|
||||
dup_order = dup_pairing_order_sql(compare_fields, rules) if duplicate_pairing == "sorted" else ""
|
||||
tag = uuid4().hex[:8]
|
||||
tb = f"_netx_cmp_b_{tag}"
|
||||
ta = f"_netx_cmp_a_{tag}"
|
||||
|
|
@ -500,7 +544,7 @@ def run_sql_sheet_compare(
|
|||
data_json AS data,
|
||||
row_number() OVER (
|
||||
PARTITION BY ({rk})
|
||||
ORDER BY seq ASC, id ASC
|
||||
ORDER BY {dup_order}seq ASC, id ASC
|
||||
) AS dup_rn
|
||||
FROM biz_state_metric_row
|
||||
WHERE batch_id = :bid
|
||||
|
|
|
|||
|
|
@ -8,7 +8,10 @@ from typing import Any
|
|||
from fastapi import HTTPException
|
||||
|
||||
FILTER_OPS = {"eq", "==", "ci_eq", "ne", "!=", "in", "not_in", "nin", "contains", "empty", "not_empty", "nonempty", "regex", "age_timer"}
|
||||
COMPARE_MODES = {"", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel"}
|
||||
COMPARE_MODES = {"", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel",
|
||||
"not_worse", "numeric_min", "numeric_max"}
|
||||
SIDES = {"", "both", "before", "after"}
|
||||
MAX_VALUE_MAP = 500
|
||||
NORMALIZE_MODES = {"", "none", "strip", "lower", "upper", "mac", "empty_as_blank"}
|
||||
|
||||
|
||||
|
|
@ -34,6 +37,8 @@ def validate_filters(value: Any, path: str, depth: int = 0) -> None:
|
|||
here = f"{path}[{i}]"
|
||||
if not isinstance(node, dict) or not node:
|
||||
invalid(here, "filter_required")
|
||||
if depth == 0 and str(node.get("side") or "").strip().lower() not in SIDES:
|
||||
invalid(here, "unknown_filter_side")
|
||||
groups = [key for key in ("any", "all") if key in node]
|
||||
if groups:
|
||||
if len(groups) != 1 or any(node.get(key) is not None for key in ("field", "op", "value")):
|
||||
|
|
@ -73,6 +78,12 @@ def validate_sheet(sheet: Any, path: str, *, partial: bool = False) -> None:
|
|||
for key in ("key_fields", "iface_fields", "compare_fields", "display_fields", "ignore_fields"):
|
||||
if key in sheet or (key == "key_fields" and not partial):
|
||||
_strings(sheet.get(key), f"{path}.{key}", required=key == "key_fields")
|
||||
for opt, allowed in (("unmapped_port_policy", {"by_name", "ignore_port"}),
|
||||
("duplicate_pairing", {"sorted", "ordered"})):
|
||||
if str(sheet.get(opt) or "").strip() and str(sheet[opt]).strip().lower() not in allowed:
|
||||
invalid(path, f"unknown_{opt}")
|
||||
if sheet.get("builtin_vendor_canonical") is not None and not isinstance(sheet["builtin_vendor_canonical"], bool):
|
||||
invalid(path, "boolean_builtin_vendor_canonical_required")
|
||||
if "ignore_port_changes" in sheet and sheet["ignore_port_changes"] is not None and not isinstance(sheet["ignore_port_changes"], bool):
|
||||
invalid(path, "boolean_port_policy_required")
|
||||
if "row_filters" in sheet:
|
||||
|
|
@ -87,9 +98,30 @@ def validate_sheet(sheet: Any, path: str, *, partial: bool = False) -> None:
|
|||
if not isinstance(rule, dict) or not isinstance(rule.get("field"), str) or not rule["field"].strip():
|
||||
invalid(here, "rule_field_required")
|
||||
field = rule["field"].strip()
|
||||
if field in seen:
|
||||
rule_side = str(rule.get("side") or "").strip().lower()
|
||||
if rule_side not in SIDES:
|
||||
invalid(here, "unknown_rule_side")
|
||||
marker = (field, "" if rule_side == "both" else rule_side)
|
||||
if marker in seen:
|
||||
invalid(here, "duplicate_field_rule")
|
||||
seen.add(field)
|
||||
seen.add(marker)
|
||||
vmap = rule.get("value_map")
|
||||
if vmap is not None:
|
||||
if not isinstance(vmap, dict) or any(
|
||||
not isinstance(k, str) or not isinstance(v, str) for k, v in vmap.items()
|
||||
):
|
||||
invalid(here, "value_map_string_pairs_required")
|
||||
if len(vmap) > MAX_VALUE_MAP:
|
||||
invalid(here, "value_map_too_large")
|
||||
if rule.get("apply_to_key") is not None and not isinstance(rule["apply_to_key"], bool):
|
||||
invalid(here, "boolean_apply_to_key_required")
|
||||
mode = str(rule.get("compare") or "").strip().lower()
|
||||
if mode == "not_worse":
|
||||
rank = rule.get("rank")
|
||||
if not isinstance(rank, list) or len(rank) < 2 or any(not isinstance(x, str) or not x.strip() for x in rank):
|
||||
invalid(here, "rank_list_required")
|
||||
elif rule.get("rank") is not None:
|
||||
invalid(here, "rank_only_for_not_worse")
|
||||
if str(rule.get("compare") or "").strip().lower() not in COMPARE_MODES:
|
||||
invalid(here, "unknown_compare_mode")
|
||||
if str(rule.get("normalize") or "").strip().lower() not in NORMALIZE_MODES:
|
||||
|
|
@ -124,7 +156,13 @@ def validate_template_body(body: dict[str, Any], *, partial: bool = False) -> No
|
|||
path = f"iface_normalize_rules[{i}]"
|
||||
if not isinstance(rule, dict) or any(not isinstance(rule.get(k), str) or not rule[k].strip() for k in ("from", "to")):
|
||||
invalid(path, "alias_pair_required")
|
||||
key = rule["from"].strip().lower()
|
||||
rside = str(rule.get("side") or "").strip().lower()
|
||||
if rside not in SIDES:
|
||||
invalid(path, "unknown_alias_side")
|
||||
if rule.get("vendor") is not None and not isinstance(rule["vendor"], str):
|
||||
invalid(path, "alias_vendor_string_required")
|
||||
key = (rule["from"].strip().lower(), "" if rside == "both" else rside,
|
||||
str(rule.get("vendor") or "").strip().lower())
|
||||
if key in seen:
|
||||
invalid(path, "duplicate_alias")
|
||||
seen.add(key)
|
||||
|
|
|
|||
|
|
@ -131,9 +131,25 @@
|
|||
],
|
||||
"row_filters": [
|
||||
{
|
||||
"op": "age_timer",
|
||||
"field": "age",
|
||||
"value": ""
|
||||
"any": [
|
||||
{
|
||||
"field": "entry_type",
|
||||
"op": "eq",
|
||||
"value": "dynamic"
|
||||
},
|
||||
{
|
||||
"all": [
|
||||
{
|
||||
"field": "entry_type",
|
||||
"op": "empty"
|
||||
},
|
||||
{
|
||||
"field": "age",
|
||||
"op": "age_timer"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"field_rules": [
|
||||
|
|
@ -415,11 +431,7 @@
|
|||
],
|
||||
"compare_fields": [
|
||||
"admin",
|
||||
"input_bps",
|
||||
"output_bps",
|
||||
"port_status",
|
||||
"in_util",
|
||||
"out_util",
|
||||
"ip_mtu",
|
||||
"mtu",
|
||||
"mpls_mtu",
|
||||
|
|
@ -454,7 +466,7 @@
|
|||
},
|
||||
{
|
||||
"field": "output_bps",
|
||||
"compare": "numeric",
|
||||
"compare": "percent",
|
||||
"tolerance": 5
|
||||
},
|
||||
{
|
||||
|
|
@ -593,8 +605,8 @@
|
|||
"field_rules": [
|
||||
{
|
||||
"field": "rx_power",
|
||||
"compare": "percent",
|
||||
"tolerance": 5
|
||||
"compare": "numeric",
|
||||
"tolerance": 2
|
||||
}
|
||||
]
|
||||
},
|
||||
|
|
@ -668,7 +680,9 @@
|
|||
"vpn_sid",
|
||||
"neighbor_sid"
|
||||
],
|
||||
"iface_fields": [],
|
||||
"iface_fields": [
|
||||
"ac_port"
|
||||
],
|
||||
"compare_fields": [],
|
||||
"display_fields": [
|
||||
"mac",
|
||||
|
|
|
|||
454
netx_api/biz_state/identity_pipeline.py
Normal file
454
netx_api/biz_state/identity_pipeline.py
Normal file
|
|
@ -0,0 +1,454 @@
|
|||
"""One identity pipeline for every compare path.
|
||||
|
||||
A row's *match key* must come out identical whether it is computed while a run
|
||||
executes, while its results are searched / hydrated / exported, while a mapping
|
||||
is validated, or while a cutover monitor evaluates the same sheet. All of those
|
||||
paths therefore build a :class:`Pipeline` and call the same methods instead of
|
||||
re-implementing "normalize → map → key".
|
||||
|
||||
Stages for one row on one side (``before`` or ``after``)::
|
||||
|
||||
raw row ──normalize──▶ iface aliases (+vendor canonical) ▶ key normalize ▶ value_map
|
||||
──filter────▶ ``row_filters`` (v2: on the normalized row; legacy: on the raw row)
|
||||
──map───────▶ port map (before side only)
|
||||
──key───────▶ ``key_tuple`` over the match key fields
|
||||
|
||||
``legacy=True`` reproduces the pre-pipeline behaviour (snapshot v1 runs): the
|
||||
template alias rules are the only normalization, keys are never normalized,
|
||||
there is no value map and the filters see the raw row.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
from .compare_rules import (
|
||||
normalize_value,
|
||||
row_matches_filter,
|
||||
scalar_text,
|
||||
)
|
||||
from .iface_normalize import (
|
||||
normalize_iface_name,
|
||||
normalize_iface_rules,
|
||||
normalize_port_map,
|
||||
resolve_mapped_iface,
|
||||
rules_for_side,
|
||||
vendor_canonical_rules,
|
||||
)
|
||||
|
||||
SIDES = ("before", "after")
|
||||
DIRECTIONAL_MODES = {"not_worse", "numeric_min", "numeric_max"}
|
||||
UNMAPPED_POLICIES = ("by_name", "ignore_port")
|
||||
DUPLICATE_PAIRING = ("sorted", "ordered")
|
||||
|
||||
|
||||
def filter_fields(filt: Mapping[str, Any] | None) -> set[str]:
|
||||
"""Every field name a (possibly nested) row filter reads."""
|
||||
out: set[str] = set()
|
||||
if not isinstance(filt, Mapping):
|
||||
return out
|
||||
for group in ("any", "all"):
|
||||
if group in filt:
|
||||
for kid in filt.get(group) or []:
|
||||
out |= filter_fields(kid)
|
||||
return out
|
||||
name = str(filt.get("field") or "").strip()
|
||||
if name:
|
||||
out.add(name)
|
||||
return out
|
||||
|
||||
|
||||
def _vendor_key(vendor: str = "", device_type: str = "") -> str:
|
||||
from ..topology_lldp import resolve_vendor_key
|
||||
|
||||
key = resolve_vendor_key(vendor, device_type)
|
||||
return "" if key == "generic" else key
|
||||
|
||||
|
||||
@dataclass(eq=False)
|
||||
class SideSpec:
|
||||
side: str
|
||||
vendor: str = ""
|
||||
canonical: list[dict[str, str]] = field(default_factory=list)
|
||||
alias_rules: list[dict[str, str]] = field(default_factory=list)
|
||||
# field -> {lowercased source value: canonical value}
|
||||
value_maps: dict[str, dict[str, str]] = field(default_factory=dict)
|
||||
# key field -> normalize mode (only fields whose rule asks for one)
|
||||
key_norms: dict[str, str] = field(default_factory=dict)
|
||||
strip_inner_space: bool = True
|
||||
|
||||
def iface(self, name: Any) -> str:
|
||||
text = str(name or "").strip()
|
||||
if not text:
|
||||
return ""
|
||||
if self.strip_inner_space:
|
||||
text = "".join(text.split())
|
||||
if self.canonical:
|
||||
text = normalize_iface_name(text, self.canonical)
|
||||
if self.alias_rules:
|
||||
text = normalize_iface_name(text, self.alias_rules)
|
||||
return text
|
||||
|
||||
@property
|
||||
def has_iface_rules(self) -> bool:
|
||||
return bool(self.canonical or self.alias_rules)
|
||||
|
||||
def describe(self) -> dict[str, Any]:
|
||||
"""JSON-able record of the rules in force (stored in the run snapshot)."""
|
||||
return {
|
||||
"vendor": self.vendor,
|
||||
"canonical": list(self.canonical),
|
||||
"alias_rules": list(self.alias_rules),
|
||||
"value_maps": {k: dict(v) for k, v in self.value_maps.items()},
|
||||
"key_norms": dict(self.key_norms),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(eq=False)
|
||||
class Pipeline:
|
||||
key_fields: list[str]
|
||||
iface_fields: list[str]
|
||||
compare_fields: list[str]
|
||||
field_rules: list[dict[str, Any]]
|
||||
row_filters: list[dict[str, Any]]
|
||||
port_map: dict[str, str]
|
||||
before: SideSpec
|
||||
after: SideSpec
|
||||
legacy: bool = False
|
||||
ignore_port_changes: bool | None = None
|
||||
unmapped_port_policy: str = "by_name"
|
||||
duplicate_pairing: str = "ordered"
|
||||
_iface_set: set[str] = field(default_factory=set, repr=False)
|
||||
_transformed: set[str] = field(default_factory=set, repr=False)
|
||||
_targets: set[str] = field(default_factory=set, repr=False)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self._iface_set = set(self.iface_fields)
|
||||
self._transformed = set(self._iface_set)
|
||||
for spec in (self.before, self.after):
|
||||
self._transformed |= set(spec.value_maps) | set(spec.key_norms)
|
||||
self._targets = set(self.port_map.values())
|
||||
|
||||
# -- side helpers ----------------------------------------------------
|
||||
def spec(self, side: str) -> SideSpec:
|
||||
return self.before if side == "before" else self.after
|
||||
|
||||
# -- stage 1: normalize ----------------------------------------------
|
||||
def normalize(self, row: Mapping[str, Any], side: str) -> dict[str, Any]:
|
||||
out = dict(row)
|
||||
spec = self.spec(side)
|
||||
for f in self.iface_fields:
|
||||
val = scalar_text(out.get(f)).strip()
|
||||
if val:
|
||||
out[f] = spec.iface(val)
|
||||
if self.legacy:
|
||||
return out
|
||||
for f, mode in spec.key_norms.items():
|
||||
if f in out:
|
||||
out[f] = normalize_value(out[f], mode)
|
||||
for f, vmap in spec.value_maps.items():
|
||||
if f not in out:
|
||||
continue
|
||||
raw = scalar_text(out[f]).strip()
|
||||
hit = vmap.get(raw.lower())
|
||||
if hit is not None:
|
||||
out[f] = hit
|
||||
return out
|
||||
|
||||
def normalize_name(self, name: Any, side: str) -> str:
|
||||
"""Interface-name normalization only (port-map keys / values, selectors)."""
|
||||
return self.spec(side).iface(name)
|
||||
|
||||
# -- stage 2: filter -------------------------------------------------
|
||||
def filters_for(self, side: str) -> list[dict[str, Any]]:
|
||||
out = []
|
||||
for f in self.row_filters:
|
||||
fs = str(f.get("side") or "").strip().lower()
|
||||
if not fs or fs == "both" or fs == side:
|
||||
out.append(f)
|
||||
return out
|
||||
|
||||
def keep(self, row: Mapping[str, Any], side: str) -> bool:
|
||||
"""Evaluate row filters. v2 expects the *normalized* row, legacy the raw one."""
|
||||
return all(row_matches_filter(row, f) for f in self.filters_for(side))
|
||||
|
||||
def accepts(self, raw: Mapping[str, Any], norm: Mapping[str, Any], side: str) -> bool:
|
||||
"""Row-filter verdict: legacy reads the raw row, v2 the normalized one."""
|
||||
return self.keep(raw if self.legacy else norm, side)
|
||||
|
||||
def pushdown_filters(self, side: str) -> list[dict[str, Any]]:
|
||||
"""Filters that read only untransformed fields, so they may run in SQL on raw rows."""
|
||||
if self.legacy:
|
||||
return list(self.row_filters)
|
||||
return [
|
||||
f for f in self.row_filters
|
||||
if not str(f.get("side") or "").strip()
|
||||
and not (filter_fields(f) & self._transformed)
|
||||
]
|
||||
|
||||
def prepare(self, rows: Sequence[Mapping[str, Any]], side: str) -> list[tuple[dict[str, Any], dict[str, Any]]]:
|
||||
"""``[(original row, normalized row)]`` after filtering, in input order."""
|
||||
out: list[tuple[dict[str, Any], dict[str, Any]]] = []
|
||||
has_filters = bool(self.filters_for(side))
|
||||
for raw in rows:
|
||||
if not isinstance(raw, dict):
|
||||
continue
|
||||
norm = self.normalize(raw, side)
|
||||
if has_filters and not self.accepts(raw, norm, side):
|
||||
continue
|
||||
out.append((raw, norm))
|
||||
return out
|
||||
|
||||
# -- stage 3: port map -----------------------------------------------
|
||||
def apply_port_map(self, norm_row: Mapping[str, Any]) -> dict[str, Any]:
|
||||
out = dict(norm_row)
|
||||
if not self.port_map:
|
||||
return out
|
||||
for f in self.iface_fields:
|
||||
val = scalar_text(out.get(f)).strip()
|
||||
if val:
|
||||
out[f] = resolve_mapped_iface(val, self.port_map)
|
||||
return out
|
||||
|
||||
def is_mapped(self, norm_row: Mapping[str, Any], side: str) -> bool:
|
||||
"""True when every interface value of the row is covered by the port map."""
|
||||
if not self.iface_fields:
|
||||
return True
|
||||
values = [scalar_text(norm_row.get(f)).strip() for f in self.iface_fields]
|
||||
values = [v for v in values if v]
|
||||
if not values:
|
||||
return False
|
||||
for v in values:
|
||||
if side == "before":
|
||||
if resolve_mapped_iface(v, self.port_map) == v and v not in self.port_map:
|
||||
return False
|
||||
else:
|
||||
if not self._after_value_is_target(v):
|
||||
return False
|
||||
return True
|
||||
|
||||
def _after_value_is_target(self, value: str) -> bool:
|
||||
if value in self._targets:
|
||||
return True
|
||||
parts = value.split(".")
|
||||
return any(".".join(parts[:i]) in self._targets for i in range(len(parts) - 1, 0, -1))
|
||||
|
||||
def transform(self, row: Mapping[str, Any], side: str) -> dict[str, Any]:
|
||||
norm = self.normalize(row, side)
|
||||
return self.apply_port_map(norm) if side == "before" else norm
|
||||
|
||||
# -- stage 4: key ----------------------------------------------------
|
||||
def effective_match_keys(self, *, unique_without_iface: bool | None = None) -> list[str]:
|
||||
"""Match-key fields once the port policy is resolved.
|
||||
|
||||
With a port map the full key is used (the map aligns the names). Without
|
||||
one, ``ignore_port_changes`` may drop the interface fields; ``None`` means
|
||||
"auto", resolved by the caller via ``unique_without_iface``.
|
||||
"""
|
||||
keys = list(self.key_fields)
|
||||
if self.port_map or not self._iface_set:
|
||||
return keys
|
||||
candidate = [k for k in keys if k not in self._iface_set]
|
||||
if not candidate:
|
||||
return keys
|
||||
policy = self.ignore_port_changes
|
||||
if policy is True:
|
||||
return candidate
|
||||
if policy is False:
|
||||
return keys
|
||||
return candidate if unique_without_iface else keys
|
||||
|
||||
def ignores_ports_by_policy(self) -> bool:
|
||||
return bool(self.port_map) and self.unmapped_port_policy == "ignore_port" and bool(self._iface_set)
|
||||
|
||||
def key_of(
|
||||
self, norm_row: Mapping[str, Any], side: str, match_keys: Sequence[str],
|
||||
) -> tuple[dict[str, Any], tuple[str, ...]]:
|
||||
"""``(row after port map, match key)`` for a *normalized* row."""
|
||||
row = self.apply_port_map(norm_row) if side == "before" else dict(norm_row)
|
||||
if self.ignores_ports_by_policy():
|
||||
if not self.is_mapped(norm_row, side):
|
||||
reduced = [k for k in match_keys if k not in self._iface_set] or list(match_keys)
|
||||
return row, ("U",) + tuple(scalar_text(row.get(k)).strip() for k in reduced)
|
||||
return row, ("M",) + tuple(scalar_text(row.get(k)).strip() for k in match_keys)
|
||||
return row, tuple(scalar_text(row.get(k)).strip() for k in match_keys)
|
||||
|
||||
def match_key(
|
||||
self, row: Mapping[str, Any], side: str, match_keys: Sequence[str] | None = None,
|
||||
) -> tuple[str, ...]:
|
||||
"""Match key of a *raw* row (normalize → port map → key)."""
|
||||
keys = list(match_keys) if match_keys is not None else self.effective_match_keys()
|
||||
return self.key_of(self.normalize(row, side), side, keys)[1]
|
||||
|
||||
# -- duplicate pairing -----------------------------------------------
|
||||
def pair_sort_key(self, norm_row: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
rules = {str(r.get("field")): r for r in self.field_rules if isinstance(r, dict)}
|
||||
out = []
|
||||
for f in self.compare_fields:
|
||||
mode = str((rules.get(f) or {}).get("normalize") or "strip")
|
||||
out.append(normalize_value(norm_row.get(f), mode))
|
||||
return tuple(out)
|
||||
|
||||
# -- SQL eligibility -------------------------------------------------
|
||||
def sql_blockers(self, used_fields: set[str]) -> str:
|
||||
"""Reason the SQL engine must not run this sheet ('' when it may)."""
|
||||
if self.legacy:
|
||||
return ""
|
||||
for spec in (self.before, self.after):
|
||||
if set(spec.value_maps) & used_fields:
|
||||
return "value_map"
|
||||
if set(spec.key_norms) & set(self.key_fields):
|
||||
return "key_normalize"
|
||||
if used_fields & self._iface_set and (self.before.has_iface_rules or self.after.has_iface_rules):
|
||||
return "iface_normalize"
|
||||
if any(str(f.get("side") or "").strip() for f in self.row_filters):
|
||||
return "row_filters_side"
|
||||
if any(filter_fields(f) & self._transformed for f in self.row_filters):
|
||||
return "row_filters_post_normalize"
|
||||
if any(
|
||||
str(r.get("compare") or "").strip().lower() in DIRECTIONAL_MODES
|
||||
for r in self.field_rules if isinstance(r, dict)
|
||||
):
|
||||
return "directional_compare"
|
||||
return ""
|
||||
|
||||
def describe(self) -> dict[str, Any]:
|
||||
return {
|
||||
"legacy": self.legacy,
|
||||
"before": self.before.describe(),
|
||||
"after": self.after.describe(),
|
||||
"unmapped_port_policy": self.unmapped_port_policy,
|
||||
"duplicate_pairing": self.duplicate_pairing,
|
||||
}
|
||||
|
||||
|
||||
def _field_rule_specs(
|
||||
field_rules: Sequence[Mapping[str, Any]], key_fields: Sequence[str], side: str,
|
||||
) -> tuple[dict[str, dict[str, str]], dict[str, str]]:
|
||||
value_maps: dict[str, dict[str, str]] = {}
|
||||
key_norms: dict[str, str] = {}
|
||||
key_set = set(key_fields)
|
||||
for rule in field_rules or []:
|
||||
if not isinstance(rule, Mapping):
|
||||
continue
|
||||
name = str(rule.get("field") or "").strip()
|
||||
if not name:
|
||||
continue
|
||||
rs = str(rule.get("side") or "").strip().lower()
|
||||
if rs and rs != "both" and rs != side:
|
||||
continue
|
||||
apply_to_key = rule.get("apply_to_key")
|
||||
apply_to_key = (name in key_set) if apply_to_key is None else bool(apply_to_key)
|
||||
is_key = name in key_set
|
||||
if is_key and not apply_to_key:
|
||||
continue
|
||||
mode = str(rule.get("normalize") or "").strip().lower()
|
||||
vmap = rule.get("value_map")
|
||||
if is_key and mode and mode not in ("none", "strip"):
|
||||
key_norms[name] = mode
|
||||
if isinstance(vmap, Mapping) and vmap:
|
||||
pre = mode if (mode and mode not in ("none", "strip")) else "strip"
|
||||
built: dict[str, str] = {}
|
||||
for src, dst in vmap.items():
|
||||
built[normalize_value(src, pre).lower()] = scalar_text(dst).strip()
|
||||
value_maps[name] = built
|
||||
return value_maps, key_norms
|
||||
|
||||
|
||||
def build_pipeline(
|
||||
sheet: Mapping[str, Any],
|
||||
*,
|
||||
iface_rules: Sequence[Mapping[str, Any]] | None,
|
||||
port_map: Mapping[str, str] | None,
|
||||
before_vendor: str = "",
|
||||
after_vendor: str = "",
|
||||
legacy: bool = False,
|
||||
normalize_map: bool = True,
|
||||
) -> Pipeline:
|
||||
"""Build the pipeline for one sheet.
|
||||
|
||||
``port_map`` is the map as the user wrote it. Unless ``normalize_map`` is
|
||||
false (legacy runs, whose maps were written post-normalization) its keys are
|
||||
normalized with the *before* side rules and its values with the *after* side
|
||||
rules.
|
||||
"""
|
||||
key_fields = [str(k).strip() for k in (sheet.get("key_fields") or []) if str(k).strip()]
|
||||
iface_fields = [str(k).strip() for k in (sheet.get("iface_fields") or []) if str(k).strip()]
|
||||
field_rules = [dict(r) for r in (sheet.get("field_rules") or []) if isinstance(r, Mapping)]
|
||||
from .compare_rules import effective_compare_fields
|
||||
|
||||
compare_fields = effective_compare_fields(list(sheet.get("compare_fields") or []), field_rules)
|
||||
canonical_on = sheet.get("builtin_vendor_canonical", True) is not False and not legacy
|
||||
all_rules = normalize_iface_rules(list(iface_rules or []))
|
||||
|
||||
specs: dict[str, SideSpec] = {}
|
||||
for side, vendor_raw in (("before", before_vendor), ("after", after_vendor)):
|
||||
vendor = _vendor_key(vendor_raw) if vendor_raw else ""
|
||||
if legacy:
|
||||
alias = normalize_iface_rules([{"from": r["from"], "to": r["to"]} for r in all_rules])
|
||||
value_maps, key_norms = {}, {}
|
||||
else:
|
||||
alias = rules_for_side(all_rules, side, vendor)
|
||||
value_maps, key_norms = _field_rule_specs(field_rules, key_fields, side)
|
||||
specs[side] = SideSpec(
|
||||
side=side, vendor=vendor,
|
||||
canonical=vendor_canonical_rules(vendor) if canonical_on else [],
|
||||
alias_rules=alias, value_maps=value_maps, key_norms=key_norms,
|
||||
strip_inner_space=not legacy,
|
||||
)
|
||||
|
||||
pmap = dict(port_map or {})
|
||||
if pmap and normalize_map and not legacy:
|
||||
pmap, _ = normalize_port_map(
|
||||
pmap, None,
|
||||
key_normalizer=specs["before"].iface, value_normalizer=specs["after"].iface,
|
||||
)
|
||||
ignore_ports = sheet.get("ignore_port_changes")
|
||||
policy = str(sheet.get("unmapped_port_policy") or "by_name").strip().lower()
|
||||
pairing = str(sheet.get("duplicate_pairing") or ("ordered" if legacy else "sorted")).strip().lower()
|
||||
return Pipeline(
|
||||
key_fields=key_fields,
|
||||
iface_fields=iface_fields,
|
||||
compare_fields=compare_fields,
|
||||
field_rules=field_rules,
|
||||
row_filters=[dict(f) for f in (sheet.get("row_filters") or []) if isinstance(f, Mapping)],
|
||||
port_map=pmap,
|
||||
before=specs["before"],
|
||||
after=specs["after"],
|
||||
legacy=legacy,
|
||||
ignore_port_changes=None if ignore_ports is None else bool(ignore_ports),
|
||||
unmapped_port_policy=policy if policy in UNMAPPED_POLICIES else "by_name",
|
||||
duplicate_pairing="ordered" if legacy else (pairing if pairing in DUPLICATE_PAIRING else "sorted"),
|
||||
)
|
||||
|
||||
|
||||
def legacy_pipeline(
|
||||
*,
|
||||
key_fields: Sequence[str],
|
||||
iface_fields: Sequence[str],
|
||||
compare_fields: Sequence[str],
|
||||
field_rules: Sequence[Mapping[str, Any]] | None = None,
|
||||
port_map: Mapping[str, str] | None = None,
|
||||
iface_normalize_rules: Sequence[Mapping[str, Any]] | None = None,
|
||||
ignore_port_changes: bool | None = None,
|
||||
row_filters: Sequence[Mapping[str, Any]] | None = None,
|
||||
duplicate_pairing: str = "ordered",
|
||||
) -> Pipeline:
|
||||
"""Pipeline with the pre-v2 semantics, for snapshot-v1 runs and bare callers."""
|
||||
alias = normalize_iface_rules(
|
||||
[{"from": r.get("from"), "to": r.get("to")} for r in (iface_normalize_rules or [])]
|
||||
)
|
||||
return Pipeline(
|
||||
key_fields=[str(k) for k in key_fields],
|
||||
iface_fields=[str(f) for f in iface_fields if str(f).strip()],
|
||||
compare_fields=[str(f) for f in compare_fields],
|
||||
field_rules=[dict(r) for r in (field_rules or []) if isinstance(r, Mapping)],
|
||||
row_filters=[dict(f) for f in (row_filters or []) if isinstance(f, Mapping)],
|
||||
port_map=dict(port_map or {}),
|
||||
before=SideSpec(side="before", alias_rules=list(alias), strip_inner_space=False),
|
||||
after=SideSpec(side="after", alias_rules=list(alias), strip_inner_space=False),
|
||||
legacy=True,
|
||||
ignore_port_changes=ignore_port_changes,
|
||||
duplicate_pairing=duplicate_pairing if duplicate_pairing in DUPLICATE_PAIRING else "ordered",
|
||||
)
|
||||
|
|
@ -39,7 +39,14 @@ def normalize_iface_rules(raw: Any) -> list[dict[str, str]]:
|
|||
fr = str(item.get("from") or item.get("src") or "").strip()
|
||||
to = str(item.get("to") or item.get("dst") or "").strip()
|
||||
if fr and to:
|
||||
out.append({"from": fr, "to": to})
|
||||
rule = {"from": fr, "to": to}
|
||||
side = str(item.get("side") or "").strip().lower()
|
||||
if side in ("before", "after"):
|
||||
rule["side"] = side
|
||||
vendor = str(item.get("vendor") or "").strip().lower()
|
||||
if vendor:
|
||||
rule["vendor"] = vendor
|
||||
out.append(rule)
|
||||
# Longest prefix first so XXVGE wins over XGE / GE.
|
||||
out.sort(key=lambda r: len(r["from"]), reverse=True)
|
||||
return out
|
||||
|
|
@ -107,6 +114,47 @@ def apply_iface_normalize_rows(
|
|||
]
|
||||
|
||||
|
||||
def normalize_port_map(
|
||||
port_map: Mapping[str, str] | None,
|
||||
rules: Sequence[Mapping[str, str]] | None,
|
||||
value_rules: Sequence[Mapping[str, str]] | None = None,
|
||||
*,
|
||||
key_normalizer: Any = None,
|
||||
value_normalizer: Any = None,
|
||||
) -> tuple[dict[str, str], list[dict[str, str]]]:
|
||||
"""Normalize port-map keys and values with the same rules as iface fields.
|
||||
|
||||
``value_rules`` (default: ``rules``) normalizes the after-side values, for
|
||||
per-side rules. ``key_normalizer`` / ``value_normalizer`` are callables that
|
||||
replace the rule-based normalization (used by the identity pipeline).
|
||||
|
||||
Returns ``(normalized_map, conflicts)``. A conflict is two raw keys that
|
||||
normalize to the same key but point at different (normalized) targets; the
|
||||
first one wins in ``normalized_map``.
|
||||
"""
|
||||
out: dict[str, str] = {}
|
||||
origin: dict[str, str] = {}
|
||||
conflicts: list[dict[str, str]] = []
|
||||
for raw_k, raw_v in (port_map or {}).items():
|
||||
k = key_normalizer(raw_k) if key_normalizer else normalize_iface_name(raw_k, rules)
|
||||
v = (
|
||||
value_normalizer(raw_v) if value_normalizer
|
||||
else normalize_iface_name(raw_v, rules if value_rules is None else value_rules)
|
||||
)
|
||||
if not k or not v:
|
||||
continue
|
||||
if k in out:
|
||||
if out[k] != v:
|
||||
conflicts.append({
|
||||
"before_if": k, "kept_from": origin[k], "kept_to": out[k],
|
||||
"dropped_from": str(raw_k), "dropped_to": v,
|
||||
})
|
||||
continue
|
||||
out[k] = v
|
||||
origin[k] = str(raw_k)
|
||||
return out, conflicts
|
||||
|
||||
|
||||
def resolve_mapped_iface(name: str, port_map: Mapping[str, str] | None) -> str:
|
||||
"""Exact map hit, else longest mapped parent + remaining suffix (QinQ-safe).
|
||||
|
||||
|
|
@ -142,3 +190,53 @@ def default_zte_iface_normalize_rules() -> list[dict[str, str]]:
|
|||
{"from": "SG", "to": "smartgroup"},
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Built-in per-vendor canonical names (same-vendor synonyms only; cross-vendor
|
||||
# numbering is the port map's job). Applied before template alias rules.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
VENDOR_CANONICAL: dict[str, list[dict[str, str]]] = {
|
||||
"huawei": [
|
||||
{"from": "XGigabitEthernet", "to": "XGE"}, {"from": "GigabitEthernet", "to": "GE"},
|
||||
{"from": "100GE", "to": "100GE"}, {"from": "40GE", "to": "40GE"}, {"from": "25GE", "to": "25GE"},
|
||||
{"from": "10GE", "to": "10GE"}, {"from": "Eth-Trunk", "to": "Eth-Trunk"},
|
||||
],
|
||||
"cisco": [
|
||||
{"from": "GigabitEthernet", "to": "Gi"}, {"from": "TenGigabitEthernet", "to": "Te"},
|
||||
{"from": "TenGigE", "to": "Te"}, {"from": "HundredGigE", "to": "Hu"}, {"from": "FortyGigE", "to": "Fo"},
|
||||
{"from": "TwentyFiveGigE", "to": "Twe"}, {"from": "FastEthernet", "to": "Fa"},
|
||||
{"from": "Bundle-Ether", "to": "BE"}, {"from": "Loopback", "to": "Lo"},
|
||||
],
|
||||
"h3c": [
|
||||
{"from": "Ten-GigabitEthernet", "to": "XGE"}, {"from": "GigabitEthernet", "to": "GE"},
|
||||
{"from": "FortyGigE", "to": "FGE"}, {"from": "HundredGigE", "to": "HGE"},
|
||||
{"from": "Bridge-Aggregation", "to": "BAGG"},
|
||||
],
|
||||
"zte": [
|
||||
{"from": "xxvgei", "to": "xxvgei"}, {"from": "xgei", "to": "xgei"}, {"from": "cgei", "to": "cgei"},
|
||||
{"from": "gei", "to": "gei"}, {"from": "smartgroup", "to": "smartgroup"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def vendor_canonical_rules(vendor: str) -> list[dict[str, str]]:
|
||||
return normalize_iface_rules(VENDOR_CANONICAL.get(str(vendor or "").strip().lower(), []))
|
||||
|
||||
|
||||
def rules_for_side(
|
||||
rules: Sequence[Mapping[str, str]] | None, side: str, vendor: str = "",
|
||||
) -> list[dict[str, str]]:
|
||||
"""Template alias rules that apply to ``side`` (``before``/``after``) and ``vendor``."""
|
||||
vendor_n = str(vendor or "").strip().lower()
|
||||
out: list[dict[str, str]] = []
|
||||
for r in rules or []:
|
||||
rs = str(r.get("side") or "").strip().lower()
|
||||
if rs and rs != side:
|
||||
continue
|
||||
rv = str(r.get("vendor") or "").strip().lower()
|
||||
if rv and rv != vendor_n:
|
||||
continue
|
||||
out.append({"from": str(r.get("from") or ""), "to": str(r.get("to") or "")})
|
||||
return normalize_iface_rules(out)
|
||||
|
|
|
|||
|
|
@ -85,6 +85,9 @@ class ParseProfile:
|
|||
enrich_joins: list[EnrichJoin] = field(default_factory=list)
|
||||
# light: default shared SSH lane; heavy: dedicated long-timeout connection
|
||||
collect_lane: str = "light"
|
||||
# Which slice of the metric this command produces, e.g. {"afi": "ipv4"}.
|
||||
# Empty = the command covers the whole metric (or its slicing is unknown).
|
||||
slice: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
_LLDP_FIELDS: list[FieldDef] = [
|
||||
|
|
@ -1651,6 +1654,11 @@ _HEAVY_LANE_METRICS = frozenset(
|
|||
)
|
||||
|
||||
|
||||
def metric_lane(metric_id: str) -> str:
|
||||
"""``heavy`` for huge CLI dumps (own SSH connection), else ``light``."""
|
||||
return "heavy" if str(metric_id or "").strip() in _HEAVY_LANE_METRICS else "light"
|
||||
|
||||
|
||||
def _apply_collect_lanes(profiles: list[ParseProfile]) -> list[ParseProfile]:
|
||||
for p in profiles:
|
||||
if str(p.metric_id or "").strip() in _HEAVY_LANE_METRICS:
|
||||
|
|
@ -1752,12 +1760,41 @@ def _validate_profiles(profiles: list[ParseProfile]) -> None:
|
|||
)
|
||||
|
||||
|
||||
# profile_id -> slice. Compare sheets filter on these fields (``afi``/``af``/``direction``),
|
||||
# so a missing command is detected as "slice not collected" instead of "business lost".
|
||||
_PROFILE_SLICES: dict[str, dict[str, str]] = {
|
||||
**{f"zte.bgp_{afi}_summary": {"afi": afi} for afi in ("vpnv4", "ipv4", "vpnv6", "ipv6", "evpn", "vpls")},
|
||||
"zte.bgp_vpnv4_vrf_summary": {"afi": "vpnv4"},
|
||||
"zte.bgp_vpnv6_vrf_summary": {"afi": "vpnv6"},
|
||||
"zte.vrrp_ipv4": {"af": "ipv4"},
|
||||
"zte.vrrp_ipv6": {"af": "ipv6"},
|
||||
**{
|
||||
f"zte.bgp_{afi}_neighbor_{direction}": {"afi": afi, "direction": direction}
|
||||
for afi in ("vpnv4", "vpnv6", "ipv4", "ipv6")
|
||||
for direction in ("in", "out")
|
||||
},
|
||||
**{
|
||||
f"zte.bgp_{afi}_vrf_neighbor_{direction}": {"afi": afi, "direction": direction}
|
||||
for afi in ("vpnv4", "vpnv6")
|
||||
for direction in ("in", "out")
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _apply_profile_slices(profiles: list[ParseProfile]) -> list[ParseProfile]:
|
||||
for p in profiles:
|
||||
declared = _PROFILE_SLICES.get(p.profile_id)
|
||||
if declared and not p.slice:
|
||||
p.slice = dict(declared)
|
||||
return profiles
|
||||
|
||||
|
||||
def _finalize_profiles(profiles: list[ParseProfile]) -> list[ParseProfile]:
|
||||
"""Normalize discover→aux, validate, apply collect lanes."""
|
||||
for p in profiles:
|
||||
_normalize_discover_aux(p)
|
||||
_validate_profiles(profiles)
|
||||
return _apply_collect_lanes(profiles)
|
||||
return _apply_profile_slices(_apply_collect_lanes(profiles))
|
||||
|
||||
|
||||
def all_profiles() -> list[ParseProfile]:
|
||||
|
|
|
|||
|
|
@ -6,13 +6,17 @@ from collections import defaultdict
|
|||
from datetime import datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_, select
|
||||
from sqlalchemy import func, or_, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ..models import (
|
||||
BizCompareDiff,
|
||||
BizCompareJob,
|
||||
BizCompareRun,
|
||||
BizMigrationBatch,
|
||||
BizMigrationDiff,
|
||||
BizMigrationProject,
|
||||
BizMigrationRedTicket,
|
||||
BizMigrationRun,
|
||||
BizStateBatch,
|
||||
BizStateBatchCommand,
|
||||
|
|
@ -76,9 +80,103 @@ def protected_batch_map(
|
|||
add(before_id, before_reason)
|
||||
add(after_id, after_reason)
|
||||
|
||||
# Every batch a *kept* acceptance/manual run (or the newest auto run of a wave)
|
||||
# compared, not only the first binding. Older auto runs are bounded by
|
||||
# ``purge_auto_runs`` and keep protecting just their primary pair above.
|
||||
for run_batches in _kept_run_metric_batches(db):
|
||||
for bid in run_batches:
|
||||
add(bid, "migration_run_metric")
|
||||
|
||||
return dict(out)
|
||||
|
||||
|
||||
def _kept_run_metric_batches(db: Session, *, limit: int = 500) -> list[set[str]]:
|
||||
newest_auto = (
|
||||
db.query(func.max(BizMigrationRun.created_at).label("at"), BizMigrationRun.batch_id.label("bid"))
|
||||
.filter(BizMigrationRun.purpose == "auto").group_by(BizMigrationRun.batch_id).subquery()
|
||||
)
|
||||
latest_auto_ids = [
|
||||
r[0] for r in db.query(BizMigrationRun.id).join(
|
||||
newest_auto,
|
||||
(BizMigrationRun.batch_id == newest_auto.c.bid) & (BizMigrationRun.created_at == newest_auto.c.at),
|
||||
).limit(limit).all()
|
||||
]
|
||||
kept_ids = [
|
||||
r[0] for r in db.query(BizMigrationRun.id).filter(BizMigrationRun.purpose != "auto")
|
||||
.order_by(BizMigrationRun.created_at.desc()).limit(limit).all()
|
||||
] + latest_auto_ids
|
||||
out: list[set[str]] = []
|
||||
for start in range(0, len(kept_ids), 100):
|
||||
for (summary,) in db.query(BizMigrationRun.summary_json).filter(
|
||||
BizMigrationRun.id.in_(kept_ids[start:start + 100])
|
||||
):
|
||||
batches: set[str] = set()
|
||||
for item in ((summary or {}).get("metric_batches") or {}).values():
|
||||
if isinstance(item, dict):
|
||||
batches.update(str(item.get(k) or "") for k in ("old_batch_id", "new_batch_id"))
|
||||
batches.discard("")
|
||||
if batches:
|
||||
out.append(batches)
|
||||
return out
|
||||
|
||||
|
||||
def purge_auto_runs(db: Session, *, migration_keep: int = 300, compare_keep: int = 50) -> dict[str, int]:
|
||||
"""Bound auto-generated run history.
|
||||
|
||||
Cutover monitoring writes a run (+ one diff row per object) every minute and
|
||||
auto-compare writes one per new sample. Acceptance / manual runs and runs a
|
||||
red ticket points at are kept forever; only the newest ``keep`` auto runs of
|
||||
each wave / job survive.
|
||||
"""
|
||||
dropped = {"migration_runs": 0, "compare_runs": 0}
|
||||
ticket_runs = {r[0] for r in db.query(BizMigrationRedTicket.run_id).filter(
|
||||
BizMigrationRedTicket.run_id != "").distinct()}
|
||||
accepted = {r[0] for r in db.query(BizMigrationBatch.accept_run_id).filter(
|
||||
BizMigrationBatch.accept_run_id != "")}
|
||||
keep_forever = ticket_runs | accepted
|
||||
|
||||
def surplus(model, group_col, extra_filter, keep: int) -> list[str]:
|
||||
over = [g for g, n in db.query(group_col, func.count(model.id)).filter(*extra_filter)
|
||||
.group_by(group_col).all() if n > keep]
|
||||
ids: list[str] = []
|
||||
for g in over:
|
||||
rows = db.query(model.id).filter(group_col == g, *extra_filter).order_by(
|
||||
model.created_at.desc(), model.id.desc()).offset(keep).all()
|
||||
ids.extend(r[0] for r in rows if r[0] not in keep_forever)
|
||||
return ids
|
||||
|
||||
mig_ids = surplus(BizMigrationRun, BizMigrationRun.batch_id, [BizMigrationRun.purpose == "auto"],
|
||||
max(1, int(migration_keep)))
|
||||
for start in range(0, len(mig_ids), 500):
|
||||
chunk = mig_ids[start:start + 500]
|
||||
db.query(BizMigrationDiff).filter(BizMigrationDiff.run_id.in_(chunk)).delete(synchronize_session=False)
|
||||
db.query(BizMigrationRun).filter(BizMigrationRun.id.in_(chunk)).delete(synchronize_session=False)
|
||||
dropped["migration_runs"] = len(mig_ids)
|
||||
|
||||
# Auto-mode jobs only. A run's origin lives in summary_json["purpose"]; unmarked runs
|
||||
# are manual and stay. Only the surplus beyond ``compare_keep`` is inspected.
|
||||
cmp_ids: list[str] = []
|
||||
keep_n = max(1, int(compare_keep))
|
||||
for (job_id,) in db.query(BizCompareJob.id).filter(BizCompareJob.mode == "auto").all():
|
||||
surplus_ids = [r[0] for r in db.query(BizCompareRun.id).filter(
|
||||
BizCompareRun.job_id == job_id, BizCompareRun.status.notin_(("queued", "running")),
|
||||
).order_by(BizCompareRun.created_at.desc(), BizCompareRun.id.desc()).offset(keep_n).all()]
|
||||
for start in range(0, len(surplus_ids), 200):
|
||||
for rid, summary in db.query(BizCompareRun.id, BizCompareRun.summary_json).filter(
|
||||
BizCompareRun.id.in_(surplus_ids[start:start + 200])
|
||||
):
|
||||
if (summary or {}).get("purpose") == "auto":
|
||||
cmp_ids.append(rid)
|
||||
for start in range(0, len(cmp_ids), 500):
|
||||
chunk = cmp_ids[start:start + 500]
|
||||
db.query(BizCompareDiff).filter(BizCompareDiff.run_id.in_(chunk)).delete(synchronize_session=False)
|
||||
db.query(BizCompareRun).filter(BizCompareRun.id.in_(chunk)).delete(synchronize_session=False)
|
||||
dropped["compare_runs"] = len(cmp_ids)
|
||||
if mig_ids or cmp_ids:
|
||||
db.commit()
|
||||
return dropped
|
||||
|
||||
|
||||
def protected_batch_ids(db: Session, *, task_id: str = "") -> set[str]:
|
||||
return set(protected_batch_map(db, task_id=task_id).keys())
|
||||
|
||||
|
|
|
|||
|
|
@ -549,6 +549,12 @@ class FieldRuleIn(BaseModel):
|
|||
normalize: str = ""
|
||||
ignore: bool = False
|
||||
tolerance: float | None = None
|
||||
# Cross-vendor identity rules (see identity_pipeline).
|
||||
value_map: dict[str, str] | None = None
|
||||
side: str = ""
|
||||
apply_to_key: bool | None = None
|
||||
# Order for compare=not_worse: earlier entries are better.
|
||||
rank: list[str] | None = None
|
||||
|
||||
|
||||
class RowFilterIn(BaseModel):
|
||||
|
|
@ -575,6 +581,9 @@ class TemplateMetricIn(BaseModel):
|
|||
row_filters: list[dict[str, Any]] = Field(default_factory=list)
|
||||
field_rules: list[FieldRuleIn] = Field(default_factory=list)
|
||||
ignore_port_changes: bool | None = None
|
||||
unmapped_port_policy: str = ""
|
||||
duplicate_pairing: str = ""
|
||||
builtin_vendor_canonical: bool | None = None
|
||||
|
||||
|
||||
class TemplateIn(BaseModel):
|
||||
|
|
@ -623,6 +632,16 @@ class MappingIn(BaseModel):
|
|||
rows: list[MappingRowIn] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PreflightIn(BaseModel):
|
||||
template_id: str = ""
|
||||
enabled_sheet_ids: list[str] = Field(default_factory=list)
|
||||
mapping_id: str = ""
|
||||
# Unsaved editor rows win over ``mapping_id`` when given.
|
||||
mapping_rows: list[MappingRowIn] | None = None
|
||||
before_batch_id: str
|
||||
after_batch_id: str
|
||||
|
||||
|
||||
class ValidateMappingIn(BaseModel):
|
||||
mapping_id: str
|
||||
before_batch_id: str
|
||||
|
|
@ -711,6 +730,11 @@ def api_validate_mapping(body: ValidateMappingIn, db: Session = Depends(get_db))
|
|||
)
|
||||
|
||||
|
||||
@router.post("/compare/preflight")
|
||||
def api_compare_preflight(body: PreflightIn, db: Session = Depends(get_db)) -> dict[str, Any]:
|
||||
return cmp_svc.preflight_compare(db, body.model_dump())
|
||||
|
||||
|
||||
@router.get("/compare/jobs")
|
||||
def api_list_jobs(db: Session = Depends(get_db)) -> dict[str, Any]:
|
||||
return {"items": cmp_svc.list_jobs(db)}
|
||||
|
|
|
|||
|
|
@ -47,6 +47,15 @@ def _maybe_purge_retention() -> None:
|
|||
from .biz_state.retention import purge_all_tasks
|
||||
|
||||
info = purge_all_tasks(db)
|
||||
from .biz_state.retention import purge_auto_runs
|
||||
|
||||
runs = purge_auto_runs(
|
||||
db,
|
||||
migration_keep=int(getattr(settings, "biz_migration_auto_run_keep", 300) or 300),
|
||||
compare_keep=int(getattr(settings, "biz_compare_auto_run_keep", 50) or 50),
|
||||
)
|
||||
if any(runs.values()):
|
||||
_log.info("biz_state periodic run purge %s", runs)
|
||||
if info.get("dropped"):
|
||||
_log.info(
|
||||
"biz_state periodic retention dropped=%s tasks=%s",
|
||||
|
|
|
|||
|
|
@ -137,6 +137,13 @@ class Settings(BaseSettings):
|
|||
biz_state_dedicated_workers: bool = True
|
||||
# Global ceiling for simultaneous running batches (across all workers).
|
||||
biz_state_max_concurrent_tasks: int = 16
|
||||
# Auto-generated run history kept per cutover wave / auto compare job (acceptance + manual runs are never purged).
|
||||
biz_migration_auto_run_keep: int = 300
|
||||
biz_compare_auto_run_keep: int = 50
|
||||
# Rows of immutable baseline batches cached across cutover evaluations (0 disables).
|
||||
biz_migration_baseline_cache_rows: int = 500000
|
||||
# Concurrent SSH collections per network element (0 = unlimited).
|
||||
biz_state_max_sessions_per_ne: int = 2
|
||||
biz_state_worker_collect_threads: int = 8
|
||||
biz_state_persist_workers: int = 4
|
||||
# TextFSM / normalize workers (off SSH collect threads). Default 8.
|
||||
|
|
|
|||
12
tests/conftest.py
Normal file
12
tests/conftest.py
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
"""Shared test hygiene."""
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fresh_baseline_row_cache():
|
||||
"""Test databases reuse batch ids; the process-wide baseline cache must not leak between tests."""
|
||||
from netx_api.biz_migration.service import BASELINE_ROWS
|
||||
|
||||
BASELINE_ROWS.invalidate()
|
||||
yield
|
||||
BASELINE_ROWS.invalidate()
|
||||
|
|
@ -143,7 +143,7 @@ def test_template_usage_counts_are_batched(db, monkeypatch):
|
|||
event.listen(db.get_bind(), "before_cursor_execute", lambda c, cur, statement, p, ctx, many: sql.append(statement))
|
||||
counts = {t["id"]: t["job_count"] for t in svc.list_templates(db)}
|
||||
assert counts == {used["id"]: 2, unused["id"]: 0}
|
||||
assert len(sql) == 2 # one template query and one grouped reference query
|
||||
assert len(sql) == 3 # template query + one grouped query each for jobs and monitor templates
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", ["queued", "running", "failed", "cancelled", "partial"])
|
||||
|
|
@ -188,3 +188,93 @@ def test_legacy_invalid_template_cannot_execute_as_partial_or_fail_open(db, bad)
|
|||
assert exc.value.detail["error"] == "invalid_template"
|
||||
assert "metrics[1]" in exc.value.detail["path"]
|
||||
assert db.query(BizCompareRun).count() == 0
|
||||
|
||||
|
||||
def _builtin_sheet(row, metric_id):
|
||||
return next(s for s in svc.template_metrics(row) if s["metric_id"] == metric_id)
|
||||
|
||||
|
||||
def _rule(sheet, field):
|
||||
return next(r for r in sheet["field_rules"] if r["field"] == field)
|
||||
|
||||
|
||||
def test_packaged_status_template_uses_corrected_defaults(db):
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
detail = _builtin_sheet(row, "interface_detail")
|
||||
assert _rule(detail, "output_bps")["compare"] == "percent"
|
||||
for f in ("input_bps", "output_bps", "in_util", "out_util"):
|
||||
assert f not in detail["compare_fields"] and f in detail["display_fields"]
|
||||
rx = _rule(_builtin_sheet(row, "optical_brief"), "rx_power")
|
||||
assert (rx["compare"], rx["tolerance"]) == ("numeric", 2)
|
||||
assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["ac_port"]
|
||||
|
||||
|
||||
def _store_old_packaged_values(db, row):
|
||||
sheets = svc.template_metrics(row)
|
||||
for s in sheets:
|
||||
if s["metric_id"] == "interface_detail":
|
||||
_rule(s, "output_bps").update(compare="numeric", tolerance=5)
|
||||
s["compare_fields"] = list(svc._INTERFACE_DETAIL_OLD_COMPARE)
|
||||
elif s["metric_id"] == "optical_brief":
|
||||
_rule(s, "rx_power").update(compare="percent", tolerance=5)
|
||||
elif s["metric_id"] == "l2vpn_mac":
|
||||
s["iface_fields"] = []
|
||||
svc._apply_sheets_to_row(row, sheets)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_old_packaged_values_are_upgraded_in_place(db):
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
_store_old_packaged_values(db, row)
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
detail = _builtin_sheet(row, "interface_detail")
|
||||
assert _rule(detail, "output_bps")["compare"] == "percent"
|
||||
assert "output_bps" not in detail["compare_fields"]
|
||||
assert _rule(_builtin_sheet(row, "optical_brief"), "rx_power")["compare"] == "numeric"
|
||||
assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["ac_port"]
|
||||
|
||||
|
||||
def test_operator_customised_values_survive_upgrade(db):
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
_store_old_packaged_values(db, row)
|
||||
sheets = svc.template_metrics(row)
|
||||
for s in sheets:
|
||||
if s["metric_id"] == "optical_brief":
|
||||
_rule(s, "rx_power").update(compare="percent", tolerance=1) # operator tightened it
|
||||
elif s["metric_id"] == "interface_detail":
|
||||
s["compare_fields"] = ["admin", "output_bps"] # operator trimmed the list
|
||||
elif s["metric_id"] == "l2vpn_mac":
|
||||
s["iface_fields"] = ["neighbor"]
|
||||
svc._apply_sheets_to_row(row, sheets)
|
||||
db.commit()
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
rx = _rule(_builtin_sheet(row, "optical_brief"), "rx_power")
|
||||
assert (rx["compare"], rx["tolerance"]) == ("percent", 1)
|
||||
assert _builtin_sheet(row, "interface_detail")["compare_fields"] == ["admin", "output_bps"]
|
||||
assert _builtin_sheet(row, "l2vpn_mac")["iface_fields"] == ["neighbor"]
|
||||
|
||||
|
||||
def test_packaged_arp_filter_uses_canonical_entry_type_and_old_filter_is_upgraded(db):
|
||||
from netx_api.biz_state.compare_rules import arp_dynamic_row_filters
|
||||
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
assert _builtin_sheet(row, "arp")["row_filters"] == arp_dynamic_row_filters()
|
||||
sheets = svc.template_metrics(row)
|
||||
for s in sheets:
|
||||
if s["metric_id"] == "arp":
|
||||
s["row_filters"] = [{"op": "age_timer", "field": "age", "value": ""}]
|
||||
svc._apply_sheets_to_row(row, sheets)
|
||||
db.commit()
|
||||
assert _builtin_sheet(svc.ensure_default_zte_status_template(db), "arp")["row_filters"] == arp_dynamic_row_filters()
|
||||
|
||||
|
||||
def test_customised_arp_filter_is_left_alone(db):
|
||||
row = svc.ensure_default_zte_status_template(db)
|
||||
sheets = svc.template_metrics(row)
|
||||
mine = [{"op": "eq", "field": "entry_type", "value": "static"}]
|
||||
for s in sheets:
|
||||
if s["metric_id"] == "arp":
|
||||
s["row_filters"] = mine
|
||||
svc._apply_sheets_to_row(row, sheets)
|
||||
db.commit()
|
||||
assert _builtin_sheet(svc.ensure_default_zte_status_template(db), "arp")["row_filters"] == mine
|
||||
|
|
|
|||
284
tests/test_biz_migration_async.py
Normal file
284
tests/test_biz_migration_async.py
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
"""H2/H3/H5: background evaluation, manual route tables, duplicate-key scope."""
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_migration import service as mig
|
||||
from netx_api.models import BizMigrationBatch, BizMigrationRun, BizStateBatchCommand, BizStateMetricRow
|
||||
import tests.test_biz_migration_evidence as evidence
|
||||
|
||||
|
||||
class _Inline:
|
||||
"""threading.Thread stand-in: remember the target, run it on demand."""
|
||||
|
||||
started: list = []
|
||||
|
||||
def __init__(self, target=None, **_):
|
||||
self.target = target
|
||||
|
||||
def start(self):
|
||||
_Inline.started.append(self.target)
|
||||
|
||||
|
||||
class _Fixture(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.ctx = evidence.EvidencePersistTests("test_evaluate_persists_display_ab_and_evidence_with_command")
|
||||
self.ctx.setUp()
|
||||
self.db = self.ctx.db
|
||||
self.factory = sessionmaker(bind=self.db.get_bind(), expire_on_commit=False)
|
||||
self.session_patch = mock.patch("netx_api.db.SessionLocal", self.factory)
|
||||
self.session_patch.start()
|
||||
_Inline.started = []
|
||||
proj = self.ctx._create_project()
|
||||
c = self.ctx
|
||||
for bid, tid in (("bl_old", c.old_p.id), ("bl_new", c.new_p.id), ("cur_old", proj["old_hf_task_id"]),
|
||||
("cur_new", proj["new_hf_task_id"])):
|
||||
c._mk_batch(bid, tid)
|
||||
c._seed_if_row(batch_id="bl_old", task_id=c.old_p.id, ne_id="ne-old", command_id="c1",
|
||||
raw_command="show interface brief", interface="GE-old")
|
||||
c._seed_if_row(batch_id="cur_new", task_id=proj["new_hf_task_id"], ne_id="ne-new", command_id="c2",
|
||||
raw_command="show interface brief", interface="xgei-new")
|
||||
mig.patch_project(self.db, proj["id"], {"old_baseline_batch_id": "bl_old", "new_baseline_batch_id": "bl_new"})
|
||||
self.batch = mig.create_batch(self.db, proj["id"], {"batch_label": "w", "expect_set": {"ports": ["gei-old"]}})
|
||||
mig.patch_batch(self.db, self.batch["id"], {"status": "active"})
|
||||
self.project_id = proj["id"]
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session_patch.stop()
|
||||
self.ctx.tearDown()
|
||||
|
||||
def run_background(self):
|
||||
for target in list(_Inline.started):
|
||||
target()
|
||||
_Inline.started.clear()
|
||||
|
||||
|
||||
class AsyncEvaluateTests(_Fixture):
|
||||
def test_evaluate_returns_a_running_run_immediately_then_completes(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
started = mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
self.assertEqual(started["status"], "running")
|
||||
self.assertEqual(self.db.get(BizMigrationRun, started["id"]).status, "running")
|
||||
self.assertEqual(len(_Inline.started), 1)
|
||||
self.run_background()
|
||||
self.db.expire_all()
|
||||
run = self.db.get(BizMigrationRun, started["id"])
|
||||
self.assertEqual(run.status, "success")
|
||||
self.assertEqual(run.purpose, "manual")
|
||||
self.assertIn("sheet_cards", run.summary_json)
|
||||
self.assertEqual(self.db.query(BizMigrationRun).count(), 1) # the claim row was filled, not duplicated
|
||||
|
||||
def test_second_request_while_one_runs_is_409(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
with self.assertRaises(HTTPException) as exc:
|
||||
mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
with self.assertRaises(HTTPException) as exc_finish:
|
||||
mig.enqueue_finish(self.db, self.batch["id"])
|
||||
self.assertEqual(exc.exception.status_code, 409)
|
||||
self.assertEqual(exc.exception.detail, "evaluation_in_progress")
|
||||
self.assertEqual(exc_finish.exception.status_code, 409)
|
||||
|
||||
def test_failure_is_recorded_on_the_run_and_releases_the_project(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
started = mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
with mock.patch.object(mig, "run_evaluate", side_effect=HTTPException(status_code=400, detail="boom")):
|
||||
self.run_background()
|
||||
self.db.expire_all()
|
||||
run = self.db.get(BizMigrationRun, started["id"])
|
||||
self.assertEqual((run.status, run.message), ("failed", "boom"))
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
mig.enqueue_evaluate(self.db, self.batch["id"]) # no longer blocked
|
||||
|
||||
def test_restart_cancels_orphaned_runs(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
started = mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
self.assertEqual(mig.recover_interrupted_evaluations_on_startup(self.db), 1)
|
||||
self.db.expire_all()
|
||||
run = self.db.get(BizMigrationRun, started["id"])
|
||||
self.assertEqual((run.status, run.message), ("cancelled", "interrupted_by_restart"))
|
||||
# a cancelled claim must not be resurrected by a late worker
|
||||
with self.assertRaises(HTTPException) as exc:
|
||||
mig.run_evaluate(self.db, batch_id=self.batch["id"], _run_id=started["id"])
|
||||
self.assertEqual(exc.exception.status_code, 409)
|
||||
|
||||
def test_background_evaluation_locks_the_project_only_for_the_final_write(self):
|
||||
from sqlalchemy.orm import Query
|
||||
|
||||
calls: list[str] = []
|
||||
original = Query.with_for_update
|
||||
|
||||
def spy(self_, *a, **k):
|
||||
calls.append("lock")
|
||||
return original(self_, *a, **k)
|
||||
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
mig.enqueue_evaluate(self.db, self.batch["id"]) # claim = one short lock
|
||||
with mock.patch.object(Query, "with_for_update", spy):
|
||||
self.run_background()
|
||||
self.assertEqual(len(calls), 1, "computation must be unlocked; only the write takes the project lock")
|
||||
calls.clear()
|
||||
with mock.patch.object(Query, "with_for_update", spy):
|
||||
self.db.query(BizMigrationRun).delete()
|
||||
self.db.commit()
|
||||
mig.run_evaluate(self.db, batch_id=self.batch["id"]) # synchronous callers keep the old behaviour
|
||||
self.assertGreaterEqual(len(calls), 1)
|
||||
|
||||
def test_finish_runs_in_the_background_and_closes_the_wave_only_on_success(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
started = mig.enqueue_finish(self.db, self.batch["id"])
|
||||
self.assertEqual((started["status"], started["run"]["purpose"]), ("running", "acceptance"))
|
||||
self.assertEqual(self.db.get(BizMigrationBatch, self.batch["id"]).status, "active")
|
||||
self.run_background()
|
||||
self.db.expire_all()
|
||||
batch = self.db.get(BizMigrationBatch, self.batch["id"])
|
||||
self.assertEqual(batch.status, "review")
|
||||
self.assertEqual(batch.accept_run_id, started["run_id"])
|
||||
self.assertEqual(self.db.get(BizMigrationRun, started["run_id"]).status, "success")
|
||||
|
||||
def test_finish_failure_keeps_the_wave_operable(self):
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
started = mig.enqueue_finish(self.db, self.batch["id"])
|
||||
with mock.patch.object(mig, "run_evaluate", side_effect=HTTPException(status_code=400, detail="x")):
|
||||
self.run_background()
|
||||
self.db.expire_all()
|
||||
self.assertEqual(self.db.get(BizMigrationBatch, self.batch["id"]).status, "active")
|
||||
self.assertEqual(self.db.get(BizMigrationRun, started["run_id"]).status, "failed")
|
||||
|
||||
def test_auto_monitor_stays_out_of_a_running_evaluation(self):
|
||||
from netx_api.biz_migration import auto_monitor
|
||||
|
||||
with mock.patch.object(mig.threading, "Thread", _Inline):
|
||||
mig.enqueue_evaluate(self.db, self.batch["id"])
|
||||
with mock.patch.object(mig, "run_evaluate") as run_evaluate:
|
||||
auto_monitor.try_auto_monitor_for_task(self.db, self.ctx.old_p.id)
|
||||
run_evaluate.assert_not_called()
|
||||
|
||||
|
||||
class ManualRouteAndDuplicateScopeTests(_Fixture):
|
||||
"""Share the fixture, add route sheets / duplicate rows to the compare template."""
|
||||
|
||||
def _add_sheet(self, sheet):
|
||||
self.ctx.ct.metrics_json = [*self.ctx.ct.metrics_json, sheet]
|
||||
self.ctx.mt.collect_metric_ids_json = []
|
||||
self.db.commit()
|
||||
|
||||
def test_uncollected_route_table_is_manual_pending_and_does_not_fail_acceptance(self):
|
||||
self._add_sheet({"sheet_id": "bgp_route", "title": "routes", "metric_id": "bgp_route",
|
||||
"key_fields": ["network"], "iface_fields": [], "compare_fields": [], "row_filters": [],
|
||||
"field_rules": []})
|
||||
out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance")
|
||||
card = next(c for c in out["summary"]["sheet_cards"] if c["metric_id"] == "bgp_route")
|
||||
self.assertTrue(card["manual_pending"])
|
||||
self.assertFalse(card.get("current_missing"))
|
||||
self.assertIn("routes", out["summary"]["manual_pending_metrics"])
|
||||
self.assertNotIn("routes", out["summary"]["current_missing_metrics"])
|
||||
|
||||
def test_duplicate_keys_outside_the_wave_are_warnings_not_failures(self):
|
||||
c = self.ctx
|
||||
for i, cmd in enumerate(("d1", "d2")):
|
||||
self.db.add(BizStateMetricRow(id=f"dup-{i}", batch_id="bl_old", metric_id="interface_brief", seq=10 + i,
|
||||
data_json={"interface": "GE-other", "admin": "up", "phy": "up", "prot": "up"}))
|
||||
self.db.commit()
|
||||
out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance")
|
||||
summary = out["summary"]
|
||||
self.assertEqual(summary["duplicate_keys"], 0)
|
||||
self.assertEqual(summary["duplicate_warnings"][0]["count"] >= 1, True)
|
||||
|
||||
def test_duplicate_keys_inside_the_wave_still_fail(self):
|
||||
for i in range(2):
|
||||
self.db.add(BizStateMetricRow(id=f"dup-{i}", batch_id="bl_old", metric_id="interface_brief", seq=10 + i,
|
||||
data_json={"interface": "GE-old", "admin": "up", "phy": "up", "prot": "up"}))
|
||||
self.db.commit()
|
||||
out = mig.run_evaluate(self.db, batch_id=self.batch["id"], acceptance=True, purpose="acceptance")
|
||||
self.assertGreaterEqual(out["summary"]["duplicate_keys"], 1)
|
||||
self.assertFalse(out["summary"]["coverage_complete"])
|
||||
|
||||
|
||||
class BaselineCacheTests(_Fixture):
|
||||
def test_second_evaluation_does_not_read_baseline_rows_again(self):
|
||||
mig.BASELINE_ROWS.invalidate()
|
||||
real = mig._load_metric_rows
|
||||
loaded: list[str] = []
|
||||
|
||||
def spy(db, *, batch_id, metric_id, **kw):
|
||||
loaded.append(batch_id)
|
||||
return real(db, batch_id=batch_id, metric_id=metric_id, **kw)
|
||||
|
||||
with mock.patch.object(mig, "_load_metric_rows", spy):
|
||||
mig.run_evaluate(self.db, batch_id=self.batch["id"])
|
||||
first = list(loaded)
|
||||
loaded.clear()
|
||||
mig.run_evaluate(self.db, batch_id=self.batch["id"])
|
||||
self.assertIn("bl_old", first)
|
||||
self.assertNotIn("bl_old", loaded)
|
||||
self.assertNotIn("bl_new", loaded)
|
||||
self.assertIn("cur_new", loaded) # current samples are always re-read
|
||||
|
||||
def test_lru_is_bounded_by_total_rows_and_can_be_invalidated(self):
|
||||
cache = mig.BaselineRowCache(max_rows=5)
|
||||
rows = lambda n: [{"i": i} for i in range(n)] # noqa: E731
|
||||
cache.get_or_load("a", "m", lambda: rows(3))
|
||||
cache.get_or_load("b", "m", lambda: rows(3)) # evicts a (3 + 3 > 5)
|
||||
calls: list[str] = []
|
||||
cache.get_or_load("a", "m", lambda: calls.append("a") or rows(3))
|
||||
self.assertEqual(calls, ["a"])
|
||||
cache.get_or_load("big", "m", lambda: rows(9)) # larger than the cap: never cached
|
||||
cache.get_or_load("big", "m", lambda: calls.append("big") or rows(9))
|
||||
self.assertEqual(calls, ["a", "big"])
|
||||
cache.invalidate("a")
|
||||
cache.get_or_load("a", "m", lambda: calls.append("a2") or rows(1))
|
||||
self.assertEqual(calls[-1], "a2")
|
||||
|
||||
|
||||
class RedTicketResolvableTests(_Fixture):
|
||||
def _diff(self, run_id, key, color, verdict):
|
||||
from netx_api.models import BizMigrationDiff
|
||||
|
||||
self.db.add(BizMigrationDiff(id=f"d-{run_id}-{key}", run_id=run_id, metric_id="interface_brief", color=color,
|
||||
verdict=verdict, key_json={"match_old_key": key, "sheet_id": "interface_brief",
|
||||
"key_str": key}))
|
||||
|
||||
def test_ticket_for_a_key_that_turned_green_is_flagged_not_closed(self):
|
||||
from netx_api.models import BizMigrationRedTicket
|
||||
|
||||
pid = self.project_id
|
||||
for key in ("gei-1", "gei-2"):
|
||||
self.db.add(BizMigrationRedTicket(
|
||||
id=f"t-{key}", project_id=pid, batch_id="old", run_id="r0", metric_id="interface_brief",
|
||||
match_key_str=key, key_str=key, status="open", detail_json={"sheet_id": "interface_brief", "match_old_key": key}))
|
||||
self.db.add(BizMigrationRun(id="r1", project_id=pid, batch_id=self.batch["id"], purpose="acceptance"))
|
||||
self._diff("r1", "gei-1", "green", "migrated")
|
||||
self._diff("r1", "gei-2", "red", "anomaly")
|
||||
self.db.commit()
|
||||
mig._persist_red_tickets_from_run(self.db, project_id=pid, batch_id=self.batch["id"], run_id="r1")
|
||||
self.db.commit()
|
||||
from netx_api.models import BizMigrationRedTicket as T
|
||||
|
||||
flagged = {t.id: mig._red_ticket_to_dict(t) for t in self.db.query(T)}
|
||||
self.assertTrue(flagged["t-gei-1"]["resolvable"])
|
||||
self.assertFalse(flagged["t-gei-2"]["resolvable"])
|
||||
self.assertEqual(flagged["t-gei-1"]["status"], "open")
|
||||
|
||||
out = mig.resolve_resolvable_red_tickets(self.db, pid)
|
||||
self.assertEqual(out["ids"], ["t-gei-1"])
|
||||
self.assertEqual(self.db.get(T, "t-gei-1").status, "resolved")
|
||||
self.assertEqual(self.db.get(T, "t-gei-2").status, "open")
|
||||
|
||||
|
||||
class UncoveredSliceTests(_Fixture):
|
||||
def test_a_slice_one_side_never_collected_is_not_collected_not_lost(self):
|
||||
sheet = {"sheet_id": "bgp_peer.ipv4", "metric_id": "bgp_peer", "key_fields": ["neighbor"],
|
||||
"iface_fields": [], "compare_fields": ["state"],
|
||||
"row_filters": [{"op": "eq", "field": "afi", "value": "ipv4"}], "field_rules": []}
|
||||
for batch, profile in (("cur_old", "zte.bgp_ipv4_summary"), ("cur_new", "zte.bgp_vpnv4_summary")):
|
||||
self.db.add(BizStateBatchCommand(id=f"slice-{batch}", batch_id=batch, metric_id="bgp_peer",
|
||||
profile_id=profile, parse_status="ok", row_count=0, raw_line_count=0))
|
||||
self.db.commit()
|
||||
self.assertEqual(mig._uncovered_slice_sides(self.db, sheet, "bgp_peer", "cur_old", "cur_new"), ["new"])
|
||||
self.assertEqual(mig._uncovered_slice_sides(self.db, {**sheet, "row_filters": []}, "bgp_peer",
|
||||
"cur_old", "cur_new"), [])
|
||||
|
|
@ -144,16 +144,34 @@ class HfDecoupleFlowTests(unittest.TestCase):
|
|||
self.assertNotEqual(out["old_hf_task_id"], "old_p")
|
||||
self.assertNotEqual(out["new_hf_task_id"], "new_p")
|
||||
|
||||
def test_same_interval_metrics_are_separate_paired_tasks(self) -> None:
|
||||
def test_same_interval_light_metrics_share_one_paired_task(self) -> None:
|
||||
out = self._create_project(collect_metric_ids=["interface_brief", "arp"])
|
||||
old, new = out["old_hf_bindings"], out["new_hf_bindings"]
|
||||
self.assertEqual(len(old), 2)
|
||||
self.assertEqual({b["interval_sec"] for b in old}, {60})
|
||||
for o, n in zip(old, new):
|
||||
self.assertEqual(len(o["metric_ids"]), 1)
|
||||
self.assertTrue(o["collect_group_id"])
|
||||
self.assertEqual(o["collect_group_id"], n["collect_group_id"])
|
||||
self.assertEqual(o["metric_ids"], n["metric_ids"])
|
||||
self.assertEqual((len(old), len(new)), (1, 1))
|
||||
self.assertEqual(old[0]["metric_ids"], ["interface_brief", "arp"])
|
||||
self.assertEqual(old[0]["interval_sec"], 60)
|
||||
self.assertTrue(old[0]["collect_group_id"])
|
||||
self.assertEqual(old[0]["collect_group_id"], new[0]["collect_group_id"])
|
||||
from netx_api.models import BizStateTaskItem
|
||||
|
||||
items = self.db.query(BizStateTaskItem).filter(BizStateTaskItem.task_id == old[0]["task_id"]).count()
|
||||
self.assertEqual(items, 2) # one SSH login runs both commands
|
||||
|
||||
def test_different_interval_or_lane_or_manual_route_stay_apart(self) -> None:
|
||||
out = self._create_project(collect_metric_ids=["interface_brief", "arp", "interface_detail", "ip_route"],
|
||||
metric_interval_sec={"arp": 300})
|
||||
by_metrics = {tuple(b["metric_ids"]): b for b in out["old_hf_bindings"]}
|
||||
self.assertEqual(set(by_metrics), {("interface_brief",), ("arp",), ("interface_detail",), ("ip_route",)})
|
||||
self.assertEqual(by_metrics[("arp",)]["interval_sec"], 300)
|
||||
self.assertTrue(by_metrics[("ip_route",)]["manual_only"])
|
||||
self.assertEqual(len({b["collect_group_id"] for b in out["old_hf_bindings"]}), 4)
|
||||
|
||||
def test_pair_lookup_works_for_a_metric_inside_a_merged_task(self) -> None:
|
||||
out = self._create_project(collect_metric_ids=["interface_brief", "arp"])
|
||||
proj = self.db.get(BizMigrationProject, out["id"])
|
||||
self.assertEqual(mig._hf_task_id_for_metric(proj, "old", "arp"), out["old_hf_bindings"][0]["task_id"])
|
||||
self.assertEqual(mig._hf_task_id_for_metric(proj, "new", "interface_brief"),
|
||||
out["new_hf_bindings"][0]["task_id"])
|
||||
|
||||
def test_dynamic_routes_manual_only_and_removal_cancels_queued_pairs(self) -> None:
|
||||
from netx_api.models import BizStateTaskItem, BizStateTaskItemBinding
|
||||
|
|
@ -563,3 +581,77 @@ class HfDecoupleFlowTests(unittest.TestCase):
|
|||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class SliceUnionAndSameDeviceTests(HfDecoupleFlowTests):
|
||||
"""V2 (both devices collect the same slices) and H7 (same device on both sides)."""
|
||||
|
||||
def _portrait_items(self, task_id, *profile_ids):
|
||||
from netx_api.models import BizStateTaskItem
|
||||
|
||||
for i, pid in enumerate(profile_ids):
|
||||
self.db.add(BizStateTaskItem(id=f"{task_id}-{i}", task_id=task_id, source_profile_id=pid,
|
||||
kind="catalog", enabled=True, sort_order=i))
|
||||
self.db.commit()
|
||||
|
||||
def _hf_profiles(self, task_id):
|
||||
from netx_api.models import BizStateTaskItem
|
||||
|
||||
return sorted(r.source_profile_id for r in
|
||||
self.db.query(BizStateTaskItem).filter(BizStateTaskItem.task_id == task_id))
|
||||
|
||||
def test_union_gives_the_device_that_lacks_a_slice_this_vendors_profile(self) -> None:
|
||||
old = [{"source_profile_id": "zte.bgp_ipv4_summary", "bindings": []},
|
||||
{"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}]
|
||||
new = [{"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}]
|
||||
old2, new2 = mig._union_slice_items("bgp_peer", old, new, "zte", "zte")
|
||||
self.assertEqual({i["source_profile_id"] for i in new2}, {"zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary"})
|
||||
self.assertEqual(old2, old)
|
||||
|
||||
def test_missing_vendor_profile_is_a_clear_error_not_a_silent_gap(self) -> None:
|
||||
old = [{"source_profile_id": "zte.bgp_ipv4_summary", "bindings": []}]
|
||||
new = [{"source_profile_id": "zte.bgp_vpnv4_summary", "bindings": []}]
|
||||
with self.assertRaises(Exception) as ctx:
|
||||
mig._union_slice_items("bgp_peer", old, new, "zte", "huawei")
|
||||
self.assertEqual(ctx.exception.detail, "no_profile_for_slice:bgp_peer:afi=ipv4:huawei")
|
||||
|
||||
def test_wholemetric_profiles_need_no_union(self) -> None:
|
||||
item = [{"source_profile_id": "zte.interface_brief", "bindings": []}]
|
||||
self.assertEqual(mig._union_slice_items("interface_brief", item, [], "zte", "zte"), (item, []))
|
||||
|
||||
def test_project_hf_tasks_collect_the_union_of_both_portraits(self) -> None:
|
||||
self._portrait_items("old_p", "zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary")
|
||||
self._portrait_items("new_p", "zte.bgp_vpnv4_summary")
|
||||
out = self._create_project(collect_metric_ids=["bgp_peer"])
|
||||
want = ["zte.bgp_ipv4_summary", "zte.bgp_vpnv4_summary"]
|
||||
self.assertEqual(self._hf_profiles(out["old_hf_bindings"][0]["task_id"]), want)
|
||||
self.assertEqual(self._hf_profiles(out["new_hf_bindings"][0]["task_id"]), want)
|
||||
|
||||
def test_same_device_project_uses_one_task_for_both_sides(self) -> None:
|
||||
self.new_portrait.ne_id = self.old_portrait.ne_id
|
||||
self.db.commit()
|
||||
out = self._create_project(collect_metric_ids=["interface_brief", "arp"])
|
||||
old, new = out["old_hf_bindings"], out["new_hf_bindings"]
|
||||
self.assertEqual(len(old), 1)
|
||||
self.assertEqual(old[0]["task_id"], new[0]["task_id"])
|
||||
self.assertEqual(old[0]["collect_group_id"], new[0]["collect_group_id"])
|
||||
self.assertEqual(self.db.query(BizStateTask).filter(BizStateTask.purpose == mig.PURPOSE_CUTOVER_HF).count(), 1)
|
||||
|
||||
def test_same_device_sample_is_that_devices_latest_finished_round(self) -> None:
|
||||
self.new_portrait.ne_id = self.old_portrait.ne_id
|
||||
self.db.commit()
|
||||
out = self._create_project(collect_metric_ids=["interface_brief"])
|
||||
proj = self.db.get(BizMigrationProject, out["id"])
|
||||
tid = out["old_hf_bindings"][0]["task_id"]
|
||||
self.assertEqual(mig._current_pair_for_metric(self.db, proj, "interface_brief")[2]["sample_issues"],
|
||||
["collect_pair_missing"])
|
||||
batch = self._mk_batch("solo-1", tid)
|
||||
old, new, meta = mig._current_pair_for_metric(self.db, proj, "interface_brief")
|
||||
self.assertIs(old, new)
|
||||
self.assertEqual(old.id, batch.id)
|
||||
self.assertEqual(meta["old_collect_status"], "success")
|
||||
|
||||
def test_two_devices_are_still_rejected_when_they_are_in_fact_one(self) -> None:
|
||||
# different NE ids but resolving to one task id must keep failing loudly
|
||||
out = self._create_project(collect_metric_ids=["interface_brief"])
|
||||
self.assertNotEqual(out["old_hf_bindings"][0]["task_id"], out["new_hf_bindings"][0]["task_id"])
|
||||
|
|
|
|||
|
|
@ -406,3 +406,21 @@ def test_diff_filters_apply_before_pagination_and_escape_wildcards(db):
|
|||
def test_invalid_correction_rules_fail_before_evaluation(condition):
|
||||
with pytest.raises(HTTPException):
|
||||
validate_monitor_rules({}, [{"metric_id": "arp", "success": [{"old_groups": [[condition]], "new": ["added"]}]}])
|
||||
|
||||
|
||||
@pytest.mark.parametrize("raw", [None, True, " v "])
|
||||
def test_odd_key_values_still_match_expected_key(raw):
|
||||
"""A6: diff-derived key strings must use the same form as the expect set."""
|
||||
from netx_api.biz_migration.monitor_templates import preset_override_for_metric
|
||||
from netx_api.biz_state.compare_rules import scalar_text
|
||||
|
||||
old = {"ip": "a", "vrf": raw, "mac": "m", "port": "old"}
|
||||
new = {"ip": "a", "vrf": raw, "mac": "m", "port": "new"}
|
||||
out = evaluate(
|
||||
key_fields=["ip", "vrf"], compare_fields=["mac"], iface_fields=[],
|
||||
old_baseline_rows=[old], old_current_rows=[], new_baseline_rows=[], new_current_rows=[new],
|
||||
port_map={}, window_active=False, acceptance=True,
|
||||
sheet_override=preset_override_for_metric("arp"),
|
||||
expect=parse_expect_set({"items": [{"metric_id": "arp", "key": ["a", scalar_text(raw).strip()]}]}),
|
||||
)
|
||||
assert [r["verdict"] for r in out["rows"]] == ["migrated"]
|
||||
|
|
|
|||
156
tests/test_biz_migration_presets.py
Normal file
156
tests/test_biz_migration_presets.py
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
"""Default monitor presets: port success condition and legacy upgrade (plan A4)."""
|
||||
from copy import deepcopy
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_migration import monitor_templates as mt
|
||||
from netx_api.biz_migration.evaluate import dual_verdict_ex
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import BizCompareTemplate, BizMonitorTemplate
|
||||
|
||||
NEW_UP = {"admin": "up", "phy": "up"}
|
||||
|
||||
OLD_SIDES = {
|
||||
"fibre_pulled_no_shutdown": ("changed", {"admin": "up", "phy": "down"}),
|
||||
"shutdown_link_still_up": ("changed", {"admin": "down", "phy": "up"}),
|
||||
"both_down": ("changed", {"admin": "down", "phy": "down"}),
|
||||
"port_deleted": ("removed", {}),
|
||||
}
|
||||
|
||||
|
||||
def verdict(old_kind, old_row, new_row, *, acceptance, ov=None):
|
||||
ov = ov or mt.preset_override_for_metric("interface_brief")
|
||||
return dual_verdict_ex(
|
||||
old_kind=old_kind, new_kind="changed", in_expect=True,
|
||||
window_active=not acceptance, acceptance=acceptance,
|
||||
old_status="none", new_status="up",
|
||||
success_patterns=ov["success"], anomaly_patterns=ov["anomaly"],
|
||||
old_row=old_row, new_row=new_row,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("acceptance", [False, True], ids=["window", "acceptance"])
|
||||
@pytest.mark.parametrize("case", OLD_SIDES)
|
||||
def test_old_port_out_of_service_with_new_up_is_migrated(case, acceptance):
|
||||
kind, row = OLD_SIDES[case]
|
||||
v, color, _hit = verdict(kind, row, NEW_UP, acceptance=acceptance)
|
||||
assert (v, color) == ("migrated", "green")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("acceptance", [False, True], ids=["window", "acceptance"])
|
||||
def test_new_port_phy_down_is_never_green(acceptance):
|
||||
v, color, _ = verdict("changed", {"admin": "down", "phy": "down"},
|
||||
{"admin": "up", "phy": "down"}, acceptance=acceptance)
|
||||
assert color != "green"
|
||||
|
||||
|
||||
def test_old_port_still_in_service_is_not_migrated():
|
||||
v, color, _ = verdict("unchanged", {"admin": "up", "phy": "up"}, NEW_UP, acceptance=True)
|
||||
assert color != "green"
|
||||
|
||||
|
||||
def test_legacy_form_still_requires_both_down():
|
||||
legacy = mt.preset_override_for_metric("interface_brief", legacy_port_and=True)
|
||||
v, color, _ = verdict("changed", {"admin": "up", "phy": "down"}, NEW_UP,
|
||||
acceptance=True, ov=legacy)
|
||||
assert color != "green" # the pre-A4 bug this task fixes
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _template(db, overrides):
|
||||
db.add(BizMonitorTemplate(id="m", name="m", compare_template_id="", sheet_overrides_json=overrides))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_upgrade_replaces_exact_legacy_preset_only(db):
|
||||
legacy = mt.preset_override_for_metric("interface_brief", legacy_port_and=True)
|
||||
arp = mt.preset_override_for_metric("arp")
|
||||
_template(db, [legacy, arp])
|
||||
assert mt.upgrade_legacy_port_overrides(db) == 1
|
||||
stored = db.get(BizMonitorTemplate, "m").sheet_overrides_json
|
||||
assert stored[0] == mt.preset_override_for_metric("interface_brief")
|
||||
assert stored[1] == arp
|
||||
assert mt.upgrade_legacy_port_overrides(db) == 0 # idempotent
|
||||
|
||||
|
||||
def test_customised_override_is_kept_and_warned(db):
|
||||
custom = mt.preset_override_for_metric("interface_brief", legacy_port_and=True)
|
||||
custom["down_values"] = ["down", "lowerlayerdown"]
|
||||
_template(db, [deepcopy(custom)])
|
||||
assert mt.upgrade_legacy_port_overrides(db) == 0
|
||||
assert db.get(BizMonitorTemplate, "m").sheet_overrides_json == [custom]
|
||||
db.add(BizCompareTemplate(id="ct", name="ct", metrics_json=[
|
||||
{"sheet_id": "interface_brief", "metric_id": "interface_brief",
|
||||
"key_fields": ["interface"], "compare_fields": ["admin"]}]))
|
||||
db.commit()
|
||||
warnings = mt.validate_sheet_overrides(db, compare_template_id="ct", sheet_overrides=[custom])
|
||||
assert {"sheet_id": "", "metric_id": "interface_brief",
|
||||
"reason": "port_success_legacy_and_semantics"} in warnings
|
||||
assert not any(w["reason"] == "port_success_legacy_and_semantics" for w in
|
||||
mt.validate_sheet_overrides(db, compare_template_id="ct", sheet_overrides=[
|
||||
mt.preset_override_for_metric("interface_brief")]))
|
||||
|
||||
|
||||
def test_key_tuple_is_the_single_canonical_key_form():
|
||||
from netx_api.biz_state.compare_engine import key_tuple
|
||||
|
||||
assert key_tuple({"a": None, "b": True, "c": " x "}, ["a", "b", "c", "missing"]) == ("", "true", "x", "")
|
||||
|
||||
|
||||
# ---- H8: scope_by_port_map is declared, not guessed from the metric name ----------------------
|
||||
|
||||
def _scoped(override):
|
||||
from netx_api.biz_migration.evaluate import evaluate_metric_dual, parse_expect_set
|
||||
|
||||
return evaluate_metric_dual(
|
||||
metric_id="custom_t", key_fields=["ifname"], iface_fields=["ifname"], compare_fields=[],
|
||||
old_baseline_rows=[{"ifname": "a"}], old_current_rows=[], new_baseline_rows=[],
|
||||
new_current_rows=[{"ifname": "b"}], port_map={"a": "b"}, expect=parse_expect_set({}),
|
||||
window_active=True, sheet_override=override, allow_mapping_scope=True,
|
||||
)
|
||||
|
||||
|
||||
def test_a_port_map_scopes_an_interface_sheet_unless_the_override_says_otherwise():
|
||||
assert _scoped({})["progress_total"] == 1
|
||||
assert _scoped({"scope_by_port_map": True})["progress_total"] == 1
|
||||
out = _scoped({"scope_by_port_map": False})
|
||||
assert out["progress_total"] == 0 and out["effective_scope_by_port_map"] is False
|
||||
|
||||
|
||||
def test_presets_turn_scoping_off_for_bgp_and_pseudowire_only():
|
||||
assert mt.preset_override_for_metric("bgp_peer")["scope_by_port_map"] is False
|
||||
assert mt.preset_override_for_metric("l2vpn_pw")["scope_by_port_map"] is False
|
||||
assert "scope_by_port_map" not in mt.preset_override_for_metric("interface_brief")
|
||||
assert "scope_by_port_map" not in mt.preset_override_for_metric("arp")
|
||||
|
||||
|
||||
def test_stored_overrides_get_the_old_name_based_default_written_down(db):
|
||||
_template(db, [{"metric_id": "bgp_peer", "sheet_id": "bgp_peer.vpnv4"}, {"metric_id": "arp"},
|
||||
{"metric_id": "bgp_peer", "sheet_id": "x", "scope_by_port_map": True}])
|
||||
assert mt.fill_scope_by_port_map(db) == 1
|
||||
stored = db.get(BizMonitorTemplate, "m").sheet_overrides_json
|
||||
assert stored[0]["scope_by_port_map"] is False
|
||||
assert "scope_by_port_map" not in stored[1]
|
||||
assert stored[2]["scope_by_port_map"] is True # an explicit choice is kept
|
||||
assert mt.fill_scope_by_port_map(db) == 0
|
||||
|
||||
|
||||
def test_scope_flag_must_be_boolean():
|
||||
from fastapi import HTTPException
|
||||
|
||||
from netx_api.biz_migration.validation import validate_monitor_rules
|
||||
|
||||
with pytest.raises(HTTPException):
|
||||
validate_monitor_rules({}, [{"metric_id": "arp", "scope_by_port_map": "no"}])
|
||||
validate_monitor_rules({}, [{"metric_id": "arp", "scope_by_port_map": False}])
|
||||
96
tests/test_biz_migration_retention.py
Normal file
96
tests/test_biz_migration_retention.py
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
"""H1: bounded auto-run history and run-aware batch protection."""
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_state.retention import protected_batch_map, purge_auto_runs
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareDiff, BizCompareJob, BizCompareRun, BizMigrationBatch, BizMigrationDiff, BizMigrationRedTicket,
|
||||
BizMigrationRun, BizStateBatch, BizStateTask,
|
||||
)
|
||||
from netx_api.timeutil import utcnow_naive
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _runs(db, n, *, purpose="auto", batch="wave", prefix="r"):
|
||||
base = utcnow_naive() - timedelta(hours=10)
|
||||
for i in range(n):
|
||||
db.add(BizMigrationRun(id=f"{prefix}{i}", project_id="p", batch_id=batch, purpose=purpose,
|
||||
old_batch_id=f"o-{prefix}{i}", new_batch_id=f"n-{prefix}{i}",
|
||||
created_at=base + timedelta(minutes=i), summary_json={}))
|
||||
db.add(BizMigrationDiff(id=f"d-{prefix}{i}", run_id=f"{prefix}{i}", metric_id="m"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_only_the_newest_auto_runs_survive_and_acceptance_runs_stay(db):
|
||||
_runs(db, 400)
|
||||
_runs(db, 1, purpose="acceptance", prefix="acc")
|
||||
out = purge_auto_runs(db, migration_keep=300)
|
||||
assert out["migration_runs"] == 100
|
||||
assert db.query(BizMigrationRun).count() == 301
|
||||
assert db.get(BizMigrationRun, "acc0") is not None
|
||||
assert db.get(BizMigrationRun, "r399") is not None and db.get(BizMigrationRun, "r99") is None
|
||||
assert db.query(BizMigrationDiff).count() == 301 # diffs follow their run
|
||||
|
||||
|
||||
def test_runs_pinned_by_red_tickets_or_acceptance_are_never_dropped(db):
|
||||
_runs(db, 12)
|
||||
db.add(BizMigrationRedTicket(id="t", project_id="p", batch_id="wave", run_id="r0"))
|
||||
db.add(BizMigrationBatch(id="wave", project_id="p", accept_run_id="r1"))
|
||||
db.commit()
|
||||
purge_auto_runs(db, migration_keep=5)
|
||||
assert db.get(BizMigrationRun, "r0") is not None and db.get(BizMigrationRun, "r1") is not None
|
||||
assert db.get(BizMigrationRun, "r2") is None and db.query(BizMigrationRun).count() == 7
|
||||
|
||||
|
||||
def test_keep_is_per_wave(db):
|
||||
_runs(db, 6, batch="w1", prefix="a")
|
||||
_runs(db, 6, batch="w2", prefix="b")
|
||||
assert purge_auto_runs(db, migration_keep=4)["migration_runs"] == 4
|
||||
assert db.query(BizMigrationRun).filter_by(batch_id="w1").count() == 4
|
||||
|
||||
|
||||
def test_compare_keeps_manual_runs_and_trims_auto_runs_of_auto_jobs(db):
|
||||
db.add(BizCompareJob(id="job", name="j", mode="auto"))
|
||||
base = utcnow_naive() - timedelta(hours=5)
|
||||
for i in range(8):
|
||||
db.add(BizCompareRun(id=f"c{i}", job_id="job", status="success", created_at=base + timedelta(minutes=i),
|
||||
summary_json={"purpose": "auto" if i != 0 else "manual"}))
|
||||
db.add(BizCompareDiff(id=f"cd{i}", run_id=f"c{i}", metric_id="m"))
|
||||
db.commit()
|
||||
out = purge_auto_runs(db, compare_keep=3)
|
||||
assert out["compare_runs"] == 4 # c1..c4; c0 is manual and survives, c5..c7 are the newest
|
||||
assert {r.id for r in db.query(BizCompareRun)} == {"c0", "c5", "c6", "c7"}
|
||||
assert db.query(BizCompareDiff).count() == 4
|
||||
|
||||
|
||||
def _batch(db, bid, task="t"):
|
||||
db.add(BizStateBatch(id=bid, task_id=task, status="success"))
|
||||
|
||||
|
||||
def test_protection_follows_surviving_runs_and_all_metric_batches(db):
|
||||
db.add(BizStateTask(id="t", source="managed", ne_id="n"))
|
||||
for bid in ("o-r0", "n-r0", "hf-a", "hf-b", "o-r1", "n-r1", "o-acc0", "n-acc0"):
|
||||
_batch(db, bid)
|
||||
db.commit()
|
||||
_runs(db, 2)
|
||||
_runs(db, 1, purpose="acceptance", prefix="acc")
|
||||
acc = db.get(BizMigrationRun, "acc0")
|
||||
acc.summary_json = {"metric_batches": {"arp": {"old_batch_id": "hf-a", "new_batch_id": "hf-b"}}}
|
||||
db.commit()
|
||||
reasons = protected_batch_map(db)
|
||||
assert "hf-a" in reasons and "hf-b" in reasons # beyond the first binding
|
||||
purge_auto_runs(db, migration_keep=1) # r0 goes away
|
||||
assert "o-r0" not in protected_batch_map(db) and "o-r1" in protected_batch_map(db)
|
||||
|
|
@ -95,7 +95,8 @@ class BizStateClaimTests(unittest.TestCase):
|
|||
self.assertEqual(r2.get("reason"), "already_collecting")
|
||||
|
||||
def test_claim_ne_mutex_skips_same_ne(self) -> None:
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=10):
|
||||
# Session cap 1 reproduces the classic "one collection per NE" mutex.
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=10), patch.object(claim_mod, "max_sessions_per_ne", return_value=1):
|
||||
ra = claim_mod.enqueue_collect("ta", manual=True)
|
||||
rb = claim_mod.enqueue_collect("tb", manual=True)
|
||||
rc = claim_mod.enqueue_collect("tc", manual=True)
|
||||
|
|
@ -111,6 +112,47 @@ class BizStateClaimTests(unittest.TestCase):
|
|||
self.assertEqual(self.db.get(BizStateBatch, rb["batch_id"]).status, "queued")
|
||||
self.assertEqual(self.db.get(BizStateBatch, rc["batch_id"]).status, "running")
|
||||
|
||||
def test_a_ne_runs_up_to_its_session_cap_and_no_more(self) -> None:
|
||||
self.db.add_all([BizStateTask(id=f"x{i}", source="managed", ne_id="ne-a", status="running") for i in range(3)])
|
||||
for i in range(3):
|
||||
self.db.add(BizStateTaskItem(id=uuid4().hex, task_id=f"x{i}", source_profile_id="zte.arp",
|
||||
kind="catalog", enabled=True))
|
||||
self.db.commit()
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=10), patch.object(claim_mod, "max_sessions_per_ne", return_value=2):
|
||||
queued = [claim_mod.enqueue_collect(f"x{i}", manual=True)["batch_id"] for i in range(3)]
|
||||
first = claim_mod.claim_queued_batches(10)
|
||||
again = claim_mod.claim_queued_batches(10)
|
||||
self.assertEqual(len(first), 2) # three batches queued for ne-a, only two sessions allowed
|
||||
self.assertEqual(again, [])
|
||||
self.db.expire_all()
|
||||
states = sorted(self.db.get(BizStateBatch, b).status for b in queued)
|
||||
self.assertEqual(states, ["queued", "running", "running"])
|
||||
|
||||
def test_default_session_cap_is_two(self) -> None:
|
||||
self.assertEqual(claim_mod.max_sessions_per_ne(), 2)
|
||||
|
||||
def test_single_task_group_is_a_valid_round_for_a_same_device_cutover(self) -> None:
|
||||
task = self.db.get(BizStateTask, "ta")
|
||||
task.collect_group_id = "solo"
|
||||
self.db.commit()
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=10):
|
||||
r = claim_mod.enqueue_collect("ta", manual=True)
|
||||
self.assertTrue(r["queued"], r)
|
||||
claimed = claim_mod.claim_queued_batches(10)
|
||||
self.assertEqual([c["task_id"] for c in claimed], ["ta"])
|
||||
|
||||
def test_a_pair_needs_both_sides_before_it_is_claimed(self) -> None:
|
||||
for tid, group in (("ta", "pair"), ("tc", "pair")):
|
||||
self.db.get(BizStateTask, tid).collect_group_id = group
|
||||
self.db.commit()
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=10):
|
||||
r = claim_mod.enqueue_collect("ta", manual=True)
|
||||
self.assertTrue(r["queued"], r)
|
||||
self.db.expire_all()
|
||||
self.db.query(BizStateBatch).filter(BizStateBatch.task_id == "tc").delete()
|
||||
self.db.commit()
|
||||
self.assertEqual(claim_mod.claim_queued_batches(10), [])
|
||||
|
||||
def test_claim_respects_slot_ceiling(self) -> None:
|
||||
with patch.object(claim_mod, "max_concurrent_tasks", return_value=1):
|
||||
claim_mod.enqueue_collect("ta", manual=True)
|
||||
|
|
|
|||
|
|
@ -90,6 +90,33 @@ class CompareSnapshotTests(unittest.TestCase):
|
|||
self.assertEqual(detail["sheets"][0]["compare_fields"], ["state"])
|
||||
self.assertEqual(self.run.summary_json["config_snapshot"], original)
|
||||
|
||||
def _use_raw_port_map(self, version: int):
|
||||
# Map written exactly as the device prints it (not the normalized form).
|
||||
self.pair.before_if, self.pair.after_if = "GE-1", "GE-2"
|
||||
self.db.commit()
|
||||
self.run = _create_running_run(
|
||||
self.db, job=self.job, tpl=self.tpl, before_batch_id="before",
|
||||
after_batch_id="after", sheets_cfg=template_metrics(self.tpl),
|
||||
)
|
||||
snap = deepcopy(self.run.summary_json)
|
||||
snap["config_snapshot"]["version"] = version
|
||||
self.run.summary_json = snap
|
||||
self.db.commit()
|
||||
|
||||
def test_v2_run_normalizes_raw_port_map_names(self):
|
||||
self._use_raw_port_map(2)
|
||||
detail = self.execute()
|
||||
self.assertEqual(detail["summary"]["unchanged"], 1)
|
||||
self.assertEqual(detail["summary"]["added"], 0)
|
||||
self.assertEqual(detail["summary"]["removed"], 0)
|
||||
|
||||
def test_v1_run_keeps_legacy_port_map_matching(self):
|
||||
self._use_raw_port_map(1)
|
||||
detail = self.execute()
|
||||
self.assertEqual(detail["summary"]["unchanged"], 0)
|
||||
self.assertEqual(detail["summary"]["added"], 1)
|
||||
self.assertEqual(detail["summary"]["removed"], 1)
|
||||
|
||||
def test_compact_hydration_restores_original_normalized_values(self):
|
||||
self.execute()
|
||||
self.edit_current_settings()
|
||||
|
|
@ -98,6 +125,15 @@ class CompareSnapshotTests(unittest.TestCase):
|
|||
self.assertEqual(diff["mapped_before"]["interface"], "gei-2.100")
|
||||
self.assertEqual(diff["after"]["interface"], "gei-2.100")
|
||||
|
||||
def test_diff_rows_expose_the_before_side_name_as_printed(self):
|
||||
self.execute()
|
||||
diff = list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged")["items"][0]
|
||||
self.assertEqual(diff["key"]["interface"], "gei-2.100")
|
||||
self.assertEqual(diff["raw_key_before"], {"id": "1", "interface": "GE-1.100"})
|
||||
with zipfile.ZipFile(io.BytesIO(export_run_zip(self.db, self.run.id))) as archive:
|
||||
rows = list(csv.DictReader(io.StringIO(archive.read("tables/sheet.csv").decode("utf-8-sig"))))
|
||||
self.assertEqual(rows[0]["interface__raw_before"], "GE-1.100")
|
||||
|
||||
def test_live_search_does_not_inherit_new_filters_or_transforms(self):
|
||||
with patch("netx_api.biz_state.compare_service._UNCHANGED_SAMPLE_MAX", 0):
|
||||
self.execute()
|
||||
|
|
@ -132,7 +168,7 @@ class CompareSnapshotTests(unittest.TestCase):
|
|||
self.edit_current_settings()
|
||||
detail = get_run(self.db, self.run.id)
|
||||
self.assertEqual(detail["job_name"], "Original job")
|
||||
self.assertEqual(detail["config_snapshot_version"], 1)
|
||||
self.assertEqual(detail["config_snapshot_version"], 2)
|
||||
self.assertEqual(detail["template"]["iface_normalize_rules"], [{"from": "GE", "to": "gei"}])
|
||||
self.assertNotIn("config_snapshot", detail["summary"])
|
||||
detail["template"]["iface_normalize_rules"].clear()
|
||||
|
|
|
|||
|
|
@ -65,7 +65,9 @@ def test_pair_claim_requires_two_slots(db):
|
|||
assert len(claim.claim_queued_batches(2)) == 2
|
||||
|
||||
|
||||
def test_busy_device_keeps_both_queued_and_no_ssh_overlap(db):
|
||||
def test_busy_device_keeps_both_queued_and_no_ssh_overlap(db, monkeypatch):
|
||||
# These cases are about one collection per device; the cap defaults to 2.
|
||||
monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1)
|
||||
db.add(BizStateBatch(id="portrait", task_id="other", ne_id="old", status="running"))
|
||||
db.commit()
|
||||
claim.enqueue_collect("old")
|
||||
|
|
@ -125,7 +127,9 @@ def test_inline_pair_uses_two_workers_without_waiting_for_one_side(db, monkeypat
|
|||
assert set(synchronized) == {"old", "new"}
|
||||
|
||||
|
||||
def test_faster_metrics_claim_before_manual_route_round(db):
|
||||
def test_faster_metrics_claim_before_manual_route_round(db, monkeypatch):
|
||||
# These cases are about one collection per device; the cap defaults to 2.
|
||||
monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1)
|
||||
for tid in ("ro", "rn"):
|
||||
db.add(BizStateTask(id=tid, ne_id="old" if tid == "ro" else "new",
|
||||
status="paused", collect_group_id="routes", collect_manual_only=True))
|
||||
|
|
@ -137,7 +141,9 @@ def test_faster_metrics_claim_before_manual_route_round(db):
|
|||
assert {job["task_id"] for job in jobs} == {"old", "new"}
|
||||
|
||||
|
||||
def test_manual_route_round_cannot_starve_behind_continuous_fast_metrics(db):
|
||||
def test_manual_route_round_cannot_starve_behind_continuous_fast_metrics(db, monkeypatch):
|
||||
# These cases are about one collection per device; the cap defaults to 2.
|
||||
monkeypatch.setattr(claim, "max_sessions_per_ne", lambda: 1)
|
||||
for tid in ("ro", "rn"):
|
||||
db.add(BizStateTask(id=tid, ne_id="old" if tid == "ro" else "new",
|
||||
status="paused", collect_group_id="routes", collect_manual_only=True))
|
||||
|
|
|
|||
167
tests/test_compare_preflight.py
Normal file
167
tests/test_compare_preflight.py
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
"""Mapping pre-flight (B5), N:1 reporting (B7) and command-slice coverage (V2)."""
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_migration import service as mig
|
||||
from netx_api.biz_state import compare_service as svc
|
||||
from netx_api.biz_state.compare_quality import classify_sheet, covers_slice, sheet_slice
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareTemplate, BizMigrationProject, BizMonitorTemplate, BizPortMapping, BizPortMappingRow,
|
||||
BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
SHEET = {"sheet_id": "ifb", "metric_id": "iface_t", "key_fields": ["interface"], "iface_fields": ["interface"],
|
||||
"compare_fields": ["state"], "field_rules": [], "row_filters": []}
|
||||
|
||||
|
||||
def _batches(db, before_ifaces, after_ifaces, *, receipts=True):
|
||||
db.add_all([BizStateTask(id="t1", source="managed", ne_id="a", vendor="ZTE"),
|
||||
BizStateTask(id="t2", source="managed", ne_id="b", vendor="ZTE"),
|
||||
BizStateBatch(id="b1", task_id="t1", status="success"),
|
||||
BizStateBatch(id="b2", task_id="t2", status="success"),
|
||||
BizCompareTemplate(id="tpl", name="t", metrics_json=[SHEET],
|
||||
iface_normalize_json=[{"from": "GE", "to": "gei"}])])
|
||||
for batch, names in (("b1", before_ifaces), ("b2", after_ifaces)):
|
||||
for i, name in enumerate(names):
|
||||
db.add(BizStateMetricRow(id=f"{batch}-{i}", batch_id=batch, metric_id="iface_t", seq=i,
|
||||
data_json={"interface": name, "state": "up"}))
|
||||
if receipts:
|
||||
db.add(BizStateBatchCommand(id=f"c-{batch}", batch_id=batch, metric_id="iface_t",
|
||||
profile_id="zte.x", parse_status="ok", row_count=len(names),
|
||||
raw_line_count=len(names) + 3))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _preflight(db, **extra):
|
||||
return svc.preflight_compare(db, {"template_id": "tpl", "before_batch_id": "b1", "after_batch_id": "b2", **extra})
|
||||
|
||||
|
||||
def test_exact_parent_same_name_and_suspect_are_classified(db):
|
||||
_batches(db, ["GE0/1/0/1", "GE0/1/0/2.100", "GE0/1/0/3", "GE0/1/0/9"],
|
||||
["gei0/2/0/1", "gei0/2/0/2.100", "gei0/1/0/3"])
|
||||
report = _preflight(db, mapping_rows=[{"before_if": "GE0/1/0/1", "after_if": "GE0/2/0/1"},
|
||||
{"before_if": "GE0/1/0/2", "after_if": "GE0/2/0/2"}])
|
||||
(sheet,) = report["sheets"]
|
||||
assert sheet["mapped_exact"] == ["gei0/1/0/1"]
|
||||
assert sheet["mapped_by_parent"] == ["gei0/1/0/2.100"]
|
||||
assert sheet["identity_same_name"] == ["gei0/1/0/3"]
|
||||
assert sheet["suspect_unmapped"] == ["gei0/1/0/9"]
|
||||
assert report["coverage"] == {"mapped": 2, "suspect_unmapped": 1, "identity_same_name": 1, "rate": 66.7}
|
||||
assert report["verdict"] == "risk" and "suspect_unmapped" in report["risk_reasons"]
|
||||
|
||||
|
||||
def test_unsaved_rows_are_enough_and_clean_map_is_ready(db):
|
||||
_batches(db, ["GE0/1"], ["gei0/2"])
|
||||
report = _preflight(db, mapping_rows=[{"before_if": "GE0/1", "after_if": "GE0/2"}])
|
||||
assert report["verdict"] == "ready" and report["coverage"]["rate"] == 100.0
|
||||
assert report["mapping"]["unused_map_keys"] == []
|
||||
|
||||
|
||||
def test_unused_keys_missing_targets_and_n_to_1_are_listed(db):
|
||||
_batches(db, ["GE0/1", "GE0/2"], ["gei0/9"])
|
||||
report = _preflight(db, mapping_rows=[
|
||||
{"before_if": "GE0/1", "after_if": "GE0/9"}, {"before_if": "GE0/2", "after_if": "GE0/9"},
|
||||
{"before_if": "GE0/77", "after_if": "GE0/88"}])
|
||||
m = report["mapping"]
|
||||
assert m["n_to_1_targets"] == [{"target": "gei0/9", "sources": ["gei0/1", "gei0/2"]}]
|
||||
assert m["unused_map_keys"] == ["gei0/77"]
|
||||
assert m["map_targets_missing_on_after"] == ["gei0/88"]
|
||||
assert report["verdict"] == "risk"
|
||||
|
||||
|
||||
def test_duplicate_before_rows_are_reported(db):
|
||||
_batches(db, ["GE0/1"], ["gei0/2"])
|
||||
report = _preflight(db, mapping_rows=[{"before_if": "GE0/1", "after_if": "GE0/2"},
|
||||
{"before_if": "GE0/1", "after_if": "GE0/3"}])
|
||||
assert report["mapping"]["duplicate_before"][0]["dropped_to"] == "GE0/3"
|
||||
assert report["verdict"] == "risk"
|
||||
|
||||
|
||||
def test_saved_mapping_id_is_used_when_no_rows_given(db):
|
||||
_batches(db, ["GE0/1"], ["gei0/2"])
|
||||
db.add_all([BizPortMapping(id="m", name="m"),
|
||||
BizPortMappingRow(mapping_id="m", before_if="GE0/1", after_if="GE0/2")])
|
||||
db.commit()
|
||||
assert _preflight(db, mapping_id="m")["coverage"]["mapped"] == 1
|
||||
|
||||
|
||||
def test_missing_batch_is_404(db):
|
||||
_batches(db, ["GE0/1"], ["gei0/2"])
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc.preflight_compare(db, {"template_id": "tpl", "before_batch_id": "nope", "after_batch_id": "b2"})
|
||||
assert exc.value.status_code == 404
|
||||
|
||||
|
||||
def test_n_to_1_is_rejected_by_monitoring_with_the_colliding_rows():
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
mig._reject_ambiguous_mapping({"a": "x", "b": "x", "c": "y"})
|
||||
assert exc.value.status_code == 400
|
||||
assert exc.value.detail["code"] == "mapping_target_ambiguous"
|
||||
assert exc.value.detail["n_to_1"] == [{"target": "x", "sources": ["a", "b"]}]
|
||||
assert "a, b -> x" in exc.value.detail["message"]
|
||||
mig._reject_ambiguous_mapping({"a": "x", "b": "y"})
|
||||
|
||||
|
||||
def test_project_preflight_uses_the_portrait_baselines(db):
|
||||
_batches(db, ["GE0/1", "GE0/5"], ["gei0/2"])
|
||||
db.add_all([BizPortMapping(id="m", name="m"),
|
||||
BizPortMappingRow(mapping_id="m", before_if="GE0/1", after_if="GE0/2"),
|
||||
BizMonitorTemplate(id="mt", name="mt", compare_template_id="tpl"),
|
||||
BizMigrationProject(id="p", name="p", mapping_id="m", monitor_template_id="mt", old_task_id="t1",
|
||||
new_task_id="t2", old_baseline_batch_id="b1", new_baseline_batch_id="b2")])
|
||||
db.commit()
|
||||
report = mig.preflight_project(db, "p")
|
||||
assert report["sheets"][0]["suspect_unmapped"] == ["gei0/5"]
|
||||
assert report["verdict"] == "risk"
|
||||
|
||||
|
||||
# ---- V2: slices ---------------------------------------------------------------------------
|
||||
|
||||
def receipts(*profile_ids, status="ok"):
|
||||
return [{"profile_id": p, "parse_status": status, "row_count": 1, "raw_line_count": 4} for p in profile_ids]
|
||||
|
||||
|
||||
def test_sheet_slice_reads_only_top_level_eq_filters():
|
||||
s = {"row_filters": [{"op": "eq", "field": "afi", "value": "IPv4"},
|
||||
{"op": "contains", "field": "x", "value": "y"},
|
||||
{"any": [{"op": "eq", "field": "z", "value": "1"}]}]}
|
||||
assert sheet_slice(s) == {"afi": "ipv4"}
|
||||
|
||||
|
||||
def test_missing_afi_command_means_not_collected_not_lost():
|
||||
only_vpnv4 = receipts("zte.bgp_vpnv4_summary")
|
||||
both = receipts("zte.bgp_vpnv4_summary", "zte.bgp_ipv4_summary")
|
||||
state, reasons = classify_sheet(before=both, after=only_vpnv4, before_rows=3, after_rows=0,
|
||||
wanted_slice={"afi": "ipv4"})
|
||||
assert state == "not_collected" and reasons == ["after:slice_not_collected"]
|
||||
ok, _ = classify_sheet(before=both, after=both, before_rows=3, after_rows=3, wanted_slice={"afi": "ipv4"})
|
||||
assert ok == "ok"
|
||||
|
||||
|
||||
def test_command_without_declared_slice_covers_everything():
|
||||
assert covers_slice(receipts("custom.unknown"), {"afi": "ipv4"})
|
||||
assert not covers_slice(receipts("zte.bgp_vpnv6_summary"), {"afi": "ipv4"})
|
||||
assert not covers_slice(receipts("zte.bgp_ipv4_summary", status="failed"), {"afi": "ipv4"})
|
||||
|
||||
|
||||
def test_command_coverage_compares_what_commands_produce(db):
|
||||
_batches(db, ["GE0/1"], ["gei0/2"])
|
||||
db.add(BizStateBatchCommand(id="x", batch_id="b1", metric_id="bgp_peer", profile_id="zte.bgp_ipv4_summary",
|
||||
parse_status="ok"))
|
||||
db.commit()
|
||||
cov = _preflight(db, mapping_rows=[])["command_coverage"]
|
||||
assert cov["only_before"] == ["bgp_peer{afi=ipv4}"]
|
||||
163
tests/test_compare_quality.py
Normal file
163
tests/test_compare_quality.py
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
"""Sheet data quality (V1) and directional compare modes (V3)."""
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_state import compare_service as svc
|
||||
from netx_api.biz_state.compare_rules import explain_diff, values_equal
|
||||
from netx_api.biz_state.identity_pipeline import build_pipeline
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask
|
||||
|
||||
SHEET = {"sheet_id": "peers", "metric_id": "bgp_peer", "key_fields": ["neighbor"], "iface_fields": [],
|
||||
"compare_fields": ["state"], "field_rules": [], "row_filters": []}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
session.add_all([BizStateTask(id="t", source="managed", ne_id="n"),
|
||||
BizStateBatch(id="b1", task_id="t", status="success"),
|
||||
BizStateBatch(id="b2", task_id="t", status="success")])
|
||||
session.commit()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def collect(db, batch, rows, *, lines=None, status="ok", command=True):
|
||||
if command:
|
||||
db.add(BizStateBatchCommand(id=f"c-{batch}", batch_id=batch, metric_id="bgp_peer", profile_id="zte.x",
|
||||
parse_status=status, row_count=len(rows),
|
||||
raw_line_count=len(rows) + 2 if lines is None else lines))
|
||||
for i, peer in enumerate(rows):
|
||||
db.add(BizStateMetricRow(id=f"{batch}-{i}", batch_id=batch, metric_id="bgp_peer", seq=i,
|
||||
data_json={"neighbor": peer, "state": "up"}))
|
||||
db.commit()
|
||||
|
||||
|
||||
def run(db):
|
||||
return svc._run_sheet(db, sheet=SHEET, before_batch_id="b1", after_batch_id="b2",
|
||||
pipeline=build_pipeline(SHEET, iface_rules=[], port_map={}))["summary"]
|
||||
|
||||
|
||||
def enrich(summary):
|
||||
sheet = {"sheet_id": "peers", "metric_id": "bgp_peer", "status": "done", "summary": summary}
|
||||
return svc._enrich_summary({k: summary.get(k, 0) for k in
|
||||
("added", "removed", "changed", "unchanged", "before_count", "after_count")}, [sheet])
|
||||
|
||||
|
||||
def test_empty_against_empty_is_no_data_not_a_hundred_percent(db):
|
||||
collect(db, "b1", [], lines=0)
|
||||
collect(db, "b2", [], lines=0)
|
||||
summary = run(db)
|
||||
assert summary["data_state"] == "no_data"
|
||||
out = enrich(summary)
|
||||
assert out["pass_rate"] is None and out["sheet_cards"][0]["pass_rate"] is None
|
||||
assert out["ok"] is None
|
||||
|
||||
|
||||
def test_unparsed_output_is_suspect_parse_and_leaves_the_verdict_open(db):
|
||||
collect(db, "b1", [f"p{i}" for i in range(10)])
|
||||
collect(db, "b2", [], lines=40) # 40 CLI lines, 0 rows parsed
|
||||
summary = run(db)
|
||||
assert summary["data_state"] == "suspect_parse"
|
||||
assert summary["data_state_reasons"] == ["after:parsed_0_of_40_lines"]
|
||||
out = enrich(summary)
|
||||
assert out["sheet_cards"][0]["removed"] == 10 # still shown
|
||||
assert out["ok"] is None and out["suspect_sheets"] == 1
|
||||
|
||||
|
||||
def test_side_without_any_command_is_not_collected_and_leaves_the_totals(db):
|
||||
collect(db, "b1", ["a", "b"])
|
||||
db.add(BizStateBatchCommand(id="other", batch_id="b2", metric_id="other_metric", profile_id="zte.y",
|
||||
parse_status="ok", row_count=1, raw_line_count=1))
|
||||
db.commit()
|
||||
summary = run(db)
|
||||
assert summary["data_state"] == "not_collected"
|
||||
out = enrich(summary)
|
||||
assert out["sheet_cards"][0]["pass_rate"] is None
|
||||
assert out["fail_count"] == 0 and out["ok"] is None and out["unjudged_sheets"] == 1
|
||||
|
||||
|
||||
def test_batches_without_receipts_are_trusted(db):
|
||||
collect(db, "b1", ["a"], command=False)
|
||||
collect(db, "b2", ["a"], command=False)
|
||||
summary = run(db)
|
||||
assert summary["data_state"] == "ok"
|
||||
assert enrich(summary)["pass_rate"] == 100.0
|
||||
|
||||
|
||||
def test_normal_comparison_keeps_the_verdict(db):
|
||||
collect(db, "b1", ["a", "b"])
|
||||
collect(db, "b2", ["a"])
|
||||
out = enrich(run(db))
|
||||
assert out["ok"] is False and out["fail_count"] == 1 and out["pass_rate"] == 50.0
|
||||
|
||||
|
||||
# ---- V3 -----------------------------------------------------------------------------------
|
||||
|
||||
NOT_WORSE = {"compare": "not_worse", "rank": ["up", "down"]}
|
||||
|
||||
|
||||
def test_not_worse_accepts_recovery_and_rejects_degradation():
|
||||
assert values_equal("down", "up", rule=NOT_WORSE)
|
||||
assert values_equal("up", "up", rule=NOT_WORSE)
|
||||
assert not values_equal("up", "down", rule=NOT_WORSE)
|
||||
assert explain_diff("up", "down", rule=NOT_WORSE) == "worse: up→down"
|
||||
|
||||
|
||||
def test_not_worse_falls_back_to_equality_for_unranked_values():
|
||||
assert not values_equal("up", "flapping", rule=NOT_WORSE)
|
||||
assert values_equal("flapping", "flapping", rule=NOT_WORSE)
|
||||
|
||||
|
||||
def test_numeric_min_allows_growth_and_bounded_loss():
|
||||
rule = {"compare": "numeric_min", "tolerance": 5}
|
||||
assert values_equal("1000", "1200", rule=rule)
|
||||
assert values_equal("1000", "960", rule=rule)
|
||||
assert not values_equal("1000", "900", rule=rule)
|
||||
assert explain_diff("1000", "900", rule=rule) == "drop 10% > 5%"
|
||||
|
||||
|
||||
def test_numeric_max_is_the_mirror_image():
|
||||
rule = {"compare": "numeric_max", "tolerance": 10}
|
||||
assert values_equal("100", "60", rule=rule)
|
||||
assert values_equal("100", "110", rule=rule)
|
||||
assert not values_equal("100", "150", rule=rule)
|
||||
assert explain_diff("100", "150", rule=rule) == "rise 50% > 10%"
|
||||
|
||||
|
||||
def test_directional_modes_pass_validation_and_keep_their_rank():
|
||||
from netx_api.biz_state.compare_validation import validate_template_body
|
||||
|
||||
sheet = {"metric_id": "m", "key_fields": ["k"], "compare_fields": ["s"],
|
||||
"field_rules": [{"field": "s", "compare": "not_worse", "rank": ["up", "down"]}]}
|
||||
validate_template_body({"metrics": [sheet]})
|
||||
normalized = svc._normalize_sheet(sheet)
|
||||
assert normalized["field_rules"][0]["rank"] == ["up", "down"]
|
||||
with pytest.raises(Exception):
|
||||
validate_template_body({"metrics": [{**sheet, "field_rules": [{"field": "s", "compare": "not_worse"}]}]})
|
||||
|
||||
|
||||
def test_directional_compare_runs_end_to_end_and_blocks_sql(db):
|
||||
from tests.test_identity_pipeline import pg_db
|
||||
from netx_api.biz_state.compare_engine import compare_rows
|
||||
from netx_api.biz_state.compare_sql import sql_compare_skip_reason
|
||||
|
||||
sheet = {**SHEET, "field_rules": [{"field": "state", **NOT_WORSE}]}
|
||||
out = compare_rows(before_rows=[{"neighbor": "a", "state": "down"}, {"neighbor": "b", "state": "up"}],
|
||||
after_rows=[{"neighbor": "a", "state": "up"}, {"neighbor": "b", "state": "down"}],
|
||||
key_fields=["neighbor"], iface_fields=[], compare_fields=["state"],
|
||||
field_rules=sheet["field_rules"])
|
||||
assert (out["summary"]["unchanged"], out["summary"]["changed"]) == (1, 1)
|
||||
assert sql_compare_skip_reason(pg_db(), sheet) == "directional_compare"
|
||||
|
||||
|
||||
def test_sorted_pairing_is_the_default_for_new_runs_only():
|
||||
assert build_pipeline(SHEET, iface_rules=[], port_map={}).duplicate_pairing == "sorted"
|
||||
assert build_pipeline({**SHEET, "duplicate_pairing": "ordered"}, iface_rules=[],
|
||||
port_map={}).duplicate_pairing == "ordered"
|
||||
assert build_pipeline(SHEET, iface_rules=[], port_map={}, legacy=True).duplicate_pairing == "ordered"
|
||||
204
tests/test_cutover_references.py
Normal file
204
tests/test_cutover_references.py
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
"""Reference protection and input validation for compare/cutover (plan A2, A5)."""
|
||||
import pytest
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_migration import service as mig
|
||||
from netx_api.biz_state import compare_service as svc
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareJob, BizCompareTemplate, BizMigrationProject, BizMonitorTemplate,
|
||||
BizPortMapping, BizPortMappingRow, BizStateBatch, BizStateTask,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _template(db, tid="tpl"):
|
||||
db.add(BizCompareTemplate(id=tid, name=f"name-{tid}", metrics_json=[]))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _mapping(db, mid="map"):
|
||||
db.add(BizPortMapping(id=mid, name=f"map-{mid}"))
|
||||
db.add(BizPortMappingRow(mapping_id=mid, before_if="a", after_if="b"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_mapping_used_by_job_cannot_be_deleted(db):
|
||||
_mapping(db)
|
||||
db.add(BizCompareJob(id="j", name="job-1", template_id="tpl", mapping_id="map"))
|
||||
db.commit()
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc.delete_mapping(db, "map")
|
||||
assert exc.value.status_code == 409
|
||||
assert exc.value.detail["code"] == "mapping_in_use"
|
||||
assert exc.value.detail["jobs"] == ["job-1"]
|
||||
assert db.get(BizPortMapping, "map") is not None
|
||||
|
||||
|
||||
def test_mapping_used_by_project_cannot_be_deleted(db):
|
||||
_mapping(db)
|
||||
db.add(BizMigrationProject(id="p", name="proj-1", mapping_id="map"))
|
||||
db.commit()
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc.delete_mapping(db, "map")
|
||||
assert exc.value.status_code == 409
|
||||
assert exc.value.detail["projects"] == ["proj-1"]
|
||||
assert "proj-1" in exc.value.detail["message"]
|
||||
|
||||
|
||||
def test_unreferenced_mapping_is_deleted_with_rows(db):
|
||||
_mapping(db)
|
||||
svc.delete_mapping(db, "map")
|
||||
assert db.get(BizPortMapping, "map") is None
|
||||
assert db.query(BizPortMappingRow).count() == 0
|
||||
|
||||
|
||||
def test_list_mappings_reports_reference_counts(db):
|
||||
_mapping(db)
|
||||
db.add_all([BizCompareJob(id="j1", template_id="tpl", mapping_id="map"),
|
||||
BizCompareJob(id="j2", template_id="tpl", mapping_id="map"),
|
||||
BizMigrationProject(id="p", name="p", mapping_id="map")])
|
||||
db.commit()
|
||||
(item,) = svc.list_mappings(db)
|
||||
assert (item["job_count"], item["project_count"]) == (2, 1)
|
||||
|
||||
|
||||
def test_template_used_by_monitor_template_cannot_be_deleted(db):
|
||||
_template(db)
|
||||
db.add(BizMonitorTemplate(id="mt", name="monitor-1", compare_template_id="tpl"))
|
||||
db.commit()
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc.delete_template(db, "tpl")
|
||||
assert exc.value.status_code == 409
|
||||
assert exc.value.detail["code"] == "template_in_use"
|
||||
assert exc.value.detail["monitor_templates"] == ["monitor-1"]
|
||||
assert exc.value.detail["jobs"] == []
|
||||
|
||||
|
||||
def test_template_used_by_job_reports_job_names(db):
|
||||
_template(db)
|
||||
db.add(BizCompareJob(id="j", name="job-1", template_id="tpl"))
|
||||
db.commit()
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc.delete_template(db, "tpl")
|
||||
assert exc.value.detail["jobs"] == ["job-1"]
|
||||
|
||||
|
||||
def test_list_templates_counts_monitor_templates(db, monkeypatch):
|
||||
monkeypatch.setattr(svc, "ensure_default_templates", lambda db: None)
|
||||
monkeypatch.setattr(svc, "upgrade_builtin_split_sheets", lambda db: None)
|
||||
_template(db)
|
||||
db.add_all([BizMonitorTemplate(id="m1", compare_template_id="tpl"),
|
||||
BizMonitorTemplate(id="m2", compare_template_id="tpl")])
|
||||
db.commit()
|
||||
(item,) = svc.list_templates(db)
|
||||
assert item["monitor_template_count"] == 2
|
||||
|
||||
|
||||
def test_job_with_deleted_mapping_is_rejected_not_silently_unmapped(db):
|
||||
_template(db)
|
||||
db.add(BizCompareJob(id="j", template_id="tpl", mapping_id="gone",
|
||||
before_batch_id="b", after_batch_id="a"))
|
||||
db.commit()
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
svc._validate_compare_job(db, "j")
|
||||
assert exc.value.status_code == 400
|
||||
assert exc.value.detail == "mapping_missing"
|
||||
|
||||
|
||||
def _project_tasks(db):
|
||||
db.add_all([BizStateTask(id="old_p", source="managed", ne_id="ne-old", status="running"),
|
||||
BizStateTask(id="new_p", source="managed", ne_id="ne-new", status="running"),
|
||||
BizStateTask(id="other", source="managed", ne_id="ne-x", status="running")])
|
||||
db.add_all([BizStateBatch(id="old_ok", task_id="old_p", status="success"),
|
||||
BizStateBatch(id="new_ok", task_id="new_p", status="partial"),
|
||||
BizStateBatch(id="foreign", task_id="other", status="success"),
|
||||
BizStateBatch(id="old_failed", task_id="old_p", status="failed")])
|
||||
db.commit()
|
||||
|
||||
|
||||
def _body(**extra):
|
||||
return {"name": "n", "old_task_id": "old_p", "new_task_id": "new_p", **extra}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("extra,code", [
|
||||
({"old_baseline_batch_id": "foreign"}, "old_baseline_invalid"),
|
||||
({"new_baseline_batch_id": "foreign"}, "new_baseline_invalid"),
|
||||
({"old_baseline_batch_id": "old_failed"}, "old_baseline_invalid"),
|
||||
({"new_baseline_batch_id": "old_ok"}, "new_baseline_invalid"),
|
||||
])
|
||||
def test_create_project_rejects_baseline_from_wrong_task(db, extra, code):
|
||||
_project_tasks(db)
|
||||
with pytest.raises(HTTPException) as exc:
|
||||
mig.create_project(db, _body(**extra))
|
||||
assert exc.value.status_code == 400
|
||||
assert exc.value.detail == code
|
||||
assert db.query(BizMigrationProject).count() == 0
|
||||
|
||||
|
||||
def test_baseline_ownership_accepts_success_and_partial(db):
|
||||
_project_tasks(db)
|
||||
mig._validate_baseline_ownership(db, "old", "old_ok", "old_p")
|
||||
mig._validate_baseline_ownership(db, "new", "new_ok", "new_p")
|
||||
mig._validate_baseline_ownership(db, "old", "", "old_p")
|
||||
|
||||
|
||||
# ---- H9: reads do not write; project lists do not fan out --------------------------------------
|
||||
|
||||
def _project(db, n, **extra):
|
||||
db.add(BizStateTask(id=f"to{n}", source="managed", ne_id=f"o{n}", status="running"))
|
||||
db.add(BizStateTask(id=f"tn{n}", source="managed", ne_id=f"n{n}", status="running"))
|
||||
db.add(BizStateBatch(id=f"bo{n}", task_id=f"to{n}", status="success"))
|
||||
db.add(BizStateBatch(id=f"bn{n}", task_id=f"tn{n}", status="success"))
|
||||
db.add(BizMigrationProject(id=f"p{n}", name=f"p{n}", old_task_id=f"to{n}", new_task_id=f"tn{n}",
|
||||
old_baseline_batch_id=f"bo{n}", new_baseline_batch_id=f"bn{n}", **extra))
|
||||
|
||||
|
||||
def _statements(db, work):
|
||||
from sqlalchemy import event
|
||||
|
||||
seen: list[str] = []
|
||||
listener = lambda conn, cur, stmt, params, ctx, many: seen.append(stmt) # noqa: E731
|
||||
event.listen(db.get_bind(), "before_cursor_execute", listener)
|
||||
try:
|
||||
work()
|
||||
finally:
|
||||
event.remove(db.get_bind(), "before_cursor_execute", listener)
|
||||
return seen
|
||||
|
||||
|
||||
def test_get_project_with_unbound_monitor_template_does_not_write(db):
|
||||
db.add_all([BizCompareTemplate(id="ct", name="ZTE status default", metrics_json=[]),
|
||||
BizMonitorTemplate(id="mt", name="默认割接监控", compare_template_id="ct")])
|
||||
_project(db, 1)
|
||||
db.commit()
|
||||
seen = _statements(db, lambda: mig.get_project(db, "p1"))
|
||||
assert not [s for s in seen if s.lstrip().upper().startswith(("INSERT", "UPDATE", "DELETE"))]
|
||||
assert db.get(BizMigrationProject, "p1").monitor_template_id == ""
|
||||
|
||||
|
||||
def test_project_list_query_count_does_not_grow_with_projects(db):
|
||||
db.add_all([BizCompareTemplate(id="ct", name="c", metrics_json=[]),
|
||||
BizMonitorTemplate(id="mt", name="默认割接监控", compare_template_id="ct")])
|
||||
for n in range(3):
|
||||
_project(db, n, monitor_template_id="mt")
|
||||
db.commit()
|
||||
few = len(_statements(db, lambda: mig.list_projects(db)))
|
||||
db.expire_all()
|
||||
for n in range(3, 15):
|
||||
_project(db, n, monitor_template_id="mt")
|
||||
db.commit()
|
||||
db.expire_all()
|
||||
many = len(_statements(db, lambda: mig.list_projects(db)))
|
||||
assert many <= few + 2, (few, many)
|
||||
341
tests/test_identity_pipeline.py
Normal file
341
tests/test_identity_pipeline.py
Normal file
|
|
@ -0,0 +1,341 @@
|
|||
"""Identity pipeline: one match key everywhere, per-side rules, value maps, post-normalize filters."""
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_migration.evaluate import evaluate_metric_dual, parse_expect_set
|
||||
from netx_api.biz_state import compare_service as svc
|
||||
from netx_api.biz_state.compare_engine import compare_rows
|
||||
from netx_api.biz_state.compare_sql import dup_pairing_order_sql, sql_compare_skip_reason
|
||||
from netx_api.biz_state.identity_pipeline import build_pipeline, legacy_pipeline
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareRun, BizCompareTemplate, BizStateBatch, BizStateBatchCommand, BizStateMetricRow, BizStateTask,
|
||||
)
|
||||
|
||||
ZTE_RULES = [{"from": "GE", "to": "gei"}, {"from": "XGE", "to": "xgei"}]
|
||||
|
||||
|
||||
def sheet(**over):
|
||||
base = {
|
||||
"sheet_id": "s", "metric_id": "iface_t", "key_fields": ["interface"], "iface_fields": ["interface"],
|
||||
"compare_fields": ["state"], "field_rules": [], "row_filters": [],
|
||||
}
|
||||
return {**base, **over}
|
||||
|
||||
|
||||
def pg_db():
|
||||
db = MagicMock()
|
||||
db.get_bind.return_value = MagicMock(dialect=MagicMock())
|
||||
db.get_bind.return_value.dialect.name = "postgresql"
|
||||
return db
|
||||
|
||||
|
||||
# ---- B2: per-side rules + vendor canonical -------------------------------------------------
|
||||
|
||||
def test_huawei_long_and_short_names_are_the_same_interface():
|
||||
p = build_pipeline(sheet(), iface_rules=ZTE_RULES, port_map={}, before_vendor="huawei", after_vendor="huawei")
|
||||
a = p.match_key({"interface": "GigabitEthernet0/0/1"}, "before")
|
||||
b = p.match_key({"interface": "GE0/0/1"}, "before")
|
||||
assert a == b == ("gei0/0/1",)
|
||||
|
||||
|
||||
def test_inner_spaces_are_removed():
|
||||
p = build_pipeline(sheet(), iface_rules=[], port_map={}, before_vendor="cisco")
|
||||
assert p.match_key({"interface": "Gigabit Ethernet 0/0/1"}, "before") == p.match_key(
|
||||
{"interface": "GigabitEthernet0/0/1"}, "before")
|
||||
|
||||
|
||||
def test_cisco_and_h3c_canonical_forms():
|
||||
p = build_pipeline(sheet(), iface_rules=[], port_map={}, before_vendor="cisco", after_vendor="h3c")
|
||||
assert p.normalize({"interface": "TenGigE0/0/0/1"}, "before")["interface"] == "Te0/0/0/1"
|
||||
assert p.normalize({"interface": "Ten-GigabitEthernet1/0/1"}, "after")["interface"] == "XGE1/0/1"
|
||||
|
||||
|
||||
def test_builtin_canonical_can_be_switched_off_per_sheet():
|
||||
p = build_pipeline(sheet(builtin_vendor_canonical=False), iface_rules=[], port_map={}, before_vendor="huawei")
|
||||
assert p.normalize({"interface": "GigabitEthernet0/0/1"}, "before")["interface"] == "GigabitEthernet0/0/1"
|
||||
|
||||
|
||||
def test_side_scoped_rule_only_touches_its_side():
|
||||
rules = [{"from": "GE", "to": "gei", "side": "before"}]
|
||||
p = build_pipeline(sheet(), iface_rules=rules, port_map={}, before_vendor="huawei", after_vendor="zte")
|
||||
assert p.normalize({"interface": "GE0/1"}, "before")["interface"] == "gei0/1"
|
||||
assert p.normalize({"interface": "GE0/1"}, "after")["interface"] == "GE0/1"
|
||||
|
||||
|
||||
def test_vendor_scoped_rule_follows_the_batch_vendor():
|
||||
rules = [{"from": "Eth-Trunk", "to": "ae", "vendor": "huawei"}]
|
||||
p = build_pipeline(sheet(), iface_rules=rules, port_map={}, before_vendor="huawei", after_vendor="zte")
|
||||
assert p.normalize({"interface": "Eth-Trunk1"}, "before")["interface"] == "ae1"
|
||||
assert p.normalize({"interface": "Eth-Trunk1"}, "after")["interface"] == "Eth-Trunk1"
|
||||
|
||||
|
||||
def test_port_map_keys_use_before_rules_and_values_use_after_rules():
|
||||
rules = [{"from": "GE", "to": "gei", "side": "after"}]
|
||||
p = build_pipeline(sheet(), iface_rules=rules, port_map={"GE0/1": "GE0/2"},
|
||||
before_vendor="huawei", after_vendor="zte")
|
||||
# before-side name stays "GE0/1" (rule is after-only); after-side value is normalized.
|
||||
assert p.port_map == {"GigabitEthernet0/1".replace("GigabitEthernet", "GE"): "gei0/2"}
|
||||
row = p.transform({"interface": "GigabitEthernet0/1"}, "before")
|
||||
assert row["interface"] == "gei0/2"
|
||||
|
||||
|
||||
def test_zte_to_huawei_numbering_change_with_alias_rules():
|
||||
p = build_pipeline(
|
||||
sheet(), iface_rules=ZTE_RULES, port_map={"GE0/1/0/1": "GigabitEthernet0/2/0/1"},
|
||||
before_vendor="zte", after_vendor="huawei",
|
||||
)
|
||||
out = compare_rows(
|
||||
before_rows=[{"interface": "gei0/1/0/1", "state": "up"}],
|
||||
after_rows=[{"interface": "GE0/2/0/1", "state": "up"}],
|
||||
key_fields=["interface"], iface_fields=["interface"], compare_fields=["state"], pipeline=p,
|
||||
)
|
||||
assert out["summary"]["unchanged"] == 1
|
||||
|
||||
|
||||
# ---- B3: value maps ----------------------------------------------------------------------
|
||||
|
||||
def test_value_map_aligns_vendor_state_names():
|
||||
rules = [{"field": "state", "value_map": {"estab": "established", "*down": "down"}}]
|
||||
p = build_pipeline(sheet(iface_fields=[], key_fields=["peer"], field_rules=rules), iface_rules=[],
|
||||
port_map={})
|
||||
out = compare_rows(
|
||||
before_rows=[{"peer": "1.1.1.1", "state": "Estab"}, {"peer": "2.2.2.2", "state": "*down"}],
|
||||
after_rows=[{"peer": "1.1.1.1", "state": "established"}, {"peer": "2.2.2.2", "state": "down"}],
|
||||
key_fields=["peer"], iface_fields=[], compare_fields=["state"], pipeline=p,
|
||||
)
|
||||
assert out["summary"]["unchanged"] == 2 and out["summary"]["changed"] == 0
|
||||
|
||||
|
||||
def test_value_map_can_be_scoped_to_one_side():
|
||||
rules = [{"field": "state", "side": "before", "value_map": {"estab": "established"}}]
|
||||
p = build_pipeline(sheet(iface_fields=[], key_fields=["peer"], field_rules=rules), iface_rules=[], port_map={})
|
||||
assert p.normalize({"peer": "a", "state": "Estab"}, "before")["state"] == "established"
|
||||
assert p.normalize({"peer": "a", "state": "Estab"}, "after")["state"] == "Estab"
|
||||
|
||||
|
||||
def test_mac_key_with_different_formats_pairs():
|
||||
rules = [{"field": "mac", "normalize": "mac"}]
|
||||
p = build_pipeline(sheet(metric_id="mac_t", iface_fields=[], key_fields=["mac"], compare_fields=["port"],
|
||||
field_rules=rules), iface_rules=[], port_map={})
|
||||
out = compare_rows(
|
||||
before_rows=[{"mac": "00e0-fc12-3456", "port": "1"}], after_rows=[{"mac": "00e0.fc12.3456", "port": "1"}],
|
||||
key_fields=["mac"], iface_fields=[], compare_fields=["port"], pipeline=p,
|
||||
)
|
||||
assert out["summary"]["unchanged"] == 1 and out["summary"]["added"] == out["summary"]["removed"] == 0
|
||||
|
||||
|
||||
def test_legacy_pipeline_never_normalizes_keys():
|
||||
rules = [{"field": "mac", "normalize": "mac"}]
|
||||
p = legacy_pipeline(key_fields=["mac"], iface_fields=[], compare_fields=["port"], field_rules=rules)
|
||||
out = compare_rows(
|
||||
before_rows=[{"mac": "00e0-fc12-3456", "port": "1"}], after_rows=[{"mac": "00e0.fc12.3456", "port": "1"}],
|
||||
key_fields=["mac"], iface_fields=[], compare_fields=["port"], field_rules=rules, pipeline=p,
|
||||
)
|
||||
assert out["summary"]["added"] == out["summary"]["removed"] == 1
|
||||
|
||||
|
||||
def test_value_map_and_key_normalize_block_sql():
|
||||
s = sheet(metric_id="bgp_route", iface_fields=[], key_fields=["mac"],
|
||||
field_rules=[{"field": "state", "value_map": {"a": "b"}}], compare_fields=["state"])
|
||||
p = build_pipeline(s, iface_rules=[], port_map={})
|
||||
assert sql_compare_skip_reason(pg_db(), s, pipeline=p) == "value_map"
|
||||
s2 = sheet(metric_id="m", iface_fields=[], key_fields=["mac"], compare_fields=[],
|
||||
field_rules=[{"field": "mac", "normalize": "lower"}])
|
||||
assert sql_compare_skip_reason(pg_db(), s2, pipeline=build_pipeline(s2, iface_rules=[], port_map={})) \
|
||||
== "key_normalize"
|
||||
|
||||
|
||||
# ---- B4: filters run after normalization -----------------------------------------------------
|
||||
|
||||
CONTAINS_GEI = [{"field": "interface", "op": "contains", "value": "gei"}]
|
||||
|
||||
|
||||
def test_v2_filter_sees_normalized_names_but_legacy_sees_raw():
|
||||
s = sheet(row_filters=CONTAINS_GEI)
|
||||
v2 = build_pipeline(s, iface_rules=ZTE_RULES, port_map={}, before_vendor="huawei")
|
||||
assert [raw["interface"] for raw, _ in v2.prepare([{"interface": "GE0/0/1"}], "before")] == ["GE0/0/1"]
|
||||
old = legacy_pipeline(key_fields=["interface"], iface_fields=["interface"], compare_fields=[],
|
||||
iface_normalize_rules=ZTE_RULES, row_filters=CONTAINS_GEI)
|
||||
assert old.prepare([{"interface": "GE0/0/1"}], "before") == [] # the cross-vendor bug, kept for v1
|
||||
|
||||
|
||||
def test_filter_can_target_one_side():
|
||||
s = sheet(iface_fields=[], row_filters=[{"field": "type", "op": "eq", "value": "dynamic", "side": "before"}])
|
||||
p = build_pipeline(s, iface_rules=[], port_map={})
|
||||
rows = [{"interface": "a", "type": "static"}]
|
||||
assert p.prepare(rows, "before") == []
|
||||
assert len(p.prepare(rows, "after")) == 1
|
||||
|
||||
|
||||
def test_pushdown_only_for_filters_that_read_untransformed_fields():
|
||||
s = sheet(row_filters=[CONTAINS_GEI[0], {"field": "afi", "op": "eq", "value": "ipv4"}])
|
||||
p = build_pipeline(s, iface_rules=ZTE_RULES, port_map={})
|
||||
assert p.pushdown_filters("before") == [{"field": "afi", "op": "eq", "value": "ipv4"}]
|
||||
assert sql_compare_skip_reason(pg_db(), sheet(metric_id="m", iface_fields=["interface"], key_fields=["id"],
|
||||
compare_fields=[], row_filters=CONTAINS_GEI,
|
||||
ignore_port_changes=False),
|
||||
pipeline=build_pipeline(sheet(row_filters=CONTAINS_GEI), iface_rules=[],
|
||||
port_map={})) == "row_filters_post_normalize"
|
||||
|
||||
|
||||
# ---- B6: a port map no longer pushes interface-free sheets to Python ----------------------------
|
||||
|
||||
def test_port_map_only_blocks_sql_for_sheets_that_use_interfaces():
|
||||
route = {"metric_id": "bgp_route", "key_fields": ["network"], "iface_fields": [],
|
||||
"compare_fields": ["path"], "field_rules": [], "row_filters": []}
|
||||
assert sql_compare_skip_reason(pg_db(), route, port_map={"a": "b"}) == ""
|
||||
brief = {**route, "metric_id": "interface_brief", "key_fields": ["interface"], "iface_fields": ["interface"]}
|
||||
assert sql_compare_skip_reason(pg_db(), brief, port_map={"a": "b"}) == "port_map"
|
||||
# a map on a sheet whose interface column is only displayed does not matter either
|
||||
shown = {**route, "iface_fields": ["interface"]}
|
||||
assert sql_compare_skip_reason(pg_db(), shown, port_map={"a": "b"}) == ""
|
||||
|
||||
|
||||
# ---- V4: duplicate keys pair by value ----------------------------------------------------------
|
||||
|
||||
def test_same_key_rows_pair_by_value_not_output_order():
|
||||
s = sheet(iface_fields=[], key_fields=["prefix"], compare_fields=["hop"])
|
||||
rows_b = [{"prefix": "p", "hop": "a"}, {"prefix": "p", "hop": "b"}]
|
||||
rows_a = [{"prefix": "p", "hop": "b"}, {"prefix": "p", "hop": "a"}]
|
||||
kw = dict(before_rows=rows_b, after_rows=rows_a, key_fields=["prefix"], iface_fields=[], compare_fields=["hop"])
|
||||
new = compare_rows(**kw, pipeline=build_pipeline(s, iface_rules=[], port_map={}))
|
||||
assert new["summary"]["unchanged"] == 2 and new["summary"]["changed"] == 0
|
||||
old = compare_rows(**kw) # ordered, legacy
|
||||
assert old["summary"]["changed"] == 2
|
||||
|
||||
|
||||
def test_sql_dup_order_mirrors_python_sort_key():
|
||||
sql = dup_pairing_order_sql(["hop", "metric"], {"hop": {"normalize": "lower"}})
|
||||
assert 'lower(trim(both from coalesce(data_json->>\'hop\', \'\'))) COLLATE "C" ASC' in sql.replace("(", "", 1) \
|
||||
or 'COLLATE "C"' in sql
|
||||
assert sql.endswith(", ")
|
||||
assert dup_pairing_order_sql([], {}) == ""
|
||||
|
||||
|
||||
# ---- B7: partial maps ---------------------------------------------------------------------------
|
||||
|
||||
def test_ignore_port_policy_keeps_unmapped_ports_matching_by_content():
|
||||
s = sheet(iface_fields=["interface"], key_fields=["interface", "ip"], compare_fields=["mac"],
|
||||
unmapped_port_policy="ignore_port")
|
||||
p = build_pipeline(s, iface_rules=[], port_map={"gei-1": "gei-9"})
|
||||
out = compare_rows(
|
||||
before_rows=[{"interface": "gei-1", "ip": "10.0.0.1", "mac": "m1"},
|
||||
{"interface": "gei-2", "ip": "10.0.0.2", "mac": "m2"}],
|
||||
after_rows=[{"interface": "gei-9", "ip": "10.0.0.1", "mac": "m1"},
|
||||
{"interface": "gei-7", "ip": "10.0.0.2", "mac": "m2"}],
|
||||
key_fields=["interface", "ip"], iface_fields=["interface"], compare_fields=["mac"], pipeline=p,
|
||||
)
|
||||
assert out["summary"]["unchanged"] == 2
|
||||
by_name = build_pipeline({**s, "unmapped_port_policy": "by_name"}, iface_rules=[], port_map={"gei-1": "gei-9"})
|
||||
out2 = compare_rows(
|
||||
before_rows=[{"interface": "gei-2", "ip": "10.0.0.2", "mac": "m2"}],
|
||||
after_rows=[{"interface": "gei-7", "ip": "10.0.0.2", "mac": "m2"}],
|
||||
key_fields=["interface", "ip"], iface_fields=["interface"], compare_fields=["mac"], pipeline=by_name,
|
||||
)
|
||||
assert out2["summary"]["added"] == out2["summary"]["removed"] == 1
|
||||
|
||||
|
||||
# ---- B1: every path derives the same match key --------------------------------------------------
|
||||
|
||||
@pytest.fixture()
|
||||
def db():
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session = sessionmaker(bind=engine, expire_on_commit=False)()
|
||||
yield session
|
||||
session.close()
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def _seed_two_batches(db):
|
||||
db.add_all([
|
||||
BizStateTask(id="t-old", source="managed", ne_id="o", vendor="Huawei", device_type="huawei"),
|
||||
BizStateTask(id="t-new", source="managed", ne_id="n", vendor="ZTE", device_type="zte_zxros"),
|
||||
BizStateBatch(id="b-old", task_id="t-old", status="success", vendor="Huawei"),
|
||||
BizStateBatch(id="b-new", task_id="t-new", status="success", vendor="ZTE"),
|
||||
])
|
||||
db.add_all([
|
||||
BizStateMetricRow(id="r-old", batch_id="b-old", metric_id="iface_t", seq=0,
|
||||
data_json={"interface": "GigabitEthernet0/1/0/1", "state": "up"}),
|
||||
BizStateMetricRow(id="r-new", batch_id="b-new", metric_id="iface_t", seq=0,
|
||||
data_json={"interface": "gei-0/2/0/1", "state": "up"}),
|
||||
])
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_all_paths_agree_on_the_match_key(db):
|
||||
_seed_two_batches(db)
|
||||
raw_map = {"GigabitEthernet0/1/0/1": "gei-0/2/0/1"}
|
||||
rules = [{"from": "GE", "to": "gei"}]
|
||||
s = sheet(sheet_id="s")
|
||||
tpl = BizCompareTemplate(id="tpl", name="t", metrics_json=[s], iface_normalize_json=rules)
|
||||
db.add(tpl)
|
||||
db.commit()
|
||||
from netx_api.biz_state.compare_service import _create_running_run, template_metrics
|
||||
from netx_api.models import BizCompareJob, BizPortMapping, BizPortMappingRow
|
||||
|
||||
db.add_all([BizPortMapping(id="m", name="m"),
|
||||
BizPortMappingRow(mapping_id="m", before_if="GigabitEthernet0/1/0/1", after_if="gei-0/2/0/1"),
|
||||
BizCompareJob(id="j", name="j", template_id="tpl", mapping_id="m")])
|
||||
db.commit()
|
||||
job = db.get(BizCompareJob, "j")
|
||||
run = _create_running_run(db, job=job, tpl=tpl, before_batch_id="b-old", after_batch_id="b-new",
|
||||
sheets_cfg=template_metrics(tpl))
|
||||
assert run.summary_json["config_snapshot"]["vendors"] == {"before": "huawei", "after": "zte"}
|
||||
pipe = svc._run_pipeline(db, run, s)
|
||||
expected = ("gei-0/2/0/1",)
|
||||
|
||||
# 1. execution
|
||||
result = compare_rows(before_rows=[{"interface": "GigabitEthernet0/1/0/1", "state": "up"}],
|
||||
after_rows=[{"interface": "gei-0/2/0/1", "state": "up"}],
|
||||
key_fields=["interface"], iface_fields=["interface"], compare_fields=["state"],
|
||||
pipeline=pipe)
|
||||
assert result["summary"]["unchanged"] == 1
|
||||
# 2. pipeline directly
|
||||
assert pipe.match_key({"interface": "GigabitEthernet0/1/0/1"}, "before") == expected
|
||||
assert pipe.match_key({"interface": "gei-0/2/0/1"}, "after") == expected
|
||||
# 3. search key groups
|
||||
rows, complete = svc._load_search_key_groups(
|
||||
db, batch_id="b-old", metric_id="iface_t", keys={expected}, match_keys=["interface"],
|
||||
pipeline=pipe, side="before")
|
||||
assert [r["interface"] for r in rows] == ["GigabitEthernet0/1/0/1"] and complete == {expected}
|
||||
# 4. hydrate
|
||||
item = {"before": {"interface": "GigabitEthernet0/1/0/1"}, "after": {}, "mapped_before": {},
|
||||
"after_row_id": "r-new", "before_row_id": ""}
|
||||
hydrated = svc._hydrate_diff_rows(db, [item], run=run, sheet=s)[0]
|
||||
assert hydrated["mapped_before"]["interface"] == hydrated["after"]["interface"] == "gei-0/2/0/1"
|
||||
# 5. cutover evaluation (old = before, new = after)
|
||||
dual = evaluate_metric_dual(
|
||||
metric_id="iface_t", sheet_id="s", key_fields=["interface"], iface_fields=["interface"],
|
||||
compare_fields=["state"], old_baseline_rows=[{"interface": "GigabitEthernet0/1/0/1", "state": "up"}],
|
||||
old_current_rows=[], new_baseline_rows=[{"interface": "gei-0/2/0/1", "state": "down"}],
|
||||
new_current_rows=[{"interface": "gei-0/2/0/1", "state": "up"}], port_map=pipe.port_map,
|
||||
expect=parse_expect_set({"items": [{"metric_id": "iface_t", "key": "GE0/1/0/1"}]}),
|
||||
window_active=False, acceptance=True, pipeline=pipe,
|
||||
)
|
||||
(row,) = dual["rows"]
|
||||
assert (row["match_old_key"], row["match_new_key"]) == ("gei0/1/0/1", "gei-0/2/0/1")
|
||||
assert row["old_key"] == "GigabitEthernet0/1/0/1" and row["verdict"] == "migrated"
|
||||
# 6. baseline expect picker
|
||||
assert pipe.transform({"interface": "GigabitEthernet0/1/0/1"}, "before")["interface"] == "gei-0/2/0/1"
|
||||
|
||||
|
||||
# ---- v1 replay must not change -----------------------------------------------------------------
|
||||
|
||||
def test_v1_snapshot_run_replays_with_legacy_semantics(db):
|
||||
_seed_two_batches(db)
|
||||
s = sheet(sheet_id="s", row_filters=CONTAINS_GEI)
|
||||
run = BizCompareRun(id="r1", job_id="j", template_id="", before_batch_id="b-old", after_batch_id="b-new",
|
||||
summary_json={"config_snapshot": {
|
||||
"version": 1, "template": {"iface_normalize_rules": [{"from": "GE", "to": "gei"}]},
|
||||
"sheets": [s], "port_map": {"gei-0/1/0/1": "gei-0/2/0/1"}}})
|
||||
pipe = svc._run_pipeline(db, run, s)
|
||||
assert pipe.legacy and pipe.duplicate_pairing == "ordered"
|
||||
assert pipe.port_map == {"gei-0/1/0/1": "gei-0/2/0/1"}
|
||||
assert pipe.prepare([{"interface": "GE0/1"}], "before") == []
|
||||
|
|
@ -9,6 +9,7 @@ from netx_api.biz_state.iface_normalize import (
|
|||
default_zte_iface_normalize_rules,
|
||||
normalize_iface_name,
|
||||
normalize_iface_rules,
|
||||
normalize_port_map,
|
||||
resolve_mapped_iface,
|
||||
)
|
||||
|
||||
|
|
@ -71,6 +72,32 @@ class PortMapSubifTests(unittest.TestCase):
|
|||
self.assertEqual(row["interface"], "xgei-0/1/0/1.55")
|
||||
|
||||
|
||||
class NormalizePortMapTests(unittest.TestCase):
|
||||
def test_keys_and_values_use_template_rules(self) -> None:
|
||||
out, conflicts = normalize_port_map(
|
||||
{"GE0/1/0/1": "GE0/2/0/1"},
|
||||
normalize_iface_rules([{"from": "GE", "to": "gei"}]),
|
||||
)
|
||||
self.assertEqual(out, {"gei0/1/0/1": "gei0/2/0/1"})
|
||||
self.assertEqual(conflicts, [])
|
||||
|
||||
def test_conflicting_keys_are_reported_first_wins(self) -> None:
|
||||
out, conflicts = normalize_port_map(
|
||||
{"GE0/1": "gei0/2", "gei0/1": "gei0/3"},
|
||||
normalize_iface_rules([{"from": "GE", "to": "gei"}]),
|
||||
)
|
||||
self.assertEqual(out, {"gei0/1": "gei0/2"})
|
||||
self.assertEqual(len(conflicts), 1)
|
||||
self.assertEqual(conflicts[0]["dropped_to"], "gei0/3")
|
||||
|
||||
def test_same_target_duplicate_is_not_a_conflict(self) -> None:
|
||||
_, conflicts = normalize_port_map(
|
||||
{"GE0/1": "gei0/2", "gei0/1": "gei0/2"},
|
||||
normalize_iface_rules([{"from": "GE", "to": "gei"}]),
|
||||
)
|
||||
self.assertEqual(conflicts, [])
|
||||
|
||||
|
||||
class ComparePipelineTests(unittest.TestCase):
|
||||
def test_normalize_then_map_matches(self) -> None:
|
||||
before = [{"interface": "GE-0/0/0/1.100", "admin": "up"}]
|
||||
|
|
|
|||
|
|
@ -1,8 +1,15 @@
|
|||
import { Button, Input, Modal } from "@heroui/react";
|
||||
import { useCallback, useEffect, useLayoutEffect, useMemo, useRef, useState } from "react";
|
||||
import type { KeyboardEvent as ReactKeyboardEvent } from "react";
|
||||
import { parseTemplateImport, parseIfaceNormText, toOrGroups, fromOrGroups, validateTemplateDraft } from "./lib/compareTemplateModel";
|
||||
import type { MetricSheet, FieldRule, RowFilter } from "./lib/compareTemplateModel";
|
||||
import {
|
||||
parseTemplateImport, parseIfaceNormText, formatIfaceNormText, parseValueMapText, formatValueMapText,
|
||||
sheetPayload, sheetOptions, toOrGroups, fromOrGroups, validateTemplateDraft,
|
||||
} from "./lib/compareTemplateModel";
|
||||
import type { MetricSheet, FieldRule, RowFilter, IfaceRule, Side } from "./lib/compareTemplateModel";
|
||||
import { MappingTable } from "./components/MappingTable";
|
||||
import { PreflightReport, dataStateLabel } from "./components/PreflightReport";
|
||||
import type { PreflightReportData } from "./components/PreflightReport";
|
||||
import { parseMapText, mapTextDirty } from "./lib/mappingText";
|
||||
import { ListPager } from "../../components/ListPager";
|
||||
import { AppModalShell } from "../../components/ui/AppModalShell";
|
||||
import { FieldSelect } from "../../components/ui/FieldSelect";
|
||||
|
|
@ -29,7 +36,7 @@ import {
|
|||
bizCompareUpdateJob,
|
||||
bizCompareUpdateMapping,
|
||||
bizCompareUpdateTemplate,
|
||||
bizCompareValidateMapping,
|
||||
bizCompareMappingPreflight,
|
||||
bizStateListBatches,
|
||||
bizStateListTasks,
|
||||
formatErr,
|
||||
|
|
@ -72,12 +79,19 @@ type Template = {
|
|||
key_fields: string[];
|
||||
iface_fields: string[];
|
||||
compare_fields: string[];
|
||||
iface_normalize_rules?: { from: string; to: string }[];
|
||||
iface_normalize_rules?: IfaceRule[];
|
||||
note: string;
|
||||
job_count?: number;
|
||||
monitor_template_count?: number;
|
||||
};
|
||||
|
||||
type Mapping = { id: string; name: string; rows: { before_if: string; after_if: string }[] };
|
||||
type Mapping = {
|
||||
id: string;
|
||||
name: string;
|
||||
rows: { before_if: string; after_if: string }[];
|
||||
job_count?: number;
|
||||
project_count?: number;
|
||||
};
|
||||
type StoreUnchanged = "auto" | "always" | "never" | "sample" | "keys";
|
||||
|
||||
type Job = {
|
||||
|
|
@ -102,6 +116,8 @@ type DiffRow = {
|
|||
before: Record<string, unknown> | null;
|
||||
after: Record<string, unknown> | null;
|
||||
mapped_before?: Record<string, unknown> | null;
|
||||
/** Before-side key exactly as the device printed it; only present when it differs from the match key. */
|
||||
raw_key_before?: Record<string, unknown> | null;
|
||||
changes: Record<string, { before?: unknown; after?: unknown; reason?: string }>;
|
||||
};
|
||||
|
||||
|
|
@ -525,6 +541,7 @@ function cloneSheet(s: MetricSheet): MetricSheet {
|
|||
row_filters: JSON.parse(JSON.stringify(s.row_filters || [])),
|
||||
field_rules: JSON.parse(JSON.stringify(s.field_rules || [])),
|
||||
ignore_port_changes: s.ignore_port_changes,
|
||||
...sheetOptions(s as unknown as Record<string, unknown>),
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -532,21 +549,9 @@ const TEMPLATE_EXPORT_FORMAT = "netx.biz_compare_template";
|
|||
const TEMPLATE_EXPORT_VERSION = 1;
|
||||
|
||||
function templateExportPayload(tpl: Template) {
|
||||
const metrics = templateSheets(tpl).map((s) => ({
|
||||
sheet_id: s.sheet_id || s.metric_id,
|
||||
title: s.title || s.sheet_id || s.metric_id,
|
||||
metric_id: s.metric_id,
|
||||
key_fields: [...(s.key_fields || [])],
|
||||
iface_fields: [...(s.iface_fields || [])],
|
||||
compare_fields: [...(s.compare_fields || [])],
|
||||
display_fields:
|
||||
s.display_fields && s.display_fields.length
|
||||
? [...s.display_fields]
|
||||
: [...(s.key_fields || []), ...(s.compare_fields || [])],
|
||||
row_filters: JSON.parse(JSON.stringify(s.row_filters || [])),
|
||||
field_rules: JSON.parse(JSON.stringify(s.field_rules || [])),
|
||||
ignore_port_changes: s.ignore_port_changes,
|
||||
}));
|
||||
const metrics = templateSheets(tpl).map((s) =>
|
||||
JSON.parse(JSON.stringify(sheetPayload({ ...s, display_fields: s.display_fields?.length ? s.display_fields : undefined }))),
|
||||
);
|
||||
return {
|
||||
format: TEMPLATE_EXPORT_FORMAT,
|
||||
version: TEMPLATE_EXPORT_VERSION,
|
||||
|
|
@ -558,8 +563,8 @@ function templateExportPayload(tpl: Template) {
|
|||
};
|
||||
}
|
||||
|
||||
function ifaceNormRulesToText(rules?: { from: string; to: string }[]): string {
|
||||
return (rules || []).map((r) => `${r.from},${r.to}`).join("\n");
|
||||
function ifaceNormRulesToText(rules?: IfaceRule[]): string {
|
||||
return formatIfaceNormText(rules);
|
||||
}
|
||||
|
||||
function downloadJsonFile(filename: string, data: unknown) {
|
||||
|
|
@ -572,13 +577,16 @@ function downloadJsonFile(filename: string, data: unknown) {
|
|||
URL.revokeObjectURL(url);
|
||||
}
|
||||
|
||||
const isSided = (r: FieldRule) => r.side === "before" || r.side === "after";
|
||||
|
||||
/** The rule that drives the comparison of a field (side-scoped value maps are separate rules). */
|
||||
function ruleForField(sheet: MetricSheet, field: string): FieldRule {
|
||||
return (sheet.field_rules || []).find((r) => r.field === field) || { field };
|
||||
return (sheet.field_rules || []).find((r) => r.field === field && !isSided(r)) || { field };
|
||||
}
|
||||
|
||||
function upsertFieldRule(sheet: MetricSheet, field: string, patch: Partial<FieldRule>): FieldRule[] {
|
||||
const rules = [...(sheet.field_rules || [])];
|
||||
const idx = rules.findIndex((r) => r.field === field);
|
||||
const idx = rules.findIndex((r) => r.field === field && !isSided(r));
|
||||
const next: FieldRule = { ...(idx >= 0 ? rules[idx] : { field }), ...patch, field };
|
||||
const norm = (next.normalize || "").toLowerCase();
|
||||
const cmp = (next.compare || "").toLowerCase();
|
||||
|
|
@ -586,15 +594,33 @@ function upsertFieldRule(sheet: MetricSheet, field: string, patch: Partial<Field
|
|||
!next.ignore &&
|
||||
(!cmp || cmp === "eq") &&
|
||||
(!norm || norm === "none" || norm === "strip") &&
|
||||
(next.tolerance === undefined || next.tolerance === null);
|
||||
(next.tolerance === undefined || next.tolerance === null) &&
|
||||
!Object.keys(next.value_map || {}).length &&
|
||||
!(next.rank || []).length &&
|
||||
next.apply_to_key === undefined;
|
||||
if (empty) {
|
||||
return rules.filter((r) => r.field !== field);
|
||||
return rules.filter((_r, i) => i !== idx);
|
||||
}
|
||||
if (idx >= 0) rules[idx] = next;
|
||||
else rules.push(next);
|
||||
return rules;
|
||||
}
|
||||
|
||||
/** Value map of a field for one scope: "both" lives on the main rule, before/after on their own rules. */
|
||||
function valueMapFor(sheet: MetricSheet, field: string, scope: Side): Record<string, string> {
|
||||
const rule = (sheet.field_rules || []).find((r) => r.field === field && (scope === "before" || scope === "after" ? r.side === scope : !isSided(r)));
|
||||
return rule?.value_map || {};
|
||||
}
|
||||
|
||||
function setValueMapFor(sheet: MetricSheet, field: string, scope: Side, map: Record<string, string>): FieldRule[] {
|
||||
if (scope !== "before" && scope !== "after") {
|
||||
return upsertFieldRule(sheet, field, { value_map: Object.keys(map).length ? map : undefined });
|
||||
}
|
||||
const rules = (sheet.field_rules || []).filter((r) => !(r.field === field && r.side === scope));
|
||||
if (Object.keys(map).length) rules.push({ field, side: scope, value_map: map });
|
||||
return rules;
|
||||
}
|
||||
|
||||
function emptyLeaf(field = ""): RowFilter {
|
||||
return { field, op: "eq", value: "" };
|
||||
}
|
||||
|
|
@ -611,6 +637,69 @@ const FILTER_OPS = [
|
|||
"age_timer",
|
||||
] as const;
|
||||
|
||||
/** Value-map editor for one field: text is parsed on blur so half-typed lines never clobber the rule. */
|
||||
function ValueMapCell({
|
||||
field, value, scope, onScope, onCommit, rank, onRank, showRank, t,
|
||||
}: {
|
||||
field: string;
|
||||
value: Record<string, string>;
|
||||
scope: Side;
|
||||
onScope: (s: Side) => void;
|
||||
onCommit: (map: Record<string, string>) => void;
|
||||
rank: string[];
|
||||
onRank: (rank: string[]) => void;
|
||||
showRank: boolean;
|
||||
t: (k: string, vars?: Record<string, string | number>) => string;
|
||||
}) {
|
||||
const external = formatValueMapText(value);
|
||||
const [text, setText] = useState(external);
|
||||
const [error, setError] = useState("");
|
||||
useEffect(() => {
|
||||
setText(external);
|
||||
setError("");
|
||||
}, [external, scope]);
|
||||
return (
|
||||
<div className="ct-valuemap">
|
||||
{showRank ? (
|
||||
<Input
|
||||
aria-label={`${field} ${t("bizCompare.rankLabel")}`}
|
||||
placeholder={t("bizCompare.rankPlaceholder")}
|
||||
value={rank.join(",")}
|
||||
onChange={(e) => onRank(e.target.value.split(",").map((x) => x.trim()).filter(Boolean))}
|
||||
/>
|
||||
) : null}
|
||||
<select
|
||||
className="mt-select"
|
||||
aria-label={`${field} ${t("bizCompare.valueMapScope")}`}
|
||||
value={scope === "before" || scope === "after" ? scope : "both"}
|
||||
onChange={(e) => onScope(e.target.value as Side)}
|
||||
>
|
||||
<option value="both">{t("bizCompare.scopeBoth")}</option>
|
||||
<option value="before">{t("bizCompare.scopeBefore")}</option>
|
||||
<option value="after">{t("bizCompare.scopeAfter")}</option>
|
||||
</select>
|
||||
<textarea
|
||||
className={`bs-cmp-mapping__editor ct-valuemap__text${error ? " is-invalid" : ""}`}
|
||||
aria-label={`${field} ${t("bizCompare.valueMap")}`}
|
||||
placeholder={t("bizCompare.valueMapPlaceholder")}
|
||||
rows={2}
|
||||
spellCheck={false}
|
||||
value={text}
|
||||
onChange={(e) => setText(e.target.value)}
|
||||
onBlur={() => {
|
||||
try {
|
||||
onCommit(parseValueMapText(text));
|
||||
setError("");
|
||||
} catch (e) {
|
||||
setError(String((e as Error).message));
|
||||
}
|
||||
}}
|
||||
/>
|
||||
{error ? <span className="ct-valuemap__err" role="alert">{error}</span> : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function TplRowFiltersEditor({
|
||||
groups,
|
||||
fieldOpts,
|
||||
|
|
@ -622,6 +711,8 @@ function TplRowFiltersEditor({
|
|||
t: (k: string, vars?: Record<string, string | number>) => string;
|
||||
onChange: (next: RowFilter[][]) => void;
|
||||
}) {
|
||||
// A side only has meaning on a top-level filter, i.e. when there is a single AND group.
|
||||
const sideAllowed = groups.length === 1;
|
||||
const setGroups = onChange;
|
||||
return (
|
||||
<div className="ct-filters">
|
||||
|
|
@ -701,6 +792,24 @@ function TplRowFiltersEditor({
|
|||
))}
|
||||
{filt.op && !(FILTER_OPS as readonly string[]).includes(filt.op) ? <option value={filt.op}>{filt.op}</option> : null}
|
||||
</select>
|
||||
{sideAllowed ? (
|
||||
<select
|
||||
className="mt-select"
|
||||
aria-label={`${t("bizCompare.filterSide")} ${gi + 1}.${fi + 1}`}
|
||||
value={filt.side === "before" || filt.side === "after" ? filt.side : "both"}
|
||||
onChange={(e) => {
|
||||
const next = groups.map((g) => g.map((x) => ({ ...x })));
|
||||
const side = e.target.value as Side;
|
||||
if (side === "before" || side === "after") next[gi][fi] = { ...next[gi][fi], side };
|
||||
else delete next[gi][fi].side;
|
||||
setGroups(next);
|
||||
}}
|
||||
>
|
||||
<option value="both">{t("bizCompare.scopeBoth")}</option>
|
||||
<option value="before">{t("bizCompare.scopeBefore")}</option>
|
||||
<option value="after">{t("bizCompare.scopeAfter")}</option>
|
||||
</select>
|
||||
) : null}
|
||||
<Input
|
||||
value={
|
||||
Array.isArray(filt.value) ? filt.value.join(",") : String(filt.value ?? "")
|
||||
|
|
@ -789,6 +898,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const [tplNormText, setTplNormText] = useState("");
|
||||
const [tplSheets, setTplSheets] = useState<MetricSheet[]>([]);
|
||||
const [tplSheetIdx, setTplSheetIdx] = useState(0);
|
||||
const [valueMapScope, setValueMapScope] = useState<Record<string, Side>>({});
|
||||
const [tplPaneTab, setTplPaneTab] = useState<"fields" | "filters">("fields");
|
||||
const [showMetricPicker, setShowMetricPicker] = useState(true);
|
||||
|
||||
|
|
@ -811,7 +921,8 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const [storeUnchanged, setStoreUnchanged] = useState<StoreUnchanged>("auto");
|
||||
const [mapName, setMapName] = useState("端口映射");
|
||||
const [mapText, setMapText] = useState("");
|
||||
const [validateOut, setValidateOut] = useState<any>(null);
|
||||
const [validateOut, setValidateOut] = useState<PreflightReportData | null>(null);
|
||||
const [mapView, setMapView] = useState<"text" | "table">("text");
|
||||
const [runs, setRuns] = useState<RunRecord[]>([]);
|
||||
const [runsLoading, setRunsLoading] = useState(false);
|
||||
const [runsError, setRunsError] = useState("");
|
||||
|
|
@ -1333,6 +1444,8 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
success_count?: number;
|
||||
diff_count?: number;
|
||||
pass_rate?: number | null;
|
||||
data_state?: string;
|
||||
data_state_reasons?: string[];
|
||||
}>;
|
||||
// Failures first; keep pending/running at the end so completed fails stay visible
|
||||
return [...raw].sort((a, b) => {
|
||||
|
|
@ -1349,6 +1462,9 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const activeFail = sheetFailOf(activeSheetCard || {});
|
||||
const activeSuccess = sheetSuccessOf(activeSheetCard || {});
|
||||
const activePassRate = sheetPassRateOf(activeSheetCard || {});
|
||||
const activeDataState = String(activeSheetCard?.data_state || "ok");
|
||||
const activeCoverage = (runSheets.find((sh) => sheetIdentity(sh) === sheetIdentity(activeSheetCard || {}))?.summary as
|
||||
{ mapping_coverage?: { suspect_unmapped_total?: number; suspect_unmapped?: string[] } } | undefined)?.mapping_coverage;
|
||||
const activeSheetPending = ["pending", "running", "queued"].includes(
|
||||
String(activeSheetCard?.status || ""),
|
||||
);
|
||||
|
|
@ -1501,6 +1617,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
setTplSheets([]);
|
||||
setTplSheetIdx(0);
|
||||
setTplPaneTab("fields");
|
||||
setValueMapScope({});
|
||||
setShowMetricPicker(true);
|
||||
setTplOpen(true);
|
||||
};
|
||||
|
|
@ -1515,6 +1632,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
setTplSheets(sheets.length ? sheets.map(cloneSheet) : []);
|
||||
setTplSheetIdx(0);
|
||||
setTplPaneTab("fields");
|
||||
setValueMapScope({});
|
||||
setShowMetricPicker(!sheets.length);
|
||||
setTplOpen(true);
|
||||
};
|
||||
|
|
@ -1605,18 +1723,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
name: tplName || tplSheets.map((s) => sheetLabel(s)).join("+"),
|
||||
note: tplNote,
|
||||
iface_normalize_rules: parseIfaceNormText(tplNormText),
|
||||
metrics: tplSheets.map((s) => ({
|
||||
sheet_id: sheetIdentity(s),
|
||||
title: s.title || sheetIdentity(s),
|
||||
metric_id: s.metric_id,
|
||||
key_fields: s.key_fields,
|
||||
iface_fields: s.iface_fields,
|
||||
compare_fields: s.compare_fields,
|
||||
display_fields: s.display_fields || [...s.key_fields, ...s.compare_fields],
|
||||
row_filters: s.row_filters || [],
|
||||
field_rules: s.field_rules || [],
|
||||
ignore_port_changes: s.ignore_port_changes,
|
||||
})),
|
||||
metrics: tplSheets.map((s) => sheetPayload(s, sheetIdentity(s))),
|
||||
};
|
||||
if (tplEditId) await bizCompareUpdateTemplate(tplEditId, body);
|
||||
else await bizCompareCreateTemplate(body);
|
||||
|
|
@ -1676,18 +1783,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
name: body.name,
|
||||
note: body.note,
|
||||
iface_normalize_rules: body.iface_normalize_rules || [],
|
||||
metrics: body.metrics.map((s) => ({
|
||||
sheet_id: sheetIdentity(s),
|
||||
title: s.title || sheetIdentity(s),
|
||||
metric_id: s.metric_id,
|
||||
key_fields: s.key_fields,
|
||||
iface_fields: s.iface_fields,
|
||||
compare_fields: s.compare_fields,
|
||||
display_fields: s.display_fields || [...s.key_fields, ...s.compare_fields],
|
||||
row_filters: s.row_filters || [],
|
||||
field_rules: s.field_rules || [],
|
||||
ignore_port_changes: s.ignore_port_changes,
|
||||
})),
|
||||
metrics: body.metrics.map((s) => sheetPayload(s, sheetIdentity(s))),
|
||||
});
|
||||
showOk(t("bizCompare.templateImported"));
|
||||
invalidateCutoverCache("bizMonitor:");
|
||||
|
|
@ -1717,21 +1813,6 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
};
|
||||
|
||||
const parseMapRows = () => {
|
||||
const rows: { before_if: string; after_if: string }[] = [];
|
||||
for (const line of mapText.split(/\r?\n/)) {
|
||||
const s = line.trim();
|
||||
if (!s || s.startsWith("#") || s.toLowerCase().startsWith("old") || s.toLowerCase().startsWith("before")) {
|
||||
continue;
|
||||
}
|
||||
const parts = s.split(/[,|\t]+/).map((x) => x.trim());
|
||||
if (parts.length >= 2 && parts[0] && parts[1]) {
|
||||
rows.push({ before_if: parts[0], after_if: parts[1] });
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
};
|
||||
|
||||
const loadMappingText = (id: string) => {
|
||||
const m = mappings.find((x) => x.id === id);
|
||||
if (!m) return;
|
||||
|
|
@ -1740,17 +1821,52 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
setMapText(m.rows.map((r) => `${r.before_if},${r.after_if}`).join("\n"));
|
||||
};
|
||||
|
||||
const mapParsed = useMemo(() => parseMapText(mapText), [mapText]);
|
||||
const savedMapping = mappings.find((x) => x.id === mappingId);
|
||||
const mapDirty = useMemo(() => mapTextDirty(mapText, savedMapping?.rows), [mapText, savedMapping]);
|
||||
|
||||
/** Persist the editor text. Returns the mapping id, "" for "no mapping", or null when blocked. */
|
||||
const persistMapping = async (): Promise<string | null> => {
|
||||
const parsed = mapParsed;
|
||||
if (parsed.invalidLines.length || parsed.duplicateBefore.length) {
|
||||
showError(t("bizCompare.mapInvalid", {
|
||||
lines: parsed.invalidLines.join(", ") || "-",
|
||||
dup: parsed.duplicateBefore.join(", ") || "-",
|
||||
}));
|
||||
return null;
|
||||
}
|
||||
if (!parsed.rows.length) {
|
||||
if (mappingId && mapDirty) {
|
||||
showError(t("bizCompare.mapEmpty"));
|
||||
return null;
|
||||
}
|
||||
return mappingId;
|
||||
}
|
||||
if (savedMapping) {
|
||||
const own = jobId && jobs.find((j) => j.id === jobId)?.mapping_id === savedMapping.id ? 1 : 0;
|
||||
const jobsUsing = Math.max(0, Number(savedMapping.job_count || 0) - own);
|
||||
const projectsUsing = Number(savedMapping.project_count || 0);
|
||||
if ((jobsUsing > 0 || projectsUsing > 0)
|
||||
&& !window.confirm(t("bizCompare.mapSharedConfirm", { jobs: jobsUsing, projects: projectsUsing }))) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
const rows = parsed.rows;
|
||||
let id = mappingId;
|
||||
if (mappingId) await bizCompareUpdateMapping(mappingId, { name: mapName, rows });
|
||||
else {
|
||||
const m = await bizCompareCreateMapping({ name: mapName, rows });
|
||||
id = String(m.id);
|
||||
}
|
||||
setMappingId(id);
|
||||
await refresh({ force: true });
|
||||
return id;
|
||||
};
|
||||
|
||||
const saveMapping = async () => {
|
||||
setBusy(true);
|
||||
try {
|
||||
const rows = parseMapRows();
|
||||
if (mappingId) await bizCompareUpdateMapping(mappingId, { name: mapName, rows });
|
||||
else {
|
||||
const m = await bizCompareCreateMapping({ name: mapName, rows });
|
||||
setMappingId(String(m.id));
|
||||
}
|
||||
showOk(t("bizCompare.mappingSaved"));
|
||||
await refresh({ force: true });
|
||||
if ((await persistMapping()) !== null) showOk(t("bizCompare.mappingSaved"));
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
} finally {
|
||||
|
|
@ -1758,19 +1874,36 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
};
|
||||
|
||||
/** Job create/save/run must not silently drop edited mapping text: save it first. */
|
||||
const ensureMappingSaved = async (): Promise<string | null> => {
|
||||
if (!mapDirty) return mappingId;
|
||||
const id = await persistMapping();
|
||||
if (id !== null && id !== mappingId) showOk(t("bizCompare.mappingAutoSaved"));
|
||||
return id;
|
||||
};
|
||||
|
||||
const doValidate = async () => {
|
||||
if (!mappingId || !beforeBatchId || !afterBatchId) {
|
||||
showError(t("bizCompare.validateNeed"));
|
||||
if (!beforeBatchId || !afterBatchId) {
|
||||
showError(t("bizCompare.mappingNeedBatches"));
|
||||
return;
|
||||
}
|
||||
if (mapParsed.invalidLines.length || mapParsed.duplicateBefore.length) {
|
||||
showError(t("bizCompare.mapInvalid", {
|
||||
lines: mapParsed.invalidLines.join(", ") || "-",
|
||||
dup: mapParsed.duplicateBefore.join(", ") || "-",
|
||||
}));
|
||||
return;
|
||||
}
|
||||
setBusy(true);
|
||||
try {
|
||||
const v = await bizCompareValidateMapping({
|
||||
mapping_id: mappingId,
|
||||
// The editor text is what gets checked, saved or not.
|
||||
const v = (await bizCompareMappingPreflight({
|
||||
template_id: templateId,
|
||||
enabled_sheet_ids: enabledSheetIds,
|
||||
mapping_rows: mapParsed.rows,
|
||||
before_batch_id: beforeBatchId,
|
||||
after_batch_id: afterBatchId,
|
||||
template_id: templateId,
|
||||
});
|
||||
})) as unknown as PreflightReportData;
|
||||
setValidateOut(v);
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
|
|
@ -1779,6 +1912,13 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
};
|
||||
|
||||
/** Suspected-unmapped ports go into the editor as `port,` lines; the target is left to fill in. */
|
||||
const addSuspectedPorts = (ports: string[]) => {
|
||||
const have = new Set(mapParsed.rows.map((r) => r.before_if));
|
||||
const lines = ports.filter((p) => !have.has(p)).map((p) => `${p},`);
|
||||
if (lines.length) setMapText((prev) => `${prev.replace(/\s+$/, "")}${prev.trim() ? "\n" : ""}${lines.join("\n")}`);
|
||||
};
|
||||
|
||||
const resetJobForm = (preset?: Partial<Job>) => {
|
||||
const tplId = preset?.template_id || templates[0]?.id || "";
|
||||
setName(preset?.name || t("bizCompare.defaultJobName"));
|
||||
|
|
@ -1811,10 +1951,10 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
};
|
||||
|
||||
const jobConfigBody = () => ({
|
||||
const jobConfigBody = (mid: string = mappingId) => ({
|
||||
name,
|
||||
template_id: templateId,
|
||||
mapping_id: mappingId,
|
||||
mapping_id: mid,
|
||||
before_task_id: beforeTaskId,
|
||||
after_task_id: afterTaskId || beforeTaskId,
|
||||
before_batch_id: beforeBatchId,
|
||||
|
|
@ -1903,7 +2043,9 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
setBusy(true);
|
||||
try {
|
||||
const j = await bizCompareCreateJob(jobConfigBody());
|
||||
const mid = await ensureMappingSaved();
|
||||
if (mid === null) return;
|
||||
const j = await bizCompareCreateJob(jobConfigBody(mid));
|
||||
showOk(t("bizCompare.created"));
|
||||
closeCreateJob();
|
||||
await refresh({ force: true });
|
||||
|
|
@ -2003,7 +2145,9 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
setBusy(true);
|
||||
try {
|
||||
await bizCompareUpdateJob(jobId, jobConfigBody());
|
||||
const mid = await ensureMappingSaved();
|
||||
if (mid === null) return;
|
||||
await bizCompareUpdateJob(jobId, jobConfigBody(mid));
|
||||
showOk(t("bizCompare.jobSaved"));
|
||||
await refresh({ force: true });
|
||||
} catch (e) {
|
||||
|
|
@ -2041,7 +2185,9 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const request = jobRequestRef.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
await bizCompareUpdateJob(jobId, jobConfigBody());
|
||||
const mid = await ensureMappingSaved();
|
||||
if (mid === null) return;
|
||||
await bizCompareUpdateJob(jobId, jobConfigBody(mid));
|
||||
// Async enqueue — returns immediately with status=running; poll below.
|
||||
const run = await bizCompareRunJob(jobId);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
|
|
@ -2251,17 +2397,48 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<Input value={mapName} onChange={(e) => setMapName(e.target.value)} />
|
||||
</label>
|
||||
</div>
|
||||
<label className="ui-field ui-field--full">
|
||||
<span className="ui-field__label">{t("bizCompare.mappingPairs")}</span>
|
||||
<textarea
|
||||
className="bs-cmp-mapping__editor"
|
||||
value={mapText}
|
||||
onChange={(e) => setMapText(e.target.value)}
|
||||
placeholder={t("bizCompare.mapHint")}
|
||||
rows={4}
|
||||
spellCheck={false}
|
||||
/>
|
||||
</label>
|
||||
<div className="bs-cmp-mapping__views">
|
||||
{(["text", "table"] as const).map((v) => (
|
||||
<button
|
||||
key={v}
|
||||
type="button"
|
||||
className={`mt-link-btn${mapView === v ? " is-active" : ""}`}
|
||||
aria-pressed={mapView === v}
|
||||
onClick={() => setMapView(v)}
|
||||
>
|
||||
{t(v === "text" ? "bizCompare.mapViewText" : "bizCompare.mapViewTable")}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
{mapView === "table" ? (
|
||||
<MappingTable text={mapText} onChange={setMapText} t={t} />
|
||||
) : (
|
||||
<label className="ui-field ui-field--full">
|
||||
<span className="ui-field__label">{t("bizCompare.mappingPairs")}</span>
|
||||
<textarea
|
||||
className="bs-cmp-mapping__editor"
|
||||
value={mapText}
|
||||
onChange={(e) => setMapText(e.target.value)}
|
||||
placeholder={t("bizCompare.mapHint")}
|
||||
rows={4}
|
||||
spellCheck={false}
|
||||
/>
|
||||
</label>
|
||||
)}
|
||||
<div
|
||||
className={`muted bs-cmp-mapping__status${mapParsed.invalidLines.length || mapParsed.duplicateBefore.length ? " is-error" : ""}`}
|
||||
role="status"
|
||||
data-testid="map-parse-status"
|
||||
>
|
||||
{t("bizCompare.mapParsed", { n: mapParsed.rows.length })}
|
||||
{mapParsed.invalidLines.length
|
||||
? ` · ${t("bizCompare.mapInvalidLines", { m: mapParsed.invalidLines.length, lines: mapParsed.invalidLines.join(", ") })}`
|
||||
: ""}
|
||||
{mapParsed.duplicateBefore.length
|
||||
? ` · ${t("bizCompare.mapDuplicate", { list: mapParsed.duplicateBefore.join(", ") })}`
|
||||
: ""}
|
||||
{mapDirty ? ` · ${t("bizCompare.mapUnsaved")}` : ""}
|
||||
</div>
|
||||
<div className="bs-cmp-mapping__actions">
|
||||
<Button size="sm" variant="secondary" isDisabled={busy} onPress={() => void saveMapping()}>
|
||||
{t("bizCompare.saveMapping")}
|
||||
|
|
@ -2270,11 +2447,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
{t("bizCompare.validateMapping")}
|
||||
</Button>
|
||||
</div>
|
||||
{validateOut ? (
|
||||
<pre className="muted" style={{ fontSize: 12, maxHeight: 120, overflow: "auto" }}>
|
||||
{JSON.stringify(validateOut, null, 2)}
|
||||
</pre>
|
||||
) : null}
|
||||
{validateOut ? <PreflightReport report={validateOut} t={t} onAddSuspects={addSuspectedPorts} /> : null}
|
||||
</div>
|
||||
);
|
||||
|
||||
|
|
@ -2515,6 +2688,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<td>
|
||||
<div className="pt-list-task-name">{tpl.name}</div>
|
||||
{tpl.job_count ? <div className="muted">{t("bizCompare.templateUsage", {n: tpl.job_count})}</div> : null}
|
||||
{tpl.monitor_template_count ? <div className="muted">{t("bizCompare.templateMonitorUsage", {n: tpl.monitor_template_count})}</div> : null}
|
||||
</td>
|
||||
<td>
|
||||
<div className="bs-cmp-metric-chips">
|
||||
|
|
@ -2546,7 +2720,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<Button
|
||||
size="sm"
|
||||
variant="danger"
|
||||
isDisabled={busy || Boolean(tpl.job_count) || jobs.some((j) => j.template_id === tpl.id)}
|
||||
isDisabled={busy || Boolean(tpl.job_count) || Boolean(tpl.monitor_template_count) || jobs.some((j) => j.template_id === tpl.id)}
|
||||
onPress={() => void removeTemplate(tpl.id)}
|
||||
>
|
||||
{t("bizCompare.delete")}
|
||||
|
|
@ -2827,6 +3001,23 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<FieldSelect label={t("bizCompare.portPolicy")} value={activeTplSheet.ignore_port_changes == null ? "auto" : activeTplSheet.ignore_port_changes ? "ignore" : "strict"} onChange={(e) => updateActiveSheet({ignore_port_changes: e.target.value === "auto" ? null : e.target.value === "ignore"})} hint={t("bizCompare.portPolicyHint")}>
|
||||
<option value="auto">{t("bizCompare.portPolicyAuto")}</option><option value="strict">{t("bizCompare.portPolicyStrict")}</option><option value="ignore">{t("bizCompare.portPolicyIgnore")}</option>
|
||||
</FieldSelect>
|
||||
<div className="ct-sheet-options">
|
||||
<FieldSelect label={t("bizCompare.unmappedPortPolicy")} value={activeTplSheet.unmapped_port_policy || "by_name"} hint={t("bizCompare.unmappedPortPolicyHint")}
|
||||
onChange={(e) => updateActiveSheet({ unmapped_port_policy: e.target.value as "by_name" | "ignore_port" })}>
|
||||
<option value="by_name">{t("bizCompare.unmappedByName")}</option>
|
||||
<option value="ignore_port">{t("bizCompare.unmappedIgnorePort")}</option>
|
||||
</FieldSelect>
|
||||
<FieldSelect label={t("bizCompare.duplicatePairing")} value={activeTplSheet.duplicate_pairing || "sorted"} hint={t("bizCompare.duplicatePairingHint")}
|
||||
onChange={(e) => updateActiveSheet({ duplicate_pairing: e.target.value as "sorted" | "ordered" })}>
|
||||
<option value="sorted">{t("bizCompare.pairSorted")}</option>
|
||||
<option value="ordered">{t("bizCompare.pairOrdered")}</option>
|
||||
</FieldSelect>
|
||||
<label className="ct-sheet-options__check">
|
||||
<input type="checkbox" checked={activeTplSheet.builtin_vendor_canonical !== false}
|
||||
onChange={(e) => updateActiveSheet({ builtin_vendor_canonical: e.target.checked ? undefined : false })} />
|
||||
<span>{t("bizCompare.builtinVendorCanonical")}</span>
|
||||
</label>
|
||||
</div>
|
||||
{tplPaneTab === "filters" ? (activeTplFilterGroups === null ? <div className="ct-complex-filters"><p className="muted">{t("bizCompare.complexFiltersPreserved")}</p><pre aria-label={t("bizCompare.tabFilters")}>{JSON.stringify(activeTplSheet.row_filters, null, 2)}</pre></div> : (
|
||||
<TplRowFiltersEditor
|
||||
groups={activeTplFilterGroups}
|
||||
|
|
@ -2849,6 +3040,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<th>{t("bizCompare.tolerance")}</th>
|
||||
<th>{t("bizCompare.displayField")}</th>
|
||||
<th>{t("bizCompare.normalizeField")}</th>
|
||||
<th>{t("bizCompare.valueMap")}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
|
|
@ -2862,7 +3054,8 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const cmpMode = ["number", "int", "float"].includes(rawMode) ? "numeric" : ["pct", "rel"].includes(rawMode) ? "percent" : ["ignore", "skip", "off"].includes(rawMode) ? "eq" : rawMode;
|
||||
const rawNorm = (rule.normalize || "none").trim().toLowerCase();
|
||||
const norm = rawNorm === "strip" ? "none" : rawNorm;
|
||||
const needsTol = cmpMode === "numeric" || cmpMode === "percent";
|
||||
const needsTol = ["numeric", "percent", "numeric_min", "numeric_max"].includes(cmpMode);
|
||||
const vmScope = valueMapScope[f.name] || "both";
|
||||
return (
|
||||
<tr key={f.name}>
|
||||
<td>
|
||||
|
|
@ -2923,19 +3116,23 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
value={cmpMode === "ignore" ? "eq" : cmpMode}
|
||||
aria-label={`${f.name} ${t("bizCompare.compareMode")}`}
|
||||
disabled={!isCompare}
|
||||
onChange={(e) =>
|
||||
onChange={(e) => {
|
||||
const mode = e.target.value;
|
||||
updateActiveSheet({
|
||||
field_rules: upsertFieldRule(activeTplSheet, f.name, {
|
||||
compare: e.target.value,
|
||||
tolerance:
|
||||
e.target.value === "eq" ? undefined : rule.tolerance ?? 0,
|
||||
compare: mode,
|
||||
tolerance: ["eq", "not_worse"].includes(mode) ? undefined : rule.tolerance ?? 0,
|
||||
rank: mode === "not_worse" ? (rule.rank?.length ? rule.rank : ["up", "down"]) : undefined,
|
||||
}),
|
||||
})
|
||||
}
|
||||
});
|
||||
}}
|
||||
>
|
||||
<option value="eq">{t("bizCompare.modeEq")}</option>
|
||||
<option value="numeric">{t("bizCompare.modeNumeric")}</option>
|
||||
<option value="percent">{t("bizCompare.modePercent")}</option>
|
||||
<option value="not_worse">{t("bizCompare.modeNotWorse")}</option>
|
||||
<option value="numeric_min">{t("bizCompare.modeNumericMin")}</option>
|
||||
<option value="numeric_max">{t("bizCompare.modeNumericMax")}</option>
|
||||
</select>
|
||||
</td>
|
||||
<td>
|
||||
|
|
@ -2951,7 +3148,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
? String(rule.tolerance)
|
||||
: ""
|
||||
}
|
||||
placeholder={cmpMode === "percent" ? "%" : ""}
|
||||
placeholder={["percent", "numeric_min", "numeric_max"].includes(cmpMode) ? "%" : ""}
|
||||
disabled={!isCompare || !needsTol}
|
||||
onChange={(e) => {
|
||||
const raw = e.target.value.trim();
|
||||
|
|
@ -2989,7 +3186,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
className="mt-select"
|
||||
value={norm}
|
||||
aria-label={`${f.name} ${t("bizCompare.normalizeField")}`}
|
||||
disabled={!isCompare}
|
||||
disabled={!isCompare && !isKey}
|
||||
onChange={(e) =>
|
||||
updateActiveSheet({
|
||||
field_rules: upsertFieldRule(activeTplSheet, f.name, {
|
||||
|
|
@ -3007,12 +3204,25 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
</option>
|
||||
</select>
|
||||
</td>
|
||||
<td>
|
||||
<ValueMapCell
|
||||
field={f.name}
|
||||
t={t}
|
||||
scope={vmScope}
|
||||
onScope={(sc) => setValueMapScope((prev) => ({ ...prev, [f.name]: sc }))}
|
||||
value={valueMapFor(activeTplSheet, f.name, vmScope)}
|
||||
onCommit={(map) => updateActiveSheet({ field_rules: setValueMapFor(activeTplSheet, f.name, vmScope, map) })}
|
||||
showRank={cmpMode === "not_worse"}
|
||||
rank={rule.rank || []}
|
||||
onRank={(rank) => updateActiveSheet({ field_rules: upsertFieldRule(activeTplSheet, f.name, { compare: "not_worse", rank }) })}
|
||||
/>
|
||||
</td>
|
||||
</tr>
|
||||
);
|
||||
})}
|
||||
{!activeTplFields.length ? (
|
||||
<tr>
|
||||
<td colSpan={8}>
|
||||
<td colSpan={9}>
|
||||
<div className="pt-list-empty">{t("bizCompare.noMetricFields")}</div>
|
||||
</td>
|
||||
</tr>
|
||||
|
|
@ -3767,7 +3977,11 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
{pending ? "—" : ok}
|
||||
</span>
|
||||
<span className="bs-cmp-nav__num bs-cmp-nav__num--rate">
|
||||
{pending || rate === null ? "…" : formatPassRate(rate)}
|
||||
{pending
|
||||
? "…"
|
||||
: (c as { data_state?: string }).data_state && (c as { data_state?: string }).data_state !== "ok"
|
||||
? dataStateLabel((c as { data_state?: string }).data_state, t)
|
||||
: rate === null ? "…" : formatPassRate(rate)}
|
||||
</span>
|
||||
</span>
|
||||
</button>
|
||||
|
|
@ -3777,6 +3991,19 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
</aside>
|
||||
|
||||
<div className="bs-cmp-main">
|
||||
{activeDataState !== "ok" ? (
|
||||
<p className="cx-banner" role="status" data-testid="data-state-banner">
|
||||
<strong>{dataStateLabel(activeDataState, t)}</strong>{" · "}
|
||||
{t(`bizCompare.stateBanner.${activeDataState}`)}
|
||||
{activeSheetCard?.data_state_reasons?.length ? ` (${activeSheetCard.data_state_reasons.join(", ")})` : ""}
|
||||
</p>
|
||||
) : null}
|
||||
{activeCoverage?.suspect_unmapped_total ? (
|
||||
<p className="cx-banner" role="status" data-testid="coverage-banner">
|
||||
{t("bizCompare.coverageBanner", { n: activeCoverage.suspect_unmapped_total })}
|
||||
{` ${(activeCoverage.suspect_unmapped || []).slice(0, 5).join(", ")}`}
|
||||
</p>
|
||||
) : null}
|
||||
<div
|
||||
className={`bs-cmp-strip${
|
||||
activeSheetPending
|
||||
|
|
@ -3821,9 +4048,13 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
</span>
|
||||
</span>
|
||||
<span className="bs-cmp-strip__pass-value">
|
||||
{activeSheetPending || activePassRate === null
|
||||
{activeSheetPending
|
||||
? "…"
|
||||
: formatPassRate(activePassRate)}
|
||||
: activeDataState !== "ok" && activePassRate === null
|
||||
? dataStateLabel(activeDataState, t)
|
||||
: activePassRate === null
|
||||
? "…"
|
||||
: formatPassRate(activePassRate)}
|
||||
</span>
|
||||
</div>
|
||||
<div className="bs-cmp-strip__kinds" role="group">
|
||||
|
|
@ -4159,6 +4390,11 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}`}
|
||||
>
|
||||
{cellText(d.key?.[k] ?? pre[k] ?? post[k]) || "—"}
|
||||
{d.raw_key_before?.[k] !== undefined && cellText(d.raw_key_before[k]) !== cellText(d.key?.[k]) ? (
|
||||
<span className="cx-rawkey" title={t("bizCompare.rawKeyTitle")}>
|
||||
← {cellText(d.raw_key_before[k])}
|
||||
</span>
|
||||
) : null}
|
||||
</td>
|
||||
))}
|
||||
{resultColumns.extras
|
||||
|
|
|
|||
|
|
@ -18,6 +18,9 @@ import {
|
|||
bizMigrationEnsureHighfreq,
|
||||
bizMigrationEvaluate,
|
||||
bizMigrationFinishBatch,
|
||||
bizMigrationGetRun,
|
||||
bizMigrationPreflight,
|
||||
bizMigrationResolveResolvable,
|
||||
bizMigrationGetBoard,
|
||||
bizMigrationListBaselineExpect,
|
||||
bizMigrationListBatches,
|
||||
|
|
@ -40,6 +43,8 @@ import type { CliTargetItem } from "../../types";
|
|||
import { writeClipboardText } from "../../utils/clipboard";
|
||||
import { pageCount } from "../../utils/display";
|
||||
import { formatSystemTime, localDatetimeInputToUtcIso, utcIsoToLocalDatetimeInput } from "../../utils/time";
|
||||
import { PreflightReport } from "./components/PreflightReport";
|
||||
import type { PreflightReportData } from "./components/PreflightReport";
|
||||
import { cutoverCachedGet, cutoverCachedGetSWR, invalidateCutoverCache } from "./lib/cutoverDataCache";
|
||||
import { jobChipColor, NmStatusChip, sourceChipColor } from "../../components/NmChips";
|
||||
|
||||
|
|
@ -287,6 +292,8 @@ type RedTicket = {
|
|||
old_status?: string;
|
||||
new_status?: string;
|
||||
status: string;
|
||||
/** Latest acceptance saw this object turn green; the operator may close it. */
|
||||
resolvable?: boolean;
|
||||
evidence?: DiffRow["evidence"];
|
||||
};
|
||||
type SamplingPair = {
|
||||
|
|
@ -318,6 +325,10 @@ type SheetCard = {
|
|||
new_collected_at?: string;
|
||||
duplicate_keys_before?: number;
|
||||
duplicate_keys_after?: number;
|
||||
duplicate_in_expect?: string[];
|
||||
manual_pending?: boolean;
|
||||
not_collected?: boolean;
|
||||
data_state_reasons?: string[];
|
||||
age_limit_sec?: number;
|
||||
baseline_missing_sides?: string[];
|
||||
};
|
||||
|
|
@ -389,6 +400,9 @@ export function BizMigrationPage() {
|
|||
const [projects, setProjects] = useState<Project[]>([]);
|
||||
const [mappings, setMappings] = useState<PortMapping[]>([]);
|
||||
const [busy, setBusy] = useState(false);
|
||||
// Background evaluate / acceptance: text shown while the run is being computed.
|
||||
const [evalProgress, setEvalProgress] = useState("");
|
||||
const [preflight, setPreflight] = useState<PreflightReportData | null>(null);
|
||||
const [listKeyword, setListKeyword] = useState("");
|
||||
const debouncedListKw = useDebouncedValue(listKeyword, 250);
|
||||
|
||||
|
|
@ -462,6 +476,9 @@ export function BizMigrationPage() {
|
|||
verdict_counts?: Record<string, number>;
|
||||
duplicate_keys?: number;
|
||||
uncovered_scope?: string[];
|
||||
duplicate_warnings?: Array<{ sheet_id?: string; title?: string; count: number; examples?: string[] }>;
|
||||
manual_pending_metrics?: string[];
|
||||
not_collected_metrics?: string[];
|
||||
};
|
||||
} | null;
|
||||
} | null>(null);
|
||||
|
|
@ -1434,18 +1451,53 @@ export function BizMigrationPage() {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Evaluate / acceptance run in the background on the server. Poll the run until it settles;
|
||||
* a failed or interrupted run is reported with the server's reason.
|
||||
*/
|
||||
async function waitForRun(runId: string, label: string): Promise<Record<string, unknown>> {
|
||||
const started = Date.now();
|
||||
for (;;) {
|
||||
const run = (await bizMigrationGetRun(runId)) as {
|
||||
status?: string;
|
||||
message?: string;
|
||||
summary?: { progress?: { sheet_index?: number; sheet_total?: number; sheet_title?: string } };
|
||||
};
|
||||
if (run.status && run.status !== "running") {
|
||||
setEvalProgress("");
|
||||
if (run.status !== "success") throw new Error(run.message || run.status);
|
||||
return run as Record<string, unknown>;
|
||||
}
|
||||
const p = run.summary?.progress;
|
||||
setEvalProgress(
|
||||
p?.sheet_total
|
||||
? t("bizMigration.evalProgress", { label, i: String(p.sheet_index ?? 0), n: String(p.sheet_total), title: p.sheet_title || "" })
|
||||
: t("bizMigration.evalQueued", { label }),
|
||||
);
|
||||
if (Date.now() - started > 30 * 60 * 1000) throw new Error("evaluation_timeout");
|
||||
await new Promise((resolve) => setTimeout(resolve, 1500));
|
||||
}
|
||||
}
|
||||
|
||||
async function setBatchStatus(status: string) {
|
||||
if (!batchId) return;
|
||||
setBusy(true);
|
||||
try {
|
||||
if (status === "review" || status === "done") {
|
||||
const res = (await bizMigrationFinishBatch(batchId, status === "done")) as {
|
||||
let res = (await bizMigrationFinishBatch(batchId, status === "done")) as {
|
||||
run_id?: string;
|
||||
batch?: MigBatch;
|
||||
accept_summary?: MigBatch["accept_summary"];
|
||||
open_red_count?: number;
|
||||
};
|
||||
if (res.run_id && !res.accept_summary) {
|
||||
await waitForRun(res.run_id, t("bizMigration.finishBatch"));
|
||||
}
|
||||
const list = await bizMigrationListBatches(projectId);
|
||||
setBatches((list.items || []) as MigBatch[]);
|
||||
const items = (list.items || []) as MigBatch[];
|
||||
setBatches(items);
|
||||
const closed = items.find((b) => b.id === batchId) || res.batch;
|
||||
res = { ...res, batch: closed, accept_summary: res.accept_summary || closed?.accept_summary };
|
||||
setAcceptInfo(res.accept_summary || res.batch?.accept_summary || null);
|
||||
await loadRedTickets(projectId);
|
||||
if (res.accept_summary?.passed) {
|
||||
|
|
@ -1494,17 +1546,44 @@ export function BizMigrationPage() {
|
|||
if (pinNewBatchId) body.new_batch_id = pinNewBatchId;
|
||||
const run = await bizMigrationEvaluate(batchId, body);
|
||||
const rid = String((run as { id?: string }).id || "");
|
||||
if ((run as { status?: string }).status === "running" && rid) {
|
||||
await waitForRun(rid, t("bizMigration.evaluate"));
|
||||
}
|
||||
await loadRuns(batchId);
|
||||
await loadBoard(batchId, rid);
|
||||
showOk(t("bizMigration.evaluated"));
|
||||
setDetailTab("board");
|
||||
} catch (e) {
|
||||
setEvalProgress("");
|
||||
showError(mapMigrationApiErr(e, t));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function onPreflight() {
|
||||
if (!projectId) return;
|
||||
setBusy(true);
|
||||
try {
|
||||
setPreflight((await bizMigrationPreflight(projectId)) as unknown as PreflightReportData);
|
||||
} catch (e) {
|
||||
showError(mapMigrationApiErr(e, t));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function onResolveResolvable() {
|
||||
if (!projectId) return;
|
||||
try {
|
||||
const out = (await bizMigrationResolveResolvable(projectId)) as { closed?: number };
|
||||
await loadRedTickets(projectId);
|
||||
showOk(t("bizMigration.redResolvableClosed", { n: String(out.closed ?? 0) }));
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
}
|
||||
}
|
||||
|
||||
async function onEnsureHighfreq() {
|
||||
if (!projectId) return;
|
||||
if (!collectMetricIds.length) {
|
||||
|
|
@ -2323,6 +2402,7 @@ export function BizMigrationPage() {
|
|||
{t("bizMigration.evaluate")}
|
||||
</Button>
|
||||
) : null}
|
||||
{evalProgress ? <span className="muted" role="status" data-testid="eval-progress">{evalProgress}</span> : null}
|
||||
<Button
|
||||
size="sm"
|
||||
variant="secondary"
|
||||
|
|
@ -2436,6 +2516,14 @@ export function BizMigrationPage() {
|
|||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
<p className="muted bm-hint" data-testid="port-success-hint">{t("bizMigration.portSuccessHint")}</p>
|
||||
<div className="bm-preflight">
|
||||
<Button size="sm" variant="secondary" isDisabled={busy || !project.old_baseline_batch_id || !project.new_baseline_batch_id} onPress={() => void onPreflight()}>
|
||||
{t("bizMigration.preflight")}
|
||||
</Button>
|
||||
<span className="muted bm-hint">{t("bizMigration.preflightHint")}</span>
|
||||
{preflight ? <PreflightReport report={preflight} t={t} /> : null}
|
||||
</div>
|
||||
<div className="bm-hf-collect">
|
||||
<div className="bm-mapping__label">{t("bizMigration.projectCollectMetrics")}</div>
|
||||
<p className="muted bm-hint">{t("bizMigration.pairCollectionHint")}</p>
|
||||
|
|
@ -3000,6 +3088,27 @@ export function BizMigrationPage() {
|
|||
})}
|
||||
</div>
|
||||
) : null}
|
||||
{sheetCards.some((c) => c.manual_pending) ? (
|
||||
<div className="muted bm-hint" data-testid="manual-pending-banner">
|
||||
{t("bizMigration.manualPendingBanner", {
|
||||
metrics: sheetCards.filter((c) => c.manual_pending).map((c) => sheetLabel(c)).join(", "),
|
||||
})}
|
||||
</div>
|
||||
) : null}
|
||||
{sheetCards.some((c) => c.not_collected) ? (
|
||||
<div className="muted bm-hint" data-testid="not-collected-banner">
|
||||
{t("bizMigration.notCollectedBanner", {
|
||||
metrics: sheetCards.filter((c) => c.not_collected).map((c) => `${sheetLabel(c)} (${(c.data_state_reasons || []).join(", ")})`).join("; "),
|
||||
})}
|
||||
</div>
|
||||
) : null}
|
||||
{liveSummary?.duplicate_warnings?.length ? (
|
||||
<div className="muted bm-hint" data-testid="duplicate-warning-banner">
|
||||
{t("bizMigration.duplicateWarningBanner", {
|
||||
metrics: liveSummary.duplicate_warnings.map((w) => `${w.title || w.sheet_id} (${w.count})`).join(", "),
|
||||
})}
|
||||
</div>
|
||||
) : null}
|
||||
{sheetCards.some((c) => c.current_missing) ? (
|
||||
<div className="form-error bm-hint">
|
||||
{t("bizMigration.currentMissingBanner", {
|
||||
|
|
@ -3036,6 +3145,14 @@ export function BizMigrationPage() {
|
|||
|
||||
{boardSubTab === "red" ? (
|
||||
<div className="pt-list-table-wrap bm-board__table">
|
||||
{redTickets.some((r) => r.resolvable && r.status !== "resolved") ? (
|
||||
<div className="bm-hint" role="status">
|
||||
<span>{t("bizMigration.redResolvableHint", { n: String(redTickets.filter((r) => r.resolvable && r.status !== "resolved").length) })}</span>{" "}
|
||||
<Button size="sm" variant="secondary" onPress={() => void onResolveResolvable()}>
|
||||
{t("bizMigration.redResolveAllResolvable")}
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
<table className="data-table pt-list-table">
|
||||
<thead>
|
||||
<tr>
|
||||
|
|
@ -3060,7 +3177,10 @@ export function BizMigrationPage() {
|
|||
<td>{r.old_status || "—"}</td>
|
||||
<td>{r.new_status || "—"}</td>
|
||||
<td style={{ color: COLOR.red, fontWeight: 600 }}>{verdictLabel(r.verdict)}</td>
|
||||
<td>{r.status}</td>
|
||||
<td>
|
||||
{r.status}
|
||||
{r.resolvable && r.status !== "resolved" ? <span className="muted"> · {t("bizMigration.redNowGreen")}</span> : null}
|
||||
</td>
|
||||
<td>
|
||||
{r.status !== "resolved" ? (
|
||||
<Button
|
||||
|
|
@ -3109,7 +3229,8 @@ export function BizMigrationPage() {
|
|||
const active = boardMetricId === sid;
|
||||
const skipped = Boolean(c.collect_skipped);
|
||||
const noCurrent = Boolean(c.current_missing);
|
||||
const mutedCard = skipped || noCurrent;
|
||||
const pending = Boolean(c.manual_pending || c.not_collected);
|
||||
const mutedCard = skipped || noCurrent || pending;
|
||||
const hot =
|
||||
!mutedCard && ((c.anomaly || 0) > 0 || Boolean(c.new_baseline_missing));
|
||||
return (
|
||||
|
|
@ -3127,7 +3248,9 @@ export function BizMigrationPage() {
|
|||
<span className="bs-cmp-nav__dot" />
|
||||
<span className="bs-cmp-nav__name">
|
||||
{sheetLabel(c)}
|
||||
{skipped
|
||||
{pending
|
||||
? ` · ${t(c.manual_pending ? "bizCompare.dataState.manual_pending" : "bizCompare.dataState.not_collected")}`
|
||||
: skipped
|
||||
? ` · ${t("bizMigration.collectSkipped")}`
|
||||
: noCurrent
|
||||
? ` · ${t("bizMigration.currentMissing")}`
|
||||
|
|
|
|||
110
web/src/features/cutover/components/MappingTable.tsx
Normal file
110
web/src/features/cutover/components/MappingTable.tsx
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
import { useMemo } from "react";
|
||||
|
||||
type T = (key: string, vars?: Record<string, string | number>) => string;
|
||||
|
||||
type Line = { index: number; before: string; after: string };
|
||||
|
||||
/** Lenient parse: a half-filled `port,` line is a row waiting for its target. */
|
||||
function readLines(text: string): Line[] {
|
||||
const out: Line[] = [];
|
||||
text.split(/\r?\n/).forEach((raw, index) => {
|
||||
const s = raw.trim();
|
||||
if (!s || s.startsWith("#")) return;
|
||||
const delimited = /[,|\t]/.test(s);
|
||||
const parts = (delimited ? s.split(/[,|\t]+/) : s.split(/\s+/)).map((x) => x.trim());
|
||||
out.push({ index, before: parts[0] || "", after: parts.slice(1).join(" ").trim() });
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Table view over the mapping text. The text stays the single source of truth, so
|
||||
* comments, order and the "save" flow are shared with the text editor.
|
||||
*/
|
||||
export function MappingTable({ text, onChange, t }: { text: string; onChange: (next: string) => void; t: T }) {
|
||||
const rows = useMemo(() => readLines(text), [text]);
|
||||
const seen = new Map<string, number>();
|
||||
rows.forEach((r) => seen.set(r.before, (seen.get(r.before) || 0) + 1));
|
||||
|
||||
const replaceLine = (index: number, line: string | null) => {
|
||||
const lines = text.split(/\r?\n/);
|
||||
if (line === null) lines.splice(index, 1);
|
||||
else lines[index] = line;
|
||||
onChange(lines.join("\n"));
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="cx-maptable" data-testid="mapping-table">
|
||||
<table className="data-table pt-list-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>{t("bizCompare.mapTable.before")}</th>
|
||||
<th>{t("bizCompare.mapTable.after")}</th>
|
||||
<th>{t("bizCompare.mapTable.status")}</th>
|
||||
<th />
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{rows.map((r) => {
|
||||
const status = !r.before
|
||||
? "invalid"
|
||||
: (seen.get(r.before) || 0) > 1
|
||||
? "duplicate"
|
||||
: !r.after
|
||||
? "todo"
|
||||
: "ok";
|
||||
return (
|
||||
<tr key={r.index} className={`cx-maptable__row is-${status}`}>
|
||||
<td>
|
||||
<input
|
||||
className="ui-field__input cx-maptable__input"
|
||||
aria-label={`${t("bizCompare.mapTable.before")} ${r.index + 1}`}
|
||||
value={r.before}
|
||||
onChange={(e) => replaceLine(r.index, `${e.target.value},${r.after}`)}
|
||||
/>
|
||||
</td>
|
||||
<td>
|
||||
<input
|
||||
className="ui-field__input cx-maptable__input"
|
||||
aria-label={`${t("bizCompare.mapTable.after")} ${r.index + 1}`}
|
||||
value={r.after}
|
||||
onChange={(e) => replaceLine(r.index, `${r.before},${e.target.value}`)}
|
||||
/>
|
||||
</td>
|
||||
<td>
|
||||
<span className={`cx-state cx-state--${status === "ok" ? "ok" : status === "todo" ? "no_data" : "suspect_parse"}`}>
|
||||
{t(`bizCompare.mapTable.${status}`)}
|
||||
</span>
|
||||
</td>
|
||||
<td>
|
||||
<button
|
||||
type="button"
|
||||
className="mt-icon-btn"
|
||||
aria-label={`${t("bizCompare.mapTable.remove")} ${r.index + 1}`}
|
||||
onClick={() => replaceLine(r.index, null)}
|
||||
>
|
||||
×
|
||||
</button>
|
||||
</td>
|
||||
</tr>
|
||||
);
|
||||
})}
|
||||
{!rows.length ? (
|
||||
<tr>
|
||||
<td colSpan={4} className="muted">
|
||||
{t("bizCompare.mapTable.empty")}
|
||||
</td>
|
||||
</tr>
|
||||
) : null}
|
||||
</tbody>
|
||||
</table>
|
||||
<button
|
||||
type="button"
|
||||
className="mt-link-btn"
|
||||
onClick={() => onChange(`${text.replace(/\s+$/, "")}${text.trim() ? "\n" : ""},`)}
|
||||
>
|
||||
{t("bizCompare.mapTable.add")}
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
205
web/src/features/cutover/components/PreflightReport.tsx
Normal file
205
web/src/features/cutover/components/PreflightReport.tsx
Normal file
|
|
@ -0,0 +1,205 @@
|
|||
import type { ReactNode } from "react";
|
||||
|
||||
type T = (key: string, vars?: Record<string, string | number>) => string;
|
||||
|
||||
export type PreflightSheet = {
|
||||
sheet_id: string;
|
||||
title?: string;
|
||||
metric_id: string;
|
||||
before_count?: number;
|
||||
after_count?: number;
|
||||
skipped?: string;
|
||||
data_state?: string;
|
||||
data_state_reasons?: string[];
|
||||
counts?: Record<string, number>;
|
||||
mapped_exact?: string[];
|
||||
mapped_by_parent?: string[];
|
||||
identity_same_name?: string[];
|
||||
suspect_unmapped?: string[];
|
||||
suspect_unmapped_total?: number;
|
||||
mapped_target_missing_on_after?: string[];
|
||||
};
|
||||
|
||||
export type PreflightReportData = {
|
||||
verdict: "ready" | "risk";
|
||||
risk_reasons: string[];
|
||||
coverage: { mapped: number; suspect_unmapped: number; identity_same_name: number; rate: number | null };
|
||||
mapping: {
|
||||
pairs: number;
|
||||
conflicts?: Array<Record<string, string>>;
|
||||
duplicate_before?: Array<Record<string, string>>;
|
||||
unused_map_keys?: string[];
|
||||
unused_map_keys_total?: number;
|
||||
map_targets_missing_on_after?: string[];
|
||||
map_targets_missing_on_after_total?: number;
|
||||
n_to_1_targets?: Array<{ target: string; sources: string[] }>;
|
||||
};
|
||||
sheets: PreflightSheet[];
|
||||
command_coverage?: { only_before?: string[]; only_after?: string[] };
|
||||
};
|
||||
|
||||
export function dataStateLabel(state: string | undefined, t: T): string {
|
||||
return t(`bizCompare.dataState.${state || "ok"}`);
|
||||
}
|
||||
|
||||
function List({ items, total, children }: { items?: string[]; total?: number; children?: ReactNode }) {
|
||||
if (!items?.length) return null;
|
||||
const more = (total ?? items.length) - items.length;
|
||||
return (
|
||||
<ul className="cx-preflight__list">
|
||||
{items.map((x) => (
|
||||
<li key={x}>
|
||||
<code>{x}</code>
|
||||
</li>
|
||||
))}
|
||||
{more > 0 ? <li className="muted">+{more}</li> : null}
|
||||
{children}
|
||||
</ul>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-flight result shared by the compare page ("validate mapping") and the cutover
|
||||
* project: mapping coverage first, then per-sheet data state and details.
|
||||
*/
|
||||
export function PreflightReport({
|
||||
report,
|
||||
t,
|
||||
onAddSuspects,
|
||||
}: {
|
||||
report: PreflightReportData;
|
||||
t: T;
|
||||
/** Compare editor only: append unmapped before-side ports as empty rows to the mapping text. */
|
||||
onAddSuspects?: (ports: string[]) => void;
|
||||
}) {
|
||||
const m = report.mapping;
|
||||
const suspects = report.sheets.flatMap((s) => s.suspect_unmapped || []);
|
||||
const issues: Array<[string, ReactNode]> = [];
|
||||
if (m.duplicate_before?.length) {
|
||||
issues.push([
|
||||
"duplicate",
|
||||
<List key="d" items={m.duplicate_before.map((d) => `${d.before_if} → ${d.kept_to} / ${d.dropped_to}`)} />,
|
||||
]);
|
||||
}
|
||||
if (m.conflicts?.length) {
|
||||
issues.push([
|
||||
"conflicts",
|
||||
<List key="c" items={m.conflicts.map((c) => `${c.dropped_from} ↔ ${c.kept_from}`)} />,
|
||||
]);
|
||||
}
|
||||
if (m.unused_map_keys?.length) {
|
||||
issues.push(["unused", <List key="u" items={m.unused_map_keys} total={m.unused_map_keys_total} />]);
|
||||
}
|
||||
if (m.map_targets_missing_on_after?.length) {
|
||||
issues.push([
|
||||
"missingTargets",
|
||||
<List key="t" items={m.map_targets_missing_on_after} total={m.map_targets_missing_on_after_total} />,
|
||||
]);
|
||||
}
|
||||
if (m.n_to_1_targets?.length) {
|
||||
issues.push([
|
||||
"nTo1",
|
||||
<List key="n" items={m.n_to_1_targets.map((x) => `${x.sources.join(", ")} → ${x.target}`)} />,
|
||||
]);
|
||||
}
|
||||
const rate = report.coverage.rate;
|
||||
return (
|
||||
<div className={`cx-preflight is-${report.verdict}`} data-testid="preflight-report">
|
||||
<div className="cx-preflight__head" role="status">
|
||||
<strong>{t(`bizCompare.preflight.${report.verdict}`)}</strong>
|
||||
<span>
|
||||
{t("bizCompare.preflight.coverage", {
|
||||
mapped: report.coverage.mapped,
|
||||
suspect: report.coverage.suspect_unmapped,
|
||||
rate: rate === null ? "—" : `${rate}%`,
|
||||
})}
|
||||
</span>
|
||||
<span className="muted">{t("bizCompare.preflight.pairs", { n: m.pairs })}</span>
|
||||
</div>
|
||||
|
||||
{suspects.length ? (
|
||||
<div className="cx-preflight__block">
|
||||
<h5>{t("bizCompare.preflight.suspect")}</h5>
|
||||
<List items={suspects.slice(0, 200)} total={report.coverage.suspect_unmapped} />
|
||||
{onAddSuspects ? (
|
||||
<button type="button" className="mt-link-btn" onClick={() => onAddSuspects(suspects)}>
|
||||
{t("bizCompare.preflight.addSuspects")}
|
||||
</button>
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{issues.map(([key, body]) => (
|
||||
<div className="cx-preflight__block" key={key}>
|
||||
<h5>{t(`bizCompare.preflight.issue.${key}`)}</h5>
|
||||
{body}
|
||||
</div>
|
||||
))}
|
||||
|
||||
{report.command_coverage?.only_before?.length || report.command_coverage?.only_after?.length ? (
|
||||
<div className="cx-preflight__block">
|
||||
<h5>{t("bizCompare.preflight.commandDiff")}</h5>
|
||||
<List items={(report.command_coverage.only_before || []).map((x) => `${t("bizCompare.scopeBefore")}: ${x}`)} />
|
||||
<List items={(report.command_coverage.only_after || []).map((x) => `${t("bizCompare.scopeAfter")}: ${x}`)} />
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="cx-preflight__scroll">
|
||||
<table className="data-table pt-list-table cx-preflight__table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>{t("bizCompare.field")}</th>
|
||||
<th>{t("bizCompare.preflight.rows")}</th>
|
||||
<th>{t("bizCompare.preflight.state")}</th>
|
||||
<th>{t("bizCompare.preflight.mapping")}</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{report.sheets.map((s) => {
|
||||
const state = s.data_state || "ok";
|
||||
const c = s.counts || {};
|
||||
return (
|
||||
<tr key={s.sheet_id}>
|
||||
<td>
|
||||
<code>{s.title || s.sheet_id}</code>
|
||||
</td>
|
||||
<td>{`${s.before_count ?? 0} → ${s.after_count ?? 0}`}</td>
|
||||
<td>
|
||||
<span className={`cx-state cx-state--${state}`}>{dataStateLabel(state, t)}</span>
|
||||
{s.data_state_reasons?.length ? (
|
||||
<div className="muted cx-state__why">{s.data_state_reasons.join(" · ")}</div>
|
||||
) : null}
|
||||
</td>
|
||||
<td>
|
||||
{s.skipped ? (
|
||||
<span className="muted">{t("bizCompare.preflight.skippedHeavy")}</span>
|
||||
) : s.counts ? (
|
||||
<details>
|
||||
<summary>
|
||||
{t("bizCompare.preflight.sheetCounts", {
|
||||
exact: (c.mapped_exact || 0) + (c.mapped_by_parent || 0),
|
||||
same: c.identity_same_name || 0,
|
||||
suspect: s.suspect_unmapped_total ?? c.suspect_unmapped ?? 0,
|
||||
})}
|
||||
</summary>
|
||||
<List items={s.suspect_unmapped} total={s.suspect_unmapped_total} />
|
||||
{s.mapped_target_missing_on_after?.length ? (
|
||||
<>
|
||||
<h6>{t("bizCompare.preflight.issue.missingTargets")}</h6>
|
||||
<List items={s.mapped_target_missing_on_after} />
|
||||
</>
|
||||
) : null}
|
||||
</details>
|
||||
) : (
|
||||
<span className="muted">—</span>
|
||||
)}
|
||||
</td>
|
||||
</tr>
|
||||
);
|
||||
})}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
|
@ -10,14 +10,28 @@ export type MetricSheet = {
|
|||
row_filters?: RowFilter[];
|
||||
field_rules?: FieldRule[];
|
||||
ignore_port_changes?: boolean | null;
|
||||
/** Partial port map: unmapped ports match by name (default) or by content only. */
|
||||
unmapped_port_policy?: "" | "by_name" | "ignore_port";
|
||||
/** Same-key rows pair by value (default for new runs) or by collection order. */
|
||||
duplicate_pairing?: "" | "sorted" | "ordered";
|
||||
/** false = skip the built-in per-vendor interface names (GigabitEthernet → GE …). */
|
||||
builtin_vendor_canonical?: boolean;
|
||||
};
|
||||
|
||||
export type Side = "" | "both" | "before" | "after";
|
||||
|
||||
export type FieldRule = {
|
||||
field: string;
|
||||
compare?: string;
|
||||
normalize?: string;
|
||||
ignore?: boolean;
|
||||
tolerance?: number;
|
||||
/** Cross-vendor value aliases, e.g. {"estab": "established"}; matched case-insensitively. */
|
||||
value_map?: Record<string, string>;
|
||||
side?: Side;
|
||||
apply_to_key?: boolean;
|
||||
/** Order for compare=not_worse: earlier entries are better. */
|
||||
rank?: string[];
|
||||
};
|
||||
|
||||
export type RowFilter = {
|
||||
|
|
@ -26,13 +40,50 @@ export type RowFilter = {
|
|||
value?: string | number | boolean | (string | number | boolean)[];
|
||||
any?: RowFilter[];
|
||||
all?: RowFilter[];
|
||||
/** Only evaluate this filter on one side of the comparison. */
|
||||
side?: Side;
|
||||
};
|
||||
|
||||
export type IfaceRule = { from: string; to: string; side?: Side; vendor?: string };
|
||||
|
||||
export const DIRECTIONAL_COMPARE = ["not_worse", "numeric_min", "numeric_max"] as const;
|
||||
export const MAX_VALUE_MAP = 500;
|
||||
|
||||
const SHEET_OPTION_KEYS = ["unmapped_port_policy", "duplicate_pairing", "builtin_vendor_canonical"] as const;
|
||||
|
||||
/** The sheet exactly as the API stores it: every identity/compare setting survives a round trip. */
|
||||
export function sheetPayload(s: MetricSheet, identity?: string): Record<string, unknown> {
|
||||
const id = identity ?? (s.sheet_id || s.metric_id);
|
||||
const out: Record<string, unknown> = {
|
||||
sheet_id: id,
|
||||
title: s.title || id,
|
||||
metric_id: s.metric_id,
|
||||
key_fields: s.key_fields,
|
||||
iface_fields: s.iface_fields,
|
||||
compare_fields: s.compare_fields,
|
||||
display_fields: s.display_fields || [...s.key_fields, ...s.compare_fields],
|
||||
row_filters: s.row_filters || [],
|
||||
field_rules: s.field_rules || [],
|
||||
ignore_port_changes: s.ignore_port_changes,
|
||||
};
|
||||
for (const key of SHEET_OPTION_KEYS) {
|
||||
const value = s[key];
|
||||
if (value !== undefined && value !== "") out[key] = value;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function sheetOptions(raw: Record<string, unknown>): Partial<MetricSheet> {
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const key of SHEET_OPTION_KEYS) if (raw[key] !== undefined && raw[key] !== null) out[key] = raw[key];
|
||||
return out as Partial<MetricSheet>;
|
||||
}
|
||||
|
||||
export function parseTemplateImport(raw: unknown): {
|
||||
name: string;
|
||||
note: string;
|
||||
metrics: MetricSheet[];
|
||||
iface_normalize_rules?: { from: string; to: string }[];
|
||||
iface_normalize_rules?: IfaceRule[];
|
||||
} | null {
|
||||
if (!raw || typeof raw !== "object" || Array.isArray(raw)) return null;
|
||||
const o = raw as Record<string, unknown>;
|
||||
|
|
@ -89,11 +140,12 @@ export function parseTemplateImport(raw: unknown): {
|
|||
row_filters: (m.row_filters ?? []) as RowFilter[],
|
||||
field_rules: fieldRules,
|
||||
ignore_port_changes: m.ignore_port_changes as boolean | null | undefined,
|
||||
...sheetOptions(m),
|
||||
});
|
||||
}
|
||||
if (!metrics.length) return null;
|
||||
const normRaw = o.iface_normalize_rules;
|
||||
const iface_normalize_rules: { from: string; to: string }[] = [];
|
||||
const iface_normalize_rules: IfaceRule[] = [];
|
||||
if (Array.isArray(normRaw)) {
|
||||
for (const item of normRaw) {
|
||||
if (!item || typeof item !== "object") throw new Error("iface_normalize_rules: alias_pair_required");
|
||||
|
|
@ -101,8 +153,13 @@ export function parseTemplateImport(raw: unknown): {
|
|||
const fr = String(r.from || "").trim();
|
||||
const to = String(r.to || "").trim();
|
||||
if (!fr || !to) throw new Error("iface_normalize_rules: alias_pair_required");
|
||||
if (iface_normalize_rules.some((x) => x.from.toLowerCase() === fr.toLowerCase())) throw new Error("iface_normalize_rules: duplicate_alias");
|
||||
iface_normalize_rules.push({ from: fr, to });
|
||||
const rule: IfaceRule = { from: fr, to };
|
||||
const side = String(r.side || "").trim().toLowerCase();
|
||||
if (side === "before" || side === "after") rule.side = side;
|
||||
const vendor = String(r.vendor || "").trim().toLowerCase();
|
||||
if (vendor) rule.vendor = vendor;
|
||||
if (iface_normalize_rules.some((x) => sameAlias(x, rule))) throw new Error("iface_normalize_rules: duplicate_alias");
|
||||
iface_normalize_rules.push(rule);
|
||||
}
|
||||
}
|
||||
if (normRaw != null && !Array.isArray(normRaw)) throw new Error("iface_normalize_rules: list_required");
|
||||
|
|
@ -110,35 +167,54 @@ export function parseTemplateImport(raw: unknown): {
|
|||
return { name, note, metrics, iface_normalize_rules };
|
||||
}
|
||||
|
||||
export function parseIfaceNormText(text: string): { from: string; to: string }[] {
|
||||
const out: { from: string; to: string }[] = [];
|
||||
function sameAlias(a: IfaceRule, b: IfaceRule): boolean {
|
||||
return a.from.toLowerCase() === b.from.toLowerCase() && (a.side || "") === (b.side || "") && (a.vendor || "") === (b.vendor || "");
|
||||
}
|
||||
|
||||
/** `from,to[,side[,vendor]]` per line; side is before/after, vendor a vendor key such as huawei. */
|
||||
export function parseIfaceNormText(text: string): IfaceRule[] {
|
||||
const out: IfaceRule[] = [];
|
||||
for (const [index, line] of text.split(/\r?\n/).entries()) {
|
||||
const s = line.trim();
|
||||
if (!s || s.startsWith("#")) continue;
|
||||
let fr = "";
|
||||
let to = "";
|
||||
if (s.includes(",")) {
|
||||
const i = s.indexOf(",");
|
||||
fr = s.slice(0, i).trim();
|
||||
to = s.slice(i + 1).trim();
|
||||
} else if (s.includes("\t")) {
|
||||
const i = s.indexOf("\t");
|
||||
fr = s.slice(0, i).trim();
|
||||
to = s.slice(i + 1).trim();
|
||||
} else {
|
||||
const parts = s.split(/\s+/);
|
||||
if (parts.length >= 2) {
|
||||
fr = parts[0];
|
||||
to = parts.slice(1).join(" ");
|
||||
}
|
||||
}
|
||||
const parts = (s.includes(",") ? s.split(",") : s.includes("\t") ? s.split("\t") : s.split(/\s+/)).map((x) => x.trim());
|
||||
const [fr = "", to = "", side = "", vendor = ""] = parts;
|
||||
if (!fr || !to) throw new Error(`iface_normalize_rules:${index + 1}: alias_pair_required`);
|
||||
if (out.some((x) => x.from.toLowerCase() === fr.toLowerCase())) throw new Error(`iface_normalize_rules:${index + 1}: duplicate_alias`);
|
||||
out.push({ from: fr, to });
|
||||
const sideN = side.toLowerCase();
|
||||
if (sideN && !["before", "after", "both"].includes(sideN)) throw new Error(`iface_normalize_rules:${index + 1}: unknown_alias_side`);
|
||||
const rule: IfaceRule = { from: fr, to };
|
||||
if (sideN === "before" || sideN === "after") rule.side = sideN;
|
||||
if (vendor) rule.vendor = vendor.toLowerCase();
|
||||
if (out.some((x) => sameAlias(x, rule))) throw new Error(`iface_normalize_rules:${index + 1}: duplicate_alias`);
|
||||
out.push(rule);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function formatIfaceNormText(rules?: IfaceRule[]): string {
|
||||
return (rules || [])
|
||||
.map((r) => [r.from, r.to, r.side || (r.vendor ? "both" : ""), r.vendor || ""].filter((x, i, arr) => x || i < 2 || arr.slice(i).some(Boolean)).join(","))
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
/** `source,canonical` per line (blank lines and # comments ignored) → value_map. */
|
||||
export function parseValueMapText(text: string): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const [index, line] of text.split(/\r?\n/).entries()) {
|
||||
const s = line.trim();
|
||||
if (!s || s.startsWith("#")) continue;
|
||||
const parts = (s.includes(",") ? s.split(",") : s.includes("\t") ? s.split("\t") : s.split(/\s+/)).map((x) => x.trim());
|
||||
if (parts.length < 2 || !parts[0] || !parts[1]) throw new Error(`value_map:${index + 1}: pair_required`);
|
||||
out[parts[0]] = parts.slice(1).join(" ");
|
||||
}
|
||||
if (Object.keys(out).length > MAX_VALUE_MAP) throw new Error("value_map: too_large");
|
||||
return out;
|
||||
}
|
||||
|
||||
export function formatValueMapText(map?: Record<string, string>): string {
|
||||
return Object.entries(map || {}).map(([k, v]) => `${k},${v}`).join("\n");
|
||||
}
|
||||
|
||||
function isLeafFilter(f: RowFilter): boolean {
|
||||
return Boolean(f && !f.any && !f.all && f.field);
|
||||
}
|
||||
|
|
@ -182,9 +258,17 @@ export function validateTemplateDraft(metrics: MetricSheet[]): void {
|
|||
for (const key of ["iface_fields", "compare_fields", "display_fields"] as const) if (s[key] !== undefined) stringList(s[key], `${path}.${key}`);
|
||||
const fields = new Set<string>();
|
||||
for (const r of s.field_rules || []) {
|
||||
if (!r?.field?.trim() || fields.has(r.field.trim())) throw new Error(`${path}.field_rules: invalid_or_duplicate_field`);
|
||||
fields.add(r.field.trim());
|
||||
if (!["", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel"].includes((r.compare || "").toLowerCase())) throw new Error(`${path}.field_rules: unknown_compare_mode`);
|
||||
// one rule per (field, side): a field may carry a different value_map on each side
|
||||
const marker = `${r?.field?.trim()}\u0000${r?.side === "before" || r?.side === "after" ? r.side : ""}`;
|
||||
if (!r?.field?.trim() || fields.has(marker)) throw new Error(`${path}.field_rules: invalid_or_duplicate_field`);
|
||||
fields.add(marker);
|
||||
if (!["", "eq", "ignore", "skip", "off", "numeric", "number", "int", "float", "percent", "pct", "rel", ...DIRECTIONAL_COMPARE].includes((r.compare || "").toLowerCase())) throw new Error(`${path}.field_rules: unknown_compare_mode`);
|
||||
if ((r.compare || "").toLowerCase() === "not_worse" && (!Array.isArray(r.rank) || r.rank.length < 2)) throw new Error(`${path}.${r.field}: rank_list_required`);
|
||||
if (r.value_map != null) {
|
||||
const entries = Object.entries(r.value_map);
|
||||
if (typeof r.value_map !== "object" || entries.some(([k, v]) => typeof k !== "string" || typeof v !== "string")) throw new Error(`${path}.${r.field}: value_map_string_pairs_required`);
|
||||
if (entries.length > MAX_VALUE_MAP) throw new Error(`${path}.${r.field}: value_map_too_large`);
|
||||
}
|
||||
if (!["", "none", "strip", "lower", "upper", "mac", "empty_as_blank"].includes((r.normalize || "").toLowerCase())) throw new Error(`${path}.field_rules: unknown_normalize_mode`);
|
||||
if (r.tolerance != null && (typeof r.tolerance !== "number" || !Number.isFinite(r.tolerance) || r.tolerance < 0)) throw new Error(`${path}.${r.field}: finite_nonnegative_tolerance_required`);
|
||||
}
|
||||
|
|
@ -219,6 +303,7 @@ export function fromOrGroups(groups: RowFilter[][]): RowFilter[] {
|
|||
field: f.field || "",
|
||||
op: f.op || "eq",
|
||||
value: f.value ?? "",
|
||||
...(f.side === "before" || f.side === "after" ? { side: f.side } : {}),
|
||||
}))
|
||||
.filter((f) => f.field),
|
||||
)
|
||||
|
|
|
|||
52
web/src/features/cutover/lib/mappingText.ts
Normal file
52
web/src/features/cutover/lib/mappingText.ts
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
export type MapRow = { before_if: string; after_if: string };
|
||||
|
||||
export type MapParse = {
|
||||
rows: MapRow[];
|
||||
/** 1-based line numbers that could not be read as a "before, after" pair. */
|
||||
invalidLines: number[];
|
||||
/** before_if values that appear on more than one line (first one is kept). */
|
||||
duplicateBefore: string[];
|
||||
};
|
||||
|
||||
const HEADER = /^(old|before)\w*[\s,|]+(new|after)/i;
|
||||
|
||||
/**
|
||||
* Parse the port-mapping editor text. Pairs are split on `,` `|` or Tab; a line
|
||||
* with exactly two whitespace-separated tokens is accepted too. Anything else is
|
||||
* reported in `invalidLines` instead of being dropped silently.
|
||||
*/
|
||||
export function parseMapText(text: string): MapParse {
|
||||
const rows: MapRow[] = [];
|
||||
const invalidLines: number[] = [];
|
||||
const duplicateBefore: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
String(text ?? "").split(/\r?\n/).forEach((line, index) => {
|
||||
const s = line.trim();
|
||||
if (!s || s.startsWith("#") || HEADER.test(s)) return;
|
||||
const delimited = /[,|\t]/.test(s);
|
||||
const parts = (delimited ? s.split(/[,|\t]+/) : s.split(/\s+/)).map((x) => x.trim()).filter(Boolean);
|
||||
if (parts.length !== 2) {
|
||||
invalidLines.push(index + 1);
|
||||
return;
|
||||
}
|
||||
const [before_if, after_if] = parts;
|
||||
if (seen.has(before_if)) {
|
||||
if (!duplicateBefore.includes(before_if)) duplicateBefore.push(before_if);
|
||||
return;
|
||||
}
|
||||
seen.add(before_if);
|
||||
rows.push({ before_if, after_if });
|
||||
});
|
||||
return { rows, invalidLines, duplicateBefore };
|
||||
}
|
||||
|
||||
const pairKey = (r: MapRow) => `${r.before_if}\u0000${r.after_if}`;
|
||||
|
||||
/** True when the editor text differs from the saved rows (order-insensitive). */
|
||||
export function mapTextDirty(text: string, saved: MapRow[] | undefined): boolean {
|
||||
const parsed = parseMapText(text);
|
||||
if (parsed.invalidLines.length || parsed.duplicateBefore.length) return true;
|
||||
const a = parsed.rows.map(pairKey).sort();
|
||||
const b = (saved ?? []).map(pairKey).sort();
|
||||
return a.length !== b.length || a.some((v, i) => v !== b[i]);
|
||||
}
|
||||
|
|
@ -67,7 +67,7 @@ const bizCompare = {
|
|||
rowFilters: "Row filter rules",
|
||||
rowFiltersHint:
|
||||
"Filter rows before compare. Conditions in a group are AND; groups are OR. Selecting ARP seeds editable default rules you can change.",
|
||||
rowFiltersHintShort: "AND in group · OR across groups · empty = all rows",
|
||||
rowFiltersHintShort: "AND in group · OR across groups · empty = all rows · filters run after normalization: write canonical values",
|
||||
rowFiltersEmpty: "No filters → compare all rows.",
|
||||
filterClear: "Clear",
|
||||
filterAddOr: "Add group (OR)",
|
||||
|
|
@ -111,12 +111,82 @@ const bizCompare = {
|
|||
validateNeed: "Select mapping and before/after batches first",
|
||||
mapHint:
|
||||
"One pair per line: before_if,after_if (comma / tab / pipe). Example:\ngei-0/1/1,gei-0/1/2\nxgei-0/1/0,xgei-0/2/0\n# comments start with #. Not slash-grouped A,B/C,D.",
|
||||
templateMonitorUsage: "Used by {{n}} monitor templates",
|
||||
mapParsed: "{{n}} pairs parsed",
|
||||
mapInvalidLines: "{{m}} invalid (line {{lines}})",
|
||||
mapDuplicate: "duplicate before port: {{list}}",
|
||||
mapUnsaved: "unsaved",
|
||||
mapInvalid: "Mapping text has errors and was not saved. Invalid lines: {{lines}}; duplicate before ports: {{dup}}",
|
||||
mapEmpty: "Mapping text is empty. Pick \"New mapping…\" if you want no mapping, otherwise restore the pairs before saving.",
|
||||
mapSharedConfirm: "This mapping is also used by {{jobs}} other compare jobs and {{projects}} cutover projects; saving affects them too. Continue?",
|
||||
mappingAutoSaved: "Mapping saved automatically",
|
||||
dataState: {
|
||||
"ok": "OK",
|
||||
"no_data": "No data",
|
||||
"suspect_parse": "Suspect parse",
|
||||
"not_collected": "Not collected",
|
||||
"manual_pending": "Manual pending"
|
||||
},
|
||||
stateBanner: {
|
||||
"no_data": "Both sides are empty: nothing to compare, excluded from the pass rate.",
|
||||
"suspect_parse": "The device printed output but 0 rows were parsed. Failures may be a parsing problem rather than lost business; check the parser/template first.",
|
||||
"not_collected": "One side never collected this check (or the slice). Not judged, excluded from the pass rate."
|
||||
},
|
||||
rawKeyTitle: "Before-side name as printed by the device",
|
||||
scopeBoth: "Both sides",
|
||||
scopeBefore: "Before only",
|
||||
scopeAfter: "After only",
|
||||
valueMap: "Value map",
|
||||
valueMapScope: "value map scope",
|
||||
valueMapPlaceholder: "One per line: source,canonical\nestab,established",
|
||||
rankLabel: "best→worst",
|
||||
rankPlaceholder: "up,down (earlier is better)",
|
||||
modeNotWorse: "Not worse",
|
||||
modeNumericMin: "Not below (%)",
|
||||
modeNumericMax: "Not above (%)",
|
||||
filterSide: "filter side",
|
||||
unmappedPortPolicy: "Unmapped ports",
|
||||
unmappedByName: "Match by name (default)",
|
||||
unmappedIgnorePort: "Ignore port, match by content",
|
||||
unmappedPortPolicyHint: "With a partial port map, how ports that have no mapping are paired.",
|
||||
duplicatePairing: "Same-key pairing",
|
||||
pairSorted: "Pair by sorted value (recommended)",
|
||||
pairOrdered: "Pair by collection order (legacy)",
|
||||
duplicatePairingHint: "When one key has several rows (ECMP, multipath), CLI ordering changes must not create false differences.",
|
||||
builtinVendorCanonical: "Built-in per-vendor interface names (GigabitEthernet↔GE …)",
|
||||
mappingNeedBatches: "Pick the before and after batches first",
|
||||
preflight: {
|
||||
"run": "Validate mapping",
|
||||
"ready": "Mapping looks usable",
|
||||
"risk": "Risks found — fix before running",
|
||||
"coverage": "{{mapped}} mapped · {{suspect}} suspected unmapped · coverage {{rate}}",
|
||||
"pairs": "{{n}} map pairs",
|
||||
"suspect": "Suspected unmapped (on before, no mapping, absent on after)",
|
||||
"addSuspects": "Add to the mapping editor (targets to fill in)",
|
||||
"commandDiff": "Collected commands differ between sides",
|
||||
"rows": "Rows (before→after)",
|
||||
"state": "Data state",
|
||||
"mapping": "Mapping",
|
||||
"skippedHeavy": "Large table, interfaces not enumerated",
|
||||
"sheetCounts": "{{exact}} mapped · {{same}} same name · {{suspect}} suspected unmapped",
|
||||
"issue": {
|
||||
"duplicate": "Duplicate before port (first one wins)",
|
||||
"conflicts": "Before ports that collide after normalizing",
|
||||
"unused": "Map keys that match no interface",
|
||||
"missingTargets": "Map targets missing on the after device",
|
||||
"nTo1": "Several before ports map to one target (N:1)"
|
||||
}
|
||||
},
|
||||
coverageBanner: "Mapping coverage: {{n}} before-side ports have no mapping and do not exist on the after device; the gaps may be missing map rows: ",
|
||||
mapViewText: "Text",
|
||||
mapViewTable: "Table",
|
||||
mapTable: {"before": "Before port", "after": "After port", "status": "Status", "remove": "Remove row", "add": "Add row", "empty": "No mapping rows yet", "ok": "OK", "todo": "Target to fill in", "duplicate": "Duplicate before port", "invalid": "Before port missing"},
|
||||
mappingSaved: "Mapping saved",
|
||||
mappingOptionalHint:
|
||||
"Optional. Empty = ignore local port renames (match on non-interface keys). Fill to apply port map. Main-port pairs also cover .subinterfaces (suffix kept).",
|
||||
ifaceNormalize: "Interface normalize",
|
||||
ifaceNormalizeHint:
|
||||
"Template rules, one per line: alias,canonical. Applied before port mapping. Example:\nGE,gei\nSG,smartgroup\nXGE,xgei",
|
||||
"Template rules, one per line: alias,canonical. Applied before port mapping. Example:; optional 3rd/4th columns scope a rule: before/after and vendor, e.g. Eth-Trunk,ae,after,huawei\nGE,gei\nSG,smartgroup\nXGE,xgei",
|
||||
createJob: "Create compare job",
|
||||
defaultJobName: "Cutover compare",
|
||||
jobName: "Name",
|
||||
|
|
|
|||
|
|
@ -203,6 +203,18 @@ const bizMigration = {
|
|||
startBatch: "Start",
|
||||
finishBatch: "Finish",
|
||||
evaluate: "Evaluate",
|
||||
evalProgress: "{{label}} {{i}}/{{n}} {{title}}",
|
||||
evalQueued: "{{label}} queued…",
|
||||
preflight: "Mapping pre-flight",
|
||||
preflightHint: "Checks mapping coverage and collection differences on the two portrait baselines before the window.",
|
||||
manualPendingBanner: "Waiting for manual collection (not part of the acceptance gate): {{metrics}}",
|
||||
notCollectedBanner: "Collected differently on the two devices, not judged (not red): {{metrics}}",
|
||||
duplicateWarningBanner: "Duplicate keys outside this wave (do not block acceptance; refine key fields): {{metrics}}",
|
||||
redResolvableHint: "{{n}} red tickets belong to objects that turned green in the latest acceptance.",
|
||||
redResolveAllResolvable: "Close all that turned green",
|
||||
redNowGreen: "now green",
|
||||
redResolvableClosed: "{{n}} red tickets closed",
|
||||
portSuccessHint: "A port counts as migrated when the old port is gone, physically down or admin down (any one) and the new port is admin and physically up. Pulling the fibre without shutdown counts.",
|
||||
evaluated: "Done",
|
||||
statusUpdated: "Status updated",
|
||||
board: "Board",
|
||||
|
|
|
|||
|
|
@ -66,7 +66,7 @@ const bizCompare = {
|
|||
rowFilters: "行过滤规则",
|
||||
rowFiltersHint:
|
||||
"比对前按规则筛选行。组内条件为「且」，组与组之间为「或」。勾选 ARP 等监控项时会带上默认可编辑规则，可按需改删。",
|
||||
rowFiltersHintShort: "组内且 · 组间或 · 无过滤则比对全部行",
|
||||
rowFiltersHintShort: "组内且 · 组间或 · 无过滤则比对全部行 · 过滤在归一化之后执行，请按规范值书写",
|
||||
rowFiltersEmpty: "无过滤则比对全部行。",
|
||||
filterClear: "清空",
|
||||
filterAddOr: "添加条件组（或）",
|
||||
|
|
@ -110,12 +110,82 @@ const bizCompare = {
|
|||
validateNeed: "请先选择映射与前后批次",
|
||||
mapHint:
|
||||
"每行一对：操作前接口,操作后接口（逗号/Tab/竖线分隔）。例：\ngei-0/1/1,gei-0/1/2\nxgei-0/1/0,xgei-0/2/0\n# 井号开头为注释。不是 A,B/C,D 这种斜杠分组。",
|
||||
templateMonitorUsage: "已被 {{n}} 个监控模板引用",
|
||||
mapParsed: "已解析 {{n}} 对",
|
||||
mapInvalidLines: "无效 {{m}} 行（第 {{lines}} 行）",
|
||||
mapDuplicate: "前接口重复：{{list}}",
|
||||
mapUnsaved: "未保存",
|
||||
mapInvalid: "映射文本有误，未保存。无效行：{{lines}}；重复的前接口：{{dup}}",
|
||||
mapEmpty: "映射文本为空。若不需要映射，请选择“新建映射…”；否则请恢复内容后再保存。",
|
||||
mapSharedConfirm: "该映射还被 {{jobs}} 个其他比对任务、{{projects}} 个割接项目使用，保存会同时影响它们。继续？",
|
||||
mappingAutoSaved: "已自动保存映射",
|
||||
dataState: {
|
||||
"ok": "正常",
|
||||
"no_data": "无数据",
|
||||
"suspect_parse": "疑似解析失败",
|
||||
"not_collected": "未采集",
|
||||
"manual_pending": "待手动采集"
|
||||
},
|
||||
stateBanner: {
|
||||
"no_data": "两侧都没有数据，没有可比较的内容，不计入通过率。",
|
||||
"suspect_parse": "设备输出了内容但解析出 0 行，失败行可能是解析问题而非业务丢失，请先检查解析/模板。",
|
||||
"not_collected": "某一侧没有采集该检查项（或缺少对应切片），不做判定，不计入通过率。"
|
||||
},
|
||||
rawKeyTitle: "操作前设备原始名称",
|
||||
scopeBoth: "两侧",
|
||||
scopeBefore: "仅操作前",
|
||||
scopeAfter: "仅操作后",
|
||||
valueMap: "值映射",
|
||||
valueMapScope: "值映射范围",
|
||||
valueMapPlaceholder: "每行：原值,规范值\nestab,established",
|
||||
rankLabel: "好→坏顺序",
|
||||
rankPlaceholder: "up,down（靠前更好）",
|
||||
modeNotWorse: "不劣于",
|
||||
modeNumericMin: "不低于(%)",
|
||||
modeNumericMax: "不高于(%)",
|
||||
filterSide: "过滤作用侧",
|
||||
unmappedPortPolicy: "未映射端口",
|
||||
unmappedByName: "按名称匹配（默认）",
|
||||
unmappedIgnorePort: "忽略端口，按内容匹配",
|
||||
unmappedPortPolicyHint: "只配了部分端口映射时，没有映射的端口如何配对。",
|
||||
duplicatePairing: "同键多行配对",
|
||||
pairSorted: "按取值排序后配对（推荐）",
|
||||
pairOrdered: "按采集顺序配对（旧行为）",
|
||||
duplicatePairingHint: "ECMP、多路径等同一个键出现多行时，CLI 输出顺序变化不应产生假差异。",
|
||||
builtinVendorCanonical: "启用内置厂商接口名称统一（GigabitEthernet↔GE 等）",
|
||||
mappingNeedBatches: "请先选择前后批次",
|
||||
preflight: {
|
||||
"run": "校验映射",
|
||||
"ready": "映射可用",
|
||||
"risk": "存在风险，请先处理",
|
||||
"coverage": "已映射 {{mapped}} · 疑似漏配 {{suspect}} · 覆盖率 {{rate}}",
|
||||
"pairs": "{{n}} 条映射",
|
||||
"suspect": "疑似漏配（操作前有、未映射、操作后也不存在）",
|
||||
"addSuspects": "加入映射编辑区（目标待填）",
|
||||
"commandDiff": "两侧采集命令不一致",
|
||||
"rows": "行数（前→后）",
|
||||
"state": "数据状态",
|
||||
"mapping": "映射",
|
||||
"skippedHeavy": "大表，未逐个检查接口",
|
||||
"sheetCounts": "已映射 {{exact}} · 同名 {{same}} · 疑似漏配 {{suspect}}",
|
||||
"issue": {
|
||||
"duplicate": "前接口重复（只取第一条）",
|
||||
"conflicts": "归一化后前接口冲突",
|
||||
"unused": "未命中任何接口的映射键",
|
||||
"missingTargets": "映射目标在操作后设备上不存在",
|
||||
"nTo1": "多个前接口指向同一目标（N:1）"
|
||||
}
|
||||
},
|
||||
coverageBanner: "映射覆盖：{{n}} 个操作前端口没有映射且在操作后设备上不存在，缺失可能是漏配映射：",
|
||||
mapViewText: "文本",
|
||||
mapViewTable: "表格",
|
||||
mapTable: {"before": "操作前接口", "after": "操作后接口", "status": "状态", "remove": "删除行", "add": "添加一行", "empty": "还没有映射行", "ok": "正常", "todo": "待填目标", "duplicate": "前接口重复", "invalid": "缺少前接口"},
|
||||
mappingSaved: "映射已保存",
|
||||
mappingOptionalHint:
|
||||
"可选。留空=忽略本端端口改名，按非接口 Key 对齐。填写后才做端口映射校验。主接口映射会自动覆盖 .子接口（后缀不变）。",
|
||||
ifaceNormalize: "接口归一化",
|
||||
ifaceNormalizeHint:
|
||||
"模板级规则，每行：简写,规范名。先归一化再端口映射。例：\nGE,gei\nSG,smartgroup\nXGE,xgei",
|
||||
"模板级规则，每行：简写,规范名。先归一化再端口映射。例：；可选第 3/4 列限定：before/after 与厂商，如 Eth-Trunk,ae,after,huawei\nGE,gei\nSG,smartgroup\nXGE,xgei",
|
||||
createJob: "建立比对任务",
|
||||
defaultJobName: "割接比对",
|
||||
jobName: "名称",
|
||||
|
|
|
|||
|
|
@ -200,6 +200,18 @@ const bizMigration = {
|
|||
startBatch: "启动",
|
||||
finishBatch: "收工",
|
||||
evaluate: "立即判定",
|
||||
evalProgress: "{{label}}中 {{i}}/{{n}} {{title}}",
|
||||
evalQueued: "{{label}}排队中…",
|
||||
preflight: "映射预检",
|
||||
preflightHint: "按两台设备的画像基线检查映射覆盖率与采集差异，割接前先看一眼。",
|
||||
manualPendingBanner: "待手动采集（不计入验收门槛）：{{metrics}}",
|
||||
notCollectedBanner: "两侧采集不一致、不做判定（不判红）：{{metrics}}",
|
||||
duplicateWarningBanner: "本批范围外存在重复键（不影响验收，建议完善 key 字段）：{{metrics}}",
|
||||
redResolvableHint: "{{n}} 张红单对应的对象在最近一次验收中已转绿。",
|
||||
redResolveAllResolvable: "一键关闭已转绿红单",
|
||||
redNowGreen: "已转绿",
|
||||
redResolvableClosed: "已关闭 {{n}} 张红单",
|
||||
portSuccessHint: "端口迁移成功 = 旧端口（已删除 / 物理 down / 管理 down 任一）且新端口管理、物理都 up。拔纤但未 shutdown 也算完成。",
|
||||
evaluated: "判定完成",
|
||||
statusUpdated: "状态已更新",
|
||||
board: "进度看板",
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@
|
|||
@import "./styles/pages/biz-compare-templates.css";
|
||||
@import "./styles/pages/biz-migration.css";
|
||||
@import "./styles/pages/biz-compare.css";
|
||||
@import "./styles/pages/cutover-shared.css";
|
||||
@import "./styles/pages/port-traffic.css";
|
||||
@import "./styles/legacy/overrides-dark.css";
|
||||
@import "./styles/legacy/overrides-late.css";
|
||||
|
|
|
|||
|
|
@ -52,6 +52,9 @@ export const bizCompareUpdateMapping = (mappingId: string, body: Record<string,
|
|||
export const bizCompareValidateMapping = (body: Record<string, unknown>) =>
|
||||
apiPost<Record<string, unknown>>("/v1/biz-state/compare/mappings/validate", body);
|
||||
|
||||
export const bizCompareMappingPreflight = (body: Record<string, unknown>) =>
|
||||
apiPost<Record<string, unknown>>("/v1/biz-state/compare/preflight", body);
|
||||
|
||||
export const bizCompareListJobs = () =>
|
||||
apiGet<{ items: Record<string, unknown>[] }>("/v1/biz-state/compare/jobs");
|
||||
|
||||
|
|
|
|||
|
|
@ -176,3 +176,17 @@ export const bizMigrationListDiffs = (params: {
|
|||
`/v1/biz-migration/runs/${encodeURIComponent(params.runId)}/diffs${qs ? `?${qs}` : ""}`,
|
||||
);
|
||||
};
|
||||
|
||||
|
||||
/** Evaluate / finish run in the background: poll this until ``status`` leaves ``running``. */
|
||||
export const bizMigrationGetRun = (runId: string) =>
|
||||
apiGet<Record<string, unknown>>(`/v1/biz-migration/runs/${encodeURIComponent(runId)}`);
|
||||
|
||||
export const bizMigrationPreflight = (projectId: string) =>
|
||||
apiGet<Record<string, unknown>>(`/v1/biz-migration/projects/${encodeURIComponent(projectId)}/preflight`);
|
||||
|
||||
export const bizMigrationResolveResolvable = (projectId: string) =>
|
||||
apiPost<Record<string, unknown>>(
|
||||
`/v1/biz-migration/projects/${encodeURIComponent(projectId)}/red-tickets/resolve-resolvable`,
|
||||
{},
|
||||
);
|
||||
|
|
|
|||
167
web/src/styles/pages/cutover-shared.css
Normal file
167
web/src/styles/pages/cutover-shared.css
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
/* Shared by 业务对比 and 割接监控: pre-flight report, data-state badges, value-map editor. */
|
||||
|
||||
.cx-preflight {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
padding: 10px 12px;
|
||||
border: 1px solid rgb(var(--nx-ok-rgb) / 0.4);
|
||||
border-radius: 6px;
|
||||
background: rgb(var(--nx-ok-rgb) / 0.06);
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.cx-preflight.is-risk {
|
||||
border-color: rgb(var(--nx-warn-rgb) / 0.5);
|
||||
background: rgb(var(--nx-warn-rgb) / 0.07);
|
||||
}
|
||||
|
||||
.cx-preflight__head {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: baseline;
|
||||
gap: 4px 14px;
|
||||
}
|
||||
|
||||
.cx-preflight__block h5,
|
||||
.cx-preflight__block h6 {
|
||||
margin: 0 0 4px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.cx-preflight__list {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
list-style: none;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 4px 10px;
|
||||
max-height: 140px;
|
||||
overflow: auto;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.cx-preflight__scroll {
|
||||
max-width: 100%;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
.cx-preflight code {
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.cx-preflight__table {
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.cx-preflight__table details summary {
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.cx-state {
|
||||
display: inline-block;
|
||||
padding: 0 6px;
|
||||
border-radius: 3px;
|
||||
font-size: 11px;
|
||||
line-height: 18px;
|
||||
border: 1px solid rgb(var(--nx-ok-rgb) / 0.4);
|
||||
color: var(--nx-ok);
|
||||
}
|
||||
|
||||
.cx-state--no_data,
|
||||
.cx-state--not_collected,
|
||||
.cx-state--manual_pending {
|
||||
border-color: var(--nx-line);
|
||||
color: var(--nx-text-muted);
|
||||
}
|
||||
|
||||
.cx-state--suspect_parse {
|
||||
border-color: rgb(var(--nx-warn-rgb) / 0.55);
|
||||
color: var(--nx-warn);
|
||||
}
|
||||
|
||||
.cx-state__why {
|
||||
font-size: 11px;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.cx-banner {
|
||||
margin: 0;
|
||||
padding: 6px 10px;
|
||||
border-left: 3px solid rgb(var(--nx-warn-rgb) / 0.8);
|
||||
background: rgb(var(--nx-warn-rgb) / 0.08);
|
||||
font-size: 12px;
|
||||
line-height: 1.6;
|
||||
}
|
||||
|
||||
.cx-rawkey {
|
||||
display: block;
|
||||
font-size: 11px;
|
||||
color: var(--nx-text-muted);
|
||||
}
|
||||
|
||||
.ct-valuemap {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
min-width: 150px;
|
||||
}
|
||||
|
||||
.ct-valuemap__text {
|
||||
min-height: 44px;
|
||||
}
|
||||
|
||||
.ct-valuemap__text.is-invalid {
|
||||
border-color: var(--nx-bad);
|
||||
}
|
||||
|
||||
.ct-valuemap__err {
|
||||
color: var(--nx-bad-text);
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.ct-sheet-options {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: flex-end;
|
||||
gap: 8px 16px;
|
||||
}
|
||||
|
||||
.ct-sheet-options__check {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.bs-cmp-mapping__status.is-error {
|
||||
color: var(--nx-bad-text);
|
||||
}
|
||||
|
||||
.bs-cmp-mapping__views {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.bs-cmp-mapping__views .is-active {
|
||||
font-weight: 600;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.cx-maptable {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 6px;
|
||||
max-height: 260px;
|
||||
overflow: auto;
|
||||
}
|
||||
|
||||
.cx-maptable__input {
|
||||
width: 100%;
|
||||
min-width: 90px;
|
||||
}
|
||||
|
||||
.cx-maptable__row.is-duplicate,
|
||||
.cx-maptable__row.is-invalid {
|
||||
background: rgb(var(--nx-bad-rgb) / 0.08);
|
||||
}
|
||||
|
|
@ -40,6 +40,8 @@ const diffs = [
|
|||
{ kind: "added", key: { neighbor: "10.2.1.3", vrf: "core" }, before: null, after: row("10.2.1.3", "Established"), changes: {} },
|
||||
{ kind: "unchanged", key: { neighbor: "10.2.1.4", vrf: "core" }, before: row("10.2.1.4", "Established"), after: row("10.2.1.4", "Established"), changes: {} },
|
||||
];
|
||||
const mappingPosts = [];
|
||||
const preflightPosts = [];
|
||||
let failDiffs = false;
|
||||
let active = false;
|
||||
let legacy = false;
|
||||
|
|
@ -79,6 +81,18 @@ await page.route("**/v1/**", async (route) => {
|
|||
} else if (p.endsWith("/runs/run-1")) {
|
||||
pollCounts.detail++;
|
||||
data = { ...run, config_snapshot_version: legacy ? null : 1, status: active ? "running" : "success" };
|
||||
} else if (p.endsWith("/compare/preflight") && route.request().method() === "POST") {
|
||||
preflightPosts.push(route.request().postDataJSON());
|
||||
data = {
|
||||
verdict: "risk", risk_reasons: ["suspect_unmapped"], coverage: { mapped: 2, suspect_unmapped: 1, identity_same_name: 0, rate: 66.7 },
|
||||
mapping: { pairs: 2, unused_map_keys: [], map_targets_missing_on_after: [], n_to_1_targets: [] },
|
||||
sheets: [{ sheet_id: "peer", title: "BGP 邻居", metric_id: "bgp_peer", before_count: 4, after_count: 4, data_state: "ok",
|
||||
counts: { mapped_exact: 2, mapped_by_parent: 0, identity_same_name: 0, suspect_unmapped: 1 }, suspect_unmapped: ["gei-7"], suspect_unmapped_total: 1 }],
|
||||
command_coverage: { only_before: [], only_after: [] },
|
||||
};
|
||||
} else if (p.endsWith("/mappings") && route.request().method() === "POST") {
|
||||
mappingPosts.push(route.request().postDataJSON());
|
||||
data = { id: "map-new", name: "端口映射", rows: route.request().postDataJSON().rows };
|
||||
} else if (p.endsWith("/mappings")) data = { items: [] };
|
||||
else data = { items: [] };
|
||||
await route.fulfill({ status: 200, contentType: "application/json", body: JSON.stringify(data) });
|
||||
|
|
@ -108,6 +122,49 @@ try {
|
|||
assert.deepEqual(body.enabled_sheet_ids, ["peer"]);
|
||||
assert.equal(body.before_batch_id, "b");
|
||||
assert.equal(body.after_batch_id, "a");
|
||||
// A3: mapping text typed but never "saved" must not be dropped silently.
|
||||
const mapEditor = page.locator(".bs-cmp-mapping__editor");
|
||||
const mapStatus = page.getByTestId("map-parse-status");
|
||||
await mapEditor.fill("GE0/1/0/1 gei0/2/0/1\nxge1\tgei2\nbroken line here");
|
||||
await mapStatus.getByText("已解析 2 对", { exact: false }).waitFor();
|
||||
assert.match(await mapStatus.innerText(), /无效 1 行（第 3 行）/);
|
||||
assert.match(await mapStatus.innerText(), /未保存/);
|
||||
let patched = false;
|
||||
const onPatch = (req) => { if (req.method() === "PATCH" && req.url().endsWith("/compare/jobs/job-1")) patched = true; };
|
||||
page.on("request", onPatch);
|
||||
await page.getByRole("button", { name: "保存配置", exact: true }).click();
|
||||
await page.waitForTimeout(400);
|
||||
assert.equal(patched, false, "invalid mapping text blocks saving the job");
|
||||
assert.equal(mappingPosts.length, 0);
|
||||
await mapEditor.fill("GE0/1/0/1 gei0/2/0/1\nxge1\tgei2");
|
||||
const mappedJob = page.waitForRequest((req) => req.method() === "PATCH" && req.url().endsWith("/compare/jobs/job-1"));
|
||||
await page.getByRole("button", { name: "保存配置", exact: true }).click();
|
||||
assert.equal((await mappedJob).postDataJSON().mapping_id, "map-new");
|
||||
page.off("request", onPatch);
|
||||
assert.deepEqual(mappingPosts[0].rows, [
|
||||
{ before_if: "GE0/1/0/1", after_if: "gei0/2/0/1" }, { before_if: "xge1", after_if: "gei2" }]);
|
||||
// B5: the pre-flight checks the editor text (saved or not) against the two batches.
|
||||
await page.getByRole("button", { name: "校验映射", exact: true }).click();
|
||||
const pf = page.getByTestId("preflight-report");
|
||||
await pf.waitFor();
|
||||
assert.deepEqual(preflightPosts.at(-1).mapping_rows, [
|
||||
{ before_if: "GE0/1/0/1", after_if: "gei0/2/0/1" }, { before_if: "xge1", after_if: "gei2" }]);
|
||||
assert.equal(preflightPosts.at(-1).before_batch_id, "b");
|
||||
assert.equal(preflightPosts.at(-1).after_batch_id, "a");
|
||||
assert.match(await pf.textContent(), /疑似漏配/);
|
||||
await pf.getByRole("button", { name: "加入映射编辑区（目标待填）", exact: true }).click();
|
||||
assert.match(await mapEditor.inputValue(), /gei-7,/);
|
||||
// F1: table view edits the same text.
|
||||
await mapEditor.fill("a1,b1\na1,b2\nc1,");
|
||||
await page.getByRole("button", { name: "表格", exact: true }).click();
|
||||
const tbl = page.getByTestId("mapping-table");
|
||||
await tbl.waitFor();
|
||||
assert.equal(await tbl.locator("tbody tr.is-duplicate").count(), 2);
|
||||
assert.equal(await tbl.locator("tbody tr.is-todo").count(), 1);
|
||||
await tbl.getByLabel("操作后接口 3", { exact: true }).fill("d1");
|
||||
await page.getByRole("button", { name: "文本", exact: true }).click();
|
||||
assert.equal(await mapEditor.inputValue(), "a1,b1\na1,b2\nc1,d1");
|
||||
await mapEditor.fill("");
|
||||
await page.getByLabel("模式", { exact: true }).selectOption("auto");
|
||||
assert.equal(await page.getByLabel("操作后批次", { exact: true }).count(), 0);
|
||||
await page.getByText("保持操作前批次固定；操作后任务采集完成时，自动取最新批次进行对比。", { exact: true }).waitFor();
|
||||
|
|
@ -209,6 +266,25 @@ try {
|
|||
await page.getByLabel("接口对应关系", { exact: true }).waitFor();
|
||||
await page.getByRole("button", { name: "关闭", exact: true }).click();
|
||||
|
||||
// V1/V5: a side that parsed nothing is flagged, and the before-side name is shown as printed.
|
||||
const card = run.summary.sheet_cards[0];
|
||||
Object.assign(card, { data_state: "suspect_parse", data_state_reasons: ["after:parsed_0_of_40_lines"] });
|
||||
diffs[0].raw_key_before = { neighbor: "GE0/1", vrf: "core" };
|
||||
diffs[0].key = { neighbor: "gei0/1", vrf: "core" };
|
||||
await page.reload();
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.getByRole("tab", { name: "比对结果", exact: true }).click();
|
||||
await page.locator(".bs-cmp-row").first().waitFor();
|
||||
const banner = page.getByTestId("data-state-banner");
|
||||
await banner.waitFor();
|
||||
assert.match(await banner.textContent(), /疑似解析失败/);
|
||||
assert.match(await banner.textContent(), /parsed_0_of_40_lines/);
|
||||
assert.match(await page.locator(".bs-cmp-nav__item").first().textContent(), /疑似解析失败/);
|
||||
assert.match(await page.locator(".cx-rawkey").first().textContent(), /GE0\/1/);
|
||||
Object.assign(card, { data_state: "ok", data_state_reasons: [] });
|
||||
delete diffs[0].raw_key_before;
|
||||
await page.getByRole("button", { name: "关闭", exact: true }).click();
|
||||
|
||||
active = true;
|
||||
await page.reload();
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
|
|
@ -218,7 +294,7 @@ try {
|
|||
assert.ok(pollCounts.runs <= 3 && pollCounts.detail <= 3, JSON.stringify(pollCounts));
|
||||
assert.ok(pollCounts.runs >= 2 && pollCounts.detail >= 2, "active run should keep polling");
|
||||
assert.deepEqual(errors, []);
|
||||
console.log(JSON.stringify({ passed: ["job list labels", "four config sections", "source pairing", "last sheet locked", "configuration save payload", "automatic mode", "390px configuration", "keyboard tabs", "wizard source selection", "same-task batch selection", "source change clears batch", "history counts", "execution and verdict separation", "queued progress semantics", "history loading", "history error retry", "390px history", "result layout", "saved execution settings", "legacy settings warning", "error clears stale rows", "retry", "clear filters", "768px toolbar", "old job response ignored", "single poller", "no runtime errors"], pollCounts, output }));
|
||||
console.log(JSON.stringify({ passed: ["job list labels", "four config sections", "source pairing", "last sheet locked", "configuration save payload", "automatic mode", "390px configuration", "keyboard tabs", "wizard source selection", "same-task batch selection", "source change clears batch", "history counts", "execution and verdict separation", "queued progress semantics", "history loading", "history error retry", "390px history", "result layout", "saved execution settings", "legacy settings warning", "error clears stale rows", "retry", "clear filters", "768px toolbar", "old job response ignored", "single poller", "mapping pre-flight", "mapping table view", "data state banner", "raw before key", "no runtime errors"], pollCounts, output }));
|
||||
} catch (error) {
|
||||
await page.screenshot({ path: path.join(output, "biz-compare-test-failure.png"), fullPage: true });
|
||||
console.error(JSON.stringify(await page.locator(".bs-cmp-source select").evaluateAll((els) => els.map((el) => ({ value: el.value, html: el.parentElement.outerHTML })))));
|
||||
|
|
|
|||
|
|
@ -149,6 +149,45 @@ try {
|
|||
await savedJob;
|
||||
assert.deepEqual(writes.at(-1).enabled_sheet_ids, []);
|
||||
passed.push("missing sheets block save", "incomplete sources disabled", "explicit scope repair");
|
||||
// Cross-vendor identity rules: value maps, directional compare, per-side aliases and sheet options
|
||||
// must reach the API exactly as edited.
|
||||
await page.goto(`${base}/network/cutover/compare-templates`);
|
||||
await templateRow().waitFor();
|
||||
await openEditor();
|
||||
await editor().locator("details.ct-editor__norm summary").click();
|
||||
await editor().getByLabel("接口归一化", { exact: true }).fill("GE,gei\nEth-Trunk,ae,after,huawei");
|
||||
await editor().getByRole("tab", { name: /^字段规则/ }).click();
|
||||
const stateRow = editor().locator(".ct-field-table tbody tr").filter({ has: page.getByText("state", { exact: true }) });
|
||||
await stateRow.locator('input[type="checkbox"]').nth(2).check();
|
||||
await editor().getByLabel("state 方式", { exact: true }).selectOption("not_worse");
|
||||
await editor().getByLabel("state 好→坏顺序", { exact: true }).fill("up,down");
|
||||
await editor().getByLabel("state 值映射", { exact: true }).fill("estab,established\n*down,down");
|
||||
await editor().getByLabel("state 值映射", { exact: true }).blur();
|
||||
await editor().getByLabel("state 值映射范围", { exact: true }).selectOption("before");
|
||||
await editor().getByLabel("state 值映射", { exact: true }).fill("up/up,up");
|
||||
await editor().getByLabel("state 值映射", { exact: true }).blur();
|
||||
await editor().getByLabel("未映射端口", { exact: true }).selectOption("ignore_port");
|
||||
await editor().getByLabel("同键多行配对", { exact: true }).selectOption("ordered");
|
||||
await editor().getByRole("checkbox", { name: /内置厂商接口名称统一/ }).uncheck();
|
||||
await editor().getByRole("button", { name: "保存模板", exact: true }).click();
|
||||
await page.waitForTimeout(500);
|
||||
const saved = writes.at(-1);
|
||||
assert.deepEqual(saved.iface_normalize_rules, [{ from: "GE", to: "gei" }, { from: "Eth-Trunk", to: "ae", side: "after", vendor: "huawei" }]);
|
||||
const sheetS = saved.metrics.find((m) => m.sheet_id === "s");
|
||||
const stateRules = sheetS.field_rules.filter((r) => r.field === "state");
|
||||
assert.deepEqual(stateRules.find((r) => !r.side), { field: "state", compare: "not_worse", rank: ["up", "down"], value_map: { estab: "established", "*down": "down" } });
|
||||
assert.deepEqual(stateRules.find((r) => r.side === "before"), { field: "state", side: "before", value_map: { "up/up": "up" } });
|
||||
assert.equal(sheetS.unmapped_port_policy, "ignore_port");
|
||||
assert.equal(sheetS.duplicate_pairing, "ordered");
|
||||
assert.equal(sheetS.builtin_vendor_canonical, false);
|
||||
assert.deepEqual(sheetS.row_filters, complex, "opaque filters survive the same save");
|
||||
passed.push("value map and rank saved", "per-side value map saved", "per-side alias saved", "sheet identity options saved");
|
||||
// ... and come back unchanged when the template is opened again.
|
||||
await openEditor();
|
||||
await editor().getByRole("tab", { name: /^字段规则/ }).click();
|
||||
assert.equal(await editor().getByLabel("state 值映射", { exact: true }).inputValue(), "estab,established\n*down,down");
|
||||
assert.equal(await editor().getByLabel("未映射端口", { exact: true }).inputValue(), "ignore_port");
|
||||
await page.keyboard.press("Escape");
|
||||
assert.deepEqual(errors, []);
|
||||
passed.push("no runtime errors");
|
||||
console.log(JSON.stringify({ passed, output }));
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ const browser = await chromium.launch({ headless: true, channel: process.env.NET
|
|||
const page = await browser.newPage({ viewport: { width: 1440, height: 1080 } });
|
||||
const errors = [], passed = [], requests = [], writes = [];
|
||||
page.on("pageerror", (e) => errors.push(e.message));
|
||||
let liveRunId = "live-1", uncertain = false, failBoard = false, delayedSheet = "", sampleAgeSec = 0;
|
||||
let liveRunId = "live-1", uncertain = false, failBoard = false, delayedSheet = "", sampleAgeSec = 0, richCards = false, evalPolls = 0;
|
||||
const now = new Date().toISOString();
|
||||
const project = { id: "p", name: "核心 PE 替换", old_task_id: "op", new_task_id: "np", old_hf_task_id: "oh", new_hf_task_id: "nh", old_baseline_batch_id: "ob", new_baseline_batch_id: "nb", monitor_template_id: "mt", hf_interval_sec: 60, status: "active", created_at: now,
|
||||
collect_metric_ids_effective: ["arp"], old_task: { id: "op", ne_name: "PE-OLD", ne_ip: "10.0.0.1" }, new_task: { id: "np", ne_name: "PE-NEW", ne_ip: "10.0.0.2" } };
|
||||
|
|
@ -25,7 +25,14 @@ function run(rid, pid = "p") {
|
|||
const sampleTime = sampleAgeSec ? new Date(Date.now() - sampleAgeSec * 1000).toISOString() : now;
|
||||
return { id: rid, purpose: rid === "hist" ? "manual" : "auto", created_at: now,
|
||||
summary: { progress: { ok: pid === "p2" ? 3 : 99, total: pid === "p2" ? 3 : 101 }, anomaly: 1, anomaly_outside_expect: 1, coverage_complete: !uncertain, duplicate_keys: uncertain ? 2 : 0,
|
||||
sheet_cards: sheets.map((s) => ({ ...s, progress_ok: 49, progress_total: 50, anomaly: 0, old_collected_at: sampleTime, new_collected_at: sampleTime, age_limit_sec: 180, sample_issues: uncertain ? ["sample_skew"] : [] })) } };
|
||||
duplicate_warnings: richCards ? [{ sheet_id: "isis", title: "ISIS 邻居", count: 2, examples: ["a|b"] }] : [],
|
||||
sheet_cards: [
|
||||
...sheets.map((s) => ({ ...s, progress_ok: 49, progress_total: 50, anomaly: 0, old_collected_at: sampleTime, new_collected_at: sampleTime, age_limit_sec: 180, sample_issues: uncertain ? ["sample_skew"] : [] })),
|
||||
...(richCards ? [
|
||||
{ sheet_id: "routes", metric_id: "bgp_route", title: "BGP 路由", manual_pending: true, progress_ok: 0, progress_total: 0, anomaly: 0 },
|
||||
{ sheet_id: "v6", metric_id: "bgp_peer", title: "BGP IPv6", not_collected: true, data_state: "not_collected", data_state_reasons: ["new:slice_not_collected"], progress_ok: 0, progress_total: 0, anomaly: 0 },
|
||||
] : []),
|
||||
] } };
|
||||
}
|
||||
await page.route("**/v1/**", async (route) => {
|
||||
const req = route.request(), url = new URL(req.url()), p = url.pathname;
|
||||
|
|
@ -38,6 +45,20 @@ await page.route("**/v1/**", async (route) => {
|
|||
project.collect_metric_ids_effective = project.collect_metric_ids;
|
||||
data = project;
|
||||
}
|
||||
else if (req.method() === "POST" && p.endsWith("/batches/wave/evaluate")) { evalPolls = 0; data = { id: "async-run", status: "running", purpose: "manual", summary: { progress: { phase: "queued" } } }; }
|
||||
else if (p.endsWith("/runs/async-run")) {
|
||||
evalPolls += 1;
|
||||
data = evalPolls < 3
|
||||
? { id: "async-run", status: "running", summary: { progress: { phase: "evaluating", sheet_index: 1, sheet_total: 2, sheet_title: "ARP · 公网" } } }
|
||||
: { ...run("async-run"), status: "success" };
|
||||
}
|
||||
else if (p.endsWith("/p/preflight")) data = {
|
||||
verdict: "risk", risk_reasons: ["suspect_unmapped"], coverage: { mapped: 2, suspect_unmapped: 1, identity_same_name: 0, rate: 66.7 },
|
||||
mapping: { pairs: 2, unused_map_keys: ["gei-9"], map_targets_missing_on_after: [], n_to_1_targets: [] },
|
||||
sheets: [{ sheet_id: "ifb", title: "端口状态", metric_id: "interface_brief", before_count: 3, after_count: 2, data_state: "ok",
|
||||
counts: { mapped_exact: 2, mapped_by_parent: 0, identity_same_name: 0, suspect_unmapped: 1 }, suspect_unmapped: ["gei-7"], suspect_unmapped_total: 1 }],
|
||||
command_coverage: { only_before: ["bgp_peer{afi=ipv4}"], only_after: [] },
|
||||
};
|
||||
else if (p.endsWith("/p/ensure-highfreq")) data = { project, hf_status: "running", collect: { old: [], new: [] } };
|
||||
else if (req.method() === "POST" && p.endsWith("/p/batches")) { writes.push(req.postDataJSON()); data = { ...wave, id: "created-wave", ...req.postDataJSON() }; }
|
||||
else if (p.endsWith("/biz-migration/projects")) data = { items: [project, second] };
|
||||
|
|
@ -197,6 +218,23 @@ try {
|
|||
assert.equal(await table().locator("tbody tr").count(), 100);
|
||||
passed.push("refresh failure keeps snapshot with explicit error");
|
||||
failBoard = false;
|
||||
// H2: evaluation is computed in the background; the page polls the run and shows progress.
|
||||
const evalPost = page.waitForRequest((r) => r.method() === "POST" && r.url().endsWith("/batches/wave/evaluate"));
|
||||
await dialog().getByRole("button", { name: "立即判定", exact: true }).first().click();
|
||||
await evalPost;
|
||||
await dialog().getByTestId("eval-progress").waitFor();
|
||||
assert.match(await dialog().getByTestId("eval-progress").textContent(), /1\/2|排队/);
|
||||
await dialog().getByTestId("eval-progress").waitFor({ state: "detached", timeout: 15000 });
|
||||
assert.ok(requests.filter((r) => r.p.endsWith("/runs/async-run")).length >= 3, "run polled until it settled");
|
||||
passed.push("evaluation polls a background run", "progress shown while the run computes");
|
||||
// H3/V2/H5 states on the board.
|
||||
richCards = true;
|
||||
await dialog().getByRole("button", { name: "刷新看板", exact: true }).click();
|
||||
await dialog().getByTestId("manual-pending-banner").waitFor();
|
||||
assert.match(await dialog().getByTestId("not-collected-banner").textContent(), /BGP IPv6/);
|
||||
assert.match(await dialog().getByTestId("duplicate-warning-banner").textContent(), /ISIS 邻居/);
|
||||
richCards = false;
|
||||
passed.push("manual route table is pending, not failed", "uncollected slice is not judged", "out-of-scope duplicate keys are warnings");
|
||||
await page.setViewportSize({ width: 390, height: 1000 });
|
||||
assert.ok(await dialog().locator(".bm-live-kpis").evaluate((el) => el.scrollWidth <= el.clientWidth + 1));
|
||||
assert.ok(await dialog().locator(".bm-board").evaluate((el) => el.scrollWidth <= el.clientWidth + 1));
|
||||
|
|
@ -215,6 +253,14 @@ try {
|
|||
await page.waitForTimeout(500);
|
||||
assert.deepEqual(writes.at(-1).collect_metric_ids, ['arp']);
|
||||
passed.push("route can be selected from comparison items", "route has manual mode instead of periodic interval", "dynamic metric additions saved explicitly", "removed metric no longer selected");
|
||||
// B5: mapping pre-flight on the portrait baselines.
|
||||
await dialog().getByRole("button", { name: "映射预检", exact: true }).click();
|
||||
const report = dialog().getByTestId("preflight-report");
|
||||
await report.waitFor();
|
||||
assert.match(await report.textContent(), /疑似漏配/);
|
||||
assert.match(await report.textContent(), /gei-7/);
|
||||
assert.match(await report.textContent(), /bgp_peer\{afi=ipv4\}/);
|
||||
passed.push("pre-flight lists suspected unmapped ports and collection differences");
|
||||
await dialog().getByRole("button", { name: "割接批次", exact: true }).click();
|
||||
await page.waitForFunction(() => document.querySelectorAll('.bm-expect-table tbody tr').length === 100);
|
||||
await dialog().getByRole("checkbox", { name: "ARP · 公网 业务 0", exact: true }).check();
|
||||
|
|
@ -243,6 +289,7 @@ try {
|
|||
assert.deepEqual(errors, []);
|
||||
console.log(JSON.stringify({ passed, errors }, null, 2));
|
||||
} catch (error) {
|
||||
console.log("FAILED:", String(error.message).split("\n").slice(0, 6).join(" | "));
|
||||
await page.screenshot({ path: path.join(output, "cutover-monitor-failure.png"), fullPage: true });
|
||||
console.log(JSON.stringify(await page.locator('.bm-board').evaluate((el) => {
|
||||
const layout = [];
|
||||
|
|
|
|||
|
|
@ -57,3 +57,53 @@ test("invalid tolerances, modes, duplicate sheets and scalar set membership bloc
|
|||
assert.throws(() => validateTemplateDraft([sheet(), sheet()]), /duplicate_sheet/);
|
||||
assert.throws(() => validateTemplateDraft([{ ...sheet(), row_filters: [{ field: "id", op: "in", value: "a,b" }] }]), /value_list/);
|
||||
});
|
||||
|
||||
import { sheetPayload, formatIfaceNormText, parseValueMapText, formatValueMapText } from "../src/features/cutover/lib/compareTemplateModel.ts";
|
||||
|
||||
test("sheet options survive import and the save payload", () => {
|
||||
const parsed = parseTemplateImport(template([{ ...sheet(), unmapped_port_policy: "ignore_port", duplicate_pairing: "ordered", builtin_vendor_canonical: false }]));
|
||||
const s = parsed.metrics[0];
|
||||
assert.equal(s.unmapped_port_policy, "ignore_port");
|
||||
assert.equal(s.duplicate_pairing, "ordered");
|
||||
assert.equal(s.builtin_vendor_canonical, false);
|
||||
const body = sheetPayload(s);
|
||||
assert.equal(body.unmapped_port_policy, "ignore_port");
|
||||
assert.equal(body.builtin_vendor_canonical, false);
|
||||
assert.equal("duplicate_pairing" in sheetPayload({ ...s, duplicate_pairing: "" }), false);
|
||||
});
|
||||
|
||||
test("alias rules keep side and vendor and may repeat per scope", () => {
|
||||
const rules = parseIfaceNormText("GE,gei\nGE,GigabitEthernet,before\nEth-Trunk,ae,after,huawei");
|
||||
assert.deepEqual(rules, [{ from: "GE", to: "gei" }, { from: "GE", to: "GigabitEthernet", side: "before" }, { from: "Eth-Trunk", to: "ae", side: "after", vendor: "huawei" }]);
|
||||
assert.deepEqual(parseIfaceNormText(formatIfaceNormText(rules)), rules);
|
||||
assert.throws(() => parseIfaceNormText("GE,gei\nge,x"), /duplicate_alias/);
|
||||
assert.throws(() => parseIfaceNormText("GE,gei,middle"), /unknown_alias_side/);
|
||||
});
|
||||
|
||||
test("a vendor-scoped alias with no side survives the text round trip", () => {
|
||||
const rules = parseIfaceNormText("Eth-Trunk,ae,both,huawei");
|
||||
assert.deepEqual(rules, [{ from: "Eth-Trunk", to: "ae", vendor: "huawei" }]);
|
||||
assert.equal(formatIfaceNormText(rules), "Eth-Trunk,ae,both,huawei");
|
||||
});
|
||||
|
||||
test("value maps parse, format and are bounded", () => {
|
||||
const map = parseValueMapText("estab,established\n# note\n*down,down");
|
||||
assert.deepEqual(map, { estab: "established", "*down": "down" });
|
||||
assert.deepEqual(parseValueMapText(formatValueMapText(map)), map);
|
||||
assert.throws(() => parseValueMapText("lonely"), /pair_required/);
|
||||
assert.throws(() => parseValueMapText(Array.from({ length: 501 }, (_, i) => `a${i},b`).join("\n")), /too_large/);
|
||||
});
|
||||
|
||||
test("directional compare modes need a rank and a field can carry one rule per side", () => {
|
||||
const base = sheet();
|
||||
assert.throws(() => validateTemplateDraft([{ ...base, field_rules: [{ field: "state", compare: "not_worse" }] }]), /rank_list_required/);
|
||||
validateTemplateDraft([{ ...base, field_rules: [{ field: "state", compare: "not_worse", rank: ["up", "down"] }, { field: "count", compare: "numeric_min", tolerance: 5 }] }]);
|
||||
validateTemplateDraft([{ ...base, field_rules: [{ field: "state", side: "before", value_map: { a: "b" } }, { field: "state", side: "after", value_map: { c: "d" } }] }]);
|
||||
assert.throws(() => validateTemplateDraft([{ ...base, field_rules: [{ field: "state", value_map: { a: "b" } }, { field: "state", value_map: { c: "d" } }] }]), /duplicate_field/);
|
||||
assert.throws(() => validateTemplateDraft([{ ...base, field_rules: [{ field: "state", value_map: { a: 1 } }] }]), /string_pairs/);
|
||||
});
|
||||
|
||||
test("filter side survives the editor round trip", () => {
|
||||
const groups = toOrGroups([{ field: "type", op: "eq", value: "dynamic", side: "before" }]);
|
||||
assert.deepEqual(fromOrGroups(groups), [{ field: "type", op: "eq", value: "dynamic", side: "before" }]);
|
||||
});
|
||||
|
|
|
|||
30
web/tests/mappingText.test.mjs
Normal file
30
web/tests/mappingText.test.mjs
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { test } from "node:test";
|
||||
import { parseMapText, mapTextDirty } from "../src/features/cutover/lib/mappingText.ts";
|
||||
|
||||
test("comma, pipe, tab and exactly-two-token space lines are all accepted", () => {
|
||||
const r = parseMapText("a1,b1\na2|b2\na3\tb3\na4 b4\n# note\n\nbefore_if,after_if");
|
||||
assert.deepEqual(r.rows.map((x) => [x.before_if, x.after_if]), [["a1", "b1"], ["a2", "b2"], ["a3", "b3"], ["a4", "b4"]]);
|
||||
assert.deepEqual(r.invalidLines, []);
|
||||
});
|
||||
|
||||
test("unreadable lines are reported with their line numbers instead of dropped", () => {
|
||||
const r = parseMapText("a1,b1\nonlyone\na,b,c\na5 b5 c5\nok1,ok2");
|
||||
assert.deepEqual(r.invalidLines, [2, 3, 4]);
|
||||
assert.equal(r.rows.length, 2);
|
||||
});
|
||||
|
||||
test("repeated before port is flagged and the first pair is kept", () => {
|
||||
const r = parseMapText("a,b\na,c");
|
||||
assert.deepEqual(r.duplicateBefore, ["a"]);
|
||||
assert.deepEqual(r.rows, [{ before_if: "a", after_if: "b" }]);
|
||||
});
|
||||
|
||||
test("dirty check ignores order and spacing but sees edits and invalid text", () => {
|
||||
const saved = [{ before_if: "a", after_if: "b" }, { before_if: "c", after_if: "d" }];
|
||||
assert.equal(mapTextDirty("c , d\na,b", saved), false);
|
||||
assert.equal(mapTextDirty("a,b", saved), true);
|
||||
assert.equal(mapTextDirty("a,b\nc,d\nbroken", saved), true);
|
||||
assert.equal(mapTextDirty("", undefined), false);
|
||||
assert.equal(mapTextDirty("x,y", undefined), true);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue