mirror of
https://github.com/hansjone/netx.git
synced 2026-10-11 22:00:48 +08:00
Fix business compare history, search, sampling and exports
This commit is contained in:
parent
afa8e06b2d
commit
4d5101ec3a
19 changed files with 1598 additions and 274 deletions
BIN
docs/reviews/assets/biz-compare-desktop.png
Normal file
BIN
docs/reviews/assets/biz-compare-desktop.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
BIN
docs/reviews/assets/biz-compare-error.png
Normal file
BIN
docs/reviews/assets/biz-compare-error.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 96 KiB |
BIN
docs/reviews/assets/biz-compare-legacy.png
Normal file
BIN
docs/reviews/assets/biz-compare-legacy.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
29
docs/reviews/assets/biz-compare-memory-benchmark.json
Normal file
29
docs/reviews/assets/biz-compare-memory-benchmark.json
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
{
|
||||
"date": "2026-10-10",
|
||||
"baseline_ref": "afa8e06b2de64100f963ae30b3babfcb5d7009f4",
|
||||
"python": "3.14.3",
|
||||
"dataset": {
|
||||
"rows_per_side": 1000000,
|
||||
"fields": ["id", "neighbor", "state", "afi"],
|
||||
"strata": 50,
|
||||
"unchanged_limit": 5000,
|
||||
"compact_unchanged": true,
|
||||
"interface_transforms": false,
|
||||
"all_rows_unchanged": true
|
||||
},
|
||||
"measurement": "tracemalloc peak allocated during compare_rows; excludes input allocations, DB loading and process overhead",
|
||||
"results": [
|
||||
{
|
||||
"engine": "baseline",
|
||||
"peak_engine_mib": 1144.43,
|
||||
"seconds_with_tracemalloc": 92.4
|
||||
},
|
||||
{
|
||||
"engine": "optimized",
|
||||
"peak_engine_mib": 420.49,
|
||||
"seconds_with_tracemalloc": 63.681
|
||||
}
|
||||
],
|
||||
"results_and_sample_order_identical": true,
|
||||
"test_result": "1 passed in 157.88s"
|
||||
}
|
||||
BIN
docs/reviews/assets/biz-compare-narrow.png
Normal file
BIN
docs/reviews/assets/biz-compare-narrow.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
120
docs/reviews/biz-compare-review-2026-10-10.md
Normal file
120
docs/reviews/biz-compare-review-2026-10-10.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# 业务对比检查与优化记录
|
||||
|
||||
检查日期:2026-10-10。范围:网络管理 → 割接对比 → 业务对比,包括前端任务/批次/结果界面、Python 与 PostgreSQL 对比路径、结果分页与搜索、短期列表缓存。
|
||||
|
||||
本次直接修改项目代码并添加回归测试,没有连接设备执行采集,也没有修改生产数据库。浏览器截图与交互测试使用固定 API 响应。
|
||||
|
||||
## 已修复的问题
|
||||
|
||||
### 1. 搜索会改变业务判定(P1)
|
||||
|
||||
原实现分别按搜索条件过滤割接前、后的原表,再重新对比。例如 `state=up → down`,搜索 `up` 后只剩前侧,原本的“变化”就可能变成“删除”。重复身份键的业务还可能在筛选后错位配对。
|
||||
|
||||
现在:已存储的判定直接在结果表分页筛选;需要回查未保存全量明细的成功行时,先找到候选身份键,再补齐两侧的完整分组,按原有配对规则对比,最后筛选。达到读取上限的分组整体跳过,并返回截断标记,避免产生虚假的新增或删除。
|
||||
|
||||
### 2. PostgreSQL 搜索漏用部分模板过滤条件(P1)
|
||||
|
||||
原搜索只下推 SQL 支持的过滤规则,对 `age_timer` 等不支持的规则直接忽略,可能将静态 ARP 等不属于模板范围的数据纳入结果。现在不支持下推的规则走流式读取并在 Python 中完整过滤。
|
||||
|
||||
### 3. 快速切换时旧请求覆盖当前任务(P1)
|
||||
|
||||
前后采集任务的批次列表、打开任务、加载历史结果以及轮询缺少完整的过期保护。现在对批次请求使用清理标记,对任务与结果使用请求序号;关闭弹窗、切换路由或打开其他任务后,旧响应不能覆盖当前界面。
|
||||
|
||||
### 4. 结果加载错误容易被误读(P2)
|
||||
|
||||
原查询失败时保留上一组明细,界面已经显示新筛选条件;切到未完成的表后,加载状态也可能无法结束。现在加载和失败都有独立状态,错误时清除旧明细并提供重试入口;切换表/页签后清理加载状态。
|
||||
|
||||
### 5. LLDP 搜索漏查(P2)
|
||||
|
||||
LLDP 数据存于 `biz_state_lldp_neighbor`,原搜索只读 `biz_state_metric_row`。现在原表回查与精简结果的字段搜索都支持 LLDP 表。
|
||||
|
||||
### 6. 其他已修复边界
|
||||
|
||||
- 恰好命中读取上限时,原非 PostgreSQL 路径错误地报告“已截断”;现在多读一条判断。
|
||||
- 无效或有歧义的表标识可能落到其他表;现在返回 `sheet_not_found`。
|
||||
- `%`、`_` 等搜索字符按字面值匹配,避免不同查询路径含义不一致。
|
||||
- 关闭最后一项检查后,空数组被解释为“启用全部”;现在阻止关闭最后一项并提示至少保留一项。
|
||||
- 两个 HeroUI Input 使用了 `isDisabled`,实际需要 `disabled`;过滤值与容差输入的禁用状态已修正。
|
||||
- 列表缓存缺少并发合并,慢请求可以覆盖强制刷新结果或在失效后重新填充缓存;现在共享请求、保护失效状态并抑制过期的后台通知。
|
||||
- 清理中英文文案中的重复键,解决相关类型检查错误。
|
||||
|
||||
## 性能改动与实际收益范围
|
||||
|
||||
- 合并两套轮询,下一轮在本轮请求完成后启动。模拟浏览器测试中,运行任务在 5.2 秒内请求历史列表和详情各两次,没有重复轮询,也不会因慢请求持续叠加。
|
||||
- 已保存的失败、新增、变化等结果在数据库中筛选并分页,避免每次翻页重新加载和对比原始批次。回归测试明确验证这一查询路径不调用 `compare_rows`。
|
||||
- 同一前后采集任务的批次请求共用短期缓存,列表后台刷新合并同一请求;6 项缓存测试覆盖并发、强制刷新、失败重试与失效竞态。
|
||||
- 模板查找使用 Map,避免列表搜索与绘制逐项线性查找模板。
|
||||
- 修改筛选后先回到第一页,避免先请求旧页码再重复请求第一页。
|
||||
|
||||
这些改动减少了重复工作。以下后续基准使用合成数据,没有测量生产环境延迟。
|
||||
|
||||
## 后续优化(2026-10-10)
|
||||
|
||||
### 历史执行配置与精简行还原
|
||||
|
||||
新运行在创建时将模板、启用的检查项、字段规则/过滤、接口归一化、端口映射、成功行保存策略和名称存入 `summary_json.config_snapshot`。后台执行、成功行回查、精简行还原以及导出共用该配置;排队之后修改配置,甚至删除原模板/映射,也不会改变这一运行使用的配置。无需数据库结构迁移。
|
||||
|
||||
修复了精简存储明细还原时将 `mapped_before` 直接设为原始 `before` 的问题。现在前侧依次归一化和映射,后侧归一化,保持与完整结果的显示及 CSV 一致。ZIP 新增 `config_snapshot.json`,便于核对当时配置;详情接口只返回快照模板与版本,不将完整端口映射加入轮询响应。
|
||||
|
||||
页面显示“执行配置已保存”;旧记录显示“旧记录:未保存执行配置”。旧运行无法可靠补造当时的配置,仍保留兼容查询,成功行回查及精简行还原会使用当前配置。快照也不保存原始采集行的副本,若原始批次数据被清理,依赖行 ID 的还原仍会缺少内容。
|
||||
|
||||
### Python 大表内存
|
||||
|
||||
成功样本按“层内序号、层首次出现顺序”排序,只保留样本上限内的候选和分层状态,避免先积累全部成功行。保留原分层轮询顺序;未截断的小表仍保持采集顺序。不需要接口变换时复用输入行;有变换时创建新行,不修改输入数据。稳定身份键顺序直接使用分组字典,省去再次计算全表键、额外键列表与集合。
|
||||
|
||||
每侧 1,000,000 行、50 个邻居分层、全部成功、保存 5,000 条精简样本的测试:优化前引擎新增分配峰值 **1144.43 MiB**,优化后 **420.49 MiB**,下降约 **63.3%**。完整结果与样本顺序的 SHA-256 一致。开启 `tracemalloc` 时耗时为 92.400 秒和 63.681 秒;该计时含内存追踪开销,不代表生产耗时。输入行在追踪开始前创建,峰值不包含输入行、数据库加载或 Python 进程的总内存。
|
||||
|
||||
基线为 `afa8e06b2de64100f963ae30b3babfcb5d7009f4` 中的对比引擎。原始指标见 [百万行测试结果](assets/biz-compare-memory-benchmark.json)。身份键分组仍需要随原表行数增长的内存;这次不是完全流式对比。
|
||||
|
||||
### 大表导出
|
||||
|
||||
导出改为按 `(seq, id)` 游标分批查询、分批还原精简行、逐行将 UTF-8 CSV 写入 ZIP,避免将全部明细与完整 CSV 同时保存在内存。相同 `seq` 的多条结果按 `id` 连续读取,不漏掉跨批次的同行序号记录。保留 BOM、字段顺序及逗号/引号/换行转义。最终压缩包仍由现有接口在内存中返回,压缩文件本身的大小仍计入内存。
|
||||
|
||||
## UI 改动
|
||||
|
||||
结果区分为当前检查项/全表通过率、结果类型、搜索与辅助操作三层。增加明确的分页范围、清空筛选、加载状态、错误重试和搜索无结果提示。筛选按钮增加按下状态,字段筛选增加展开状态与键盘焦点提示。后续增加了执行配置保存状态及旧记录限制提示。
|
||||
|
||||
窄屏检查项导航改为紧凑的横向排列,避免一个检查项占据整行的大块区域。浏览器检查覆盖 1440px、768px、390px;筛选栏没有横向溢出。结果表保留必要的横向滚动。
|
||||
|
||||
- [桌面结果区](assets/biz-compare-desktop.png)
|
||||
- [768px 结果区](assets/biz-compare-narrow.png)
|
||||
- [加载失败与重试](assets/biz-compare-error.png)
|
||||
- [旧记录执行配置提示](assets/biz-compare-legacy.png)
|
||||
|
||||
## 验证
|
||||
|
||||
- Python:87 项通过,1 项百万行基准默认跳过;单独启用百万行基准后通过。新增测试覆盖搜索、历史配置快照、有界抽样与游标导出。
|
||||
- 缓存:6 项 Node 测试通过。
|
||||
- 浏览器:10 个场景通过,包括配置保存状态、旧记录提示、结果布局、错误清除旧明细、重试、清空筛选、窄屏布局、旧任务响应保护、单套轮询、无运行时异常。
|
||||
- Vite 生产打包成功。
|
||||
- 本次业务对比页面的类型错误已经修复。全项目 `tsc -b` 仍被 `BizStatePage.tsx`、`BizMigrationPage.tsx` 中原有的类型/未使用变量错误阻塞。
|
||||
- 页面 ESLint 原有 34 个错误,主要是 `any` 与现有 React Hook 写法;本次未增加错误。结果请求依赖警告已修正。缓存模块没有 lint 错误。
|
||||
- PostgreSQL 路径验证了 SQL 编译与规则降级分支,没有在真实 PostgreSQL 数据库上执行查询或测量执行计划。
|
||||
|
||||
可重复执行的检查:
|
||||
|
||||
```powershell
|
||||
# 项目根目录
|
||||
.venv\Scripts\python.exe -m pytest tests/test_biz_state_compare_export.py tests/test_biz_state_compare_snapshot.py tests/test_biz_state_compare_memory.py tests/test_biz_state_compare_search.py tests/test_biz_state_compare.py tests/test_biz_state_compare_sql.py tests/test_biz_state_compare_recovery.py -q
|
||||
|
||||
# 可选:百万行基准,耗时约数分钟,内存占用较大
|
||||
$env:NETX_BENCH_ROWS = '1000000'
|
||||
$env:NETX_BENCH_REF = 'afa8e06b2de64100f963ae30b3babfcb5d7009f4'
|
||||
.venv\Scripts\python.exe -m pytest tests/test_biz_state_compare_memory.py::test_memory_benchmark_against_git_baseline -q -s
|
||||
Remove-Item Env:NETX_BENCH_ROWS, Env:NETX_BENCH_REF
|
||||
|
||||
# web 目录
|
||||
node --experimental-strip-types --test tests/cutoverDataCache.test.mjs
|
||||
node node_modules/vite/bin/vite.js build
|
||||
node node_modules/typescript/bin/tsc -b
|
||||
```
|
||||
|
||||
浏览器脚本为 `web/tests/bizCompare.browser.mjs`。先启动本地 Vite 服务,再设置 `NETX_PLAYWRIGHT_MODULE` 为可用的 Playwright 包路径;`NETX_TEST_URL` 默认是 `http://127.0.0.1:5179`,浏览器默认使用已安装的 Chrome。所有 `/v1` 请求由脚本提供固定响应,不会访问实际设备或数据库。
|
||||
|
||||
## 后续值得优化的部分
|
||||
|
||||
1. **Python 大表分组。** 有界抽样和冗余复制优化已完成,百万行基准下降约 63% 的引擎新增分配峰值。分组与输入仍随原表规模增长;进一步降低内存可研究排序流式配对或分区执行,并保持重复键配对语义。
|
||||
2. **PostgreSQL 搜索与深分页。** JSON 字段包含搜索和精确总数统计仍可能扫描大量结果。需要用真实数据查看 `EXPLAIN (ANALYZE, BUFFERS)`,再决定常用字段索引、搜索文本索引或游标分页;本次没有盲目添加生产索引。
|
||||
3. **历史配置与原始数据保留。** 新运行的执行配置快照已完成。旧运行仍无法还原未记录的当时配置;精简结果还原依赖原批次行,后续可为批次清理增加引用保护或保留策略。
|
||||
4. **重复键的业务语义。** 引擎按采集顺序配对同键行。如果业务应按集合判断,CLI 行顺序改变可能产生多条“不一致”。当前保留原有语义;可优先完善身份键,或另行实现并测试 Python/SQL 一致的集合匹配。
|
||||
5. **原表回查的数量上限。** 每侧候选命中最多 2000 条,完整分组补齐每侧预算 10000 条,最终展示最多 200 对;这不是全表匹配总数。界面会提示截断,大范围检索需要收窄条件。
|
||||
|
|
@ -2,8 +2,10 @@
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict, deque
|
||||
from typing import Any, Mapping, Sequence
|
||||
from collections import defaultdict
|
||||
from heapq import heappush, heapreplace
|
||||
from itertools import chain
|
||||
from typing import Any, Iterable, Mapping, Sequence
|
||||
|
||||
from .compare_rules import field_rule_map, values_equal, explain_diff
|
||||
from .iface_normalize import (
|
||||
|
|
@ -28,36 +30,52 @@ def stratum_key(row: Mapping[str, Any] | None) -> str:
|
|||
return "|".join(parts) if parts else "_"
|
||||
|
||||
|
||||
def stratify_take(items: list[Any], limit: int, *, key_fn) -> list[Any]:
|
||||
"""Round-robin across strata so one neighbor/direction cannot consume the whole sample."""
|
||||
lim = max(0, int(limit))
|
||||
if lim <= 0 or not items:
|
||||
class _StratifiedSample:
|
||||
"""Keep the first N round-robin ranks in O(N) space, even with many strata."""
|
||||
|
||||
def __init__(self, limit: int) -> None:
|
||||
self.limit = max(0, int(limit))
|
||||
self.count = 0
|
||||
self.strata: dict[str, tuple[int, int]] = {}
|
||||
self.heap: list[tuple[int, int, int, Any]] = []
|
||||
|
||||
def add(self, item: Any, key: str) -> None:
|
||||
serial = self.count
|
||||
self.count += 1
|
||||
if not self.limit:
|
||||
return
|
||||
state = self.strata.get(key)
|
||||
if state is None:
|
||||
# Later strata cannot beat the first item of N earlier strata.
|
||||
if len(self.strata) >= self.limit:
|
||||
return
|
||||
index, round_n = len(self.strata), 0
|
||||
else:
|
||||
index, round_n = state
|
||||
self.strata[key] = (index, round_n + 1)
|
||||
entry = (-round_n, -index, serial, item)
|
||||
if len(self.heap) < self.limit:
|
||||
heappush(self.heap, entry)
|
||||
elif entry[:2] > self.heap[0][:2]:
|
||||
heapreplace(self.heap, entry)
|
||||
|
||||
def picked(self) -> list[Any]:
|
||||
# The legacy helper preserves encounter order when no truncation occurs.
|
||||
if self.count <= self.limit:
|
||||
entries = sorted(self.heap, key=lambda e: e[2])
|
||||
else:
|
||||
entries = sorted(self.heap, key=lambda e: (-e[0], -e[1]))
|
||||
return [entry[3] for entry in entries]
|
||||
|
||||
|
||||
def stratify_take(items: Iterable[Any], limit: int, *, key_fn) -> list[Any]:
|
||||
"""Round-robin across strata without retaining every candidate."""
|
||||
sample = _StratifiedSample(limit)
|
||||
if not sample.limit:
|
||||
return []
|
||||
if len(items) <= lim:
|
||||
return list(items)
|
||||
buckets: dict[str, deque[Any]] = defaultdict(deque)
|
||||
order: list[str] = []
|
||||
for it in items:
|
||||
sk = str(key_fn(it) or "_")
|
||||
if sk not in buckets:
|
||||
order.append(sk)
|
||||
buckets[sk].append(it)
|
||||
out: list[Any] = []
|
||||
while len(out) < lim and buckets:
|
||||
drained: list[str] = []
|
||||
for sk in order:
|
||||
q = buckets.get(sk)
|
||||
if not q:
|
||||
drained.append(sk)
|
||||
continue
|
||||
out.append(q.popleft())
|
||||
if len(out) >= lim:
|
||||
break
|
||||
for sk in drained:
|
||||
buckets.pop(sk, None)
|
||||
if sk in order:
|
||||
order = [x for x in order if x != sk]
|
||||
return out
|
||||
for item in items:
|
||||
sample.add(item, str(key_fn(item) or "_"))
|
||||
return sample.picked()
|
||||
|
||||
|
||||
def apply_port_map(
|
||||
|
|
@ -198,10 +216,10 @@ def compare_rows(
|
|||
|
||||
before_norm = apply_iface_normalize_rows(
|
||||
before_rows, iface_fields=iface_list, rules=norm_rules
|
||||
)
|
||||
) if norm_rules and iface_list else before_rows
|
||||
after_norm = apply_iface_normalize_rows(
|
||||
after_rows, iface_fields=iface_list, rules=norm_rules
|
||||
)
|
||||
) if norm_rules and iface_list else after_rows
|
||||
|
||||
ignore_ports = False
|
||||
# No map → optionally ignore port renames by dropping iface from match key.
|
||||
|
|
@ -216,9 +234,16 @@ def compare_rows(
|
|||
match_keys = list(key_fields)
|
||||
else:
|
||||
# Auto heuristic (legacy default)
|
||||
before_c = [row_key(r, candidate) for r in before_norm]
|
||||
after_c = [row_key(r, candidate) for r in after_norm]
|
||||
if len(before_c) == len(set(before_c)) and len(after_c) == len(set(after_c)):
|
||||
def unique_keys(rows: list[dict[str, Any]]) -> bool:
|
||||
seen: set[tuple[str, ...]] = set()
|
||||
for row in rows:
|
||||
key = row_key(row, candidate)
|
||||
if key in seen:
|
||||
return False
|
||||
seen.add(key)
|
||||
return True
|
||||
|
||||
if unique_keys(before_norm) and unique_keys(after_norm):
|
||||
match_keys = candidate
|
||||
ignore_ports = True
|
||||
else:
|
||||
|
|
@ -226,15 +251,12 @@ def compare_rows(
|
|||
else:
|
||||
match_keys = list(key_fields)
|
||||
|
||||
before_mapped: list[dict[str, Any]] = [
|
||||
apply_port_map(r, iface_fields=iface_list, port_map=pmap) for r in before_norm
|
||||
]
|
||||
|
||||
before_groups: dict[tuple[str, ...], list[tuple[dict[str, Any], dict[str, Any]]]] = (
|
||||
defaultdict(list)
|
||||
)
|
||||
after_groups: dict[tuple[str, ...], list[dict[str, Any]]] = defaultdict(list)
|
||||
for orig, mapped in zip(before_rows, before_mapped):
|
||||
for orig, norm in zip(before_rows, before_norm):
|
||||
mapped = apply_port_map(norm, iface_fields=iface_list, port_map=pmap) if iface_list else norm
|
||||
before_groups[row_key(mapped, match_keys)].append((orig, mapped))
|
||||
for r in after_norm:
|
||||
after_groups[row_key(r, match_keys)].append(r)
|
||||
|
|
@ -246,6 +268,7 @@ def compare_rows(
|
|||
multi_before_keys: list[tuple[str, ...]] = []
|
||||
multi_after_keys: list[tuple[str, ...]] = []
|
||||
unchanged_candidates: list[tuple[dict[str, Any], dict[str, Any], dict[str, Any]]] = []
|
||||
sample = _StratifiedSample(limit_n) if include_unchanged and limit_n is not None else None
|
||||
|
||||
def _key_obj(row: dict[str, Any]) -> dict[str, Any]:
|
||||
return {f: row.get(f, "") for f in key_fields}
|
||||
|
|
@ -294,18 +317,7 @@ def compare_rows(
|
|||
)
|
||||
|
||||
# Stable key order: before encounter order, then after-only keys
|
||||
seen_keys: set[tuple[str, ...]] = set()
|
||||
ordered_keys: list[tuple[str, ...]] = []
|
||||
for orig, mapped in zip(before_rows, before_mapped):
|
||||
k = row_key(mapped, match_keys)
|
||||
if k not in seen_keys:
|
||||
seen_keys.add(k)
|
||||
ordered_keys.append(k)
|
||||
for r in after_norm:
|
||||
k = row_key(r, match_keys)
|
||||
if k not in seen_keys:
|
||||
seen_keys.add(k)
|
||||
ordered_keys.append(k)
|
||||
ordered_keys = chain(before_groups, (k for k in after_groups if k not in before_groups))
|
||||
for k in ordered_keys:
|
||||
b_list = before_groups.get(k) or []
|
||||
a_list = after_groups.get(k) or []
|
||||
|
|
@ -377,17 +389,14 @@ def compare_rows(
|
|||
else:
|
||||
unchanged += 1
|
||||
if include_unchanged:
|
||||
unchanged_candidates.append((orig, mapped, after))
|
||||
if sample is not None:
|
||||
if sample.limit:
|
||||
sample.add((orig, mapped, after), stratum_key(mapped))
|
||||
else:
|
||||
unchanged_candidates.append((orig, mapped, after))
|
||||
|
||||
if include_unchanged and unchanged_candidates:
|
||||
if limit_n is None:
|
||||
picked = unchanged_candidates
|
||||
else:
|
||||
picked = stratify_take(
|
||||
unchanged_candidates,
|
||||
limit_n,
|
||||
key_fn=lambda t: stratum_key(t[1]),
|
||||
)
|
||||
if include_unchanged:
|
||||
picked = sample.picked() if sample is not None else unchanged_candidates
|
||||
for orig, mapped, after_row in picked:
|
||||
_append_unchanged_diff(orig, mapped, after_row)
|
||||
|
||||
|
|
|
|||
|
|
@ -8,13 +8,14 @@ import logging
|
|||
import threading
|
||||
import time
|
||||
import zipfile
|
||||
from copy import deepcopy
|
||||
from datetime import datetime
|
||||
from typing import Any, Callable
|
||||
from typing import Any, Callable, Iterable, Iterator
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import and_, or_
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import String, and_, cast, func, or_, tuple_
|
||||
from sqlalchemy.orm import Session, aliased
|
||||
|
||||
from ..models import (
|
||||
BizCompareDiff,
|
||||
|
|
@ -29,7 +30,7 @@ from ..models import (
|
|||
BizStateTask,
|
||||
)
|
||||
from ..timeutil import utcnow_naive
|
||||
from .compare_engine import compare_rows, mapping_stats
|
||||
from .compare_engine import apply_port_map, compare_rows, mapping_stats, row_key
|
||||
from .compare_rules import (
|
||||
ROW_FILTER_PRESETS,
|
||||
apply_row_filters,
|
||||
|
|
@ -193,6 +194,7 @@ _FAIL_COMPACT_MIN = 50_000
|
|||
# Live search (kw): load at most this many matching rows per side, return ≤ this many pairs.
|
||||
_LIVE_SEARCH_LOAD_CAP = 2_000
|
||||
_LIVE_SEARCH_RESULT_CAP = 200
|
||||
_LIVE_SEARCH_GROUP_LOAD_CAP = 10_000
|
||||
# Success-row persist policy (see resolve_unchanged_policy)
|
||||
_STORE_UNCHANGED_MODES = frozenset({"auto", "always", "never", "sample", "keys"})
|
||||
_UNCHANGED_FULL_MAX = 20_000
|
||||
|
|
@ -269,7 +271,7 @@ def _strip_netx(row: Any) -> dict[str, Any]:
|
|||
return {k: v for k, v in row.items() if k != "_netx"}
|
||||
|
||||
|
||||
def _diff_search_text(d: dict[str, Any]) -> str:
|
||||
def _diff_search_text(d: dict[str, Any], *, limit: int | None = _SEARCH_TEXT_MAX) -> str:
|
||||
parts = [str(d.get("kind") or "")]
|
||||
for key in ("key", "before", "after", "mapped_before", "changes"):
|
||||
val = d.get(key)
|
||||
|
|
@ -278,7 +280,8 @@ def _diff_search_text(d: dict[str, Any]) -> str:
|
|||
parts.append(json.dumps(val, ensure_ascii=False, default=str, separators=(",", ":")))
|
||||
except Exception:
|
||||
parts.append(str(val))
|
||||
return " ".join(parts)[:_SEARCH_TEXT_MAX]
|
||||
value = " ".join(parts)
|
||||
return value if limit is None else value[:limit]
|
||||
|
||||
|
||||
def _top_changed_fields(diffs: list[dict[str, Any]], *, limit: int = 8) -> list[dict[str, Any]]:
|
||||
|
|
@ -422,7 +425,10 @@ def _metric_rows_by_ids(db: Session, ids: list[str]) -> dict[str, dict[str, Any]
|
|||
return out
|
||||
|
||||
|
||||
def _hydrate_diff_rows(db: Session, items: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
def _hydrate_diff_rows(
|
||||
db: Session, items: list[dict[str, Any]], *,
|
||||
run: BizCompareRun | None = None, sheet: dict[str, Any] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Fill empty before/after from metric tables when row_ids are present."""
|
||||
need: list[str] = []
|
||||
for it in items:
|
||||
|
|
@ -433,16 +439,24 @@ def _hydrate_diff_rows(db: Session, items: list[dict[str, Any]]) -> list[dict[st
|
|||
if not need:
|
||||
return items
|
||||
by_id = _metric_rows_by_ids(db, need)
|
||||
pmap, norm_rules = _run_transform_config(db, run) if run else ({}, [])
|
||||
iface_fields = list((sheet or {}).get("iface_fields") or [])
|
||||
for it in items:
|
||||
brid = str(it.get("before_row_id") or "")
|
||||
arid = str(it.get("after_row_id") or "")
|
||||
if not it.get("before") and brid and brid in by_id:
|
||||
it["before"] = by_id[brid]
|
||||
if not it.get("after") and arid and arid in by_id:
|
||||
it["after"] = by_id[arid]
|
||||
# Success compact: no mapped_before stored — UI falls back to before
|
||||
it["after"] = apply_iface_normalize_rows(
|
||||
[by_id[arid]], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
if not it.get("mapped_before") and it.get("before"):
|
||||
it["mapped_before"] = dict(it["before"])
|
||||
normalized = apply_iface_normalize_rows(
|
||||
[it["before"]], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
it["mapped_before"] = apply_port_map(
|
||||
normalized, iface_fields=iface_fields, port_map=pmap,
|
||||
)
|
||||
return items
|
||||
|
||||
|
||||
|
|
@ -471,6 +485,7 @@ def _filter_inline_diffs(
|
|||
*,
|
||||
kind: str,
|
||||
kw: str,
|
||||
field_q: dict[str, str] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
kind_n = (kind or "diff").strip().lower()
|
||||
kw_n = (kw or "").strip().lower()
|
||||
|
|
@ -484,13 +499,62 @@ def _filter_inline_diffs(
|
|||
elif kind_n != "all" and dk != kind_n:
|
||||
continue
|
||||
if kw_n:
|
||||
blob = _diff_search_text(d).lower()
|
||||
blob = _diff_search_text(d, limit=None).lower()
|
||||
if kw_n not in blob:
|
||||
continue
|
||||
if field_q and not _diff_matches_search(d, kw="", field_q=field_q):
|
||||
continue
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
|
||||
def _diff_matches_search(d: dict[str, Any], *, kw: str, field_q: dict[str, str]) -> bool:
|
||||
"""Search a completed pair, never filter its two sides before pairing."""
|
||||
if kw and kw.lower() not in _diff_search_text(d, limit=None).lower():
|
||||
return False
|
||||
return all(
|
||||
any(
|
||||
val.lower() in str((d.get(side) or {}).get(name, "")).lower()
|
||||
for side in ("key", "before", "mapped_before", "after")
|
||||
)
|
||||
for name, val in field_q.items()
|
||||
)
|
||||
|
||||
|
||||
def _search_stored_diffs(q: Any, *, metric_id: str, kw: str, field_q: dict[str, str]) -> Any:
|
||||
"""Filter persisted verdicts in SQL; join source rows for compact payloads."""
|
||||
from ..models import BizStateMetricRow
|
||||
|
||||
sides = [BizCompareDiff.key_json, BizCompareDiff.before_json,
|
||||
BizCompareDiff.mapped_before_json, BizCompareDiff.after_json]
|
||||
sources = []
|
||||
for row_id in (BizCompareDiff.before_row_id, BizCompareDiff.after_row_id):
|
||||
model = BizStateLldpNeighbor if metric_id == "lldp_neighbor" else BizStateMetricRow
|
||||
src = aliased(model)
|
||||
q = q.outerjoin(src, src.id == row_id)
|
||||
sources.append(src)
|
||||
if kw:
|
||||
expressions = [BizCompareDiff.search_text, *[cast(s, String) for s in sides]]
|
||||
for src in sources:
|
||||
if metric_id == "lldp_neighbor":
|
||||
expressions.extend(getattr(src, f) for f in
|
||||
("local_if", "remote_sys", "remote_if", "remote_ip", "protocol"))
|
||||
else:
|
||||
expressions.append(cast(src.data_json, String))
|
||||
q = q.filter(or_(*[func.lower(e).contains(kw.lower(), autoescape=True) for e in expressions]))
|
||||
for name, val in field_q.items():
|
||||
expressions = [s[name].as_string() for s in sides]
|
||||
for src in sources:
|
||||
if metric_id == "lldp_neighbor":
|
||||
if name in ("local_if", "remote_sys", "remote_if", "remote_ip", "protocol"):
|
||||
expressions.append(getattr(src, name))
|
||||
else:
|
||||
expressions.append(src.data_json[name].as_string())
|
||||
q = q.filter(or_(*[func.lower(cast(e, String)).contains(val.lower(), autoescape=True)
|
||||
for e in expressions]))
|
||||
return q
|
||||
|
||||
|
||||
def _sheet_meta_from_summary(summary: dict[str, Any], run: BizCompareRun, tpl: Any) -> list[dict[str, Any]]:
|
||||
sheets = list(summary.get("sheets") or [])
|
||||
if sheets:
|
||||
|
|
@ -2193,6 +2257,18 @@ def _create_running_run(
|
|||
) -> BizCompareRun:
|
||||
first_metric = str(sheets_cfg[0].get("metric_id") or "")
|
||||
pending_sheets = [_pending_sheet_meta(s) for s in sheets_cfg]
|
||||
mapping = db.get(BizPortMapping, job.mapping_id) if job.mapping_id else None
|
||||
config_snapshot = deepcopy({
|
||||
"version": 1,
|
||||
"template": _template_out(tpl),
|
||||
"sheets": sheets_cfg,
|
||||
"port_map": _port_map_dict(db, str(job.mapping_id or "")),
|
||||
"store_unchanged": normalize_store_unchanged(getattr(job, "store_unchanged", None)),
|
||||
"job_name": job.name or "",
|
||||
"before_task_id": job.before_task_id or "",
|
||||
"after_task_id": job.after_task_id or "",
|
||||
"mapping_name": (mapping.name if mapping else "") or "",
|
||||
})
|
||||
run = BizCompareRun(
|
||||
id=uuid4().hex,
|
||||
job_id=job.id,
|
||||
|
|
@ -2203,6 +2279,7 @@ def _create_running_run(
|
|||
metric_id=first_metric,
|
||||
status="running",
|
||||
summary_json={
|
||||
"config_snapshot": config_snapshot,
|
||||
"progress": {
|
||||
"phase": "queued",
|
||||
"sheet_index": 0,
|
||||
|
|
@ -2232,6 +2309,26 @@ def _create_running_run(
|
|||
return run
|
||||
|
||||
|
||||
def _run_config_snapshot(run: BizCompareRun) -> dict[str, Any] | None:
|
||||
snapshot = (run.summary_json or {}).get("config_snapshot")
|
||||
if isinstance(snapshot, dict) and snapshot.get("version") == 1:
|
||||
return snapshot
|
||||
return None
|
||||
|
||||
|
||||
def _run_transform_config(
|
||||
db: Session, run: BizCompareRun,
|
||||
) -> tuple[dict[str, str], list[dict[str, str]]]:
|
||||
snapshot = _run_config_snapshot(run)
|
||||
if snapshot is not None:
|
||||
return dict(snapshot.get("port_map") or {}), normalize_iface_rules(
|
||||
(snapshot.get("template") or {}).get("iface_normalize_rules") or [],
|
||||
)
|
||||
# Legacy runs did not record these settings; retain their fallback behavior.
|
||||
tpl = db.get(BizCompareTemplate, run.template_id) if run.template_id else None
|
||||
return _port_map_dict(db, str(run.mapping_id or "")), template_iface_normalize(tpl)
|
||||
|
||||
|
||||
def _set_run_progress(
|
||||
db: Session,
|
||||
run: BizCompareRun,
|
||||
|
|
@ -2309,14 +2406,15 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
run = db.get(BizCompareRun, run_id)
|
||||
if not run:
|
||||
raise HTTPException(status_code=404, detail="run_not_found")
|
||||
snapshot = _run_config_snapshot(run)
|
||||
j = db.get(BizCompareJob, run.job_id)
|
||||
if not j:
|
||||
run.status = "failed"
|
||||
run.message = "job_not_found"
|
||||
db.commit()
|
||||
raise HTTPException(status_code=404, detail="job_not_found")
|
||||
tpl = db.get(BizCompareTemplate, run.template_id)
|
||||
if not tpl:
|
||||
tpl = db.get(BizCompareTemplate, run.template_id) if snapshot is None else None
|
||||
if not tpl and snapshot is None:
|
||||
run.status = "failed"
|
||||
run.message = "template_not_found"
|
||||
db.commit()
|
||||
|
|
@ -2324,25 +2422,25 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
|
||||
before_batch_id = str(run.before_batch_id or "")
|
||||
after_batch_id = str(run.after_batch_id or "")
|
||||
sheets_cfg = _filter_enabled_sheets(
|
||||
template_metrics(tpl), getattr(j, "enabled_sheet_ids", None)
|
||||
)
|
||||
if snapshot is not None:
|
||||
sheets_cfg = deepcopy(snapshot.get("sheets") or [])
|
||||
else:
|
||||
sheets_cfg = _filter_enabled_sheets(template_metrics(tpl), getattr(j, "enabled_sheet_ids", None))
|
||||
if not sheets_cfg:
|
||||
run.status = "failed"
|
||||
run.message = "no_enabled_sheets"
|
||||
db.commit()
|
||||
raise HTTPException(status_code=400, detail="no_enabled_sheets")
|
||||
sheets_cfg = _order_sheets_small_first(
|
||||
db,
|
||||
sheets_cfg,
|
||||
before_batch_id=before_batch_id,
|
||||
after_batch_id=after_batch_id,
|
||||
)
|
||||
if snapshot is None:
|
||||
sheets_cfg = _order_sheets_small_first(
|
||||
db, sheets_cfg, before_batch_id=before_batch_id, after_batch_id=after_batch_id,
|
||||
)
|
||||
|
||||
started_mono = time.monotonic()
|
||||
store_mode = normalize_store_unchanged(getattr(j, "store_unchanged", None))
|
||||
pmap = _port_map_dict(db, run.mapping_id)
|
||||
norm_rules = template_iface_normalize(tpl)
|
||||
store_mode = normalize_store_unchanged(
|
||||
snapshot.get("store_unchanged") if snapshot is not None else getattr(j, "store_unchanged", None),
|
||||
)
|
||||
pmap, norm_rules = _run_transform_config(db, run)
|
||||
unchanged_listed_total = 0
|
||||
unchanged_truncated_any = False
|
||||
unchanged_compact_any = False
|
||||
|
|
@ -2636,6 +2734,8 @@ def _execute_compare_into_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
},
|
||||
"sheets": sheet_metas,
|
||||
}
|
||||
if snapshot is not None:
|
||||
summary_payload["config_snapshot"] = snapshot
|
||||
run = db.get(BizCompareRun, run_id) or run
|
||||
if str(run.status or "") == "cancelled":
|
||||
return get_run(db, run.id)
|
||||
|
|
@ -2842,7 +2942,7 @@ def _csv_cell(v: Any) -> str:
|
|||
return s
|
||||
|
||||
|
||||
def _sheet_csv(sheet: dict[str, Any]) -> str:
|
||||
def _sheet_csv_lines(sheet: dict[str, Any], diffs: Iterable[dict[str, Any]]) -> Iterator[str]:
|
||||
keys = list(sheet.get("key_fields") or [])
|
||||
key_set = set(keys)
|
||||
compare = [f for f in list(sheet.get("compare_fields") or []) if f not in key_set]
|
||||
|
|
@ -2861,8 +2961,8 @@ def _sheet_csv(sheet: dict[str, Any]) -> str:
|
|||
headers.append(f"{f}__post")
|
||||
else:
|
||||
headers.append(f)
|
||||
lines = [",".join(_csv_cell(h) for h in headers)]
|
||||
for d in list(sheet.get("diffs") or []):
|
||||
yield ",".join(_csv_cell(h) for h in headers)
|
||||
for d in diffs:
|
||||
kind = str(d.get("kind") or "")
|
||||
pre = dict(d.get("mapped_before") or d.get("before") or {})
|
||||
post = dict(d.get("after") or {})
|
||||
|
|
@ -2887,8 +2987,11 @@ def _sheet_csv(sheet: dict[str, Any]) -> str:
|
|||
row.append(pre.get(f, ""))
|
||||
else:
|
||||
row.append(post.get(f, pre.get(f, "")))
|
||||
lines.append(",".join(_csv_cell(x) for x in row))
|
||||
return "\ufeff" + "\n".join(lines) + "\n"
|
||||
yield ",".join(_csv_cell(x) for x in row)
|
||||
|
||||
|
||||
def _sheet_csv(sheet: dict[str, Any]) -> str:
|
||||
return "\ufeff" + "\n".join(_sheet_csv_lines(sheet, sheet.get("diffs") or [])) + "\n"
|
||||
|
||||
|
||||
def _enrich_summary(summary: dict[str, Any], sheets: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
|
|
@ -3001,8 +3104,13 @@ def get_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
r = db.get(BizCompareRun, run_id)
|
||||
if not r:
|
||||
raise HTTPException(status_code=404, detail="run_not_found")
|
||||
tpl = db.get(BizCompareTemplate, r.template_id) if r.template_id else None
|
||||
summary = dict(r.summary_json or {})
|
||||
snapshot = _run_config_snapshot(r)
|
||||
tpl = db.get(BizCompareTemplate, r.template_id) if r.template_id and snapshot is None else None
|
||||
template_out = (
|
||||
deepcopy(snapshot.get("template")) if snapshot is not None
|
||||
else (_template_out(tpl) if tpl else None)
|
||||
)
|
||||
raw_sheets = _sheet_meta_from_summary(summary, r, tpl)
|
||||
# Never return full diffs in run detail (million-row safe)
|
||||
sheets = [
|
||||
|
|
@ -3023,22 +3131,31 @@ def get_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
]
|
||||
enriched = _enrich_summary(summary, raw_sheets)
|
||||
stored = "rows" if _run_has_diff_rows(db, run_id) else "inline"
|
||||
job = db.get(BizCompareJob, r.job_id) if r.job_id else None
|
||||
mapping = db.get(BizPortMapping, r.mapping_id) if r.mapping_id else None
|
||||
job = db.get(BizCompareJob, r.job_id) if r.job_id and snapshot is None else None
|
||||
mapping = db.get(BizPortMapping, r.mapping_id) if r.mapping_id and snapshot is None else None
|
||||
before_side = _compare_side(
|
||||
db, r.before_batch_id, fallback_task_id=(job.before_task_id if job else "")
|
||||
db, r.before_batch_id, fallback_task_id=(
|
||||
snapshot.get("before_task_id", "") if snapshot is not None else (job.before_task_id if job else "")
|
||||
),
|
||||
)
|
||||
after_side = _compare_side(
|
||||
db, r.after_batch_id, fallback_task_id=(job.after_task_id if job else "")
|
||||
db, r.after_batch_id, fallback_task_id=(
|
||||
snapshot.get("after_task_id", "") if snapshot is not None else (job.after_task_id if job else "")
|
||||
),
|
||||
)
|
||||
return {
|
||||
"id": r.id,
|
||||
"job_id": r.job_id,
|
||||
"job_name": (job.name if job else "") or "",
|
||||
"job_name": (
|
||||
snapshot.get("job_name", "") if snapshot is not None else ((job.name if job else "") or "")
|
||||
),
|
||||
"template_id": r.template_id,
|
||||
"template_name": (tpl.name if tpl else "") or "",
|
||||
"template_name": (template_out or {}).get("name") or "",
|
||||
"mapping_id": r.mapping_id,
|
||||
"mapping_name": (mapping.name if mapping else "") or "",
|
||||
"mapping_name": (
|
||||
snapshot.get("mapping_name", "") if snapshot is not None else ((mapping.name if mapping else "") or "")
|
||||
),
|
||||
"config_snapshot_version": snapshot["version"] if snapshot is not None else None,
|
||||
"before_batch_id": r.before_batch_id,
|
||||
"after_batch_id": r.after_batch_id,
|
||||
"before": before_side,
|
||||
|
|
@ -3052,7 +3169,7 @@ def get_run(db: Session, run_id: str) -> dict[str, Any]:
|
|||
"mapping_stats": r.mapping_stats_json or {},
|
||||
"message": r.message,
|
||||
"created_at": r.created_at.isoformat() + "Z" if r.created_at else None,
|
||||
"template": _template_out(tpl) if tpl else None,
|
||||
"template": template_out,
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -3152,13 +3269,17 @@ def _field_qf_sql(field_q: dict[str, str], *, prefix: str = "qf") -> tuple[str,
|
|||
for i, (name, val) in enumerate(field_q.items()):
|
||||
sf = _safe_field(name)
|
||||
key = f"{prefix}_{i}"
|
||||
params[key] = f"%{val.lower()}%"
|
||||
params[key] = _literal_search_pattern(val)
|
||||
parts.append(
|
||||
f"lower(trim(both from coalesce(data_json->>'{sf}', ''))) LIKE :{key}"
|
||||
)
|
||||
return "(" + " AND ".join(parts) + ")", params
|
||||
|
||||
|
||||
def _literal_search_pattern(value: str) -> str:
|
||||
return "%" + value.lower().replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + "%"
|
||||
|
||||
|
||||
def _load_metric_rows_for_search(
|
||||
db: Session,
|
||||
*,
|
||||
|
|
@ -3188,7 +3309,9 @@ def _load_metric_rows_for_search(
|
|||
lim = max(1, min(int(cap), _LIVE_SEARCH_LOAD_CAP))
|
||||
filters = [f for f in (row_filters or []) if isinstance(f, dict)]
|
||||
|
||||
if _dialect_is_postgres(db):
|
||||
if mid != "lldp_neighbor" and _dialect_is_postgres(db) and (
|
||||
not filters or _filters_sql_compatible(filters)
|
||||
):
|
||||
filter_sql, filter_params = ("TRUE", {})
|
||||
if filters and _filters_sql_compatible(filters):
|
||||
filter_sql, filter_params = compile_row_filters_sql(filters)
|
||||
|
|
@ -3202,7 +3325,7 @@ def _load_metric_rows_for_search(
|
|||
**qf_params,
|
||||
}
|
||||
if needle:
|
||||
params["kw"] = f"%{needle.lower()}%"
|
||||
params["kw"] = _literal_search_pattern(needle)
|
||||
search_parts.append(_kw_match_sql(key_fields))
|
||||
search_sql = " AND ".join(f"({p})" for p in search_parts if p and p != "TRUE")
|
||||
if not search_sql:
|
||||
|
|
@ -3246,20 +3369,21 @@ def _load_metric_rows_for_search(
|
|||
return out, truncated
|
||||
|
||||
# Non-PG / fallback: scan with early stop (OK for tests / small sheets)
|
||||
q = (
|
||||
db.query(BizStateMetricRow)
|
||||
.filter(
|
||||
BizStateMetricRow.batch_id == bid,
|
||||
BizStateMetricRow.metric_id == mid,
|
||||
)
|
||||
.order_by(BizStateMetricRow.seq.asc(), BizStateMetricRow.id.asc())
|
||||
)
|
||||
model = BizStateLldpNeighbor if mid == "lldp_neighbor" else BizStateMetricRow
|
||||
q = db.query(model).filter(model.batch_id == bid)
|
||||
if mid == "lldp_neighbor":
|
||||
q = q.order_by(model.id.asc())
|
||||
else:
|
||||
q = q.filter(model.metric_id == mid).order_by(model.seq.asc(), model.id.asc())
|
||||
out = []
|
||||
truncated = False
|
||||
needle_l = needle.lower()
|
||||
key_set = [str(k).strip() for k in key_fields if str(k).strip()]
|
||||
for r in q.yield_per(500):
|
||||
data = dict(r.data_json or {})
|
||||
data = (
|
||||
{f: getattr(r, f) for f in ("local_if", "remote_sys", "remote_if", "remote_ip", "protocol")}
|
||||
if mid == "lldp_neighbor" else dict(r.data_json or {})
|
||||
)
|
||||
row = {
|
||||
**data,
|
||||
"_netx": {
|
||||
|
|
@ -3297,14 +3421,75 @@ def _load_metric_rows_for_search(
|
|||
if not hit:
|
||||
db.expunge(r)
|
||||
continue
|
||||
out.append(row)
|
||||
db.expunge(r)
|
||||
if len(out) >= lim:
|
||||
truncated = True
|
||||
break
|
||||
out.append(row)
|
||||
return out, truncated
|
||||
|
||||
|
||||
def _load_search_key_groups(
|
||||
db: Session, *, batch_id: str, metric_id: str, keys: set[tuple[str, ...]],
|
||||
match_keys: list[str], iface_fields: list[str], row_filters: list[dict[str, Any]],
|
||||
norm_rules: list[dict[str, str]], port_map: dict[str, str],
|
||||
) -> tuple[list[dict[str, Any]], set[tuple[str, ...]]]:
|
||||
"""Complete candidate key groups so a one-sided search cannot change verdicts.
|
||||
|
||||
Interface transforms run in Python. Other identity fields narrow the SQL
|
||||
query. If the load budget is exceeded, omit entire groups rather than compare
|
||||
partial groups and manufacture additions/deletions.
|
||||
"""
|
||||
from ..models import BizStateMetricRow
|
||||
|
||||
model = BizStateLldpNeighbor if metric_id == "lldp_neighbor" else BizStateMetricRow
|
||||
invariant = [f for f in match_keys if f not in iface_fields]
|
||||
positions = [match_keys.index(f) for f in invariant]
|
||||
coarse = sorted({tuple(k[i] for i in positions) for k in keys})
|
||||
out: list[dict[str, Any]] = []
|
||||
complete: set[tuple[str, ...]] = set()
|
||||
# Bound memory even when a candidate identifies a very large duplicate group.
|
||||
budget = _LIVE_SEARCH_GROUP_LOAD_CAP
|
||||
for start in range(0, len(coarse), 100):
|
||||
group = coarse[start:start + 100]
|
||||
group_set = set(group)
|
||||
target = {k for k in keys if tuple(k[i] for i in positions) in group_set}
|
||||
q = db.query(model).filter(model.batch_id == batch_id)
|
||||
if metric_id == "lldp_neighbor":
|
||||
q = q.order_by(model.id.asc())
|
||||
else:
|
||||
q = q.filter(model.metric_id == metric_id).order_by(model.seq.asc(), model.id.asc())
|
||||
if invariant:
|
||||
exprs = [func.trim(func.coalesce(
|
||||
cast(getattr(model, f) if metric_id == "lldp_neighbor"
|
||||
else model.data_json[f].as_string(), String), "")) for f in invariant]
|
||||
q = q.filter(tuple_(*exprs).in_(group))
|
||||
rows = q.limit(budget + 1).all()
|
||||
if len(rows) > budget:
|
||||
for r in rows:
|
||||
db.expunge(r)
|
||||
# No partial verdicts. Later groups may still fit the remaining budget.
|
||||
continue
|
||||
budget -= len(rows)
|
||||
complete.update(target)
|
||||
for r in rows:
|
||||
data = (
|
||||
{f: getattr(r, f) for f in ("local_if", "remote_sys", "remote_if", "remote_ip", "protocol")}
|
||||
if metric_id == "lldp_neighbor" else dict(r.data_json or {})
|
||||
)
|
||||
row = {**data, "_netx": {"row_id": r.id}}
|
||||
db.expunge(r)
|
||||
if row_filters and not all(row_matches_filter(row, f) for f in row_filters):
|
||||
continue
|
||||
normalized = apply_iface_normalize_rows(
|
||||
[row], iface_fields=iface_fields, rules=norm_rules,
|
||||
)[0]
|
||||
mapped = apply_port_map(normalized, iface_fields=iface_fields, port_map=port_map)
|
||||
if row_key(mapped, match_keys) in target:
|
||||
out.append(row)
|
||||
return out, complete
|
||||
|
||||
|
||||
def _live_search_sheet_diffs(
|
||||
db: Session,
|
||||
run: BizCompareRun,
|
||||
|
|
@ -3316,7 +3501,7 @@ def _live_search_sheet_diffs(
|
|||
page: int,
|
||||
page_size: int,
|
||||
) -> dict[str, Any]:
|
||||
"""Search before/after metric tables, zip-compare, filter by kind tab."""
|
||||
"""Find candidate keys, complete both sides, then compare and filter pairs."""
|
||||
mid_src = str(sheet.get("metric_id") or "").strip()
|
||||
sid = sheet_key(sheet)
|
||||
key_fields = list(sheet.get("key_fields") or [])
|
||||
|
|
@ -3328,7 +3513,7 @@ def _live_search_sheet_diffs(
|
|||
)
|
||||
row_filters = list(sheet.get("row_filters") or [])
|
||||
# Older runs may lack row_filters on sheet meta — fall back to template
|
||||
if not row_filters and run.template_id:
|
||||
if "row_filters" not in sheet and _run_config_snapshot(run) is None and run.template_id:
|
||||
tpl = db.get(BizCompareTemplate, run.template_id)
|
||||
if tpl:
|
||||
for s in template_metrics(tpl):
|
||||
|
|
@ -3363,9 +3548,7 @@ def _live_search_sheet_diffs(
|
|||
ignore_ports = sheet.get("ignore_port_changes")
|
||||
if ignore_ports is not None:
|
||||
ignore_ports = bool(ignore_ports)
|
||||
pmap = _port_map_dict(db, str(run.mapping_id or ""))
|
||||
tpl = db.get(BizCompareTemplate, run.template_id) if run.template_id else None
|
||||
norm_rules = template_iface_normalize(tpl)
|
||||
pmap, norm_rules = _run_transform_config(db, run)
|
||||
|
||||
before_rows, trunc_b = _load_metric_rows_for_search(
|
||||
db,
|
||||
|
|
@ -3385,6 +3568,31 @@ def _live_search_sheet_diffs(
|
|||
kw=free_kw,
|
||||
field_q=merged_q,
|
||||
)
|
||||
match_keys = list((sheet.get("summary") or {}).get("match_key_fields") or key_fields)
|
||||
if ignore_ports is True and not pmap:
|
||||
match_keys = [f for f in key_fields if f not in iface_fields] or key_fields
|
||||
keys: set[tuple[str, ...]] = set()
|
||||
for rows, mapping in ((before_rows, pmap), (after_rows, {})):
|
||||
for row in apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules):
|
||||
keys.add(row_key(apply_port_map(row, iface_fields=iface_fields, port_map=mapping), match_keys))
|
||||
before_rows, complete_b = _load_search_key_groups(
|
||||
db, batch_id=str(run.before_batch_id or ""), metric_id=mid_src, keys=keys,
|
||||
match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters,
|
||||
norm_rules=norm_rules, port_map=pmap,
|
||||
)
|
||||
after_rows, complete_a = _load_search_key_groups(
|
||||
db, batch_id=str(run.after_batch_id or ""), metric_id=mid_src, keys=keys,
|
||||
match_keys=match_keys, iface_fields=iface_fields, row_filters=row_filters,
|
||||
norm_rules=norm_rules, port_map={},
|
||||
)
|
||||
complete = complete_b & complete_a
|
||||
def keep_complete(rows: list[dict[str, Any]], mapping: dict[str, str]) -> list[dict[str, Any]]:
|
||||
normalized = apply_iface_normalize_rows(rows, iface_fields=iface_fields, rules=norm_rules)
|
||||
return [orig for orig, norm in zip(rows, normalized) if row_key(
|
||||
apply_port_map(norm, iface_fields=iface_fields, port_map=mapping), match_keys,
|
||||
) in complete]
|
||||
before_rows = keep_complete(before_rows, pmap)
|
||||
after_rows = keep_complete(after_rows, {})
|
||||
result = compare_rows(
|
||||
before_rows=before_rows,
|
||||
after_rows=after_rows,
|
||||
|
|
@ -3394,17 +3602,19 @@ def _live_search_sheet_diffs(
|
|||
port_map=pmap,
|
||||
field_rules=field_rules,
|
||||
iface_normalize_rules=norm_rules,
|
||||
ignore_port_changes=ignore_ports,
|
||||
ignore_port_changes=(match_keys != key_fields) if not pmap else False,
|
||||
include_unchanged=True,
|
||||
unchanged_limit=None,
|
||||
compact_unchanged=False,
|
||||
)
|
||||
kind_n = (kind or "diff").strip().lower()
|
||||
filtered = [
|
||||
d for d in list(result.get("diffs") or []) if _kind_allows(kind_n, str(d.get("kind") or ""))
|
||||
d for d in list(result.get("diffs") or [])
|
||||
if _kind_allows(kind_n, str(d.get("kind") or ""))
|
||||
and _diff_matches_search(d, kw=free_kw, field_q=merged_q)
|
||||
]
|
||||
# Cap pairs returned to keep UI snappy
|
||||
truncated = bool(trunc_b or trunc_a or len(filtered) > _LIVE_SEARCH_RESULT_CAP)
|
||||
truncated = bool(trunc_b or trunc_a or complete != keys or len(filtered) > _LIVE_SEARCH_RESULT_CAP)
|
||||
filtered = filtered[:_LIVE_SEARCH_RESULT_CAP]
|
||||
total = len(filtered)
|
||||
start = (page - 1) * page_size
|
||||
|
|
@ -3452,10 +3662,17 @@ def list_run_diffs(
|
|||
sheets = _sheet_meta_from_summary(summary, r, tpl)
|
||||
asked = (metric_id or "").strip()
|
||||
sheet = _lookup_sheet(sheets, asked) if asked else (sheets[0] if sheets else None)
|
||||
if asked and sheet is None:
|
||||
raise HTTPException(status_code=404, detail="sheet_not_found")
|
||||
mid = sheet_key(sheet) if sheet else (asked or str(r.metric_id or ""))
|
||||
|
||||
# Unified search: kw and/or field filters → live source lookup; kind tab only filters.
|
||||
if (kw_n or field_q) and sheet:
|
||||
# Only reconstruct missing success rows. Persisted verdicts must stay stable
|
||||
# while searching and avoid loading/recomparing source tables on every page.
|
||||
sheet_summary = (sheet or {}).get("summary") or {}
|
||||
success_missing = int(sheet_summary.get("unchanged") or 0) > int(
|
||||
sheet_summary.get("unchanged_listed") or 0
|
||||
)
|
||||
if (kw_n or field_q) and sheet and kind_n in ("all", "unchanged") and success_missing:
|
||||
return _live_search_sheet_diffs(
|
||||
db,
|
||||
r,
|
||||
|
|
@ -3480,6 +3697,9 @@ def list_run_diffs(
|
|||
q = q.filter(BizCompareDiff.kind == "changed")
|
||||
elif kind_n != "all":
|
||||
q = q.filter(BizCompareDiff.kind == kind_n)
|
||||
if kw_n or field_q:
|
||||
q = _search_stored_diffs(q, metric_id=str((sheet or {}).get("metric_id") or mid),
|
||||
kw=kw_n, field_q=field_q)
|
||||
total = q.count()
|
||||
rows = (
|
||||
q.order_by(BizCompareDiff.seq.asc(), BizCompareDiff.id.asc())
|
||||
|
|
@ -3487,7 +3707,7 @@ def list_run_diffs(
|
|||
.limit(size_n)
|
||||
.all()
|
||||
)
|
||||
items = _hydrate_diff_rows(db, [_diff_row_out(x) for x in rows])
|
||||
items = _hydrate_diff_rows(db, [_diff_row_out(x) for x in rows], run=r, sheet=sheet)
|
||||
return {
|
||||
"total": total,
|
||||
"page": page_n,
|
||||
|
|
@ -3505,7 +3725,7 @@ def list_run_diffs(
|
|||
inline = list((sheet or {}).get("diffs") or [])
|
||||
if not inline and mid == r.metric_id:
|
||||
inline = list(r.diffs_json or [])
|
||||
filtered = _filter_inline_diffs(inline, kind=kind_n, kw="")
|
||||
filtered = _filter_inline_diffs(inline, kind=kind_n, kw=kw_n, field_q=field_q)
|
||||
total = len(filtered)
|
||||
start = (page_n - 1) * size_n
|
||||
page_items = filtered[start : start + size_n]
|
||||
|
|
@ -3520,44 +3740,53 @@ def list_run_diffs(
|
|||
}
|
||||
|
||||
|
||||
def _iter_sheet_diffs(db: Session, run_id: str, metric_id: str) -> list[dict[str, Any]]:
|
||||
"""Load all diffs for one sheet (export). Prefer row table; fall back to inline.
|
||||
def _iter_sheet_diffs(db: Session, run_id: str, metric_id: str) -> Iterator[dict[str, Any]]:
|
||||
"""Stream one sheet using a stable (seq, id) cursor; fall back to inline.
|
||||
|
||||
Compact success rows are hydrated in chunks from metric tables.
|
||||
"""
|
||||
r = db.get(BizCompareRun, run_id)
|
||||
if not r:
|
||||
return
|
||||
tpl = db.get(BizCompareTemplate, r.template_id) if r.template_id else None
|
||||
sheets = _sheet_meta_from_summary(dict(r.summary_json or {}), r, tpl)
|
||||
sheet = next((s for s in sheets if sheet_key(s) == metric_id), None)
|
||||
if _run_has_diff_rows(db, run_id):
|
||||
out: list[dict[str, Any]] = []
|
||||
offset = 0
|
||||
cursor: tuple[int, str] | None = None
|
||||
while True:
|
||||
rows = (
|
||||
db.query(BizCompareDiff)
|
||||
.filter(BizCompareDiff.run_id == run_id, BizCompareDiff.metric_id == metric_id)
|
||||
q = db.query(BizCompareDiff).filter(
|
||||
BizCompareDiff.run_id == run_id, BizCompareDiff.metric_id == metric_id,
|
||||
)
|
||||
if cursor is not None:
|
||||
seq, diff_id = cursor
|
||||
q = q.filter(or_(BizCompareDiff.seq > seq, and_(
|
||||
BizCompareDiff.seq == seq, BizCompareDiff.id > diff_id,
|
||||
)))
|
||||
rows = (q
|
||||
.order_by(BizCompareDiff.seq.asc(), BizCompareDiff.id.asc())
|
||||
.offset(offset)
|
||||
.limit(_DIFF_CHUNK)
|
||||
.all()
|
||||
)
|
||||
if not rows:
|
||||
break
|
||||
chunk = _hydrate_diff_rows(db, [_diff_row_out(x) for x in rows])
|
||||
out.extend(chunk)
|
||||
offset += len(rows)
|
||||
chunk = _hydrate_diff_rows(db, [_diff_row_out(x) for x in rows], run=r, sheet=sheet)
|
||||
cursor = (rows[-1].seq, rows[-1].id)
|
||||
yield from chunk
|
||||
if len(rows) < _DIFF_CHUNK:
|
||||
break
|
||||
return out
|
||||
r = db.get(BizCompareRun, run_id)
|
||||
if not r:
|
||||
return []
|
||||
# Release the previous chunk before loading the next one.
|
||||
del rows, chunk
|
||||
return
|
||||
summary = dict(r.summary_json or {})
|
||||
sheets = list(summary.get("sheets") or [])
|
||||
for sh in sheets:
|
||||
if sheet_key(sh) == metric_id or (
|
||||
str(sh.get("metric_id") or "") == metric_id and sheet_key(sh) == metric_id
|
||||
):
|
||||
return list(sh.get("diffs") or [])
|
||||
yield from sh.get("diffs") or []
|
||||
return
|
||||
if metric_id == r.metric_id:
|
||||
return list(r.diffs_json or [])
|
||||
return []
|
||||
yield from r.diffs_json or []
|
||||
|
||||
|
||||
def export_run_zip(db: Session, run_id: str) -> bytes:
|
||||
|
|
@ -3586,14 +3815,17 @@ def export_run_zip(db: Session, run_id: str) -> bytes:
|
|||
f"+{card.get('added')}/-{card.get('removed')}/~{card.get('changed')}/= {card.get('unchanged')}"
|
||||
)
|
||||
zf.writestr("manifest.txt", "\n".join(manifest) + "\n")
|
||||
run = db.get(BizCompareRun, run_id)
|
||||
snapshot = _run_config_snapshot(run) if run else None
|
||||
if snapshot is not None:
|
||||
zf.writestr("config_snapshot.json", json.dumps(snapshot, ensure_ascii=False, indent=2))
|
||||
for sheet in list(detail.get("sheets") or []):
|
||||
sid = sheet_key(sheet) or "sheet"
|
||||
safe = "".join(ch if ch.isalnum() or ch in "-_." else "_" for ch in sid)[:80] or "sheet"
|
||||
sheet_full = {
|
||||
**sheet,
|
||||
"diffs": _iter_sheet_diffs(db, run_id, sid),
|
||||
}
|
||||
zf.writestr(f"tables/{safe}.csv", _sheet_csv(sheet_full))
|
||||
with zf.open(f"tables/{safe}.csv", "w", force_zip64=True) as csv_file:
|
||||
csv_file.write(b"\xef\xbb\xbf")
|
||||
for line in _sheet_csv_lines(sheet, _iter_sheet_diffs(db, run_id, sid)):
|
||||
csv_file.write((line + "\n").encode("utf-8"))
|
||||
sum_lines = ["metric_id,mode,before,after,added,removed,changed,unchanged,diff_count,pass_rate"]
|
||||
for card in list(s.get("sheet_cards") or []):
|
||||
sum_lines.append(
|
||||
|
|
|
|||
87
tests/test_biz_state_compare_export.py
Normal file
87
tests/test_biz_state_compare_export.py
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
"""Export must page lazily, keep tied sequence rows, and produce valid UTF-8 CSV."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import unittest
|
||||
import zipfile
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine, event
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from netx_api.biz_state.compare_service import _iter_sheet_diffs, export_run_zip
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import BizCompareDiff, BizCompareRun
|
||||
|
||||
|
||||
class CompareExportTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine)
|
||||
self.db = Session(self.engine)
|
||||
self.sheet = {"sheet_id": "sheet", "metric_id": "test", "key_fields": ["id"],
|
||||
"compare_fields": ["note"], "display_fields": ["id", "note"],
|
||||
"summary": {"unchanged": 7}, "status": "done"}
|
||||
self.run = BizCompareRun(id="run", metric_id="test", status="success",
|
||||
summary_json={"sheets": [self.sheet]})
|
||||
self.db.add(self.run)
|
||||
for seq, ids in ((0, "cba"), (1, "ed"), (2, "gf")):
|
||||
for rid in ids:
|
||||
note = '中文, "quoted"\nsecond line'
|
||||
self.db.add(BizCompareDiff(id=rid, run_id="run", metric_id="sheet", seq=seq,
|
||||
kind="unchanged", key_json={"id": rid},
|
||||
before_json={"id": rid, "note": note},
|
||||
after_json={"id": rid, "note": note}))
|
||||
self.db.commit()
|
||||
|
||||
def tearDown(self):
|
||||
self.db.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def test_cursor_handles_tied_sequence_numbers_across_chunks(self):
|
||||
statements = []
|
||||
def record(_conn, _cursor, statement, _parameters, _context, _executemany):
|
||||
if "ORDER BY biz_compare_diff.seq" in statement:
|
||||
statements.append(statement)
|
||||
event.listen(self.engine, "before_cursor_execute", record)
|
||||
with patch("netx_api.biz_state.compare_service._DIFF_CHUNK", 2):
|
||||
iterator = _iter_sheet_diffs(self.db, "run", "sheet")
|
||||
self.assertEqual(statements, [])
|
||||
self.assertEqual(next(iterator)["key"]["id"], "a")
|
||||
self.assertEqual(next(iterator)["key"]["id"], "b")
|
||||
self.assertEqual(len(statements), 1, "do not load the next chunk early")
|
||||
self.assertEqual([d["key"]["id"] for d in iterator], list("cdefg"))
|
||||
self.assertEqual(len(statements), 4)
|
||||
self.assertTrue(all("biz_compare_diff.seq >" in q and "biz_compare_diff.id >" in q
|
||||
for q in statements[1:]))
|
||||
|
||||
def test_export_csv_escapes_unicode_quotes_commas_and_newlines(self):
|
||||
with patch("netx_api.biz_state.compare_service._DIFF_CHUNK", 2):
|
||||
archive_bytes = export_run_zip(self.db, "run")
|
||||
with zipfile.ZipFile(io.BytesIO(archive_bytes)) as archive:
|
||||
payload = archive.read("tables/sheet.csv")
|
||||
self.assertTrue(payload.startswith(b"\xef\xbb\xbf"))
|
||||
rows = list(csv.DictReader(io.StringIO(payload.decode("utf-8-sig"))))
|
||||
self.assertEqual([r["id"] for r in rows], list("abcdefg"))
|
||||
self.assertTrue(all(r["note__pre"] == '中文, "quoted"\nsecond line' for r in rows))
|
||||
self.assertTrue(all(r["note__pre"] == r["note__post"] for r in rows))
|
||||
|
||||
def test_other_sheets_do_not_leak_into_export(self):
|
||||
self.db.add(BizCompareDiff(id="other", run_id="run", metric_id="another", seq=0,
|
||||
kind="added", key_json={"id": "other"}, after_json={"note": "other"}))
|
||||
self.db.commit()
|
||||
with patch("netx_api.biz_state.compare_service._DIFF_CHUNK", 2):
|
||||
self.assertEqual([d["key"]["id"] for d in _iter_sheet_diffs(self.db, "run", "sheet")], list("abcdefg"))
|
||||
|
||||
def test_legacy_inline_and_empty_exports(self):
|
||||
self.db.query(BizCompareDiff).delete()
|
||||
inline = {"kind": "added", "key": {"id": "legacy"}, "after": {"note": "value"}}
|
||||
self.run.summary_json = {"sheets": [{**self.sheet, "diffs": [inline]}]}
|
||||
self.db.commit()
|
||||
self.assertEqual(list(_iter_sheet_diffs(self.db, "run", "sheet")), [inline])
|
||||
self.run.summary_json = {"sheets": [self.sheet]}
|
||||
self.db.commit()
|
||||
with zipfile.ZipFile(io.BytesIO(export_run_zip(self.db, "run"))) as archive:
|
||||
self.assertEqual(archive.read("tables/sheet.csv").decode("utf-8-sig"), "kind,id,note__pre,note__post\n")
|
||||
103
tests/test_biz_state_compare_memory.py
Normal file
103
tests/test_biz_state_compare_memory.py
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
"""Bounded sampling regressions and an opt-in comparison memory benchmark."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import subprocess
|
||||
import time
|
||||
import tracemalloc
|
||||
import types
|
||||
from collections import defaultdict, deque
|
||||
from copy import deepcopy
|
||||
|
||||
import pytest
|
||||
|
||||
from netx_api.biz_state.compare_engine import _StratifiedSample, compare_rows, stratify_take
|
||||
|
||||
|
||||
def reference_sample(items, limit):
|
||||
if len(items) <= limit:
|
||||
return list(items)
|
||||
buckets = defaultdict(deque)
|
||||
for item in items:
|
||||
buckets[item[0]].append(item)
|
||||
out = []
|
||||
while buckets and len(out) < limit:
|
||||
for key in list(buckets):
|
||||
out.append(buckets[key].popleft())
|
||||
if not buckets[key]:
|
||||
del buckets[key]
|
||||
if len(out) >= limit:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def test_streaming_sample_preserves_round_robin_and_small_input_order():
|
||||
rng = random.Random(42)
|
||||
for _ in range(100):
|
||||
items = [(str(rng.randrange(20)), i) for i in range(rng.randrange(100))]
|
||||
for limit in (0, 1, 5, 20, 100):
|
||||
expected = reference_sample(items, limit)
|
||||
assert stratify_take(iter(items), limit, key_fn=lambda item: item[0]) == expected
|
||||
|
||||
|
||||
def test_sample_storage_is_bounded_for_both_many_rows_and_many_strata():
|
||||
sample = _StratifiedSample(50)
|
||||
for i in range(20_000):
|
||||
sample.add(i, str(i) if i >= 10_000 else "one-large-peer")
|
||||
assert len(sample.heap) == 50
|
||||
assert len(sample.strata) == 50
|
||||
assert sample.picked() == [0, *range(10_000, 10_049)]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("iface_fields,rules,pmap", [
|
||||
([], [], {}),
|
||||
(["interface"], [], {}),
|
||||
(["interface"], [{"from": "GE", "to": "gei"}], {"gei-1": "gei-2"}),
|
||||
])
|
||||
def test_compare_does_not_mutate_inputs_when_reusing_rows(iface_fields, rules, pmap):
|
||||
before = [{"id": "1", "interface": "GE-1", "state": "up"}]
|
||||
after = [{"id": "1", "interface": "GE-2", "state": "up"}]
|
||||
original = deepcopy((before, after))
|
||||
compare_rows(before_rows=before, after_rows=after, key_fields=["id"],
|
||||
iface_fields=iface_fields, compare_fields=["state"], port_map=pmap,
|
||||
iface_normalize_rules=rules, include_unchanged=True, unchanged_limit=1)
|
||||
assert (before, after) == original
|
||||
|
||||
|
||||
def test_memory_benchmark_against_git_baseline():
|
||||
"""Run explicitly with NETX_BENCH_ROWS=1000000; not part of routine CI."""
|
||||
n = int(os.environ.get("NETX_BENCH_ROWS", "0"))
|
||||
if not n:
|
||||
pytest.skip("opt-in large synthetic benchmark")
|
||||
baseline_ref = os.environ.get("NETX_BENCH_REF", "HEAD")
|
||||
source = subprocess.run(["git", "show", f"{baseline_ref}:netx_api/biz_state/compare_engine.py"],
|
||||
check=True, capture_output=True, text=True, encoding="utf-8").stdout
|
||||
baseline = types.ModuleType("netx_api.biz_state._benchmark_baseline")
|
||||
baseline.__package__ = "netx_api.biz_state"
|
||||
exec(compile(source, "baseline_compare_engine.py", "exec"), baseline.__dict__)
|
||||
before = [{"id": str(i), "neighbor": str(i % 50), "state": "up", "afi": "vpnv4"} for i in range(n)]
|
||||
after = [dict(row) for row in before]
|
||||
records = []
|
||||
digests = []
|
||||
for name, compare in (("baseline", baseline.compare_rows), ("optimized", compare_rows)):
|
||||
gc.collect()
|
||||
tracemalloc.start()
|
||||
started = time.perf_counter()
|
||||
result = compare(before_rows=before, after_rows=after, key_fields=["id"], iface_fields=[],
|
||||
compare_fields=["state"], include_unchanged=True, unchanged_limit=5000,
|
||||
compact_unchanged=True)
|
||||
elapsed = time.perf_counter() - started
|
||||
_, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
digests.append(hashlib.sha256(json.dumps(result, sort_keys=True).encode()).hexdigest())
|
||||
records.append({"engine": name, "rows_per_side": n, "peak_engine_mib": round(peak / 2**20, 2),
|
||||
"seconds_with_tracemalloc": round(elapsed, 3)})
|
||||
del result
|
||||
assert digests[0] == digests[1], "large-sheet results and sample order must match"
|
||||
print("\n" + json.dumps({"baseline_ref": baseline_ref, "input_allocations_included": False,
|
||||
"results": records}, indent=2))
|
||||
186
tests/test_biz_state_compare_search.py
Normal file
186
tests/test_biz_state_compare_search.py
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
"""Regression tests: searching a pair must not change its comparison verdict."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from netx_api.biz_state.compare_engine import compare_rows
|
||||
from netx_api.biz_state.compare_service import (
|
||||
_load_metric_rows_for_search,
|
||||
_literal_search_pattern,
|
||||
_persist_sheet_diffs,
|
||||
_search_stored_diffs,
|
||||
list_run_diffs,
|
||||
)
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareDiff, BizCompareRun, BizCompareTemplate, BizPortMappingRow,
|
||||
BizStateLldpNeighbor, BizStateMetricRow,
|
||||
)
|
||||
|
||||
|
||||
class CompareSearchTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine, tables=[m.__table__ for m in (
|
||||
BizCompareDiff, BizCompareRun, BizCompareTemplate, BizPortMappingRow,
|
||||
BizStateLldpNeighbor, BizStateMetricRow,
|
||||
)])
|
||||
self.db = Session(self.engine)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.db.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def seed(self, before, after, *, metric="test", keys=None, fields=None,
|
||||
iface=None, mapping=None, sample=False, compact=False, filters=None):
|
||||
keys = keys or ["id"]
|
||||
iface = iface or []
|
||||
fields = fields or ["state"]
|
||||
for side, rows in (("before", before), ("after", after)):
|
||||
for seq, row in enumerate(rows):
|
||||
rid = f"{side}-{seq}"
|
||||
if metric == "lldp_neighbor":
|
||||
self.db.add(BizStateLldpNeighbor(id=rid, batch_id=side, **row))
|
||||
else:
|
||||
self.db.add(BizStateMetricRow(id=rid, batch_id=side, metric_id=metric,
|
||||
seq=seq, data_json=dict(row)))
|
||||
row["_netx"] = {"row_id": rid}
|
||||
if mapping:
|
||||
for b, a in mapping.items():
|
||||
self.db.add(BizPortMappingRow(mapping_id="map", before_if=b, after_if=a))
|
||||
result = compare_rows(before_rows=before, after_rows=after, key_fields=keys,
|
||||
compare_fields=fields, iface_fields=iface, port_map=mapping or {},
|
||||
include_unchanged=True, compact_unchanged=compact)
|
||||
sheet = {"sheet_id": "sheet", "metric_id": metric, "key_fields": keys,
|
||||
"compare_fields": fields, "iface_fields": iface,
|
||||
"summary": {**result["summary"], "unchanged_listed": 0 if sample
|
||||
else result["summary"]["unchanged"]},
|
||||
"row_filters": filters or []}
|
||||
self.db.add(BizCompareRun(id="run", status="success", before_batch_id="before",
|
||||
after_batch_id="after", mapping_id="map" if mapping else "",
|
||||
summary_json={"sheets": [sheet]}))
|
||||
self.db.commit()
|
||||
_persist_sheet_diffs(self.db, run_id="run", metric_id="sheet",
|
||||
diffs=[d for d in result["diffs"]
|
||||
if not sample or d["kind"] != "unchanged"])
|
||||
return result
|
||||
|
||||
def search(self, **kwargs):
|
||||
return list_run_diffs(self.db, "run", metric_id="sheet", **kwargs)
|
||||
|
||||
def test_search_before_value_preserves_changed_verdict(self):
|
||||
self.seed([{"id": "1", "state": "up"}], [{"id": "1", "state": "down"}])
|
||||
with patch("netx_api.biz_state.compare_service.compare_rows") as compare:
|
||||
result = self.search(kw="up", kind="changed")
|
||||
compare.assert_not_called()
|
||||
self.assertEqual(result["total"], 1)
|
||||
self.assertEqual(result["items"][0]["after"]["state"], "down")
|
||||
self.assertEqual(self.search(kw="up", kind="removed")["total"], 0)
|
||||
|
||||
def test_field_filter_matches_after_value_on_compact_diff(self):
|
||||
self.seed([{"id": "1", "state": "up"}], [{"id": "1", "state": "down"}], compact=True)
|
||||
result = self.search(qf={"state": "down"}, kind="changed")
|
||||
self.assertEqual(result["total"], 1)
|
||||
self.assertEqual(result["items"][0]["before"]["state"], "up")
|
||||
|
||||
def test_sample_lookup_completes_pair_matching_only_one_side(self):
|
||||
self.seed([{"id": "1", "state": "up", "age": "old"}],
|
||||
[{"id": "1", "state": "up", "age": "new"}], sample=True)
|
||||
result = self.search(kw="old", kind="unchanged")
|
||||
self.assertEqual(result["source"], "live")
|
||||
self.assertEqual(result["total"], 1)
|
||||
self.assertEqual(result["items"][0]["after"]["age"], "new")
|
||||
|
||||
def test_sample_lookup_does_not_turn_changed_into_removed(self):
|
||||
self.seed([{"id": "1", "state": "up"}, {"id": "2", "state": "up"}],
|
||||
[{"id": "1", "state": "down"}, {"id": "2", "state": "up"}], sample=True)
|
||||
result = self.search(qf={"id": "1", "state": "up"}, kind="all")
|
||||
self.assertEqual([d["kind"] for d in result["items"]], ["changed"])
|
||||
|
||||
def test_sample_lookup_keeps_duplicate_group_pairing(self):
|
||||
self.seed([{"id": "1", "state": "up", "note": "first"},
|
||||
{"id": "1", "state": "down", "note": "second"}],
|
||||
[{"id": "1", "state": "up", "note": "other"},
|
||||
{"id": "1", "state": "down", "note": "other"}], sample=True)
|
||||
result = self.search(kw="second", kind="all")
|
||||
self.assertEqual([d["kind"] for d in result["items"]], ["unchanged"])
|
||||
self.assertEqual(result["items"][0]["after"]["state"], "down")
|
||||
|
||||
def test_sample_search_is_not_limited_by_persisted_search_text_size(self):
|
||||
self.seed([{"id": "1", "state": "up", "note": "x" * 5000 + "needle"}],
|
||||
[{"id": "1", "state": "up", "note": "other"}], sample=True)
|
||||
self.assertEqual(self.search(kw="needle", kind="unchanged")["total"], 1)
|
||||
|
||||
def test_sample_lookup_applies_port_mapping_before_matching(self):
|
||||
self.seed([{"id": "1", "interface": "old", "state": "up"}],
|
||||
[{"id": "1", "interface": "new", "state": "up"}],
|
||||
keys=["id", "interface"], iface=["interface"], mapping={"old": "new"}, sample=True)
|
||||
result = self.search(qf={"interface": "old"}, kind="unchanged")
|
||||
self.assertEqual(result["total"], 1)
|
||||
self.assertEqual(result["items"][0]["after"]["interface"], "new")
|
||||
|
||||
def test_lldp_source_search(self):
|
||||
self.seed([{"local_if": "gei-1", "remote_sys": "peer", "remote_if": "port", "remote_ip": "1"}],
|
||||
[{"local_if": "gei-1", "remote_sys": "peer", "remote_if": "port", "remote_ip": "1"}],
|
||||
metric="lldp_neighbor", keys=["local_if", "remote_sys"], fields=["remote_ip"],
|
||||
iface=["local_if"], sample=True)
|
||||
self.assertEqual(self.search(kw="peer", kind="unchanged")["total"], 1)
|
||||
|
||||
def test_lldp_compact_diff_field_search(self):
|
||||
self.seed([{"local_if": "gei-1", "remote_sys": "peer", "remote_ip": "1"}],
|
||||
[{"local_if": "gei-1", "remote_sys": "peer", "remote_ip": "2"}],
|
||||
metric="lldp_neighbor", keys=["local_if", "remote_sys"], fields=["remote_ip"],
|
||||
iface=["local_if"], compact=True)
|
||||
self.assertEqual(self.search(qf={"remote_ip": "2"}, kind="changed")["total"], 1)
|
||||
|
||||
def test_exact_load_cap_is_not_truncated(self):
|
||||
self.seed([{"id": "1", "state": "up"}], [{"id": "1", "state": "up"}])
|
||||
rows, truncated = _load_metric_rows_for_search(self.db, batch_id="before", metric_id="test",
|
||||
row_filters=[], key_fields=["id"], kw="up", cap=1)
|
||||
self.assertEqual(len(rows), 1)
|
||||
self.assertFalse(truncated)
|
||||
|
||||
def test_incomplete_duplicate_groups_are_omitted(self):
|
||||
self.seed([{"id": "1", "state": "up"}, {"id": "1", "state": "down"}],
|
||||
[{"id": "1", "state": "up"}, {"id": "1", "state": "down"}], sample=True)
|
||||
with patch("netx_api.biz_state.compare_service._LIVE_SEARCH_GROUP_LOAD_CAP", 1):
|
||||
result = self.search(kw="up", kind="all")
|
||||
self.assertEqual(result["items"], [])
|
||||
self.assertTrue(result["truncated"])
|
||||
|
||||
def test_postgres_search_pattern_treats_wildcards_literally(self):
|
||||
self.assertEqual(_literal_search_pattern("a_10%"), "%a\\_10\\%%")
|
||||
|
||||
def test_postgres_unsupported_filter_uses_python(self):
|
||||
self.seed([{"id": "1", "state": "up", "age": "H"}],
|
||||
[{"id": "1", "state": "up", "age": "H"}])
|
||||
with patch("netx_api.biz_state.compare_sql._dialect_is_postgres", return_value=True):
|
||||
rows, _ = _load_metric_rows_for_search(self.db, batch_id="before", metric_id="test",
|
||||
row_filters=[{"field": "age", "op": "age_timer"}], key_fields=["id"], kw="up")
|
||||
self.assertEqual(rows, [])
|
||||
|
||||
def test_unknown_sheet_is_rejected(self):
|
||||
self.seed([{"id": "1", "state": "up"}], [{"id": "1", "state": "up"}])
|
||||
with self.assertRaises(HTTPException) as error:
|
||||
list_run_diffs(self.db, "run", metric_id="missing")
|
||||
self.assertEqual(error.exception.detail, "sheet_not_found")
|
||||
|
||||
def test_stored_search_escapes_like_wildcards(self):
|
||||
self.seed([{"id": "x_1", "state": "up"}, {"id": "xy1", "state": "up"}],
|
||||
[{"id": "x_1", "state": "down"}, {"id": "xy1", "state": "down"}])
|
||||
self.assertEqual(self.search(qf={"id": "x_1"}, kind="changed")["total"], 1)
|
||||
query = _search_stored_diffs(self.db.query(BizCompareDiff), metric_id="test",
|
||||
kw="up", field_q={"id": "x_1"})
|
||||
compiled = query.statement.compile(dialect=postgresql.dialect())
|
||||
self.assertIn("LEFT OUTER JOIN", str(compiled))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
170
tests/test_biz_state_compare_snapshot.py
Normal file
170
tests/test_biz_state_compare_snapshot.py
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
"""Historical comparisons must use the settings captured when they were queued."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import unittest
|
||||
import zipfile
|
||||
from copy import deepcopy
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from netx_api.biz_state.compare_service import (
|
||||
_create_running_run, _execute_compare_into_run, cancel_compare_run,
|
||||
export_run_zip, get_run, list_run_diffs, template_metrics,
|
||||
)
|
||||
from netx_api.db import Base
|
||||
from netx_api.models import (
|
||||
BizCompareDiff, BizCompareJob, BizCompareTemplate, BizPortMapping,
|
||||
BizPortMappingRow, BizStateMetricRow,
|
||||
)
|
||||
|
||||
|
||||
class CompareSnapshotTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine)
|
||||
self.factory = sessionmaker(bind=self.engine, expire_on_commit=False)
|
||||
self.db = self.factory()
|
||||
# Progress uses a separate session in production; keep every test write local.
|
||||
self.session_patch = patch("netx_api.db.SessionLocal", self.factory)
|
||||
self.session_patch.start()
|
||||
self.sheet = {
|
||||
"sheet_id": "sheet", "metric_id": "test", "title": "Original sheet",
|
||||
"key_fields": ["id", "interface"], "iface_fields": ["interface"],
|
||||
"compare_fields": ["state"], "display_fields": ["id", "interface", "state"],
|
||||
"row_filters": [], "ignore_port_changes": False,
|
||||
}
|
||||
self.tpl = BizCompareTemplate(
|
||||
id="tpl", name="Original template", metrics_json=[deepcopy(self.sheet)],
|
||||
iface_normalize_json=[{"from": "GE", "to": "gei"}],
|
||||
)
|
||||
self.job = BizCompareJob(id="job", name="Original job", template_id="tpl",
|
||||
mapping_id="map", store_unchanged="sample")
|
||||
self.mapping = BizPortMapping(id="map", name="Original mapping")
|
||||
self.pair = BizPortMappingRow(mapping_id="map", before_if="gei-1", after_if="gei-2")
|
||||
self.db.add_all([self.tpl, self.job, self.mapping, self.pair])
|
||||
for side, iface, note in (("before", "GE-1.100", "old"), ("after", "GE-2.100", "new")):
|
||||
self.db.add(BizStateMetricRow(id=side, batch_id=side, metric_id="test", seq=0,
|
||||
data_json={"id": "1", "interface": iface,
|
||||
"state": "up", "note": note}))
|
||||
self.db.commit()
|
||||
self.run = _create_running_run(
|
||||
self.db, job=self.job, tpl=self.tpl, before_batch_id="before",
|
||||
after_batch_id="after", sheets_cfg=template_metrics(self.tpl),
|
||||
)
|
||||
|
||||
def tearDown(self):
|
||||
self.session_patch.stop()
|
||||
self.db.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def edit_current_settings(self):
|
||||
changed = {**self.sheet, "title": "Edited sheet", "compare_fields": ["note"],
|
||||
"row_filters": [{"field": "state", "op": "eq", "value": "down"}]}
|
||||
self.tpl.name = "Edited template"
|
||||
self.tpl.metrics_json = [changed]
|
||||
self.tpl.iface_normalize_json = [{"from": "GE", "to": "other"}]
|
||||
self.pair.after_if = "gei-99"
|
||||
self.job.name = "Edited job"
|
||||
self.job.store_unchanged = "never"
|
||||
self.job.enabled_sheet_ids = ["different-sheet"]
|
||||
self.mapping.name = "Edited mapping"
|
||||
self.db.commit()
|
||||
|
||||
def execute(self):
|
||||
return _execute_compare_into_run(self.db, self.run.id)
|
||||
|
||||
def test_queued_run_uses_original_sheet_rules_mapping_and_storage(self):
|
||||
original = deepcopy(self.run.summary_json["config_snapshot"])
|
||||
self.edit_current_settings()
|
||||
detail = self.execute()
|
||||
self.assertEqual(detail["status"], "success")
|
||||
self.assertEqual(detail["summary"]["unchanged"], 1)
|
||||
self.assertEqual(detail["summary"]["unchanged_listed"], 1)
|
||||
self.assertEqual(detail["summary"]["store_unchanged"], "sample")
|
||||
self.assertEqual(detail["sheets"][0]["compare_fields"], ["state"])
|
||||
self.assertEqual(self.run.summary_json["config_snapshot"], original)
|
||||
|
||||
def test_compact_hydration_restores_original_normalized_values(self):
|
||||
self.execute()
|
||||
self.edit_current_settings()
|
||||
diff = list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged")["items"][0]
|
||||
self.assertEqual(diff["before"]["interface"], "GE-1.100")
|
||||
self.assertEqual(diff["mapped_before"]["interface"], "gei-2.100")
|
||||
self.assertEqual(diff["after"]["interface"], "gei-2.100")
|
||||
|
||||
def test_live_search_does_not_inherit_new_filters_or_transforms(self):
|
||||
with patch("netx_api.biz_state.compare_service._UNCHANGED_SAMPLE_MAX", 0):
|
||||
self.execute()
|
||||
self.edit_current_settings()
|
||||
result = list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged", kw="old")
|
||||
self.assertEqual(result["source"], "live")
|
||||
self.assertEqual(result["total"], 1)
|
||||
self.assertEqual(result["items"][0]["mapped_before"]["interface"], "gei-2.100")
|
||||
|
||||
def test_snapshot_can_be_read_after_mapping_and_template_are_deleted(self):
|
||||
self.execute()
|
||||
self.db.query(BizPortMappingRow).delete()
|
||||
self.db.delete(self.mapping)
|
||||
self.db.delete(self.tpl)
|
||||
self.db.commit()
|
||||
detail = get_run(self.db, self.run.id)
|
||||
self.assertEqual(detail["template_name"], "Original template")
|
||||
self.assertEqual(detail["mapping_name"], "Original mapping")
|
||||
diff = list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged")["items"][0]
|
||||
self.assertEqual(diff["mapped_before"]["interface"], "gei-2.100")
|
||||
|
||||
def test_queued_run_can_execute_after_its_template_and_mapping_are_deleted(self):
|
||||
self.db.query(BizPortMappingRow).delete()
|
||||
self.db.delete(self.mapping)
|
||||
self.db.delete(self.tpl)
|
||||
self.db.commit()
|
||||
detail = self.execute()
|
||||
self.assertEqual(detail["status"], "success")
|
||||
self.assertEqual(detail["summary"]["unchanged"], 1)
|
||||
|
||||
def test_detail_returns_original_names_and_independent_template_copy(self):
|
||||
self.edit_current_settings()
|
||||
detail = get_run(self.db, self.run.id)
|
||||
self.assertEqual(detail["job_name"], "Original job")
|
||||
self.assertEqual(detail["config_snapshot_version"], 1)
|
||||
self.assertEqual(detail["template"]["iface_normalize_rules"], [{"from": "GE", "to": "gei"}])
|
||||
self.assertNotIn("config_snapshot", detail["summary"])
|
||||
detail["template"]["iface_normalize_rules"].clear()
|
||||
self.assertTrue(self.run.summary_json["config_snapshot"]["template"]["iface_normalize_rules"])
|
||||
|
||||
def test_export_contains_original_config_and_hydrated_csv(self):
|
||||
self.execute()
|
||||
self.edit_current_settings()
|
||||
with zipfile.ZipFile(io.BytesIO(export_run_zip(self.db, self.run.id))) as archive:
|
||||
config = json.loads(archive.read("config_snapshot.json"))
|
||||
self.assertEqual(config["port_map"], {"gei-1": "gei-2"})
|
||||
self.assertEqual(config["template"]["name"], "Original template")
|
||||
rows = list(csv.DictReader(io.StringIO(archive.read("tables/sheet.csv").decode("utf-8-sig"))))
|
||||
self.assertEqual(rows[0]["interface"], "gei-2.100")
|
||||
|
||||
def test_failed_and_cancelled_runs_keep_snapshot(self):
|
||||
original = deepcopy(self.run.summary_json["config_snapshot"])
|
||||
with patch("netx_api.biz_state.compare_service._run_sheet", side_effect=RuntimeError("test-failure")):
|
||||
with self.assertRaisesRegex(RuntimeError, "test-failure"):
|
||||
self.execute()
|
||||
self.assertEqual(self.run.summary_json["config_snapshot"], original)
|
||||
self.run.status = "running"
|
||||
self.db.commit()
|
||||
cancel_compare_run(self.db, self.run.id)
|
||||
self.assertEqual(self.run.summary_json["config_snapshot"], original)
|
||||
|
||||
def test_legacy_runs_without_snapshot_remain_readable(self):
|
||||
self.execute()
|
||||
self.run.summary_json = {k: v for k, v in self.run.summary_json.items() if k != "config_snapshot"}
|
||||
self.db.commit()
|
||||
detail = get_run(self.db, self.run.id)
|
||||
self.assertEqual(detail["status"], "success")
|
||||
self.assertIsNone(detail["config_snapshot_version"])
|
||||
self.assertEqual(list_run_diffs(self.db, self.run.id, metric_id="sheet", kind="unchanged")["total"], 1)
|
||||
self.assertEqual(self.db.query(BizCompareDiff).count(), 1)
|
||||
|
|
@ -474,6 +474,10 @@ const en = {
|
|||
sideAfter: "After",
|
||||
sidesVs: "vs",
|
||||
sidesTitle: "Compare sides",
|
||||
configSnapshot: "Execution settings saved",
|
||||
configSnapshotHint: "Template rules and port mapping are fixed for this run. Later edits do not affect historical lookup. The downloaded report includes the execution settings.",
|
||||
configLegacy: "Legacy run: settings not saved",
|
||||
configLegacyHint: "This run has no configuration snapshot. Success lookup and compact row reconstruction use the current settings, which may differ from the original execution.",
|
||||
pick: "Select…",
|
||||
sameAsBefore: "Same as before / select…",
|
||||
autoHint: "auto: new after-task batches trigger compare",
|
||||
|
|
@ -532,8 +536,14 @@ const en = {
|
|||
ranWithDuration: "Compare finished ({{s}}s)",
|
||||
unchangedNotStored: "Success rows were counted but not stored. Set “Store success rows” to sample and re-run for spot-check.",
|
||||
unchangedSampleHint: "{{total}} success rows total; browsing a stratified sample of {{listed}}. Search any route for live source lookup. Pass rate uses all {{total}}.",
|
||||
liveSearchHint: "Field filters look up source batches live; tabs only filter kind. E.g. direction=out, network=1.1.1.1.",
|
||||
liveSearchTruncatedHint: "Search results truncated — narrow the field filters.",
|
||||
liveSearchHint: "Pairs are completed from source batches before filtering. Pass rate still uses the full sheet.",
|
||||
liveSearchTruncatedHint: "Search limit reached; some entries are omitted. Narrow the filters. Incomplete key groups were skipped.",
|
||||
clearSearch: "Clear filters",
|
||||
resultLoading: "Loading comparison details…",
|
||||
resultRange: "{{from}}–{{to}} of {{total}}",
|
||||
resultLoadFailed: "Could not load comparison details",
|
||||
retry: "Retry",
|
||||
searchEmpty: "No results match these filters. Clear filters or select another result type.",
|
||||
clearKeyFilters: "Clear fields",
|
||||
storeUnchanged: "Store success rows",
|
||||
storeUnchangedAuto: "Auto (full if small / sample 5k + hydrate if large)",
|
||||
|
|
@ -567,8 +577,6 @@ const en = {
|
|||
pickBatchRun: "Select compare run",
|
||||
pickRun: "Select run…",
|
||||
runCount: "{{n}} runs",
|
||||
resultEmpty: "No matching diff rows",
|
||||
resultFilterPh: "Free text, or direction:out network:1.1.1.1",
|
||||
kindAll: "All",
|
||||
kindDiff: "Fail",
|
||||
kindAdded: "Added",
|
||||
|
|
@ -712,7 +720,6 @@ const en = {
|
|||
newBaseline: "New baseline batch",
|
||||
needProjectFields: "Enter a name and pick old/new portrait tasks",
|
||||
needExpectPorts: "Select at least one expected object for this batch",
|
||||
needBaselineFirst: "Save an old baseline batch first to list expect objects",
|
||||
needBothBaselines: "Save both old and new baseline batches before evaluating",
|
||||
errOldBaselineRequired: "Old baseline batch is required",
|
||||
errNewBaselineRequired: "New baseline batch is required",
|
||||
|
|
|
|||
|
|
@ -473,6 +473,10 @@ const zh = {
|
|||
sideAfter: "操作后",
|
||||
sidesVs: "比对",
|
||||
sidesTitle: "比对双方",
|
||||
configSnapshot: "执行配置已保存",
|
||||
configSnapshotHint: "此记录的模板规则和端口映射已固定;后续编辑不影响历史回查,下载报告包含执行配置。",
|
||||
configLegacy: "旧记录:未保存执行配置",
|
||||
configLegacyHint: "此记录创建时未保存配置快照;成功行回查和精简明细还原使用当前配置,修改配置后可能与当时结果不同。",
|
||||
pick: "选择…",
|
||||
sameAsBefore: "同操作前 / 选择…",
|
||||
autoHint: "auto:操作后任务新批次到达会自动跑比对",
|
||||
|
|
@ -531,8 +535,14 @@ const zh = {
|
|||
ranWithDuration: "比对完成(耗时 {{s}} 秒)",
|
||||
unchangedNotStored: "成功行仅统计数量未落库。可在任务配置将「成功行保存」改为抽样后重新比对(抽查用)。",
|
||||
unchangedSampleHint: "成功共 {{total}} 条,明细抽样 {{listed}} 条(分层抽查)。要查任意路由请输入筛选条件——将按原表即时判定。通过率按全部 {{total}} 计。",
|
||||
liveSearchHint: "按字段回查原表即时判定;页签只过滤种类。例:direction=out,network=1.1.1.1。",
|
||||
liveSearchTruncatedHint: "搜索结果已截断,请再收窄字段条件。",
|
||||
liveSearchHint: "从原始批次补齐同一业务的前后数据后筛选;通过率仍按全表统计。",
|
||||
liveSearchTruncatedHint: "搜索已达上限,部分业务未展示。请收窄条件;不完整的业务分组已跳过。",
|
||||
clearSearch: "清空筛选",
|
||||
resultLoading: "正在加载对比明细…",
|
||||
resultRange: "{{from}}–{{to}} / {{total}} 条",
|
||||
resultLoadFailed: "对比明细加载失败",
|
||||
retry: "重试",
|
||||
searchEmpty: "当前筛选条件下没有匹配结果,可清空筛选或切换结果类型。",
|
||||
clearKeyFilters: "清空字段",
|
||||
storeUnchanged: "成功行保存",
|
||||
storeUnchangedAuto: "自动(小表全量 / 大表抽样 5000+原表补全)",
|
||||
|
|
@ -778,7 +788,6 @@ const zh = {
|
|||
metaBaseline: "基线",
|
||||
baselineOk: "已设",
|
||||
baselineMissing: "缺",
|
||||
needBaselineFirst: "先设基线",
|
||||
enableHighfreq: "开启高频",
|
||||
highfreqReady: "高频已就绪",
|
||||
collectNow: "立即采集",
|
||||
|
|
@ -830,10 +839,6 @@ const zh = {
|
|||
currentMissingBanner: "缺当前批次:{{metrics}}",
|
||||
highfreqReadyInactive: "高频已绑定(窗口未开,未立即采集)",
|
||||
collectPartial: "部分失败(成功 {{ok}} / 失败 {{fail}})",
|
||||
confirmProjectDone: "确认完成项目并暂停全部高频?",
|
||||
projectMarkedDone: "已完成,高频已停",
|
||||
confirmDeleteProject: "确认删除项目?将停高频并清批次记录。",
|
||||
projectDeleted: "已删除",
|
||||
errMetricNeedsBindings: "监控项「{{metric}}」需要参数绑定({{params}}),割接高频暂不支持,请去掉该项",
|
||||
errNoProfileForMetric: "监控项「{{metric}}」在当前设备厂商下无可用采集 profile",
|
||||
acceptTitle: "本批验收",
|
||||
|
|
|
|||
|
|
@ -11931,6 +11931,22 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
opacity: 0.75;
|
||||
}
|
||||
|
||||
.bs-cmp-board__config {
|
||||
flex: 0 1 auto;
|
||||
min-width: 0;
|
||||
padding: 4px 7px;
|
||||
border: 1px solid var(--bs-cmp-border, rgba(148, 163, 184, 0.22));
|
||||
border-radius: 6px;
|
||||
color: var(--nm-muted, #94a3b8);
|
||||
font-size: 11px;
|
||||
line-height: 1.4;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
.bs-cmp-board__config.is-legacy {
|
||||
color: #fbbf24;
|
||||
}
|
||||
|
||||
.bs-cmp-board__actions {
|
||||
margin-left: auto;
|
||||
flex: 0 0 auto;
|
||||
|
|
@ -12497,7 +12513,7 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
flex: 1 1 280px;
|
||||
flex: 1 1 100%;
|
||||
min-width: 0;
|
||||
padding: 2px;
|
||||
border-radius: 8px;
|
||||
|
|
@ -12615,13 +12631,13 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
|
||||
.bs-cmp-filter-bar {
|
||||
display: flex;
|
||||
flex-wrap: nowrap;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
flex: 1 1 200px;
|
||||
min-width: 160px;
|
||||
max-width: 320px;
|
||||
margin-left: auto;
|
||||
flex: 1 1 100%;
|
||||
min-width: 0;
|
||||
max-width: none;
|
||||
margin-left: 0;
|
||||
min-height: 32px;
|
||||
}
|
||||
|
||||
|
|
@ -12630,6 +12646,8 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
.bs-cmp-filter-bar input {
|
||||
min-width: 0;
|
||||
width: 100%;
|
||||
flex: 1 1 220px;
|
||||
max-width: 440px;
|
||||
}
|
||||
|
||||
.bs-cmp-filter-bar .bs-sheet-count {
|
||||
|
|
@ -12717,11 +12735,57 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
}
|
||||
|
||||
.bs-cmp-result-table.is-loading {
|
||||
opacity: 0.55;
|
||||
opacity: 1;
|
||||
/* Keep header/filters clickable; only mute row hit-testing via opacity cue */
|
||||
pointer-events: auto;
|
||||
}
|
||||
|
||||
.bs-cmp-result-state {
|
||||
min-height: 180px;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 12px;
|
||||
padding: 24px;
|
||||
text-align: center;
|
||||
white-space: normal;
|
||||
overflow-wrap: anywhere;
|
||||
color: var(--bs-cmp-muted);
|
||||
}
|
||||
|
||||
.bs-cmp-result-state strong {
|
||||
color: var(--bs-cmp-ink);
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.bs-cmp-result-state.is-error strong {
|
||||
color: var(--danger, #fca5a5);
|
||||
}
|
||||
|
||||
.bs-cmp-strip__kind:focus-visible,
|
||||
.bs-cmp-strip__toggle:focus-visible,
|
||||
.bs-cmp-nav__item:focus-visible {
|
||||
outline: 2px solid var(--accent, #60a5fa);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
@media (max-width: 600px) {
|
||||
.bs-cmp-strip__sheet {
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.bs-cmp-filter-bar input {
|
||||
flex-basis: 100%;
|
||||
max-width: none;
|
||||
}
|
||||
|
||||
.bs-cmp-key-filter {
|
||||
flex: 1 1 120px;
|
||||
max-width: none;
|
||||
}
|
||||
}
|
||||
|
||||
.bs-cmp-board.is-fullscreen .bs-cmp-result-table {
|
||||
max-height: none;
|
||||
min-height: 0;
|
||||
|
|
@ -12752,6 +12816,7 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
@media (max-width: 960px) {
|
||||
.bs-cmp-board__body {
|
||||
grid-template-columns: 1fr;
|
||||
grid-template-rows: auto minmax(0, 1fr);
|
||||
}
|
||||
|
||||
.bs-cmp-board__body.is-nav-collapsed {
|
||||
|
|
@ -12760,6 +12825,7 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
|
||||
.bs-cmp-nav {
|
||||
max-height: 180px;
|
||||
align-self: start;
|
||||
}
|
||||
|
||||
.bs-cmp-nav.is-collapsed {
|
||||
|
|
@ -12769,6 +12835,8 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
.bs-cmp-nav__list {
|
||||
flex-direction: row;
|
||||
flex-wrap: nowrap;
|
||||
flex: 0 0 auto;
|
||||
align-items: flex-start;
|
||||
overflow-x: auto;
|
||||
overflow-y: hidden;
|
||||
}
|
||||
|
|
@ -12779,8 +12847,9 @@ body:has(.bs-cmp-board:fullscreen) {
|
|||
|
||||
.bs-cmp-nav__item {
|
||||
flex: 0 0 auto;
|
||||
min-width: 180px;
|
||||
grid-template-columns: 8px minmax(0, 1fr);
|
||||
width: auto;
|
||||
min-width: 240px;
|
||||
grid-template-columns: 8px minmax(0, 1fr) auto;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -845,7 +845,7 @@ function TplRowFiltersEditor({
|
|||
Array.isArray(filt.value) ? filt.value.join(",") : String(filt.value ?? "")
|
||||
}
|
||||
placeholder={t("bizCompare.filterValue")}
|
||||
isDisabled={["empty", "not_empty", "age_timer"].includes(filt.op || "")}
|
||||
disabled={["empty", "not_empty", "age_timer"].includes(filt.op || "")}
|
||||
onChange={(e) => {
|
||||
const next = groups.map((g) => g.map((x) => ({ ...x })));
|
||||
const op = next[gi][fi]?.op || "eq";
|
||||
|
|
@ -893,10 +893,6 @@ function TplRowFiltersEditor({
|
|||
);
|
||||
}
|
||||
|
||||
function metricLabel(id: string) {
|
||||
return id;
|
||||
}
|
||||
|
||||
export type BizComparePageMode = "jobs" | "templates" | "all";
|
||||
|
||||
export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePageMode }) {
|
||||
|
|
@ -988,6 +984,12 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const [diffsLoading, setDiffsLoading] = useState(false);
|
||||
const [diffsSource, setDiffsSource] = useState<DiffListSource>("");
|
||||
const [diffsTruncated, setDiffsTruncated] = useState(false);
|
||||
const [diffsError, setDiffsError] = useState("");
|
||||
const [diffsRetry, setDiffsRetry] = useState(0);
|
||||
const jobRequestRef = useRef(0);
|
||||
const runRequestRef = useRef(0);
|
||||
const listsRequestRef = useRef(0);
|
||||
const diffFiltersRef = useRef("");
|
||||
const boardRef = useRef<HTMLDivElement | null>(null);
|
||||
const tableScrollRef = useRef<HTMLDivElement | null>(null);
|
||||
const tableScrollPosRef = useRef({ top: 0, left: 0 });
|
||||
|
|
@ -996,6 +998,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const tplImportRef = useRef<HTMLInputElement | null>(null);
|
||||
|
||||
const refresh = useCallback(async (opts?: { force?: boolean }) => {
|
||||
const request = ++listsRequestRef.current;
|
||||
type Bundle = {
|
||||
taskRes: Awaited<ReturnType<typeof bizStateListTasks>>;
|
||||
tpl: Awaited<ReturnType<typeof bizCompareListTemplates>>;
|
||||
|
|
@ -1014,6 +1017,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
return { taskRes, tpl, maps, j, met };
|
||||
};
|
||||
const apply = (b: Bundle) => {
|
||||
if (request !== listsRequestRef.current) return;
|
||||
setTasks((b.taskRes.items || []) as TaskOpt[]);
|
||||
setTemplates((b.tpl.items || []) as Template[]);
|
||||
setMappings((b.maps.items || []) as Mapping[]);
|
||||
|
|
@ -1038,8 +1042,12 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [refresh]);
|
||||
|
||||
useEffect(() => () => { listsRequestRef.current += 1; }, []);
|
||||
|
||||
// Keep local tab + dismiss overlays when route mode flips (component may be reused).
|
||||
useEffect(() => {
|
||||
jobRequestRef.current += 1;
|
||||
runRequestRef.current += 1;
|
||||
if (pageMode === "templates") setPageTab("templates");
|
||||
else if (pageMode === "jobs") setPageTab("jobs");
|
||||
setTplOpen(false);
|
||||
|
|
@ -1053,39 +1061,55 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
void document.exitFullscreen().catch(() => undefined);
|
||||
}
|
||||
setBoardFs(false);
|
||||
return () => {
|
||||
jobRequestRef.current += 1;
|
||||
runRequestRef.current += 1;
|
||||
};
|
||||
}, [pageMode]);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
setBeforeBatches([]);
|
||||
void (async () => {
|
||||
if (!beforeTaskId) {
|
||||
setBeforeBatches([]);
|
||||
return;
|
||||
if (!beforeTaskId) return;
|
||||
try {
|
||||
const b = await cutoverCachedGet(`bizCompare:batches:${beforeTaskId}`,
|
||||
() => bizStateListBatches(beforeTaskId), { ttlMs: 5000 });
|
||||
if (!cancelled) setBeforeBatches((b.items || []) as BatchOpt[]);
|
||||
} catch (e) {
|
||||
if (!cancelled) showError(formatErr(e));
|
||||
}
|
||||
const b = await bizStateListBatches(beforeTaskId);
|
||||
setBeforeBatches((b.items || []) as BatchOpt[]);
|
||||
})();
|
||||
}, [beforeTaskId]);
|
||||
return () => { cancelled = true; };
|
||||
}, [beforeTaskId, showError]);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
setAfterBatches([]);
|
||||
void (async () => {
|
||||
if (!afterTaskId) {
|
||||
setAfterBatches([]);
|
||||
return;
|
||||
if (!afterTaskId) return;
|
||||
try {
|
||||
const b = await cutoverCachedGet(`bizCompare:batches:${afterTaskId}`,
|
||||
() => bizStateListBatches(afterTaskId), { ttlMs: 5000 });
|
||||
if (!cancelled) setAfterBatches((b.items || []) as BatchOpt[]);
|
||||
} catch (e) {
|
||||
if (!cancelled) showError(formatErr(e));
|
||||
}
|
||||
const b = await bizStateListBatches(afterTaskId);
|
||||
setAfterBatches((b.items || []) as BatchOpt[]);
|
||||
})();
|
||||
}, [afterTaskId]);
|
||||
return () => { cancelled = true; };
|
||||
}, [afterTaskId, showError]);
|
||||
|
||||
const templatesById = useMemo(() => new Map(templates.map((x) => [x.id, x])), [templates]);
|
||||
|
||||
const filteredJobs = useMemo(() => {
|
||||
const kw = debouncedListKw.trim().toLowerCase();
|
||||
if (!kw) return jobs;
|
||||
return jobs.filter((j) => {
|
||||
const tpl = templates.find((x) => x.id === j.template_id);
|
||||
const tpl = templatesById.get(j.template_id);
|
||||
const mids = (tpl?.metric_ids || templateSheets(tpl).map((s) => s.metric_id)).join(" ");
|
||||
return `${j.name} ${j.mode} ${j.status} ${tpl?.name || ""} ${mids}`.toLowerCase().includes(kw);
|
||||
});
|
||||
}, [jobs, templates, debouncedListKw]);
|
||||
}, [jobs, templatesById, debouncedListKw]);
|
||||
|
||||
const filteredTemplates = useMemo(() => {
|
||||
const kw = debouncedListKw.trim().toLowerCase();
|
||||
|
|
@ -1135,10 +1159,14 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const next = currentlyOn.includes(sid)
|
||||
? currentlyOn.filter((id) => id !== sid)
|
||||
: [...currentlyOn, sid];
|
||||
if (!next.length) {
|
||||
showError(t("bizCompare.needSheets"));
|
||||
return;
|
||||
}
|
||||
// Empty list means "all on" (new template sheets auto-included)
|
||||
setEnabledSheetIds(next.length === all.length ? [] : next);
|
||||
},
|
||||
[jobSheetAllIds, enabledSheetIds],
|
||||
[jobSheetAllIds, enabledSheetIds, showError, t],
|
||||
);
|
||||
|
||||
const setJobTemplateAndSheets = useCallback(
|
||||
|
|
@ -1195,28 +1223,18 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
() => runSheets.find((s) => sheetIdentity(s) === resultSheetId) || runSheets[0] || null,
|
||||
[runSheets, resultSheetId],
|
||||
);
|
||||
const activeRunSheetId = sheetIdentity(activeRunSheet);
|
||||
const activeRunSheetStatus = String(activeRunSheet?.status || "");
|
||||
|
||||
// Clear field filters when switching sheet
|
||||
useEffect(() => {
|
||||
setResultKeyFilters({});
|
||||
}, [resultSheetId]);
|
||||
|
||||
// Reset page when sheet / filter / page size changes
|
||||
useEffect(() => {
|
||||
setResultPage(1);
|
||||
}, [
|
||||
resultSheetId,
|
||||
kindFilter,
|
||||
debouncedResultKw,
|
||||
debouncedKeyFiltersJson,
|
||||
resultPageSize,
|
||||
runDetail?.id,
|
||||
]);
|
||||
|
||||
useEffect(() => {
|
||||
const runId = String(runDetail?.id || "");
|
||||
const mid = resultSheetId || sheetIdentity(activeRunSheet) || "";
|
||||
const sheetSt = String(activeRunSheet?.status || "");
|
||||
const mid = resultSheetId || activeRunSheetId || "";
|
||||
const sheetSt = activeRunSheetStatus;
|
||||
// Block only while *this* sheet is still in flight — done sheets are readable mid-run
|
||||
const sheetStillRunning = ["pending", "running", "queued"].includes(sheetSt);
|
||||
if (!runId || !mid || jobDetailTab !== "result" || sheetStillRunning) {
|
||||
|
|
@ -1224,11 +1242,30 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
setResultTotal(0);
|
||||
setDiffsSource("");
|
||||
setDiffsTruncated(false);
|
||||
setDiffsLoading(false);
|
||||
setDiffsError("");
|
||||
return;
|
||||
}
|
||||
const filterKey = JSON.stringify([runId, mid, kindFilter, debouncedResultKw,
|
||||
debouncedKeyFiltersJson, resultPageSize]);
|
||||
if (diffFiltersRef.current !== filterKey) {
|
||||
diffFiltersRef.current = filterKey;
|
||||
if (resultPage !== 1) {
|
||||
setPagedDiffs([]);
|
||||
setResultTotal(0);
|
||||
setDiffsLoading(true);
|
||||
setResultPage(1);
|
||||
return;
|
||||
}
|
||||
}
|
||||
let cancelled = false;
|
||||
void (async () => {
|
||||
setDiffsLoading(true);
|
||||
setDiffsError("");
|
||||
setPagedDiffs([]);
|
||||
setResultTotal(0);
|
||||
setDiffsSource("");
|
||||
setDiffsTruncated(false);
|
||||
try {
|
||||
const res = await bizCompareListRunDiffs({
|
||||
runId,
|
||||
|
|
@ -1253,8 +1290,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
if (resultPage > pages) setResultPage(pages);
|
||||
} catch (e) {
|
||||
if (!cancelled) {
|
||||
// Keep previous rows to avoid strip/table jump; only clear on hard empty run
|
||||
showError(formatErr(e));
|
||||
setDiffsError(formatErr(e));
|
||||
}
|
||||
} finally {
|
||||
if (!cancelled) setDiffsLoading(false);
|
||||
|
|
@ -1267,15 +1303,16 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
runDetail?.id,
|
||||
runDetail?.status,
|
||||
resultSheetId,
|
||||
activeRunSheet?.metric_id,
|
||||
activeRunSheet?.status,
|
||||
activeRunSheetId,
|
||||
activeRunSheetStatus,
|
||||
kindFilter,
|
||||
debouncedResultKw,
|
||||
debouncedKeyFiltersJson,
|
||||
debouncedKeyFilters,
|
||||
resultPage,
|
||||
resultPageSize,
|
||||
jobDetailTab,
|
||||
showError,
|
||||
diffsRetry,
|
||||
]);
|
||||
|
||||
const resultColumns = useMemo(() => {
|
||||
|
|
@ -1991,9 +2028,12 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
};
|
||||
|
||||
const openJob = async (id: string) => {
|
||||
const request = ++jobRequestRef.current;
|
||||
runRequestRef.current += 1;
|
||||
setJobId(id);
|
||||
setJobDetailTab("config");
|
||||
setRunDetail(null);
|
||||
setRuns([]);
|
||||
setResultSheetId("");
|
||||
setKindFilter("diff");
|
||||
setResultKw("");
|
||||
|
|
@ -2001,20 +2041,24 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
if (job) resetJobForm(job);
|
||||
try {
|
||||
const r = await bizCompareListRuns(id, 50);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
const items = r.items || [];
|
||||
setRuns(items);
|
||||
if (items.length) {
|
||||
const latest = await bizCompareGetRun(String((items as any[])[0].id));
|
||||
if (request !== jobRequestRef.current) return;
|
||||
setRunDetail(latest);
|
||||
// Land on batch list so stuck/cancelled runs are visible and actionable
|
||||
setJobDetailTab("runs");
|
||||
}
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
if (request === jobRequestRef.current) showError(formatErr(e));
|
||||
}
|
||||
};
|
||||
|
||||
const closeJob = () => {
|
||||
jobRequestRef.current += 1;
|
||||
runRequestRef.current += 1;
|
||||
if (document.fullscreenElement === boardRef.current) {
|
||||
void document.exitFullscreen().catch(() => undefined);
|
||||
}
|
||||
|
|
@ -2061,20 +2105,26 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
setJobDetailTab("runs");
|
||||
return;
|
||||
}
|
||||
const request = jobRequestRef.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
await bizCompareUpdateJob(jobId, jobConfigBody());
|
||||
// Async enqueue — returns immediately with status=running; poll below.
|
||||
const run = await bizCompareRunJob(jobId);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
runRequestRef.current += 1;
|
||||
setRunDetail(run);
|
||||
setJobDetailTab("runs");
|
||||
showOk(t("bizCompare.runStarted"));
|
||||
const r = await bizCompareListRuns(jobId, 50);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
setRuns(r.items || []);
|
||||
await refresh({ force: true });
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
setJobDetailTab("runs");
|
||||
if (request === jobRequestRef.current) {
|
||||
showError(formatErr(e));
|
||||
setJobDetailTab("runs");
|
||||
}
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
|
|
@ -2082,20 +2132,25 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const runProgress = (runDetail?.summary?.progress || {}) as RunProgressInfo;
|
||||
const runEngine = String(runProgress.engine || "").toLowerCase();
|
||||
|
||||
// Poll active compare runs so the modal can be closed and reopened safely.
|
||||
// One poller; schedule the next tick after requests settle to prevent overlap.
|
||||
useEffect(() => {
|
||||
if (!runIsActive || !runDetail?.id) return;
|
||||
if (!jobId || (!runIsActive && !(jobHasActiveRun && jobDetailTab === "runs"))) return;
|
||||
let cancelled = false;
|
||||
let notified = false;
|
||||
let timer: number | undefined;
|
||||
const jobRequest = jobRequestRef.current;
|
||||
const runId = String(runDetail?.id || "");
|
||||
const tick = async () => {
|
||||
const runRequest = runRequestRef.current;
|
||||
try {
|
||||
const d = await bizCompareGetRun(String(runDetail.id));
|
||||
if (cancelled) return;
|
||||
setRunDetail(d);
|
||||
if (jobId) {
|
||||
const r = await bizCompareListRuns(jobId);
|
||||
if (!cancelled) setRuns(r.items || []);
|
||||
}
|
||||
const [r, d] = await Promise.all([
|
||||
bizCompareListRuns(jobId, 50),
|
||||
runIsActive && runId ? bizCompareGetRun(runId) : Promise.resolve(null),
|
||||
]);
|
||||
if (cancelled || jobRequest !== jobRequestRef.current) return;
|
||||
setRuns(r.items || []);
|
||||
if (!d || runRequest !== runRequestRef.current) return;
|
||||
setRunDetail((prev: typeof runDetail) => String(prev?.id || "") === runId ? d : prev);
|
||||
const st = String(d.status || "");
|
||||
if (!notified && st === "success") {
|
||||
notified = true;
|
||||
|
|
@ -2111,62 +2166,45 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
}
|
||||
} catch (e) {
|
||||
if (!cancelled) showError(formatErr(e));
|
||||
} finally {
|
||||
if (!cancelled) timer = window.setTimeout(() => void tick(), 2000);
|
||||
}
|
||||
};
|
||||
void tick();
|
||||
const id = window.setInterval(() => void tick(), 2000);
|
||||
return () => {
|
||||
cancelled = true;
|
||||
window.clearInterval(id);
|
||||
window.clearTimeout(timer);
|
||||
};
|
||||
}, [runDetail?.id, runIsActive, jobId, showOk, showError, t]);
|
||||
|
||||
// Keep the runs list fresh while any batch on this job is active
|
||||
useEffect(() => {
|
||||
if (!jobId || !jobHasActiveRun || jobDetailTab !== "runs") return;
|
||||
let cancelled = false;
|
||||
const tick = async () => {
|
||||
try {
|
||||
const r = await bizCompareListRuns(jobId, 50);
|
||||
if (!cancelled) setRuns(r.items || []);
|
||||
const curId = String(runDetail?.id || "");
|
||||
if (curId) {
|
||||
const d = await bizCompareGetRun(curId);
|
||||
if (!cancelled) setRunDetail(d);
|
||||
}
|
||||
} catch {
|
||||
/* ignore list poll errors */
|
||||
}
|
||||
};
|
||||
void tick();
|
||||
const id = window.setInterval(() => void tick(), 2500);
|
||||
return () => {
|
||||
cancelled = true;
|
||||
window.clearInterval(id);
|
||||
};
|
||||
}, [jobId, jobHasActiveRun, jobDetailTab, runDetail?.id]);
|
||||
}, [runDetail?.id, runIsActive, jobId, jobHasActiveRun, jobDetailTab, showOk, showError, t]);
|
||||
|
||||
const loadRun = async (runId: string) => {
|
||||
const request = ++runRequestRef.current;
|
||||
const jobRequest = jobRequestRef.current;
|
||||
try {
|
||||
const d = await bizCompareGetRun(runId);
|
||||
if (request !== runRequestRef.current || jobRequest !== jobRequestRef.current) return;
|
||||
setRunDetail(d);
|
||||
setKindFilter("diff");
|
||||
setResultKw("");
|
||||
setResultKeyFilters({});
|
||||
setJobDetailTab("result");
|
||||
} catch (e) {
|
||||
showError(formatErr(e));
|
||||
if (request === runRequestRef.current && jobRequest === jobRequestRef.current) showError(formatErr(e));
|
||||
}
|
||||
};
|
||||
|
||||
const cancelRun = async (runId: string) => {
|
||||
if (!runId) return;
|
||||
if (!window.confirm(t("bizCompare.confirmCancelRun"))) return;
|
||||
const request = jobRequestRef.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
const d = await bizCompareCancelRun(runId);
|
||||
if (String(runDetail?.id || "") === runId) setRunDetail(d);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
setRunDetail((prev: typeof runDetail) => String(prev?.id || "") === runId ? d : prev);
|
||||
if (jobId) {
|
||||
const r = await bizCompareListRuns(jobId, 50);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
setRuns(r.items || []);
|
||||
}
|
||||
showOk(t("bizCompare.runCancelled"));
|
||||
|
|
@ -2180,19 +2218,24 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
const removeRun = async (runId: string) => {
|
||||
if (!runId) return;
|
||||
if (!window.confirm(t("bizCompare.confirmDeleteRun"))) return;
|
||||
const request = jobRequestRef.current;
|
||||
const runRequest = runRequestRef.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
const wasCurrent = String(runDetail?.id || "") === runId;
|
||||
await bizCompareDeleteRun(runId);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
let nextRuns: typeof runs = [];
|
||||
if (jobId) {
|
||||
const r = await bizCompareListRuns(jobId, 50);
|
||||
if (request !== jobRequestRef.current) return;
|
||||
nextRuns = r.items || [];
|
||||
} else {
|
||||
nextRuns = (runs || []).filter((r) => String(r.id) !== runId);
|
||||
}
|
||||
setRuns(nextRuns);
|
||||
if (wasCurrent) {
|
||||
if (wasCurrent && runRequest === runRequestRef.current) {
|
||||
runRequestRef.current += 1;
|
||||
setRunDetail(null);
|
||||
setResultSheetId("");
|
||||
setJobDetailTab("runs");
|
||||
|
|
@ -2542,7 +2585,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
</thead>
|
||||
<tbody>
|
||||
{filteredJobs.map((j) => {
|
||||
const tpl = templates.find((x) => x.id === j.template_id);
|
||||
const tpl = templatesById.get(j.template_id);
|
||||
const n = templateSheets(tpl).length;
|
||||
return (
|
||||
<tr key={j.id}>
|
||||
|
|
@ -2882,7 +2925,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
: ""
|
||||
}
|
||||
placeholder={cmpMode === "percent" ? "%" : ""}
|
||||
isDisabled={!isCompare || !needsTol}
|
||||
disabled={!isCompare || !needsTol}
|
||||
onChange={(e) => {
|
||||
const raw = e.target.value.trim();
|
||||
const tol = raw === "" ? undefined : Number(raw);
|
||||
|
|
@ -3113,7 +3156,12 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
className={jobDetailTab === "runs" ? "is-active" : undefined}
|
||||
onPress={() => {
|
||||
setJobDetailTab("runs");
|
||||
if (jobId) void bizCompareListRuns(jobId, 50).then((r) => setRuns(r.items || []));
|
||||
const request = jobRequestRef.current;
|
||||
if (jobId) void bizCompareListRuns(jobId, 50).then((r) => {
|
||||
if (request === jobRequestRef.current) setRuns(r.items || []);
|
||||
}).catch((e) => {
|
||||
if (request === jobRequestRef.current) showError(formatErr(e));
|
||||
});
|
||||
}}
|
||||
>
|
||||
{t("bizCompare.tabRuns")}
|
||||
|
|
@ -3354,6 +3402,18 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
})()}
|
||||
</div>
|
||||
) : null}
|
||||
{runDetail ? (
|
||||
<span
|
||||
className={`bs-cmp-board__config${runDetail.config_snapshot_version ? "" : " is-legacy"}`}
|
||||
title={t(runDetail.config_snapshot_version
|
||||
? "bizCompare.configSnapshotHint"
|
||||
: "bizCompare.configLegacyHint")}
|
||||
>
|
||||
{t(runDetail.config_snapshot_version
|
||||
? "bizCompare.configSnapshot"
|
||||
: "bizCompare.configLegacy")}
|
||||
</span>
|
||||
) : null}
|
||||
{!boardFs ? (
|
||||
<>
|
||||
<select
|
||||
|
|
@ -3757,6 +3817,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
className={`bs-cmp-strip__kind bs-cmp-strip__kind--${cls}${
|
||||
kindFilter === id ? " is-active" : ""
|
||||
}`}
|
||||
aria-pressed={kindFilter === id}
|
||||
onClick={() => setKindFilter(id as KindFilter)}
|
||||
>
|
||||
{id === "diff"
|
||||
|
|
@ -3782,16 +3843,22 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
<div className="bs-cmp-filter-bar">
|
||||
<Input
|
||||
value={resultKw}
|
||||
aria-label={t("bizCompare.resultFilterPh")}
|
||||
placeholder={t("bizCompare.resultFilterPh")}
|
||||
onChange={(e) => setResultKw(e.target.value)}
|
||||
/>
|
||||
<span className="muted bs-sheet-count">
|
||||
{diffsLoading ? "…" : `${pagedDiffs.length}/${resultTotal}`}
|
||||
<span className="muted bs-sheet-count" role="status" aria-live="polite">
|
||||
{diffsLoading ? "…" : t("bizCompare.resultRange", {
|
||||
from: resultTotal ? (resultPage - 1) * resultPageSize + 1 : 0,
|
||||
to: Math.min(resultPage * resultPageSize, resultTotal),
|
||||
total: resultTotal,
|
||||
})}
|
||||
</span>
|
||||
{resultSearchKeyFields.length ? (
|
||||
<button
|
||||
type="button"
|
||||
className={`bs-cmp-strip__toggle${keyFiltersVisible ? " is-active" : ""}`}
|
||||
aria-expanded={keyFiltersVisible}
|
||||
onClick={() => setKeyFiltersOpen((v) => !v)}
|
||||
>
|
||||
{t("bizCompare.keyFiltersToggle")}
|
||||
|
|
@ -3809,6 +3876,14 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
>
|
||||
{t("bizCompare.displayColsToggle")}
|
||||
</button>
|
||||
{(resultKw.trim() || activeKeyFilterCount > 0) ? (
|
||||
<Button size="sm" variant="ghost" onPress={() => {
|
||||
setResultKw("");
|
||||
setResultKeyFilters({});
|
||||
}}>
|
||||
{t("bizCompare.clearSearch")}
|
||||
</Button>
|
||||
) : null}
|
||||
</div>
|
||||
{resultSearchKeyFields.length && keyFiltersVisible ? (
|
||||
<div className="bs-cmp-key-filters">
|
||||
|
|
@ -3856,10 +3931,10 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
) : null}
|
||||
{!isLiveSearch &&
|
||||
kindFilter === "unchanged" &&
|
||||
(runDetail?.summary?.unchanged_truncated ||
|
||||
(activeRunSheet?.summary?.unchanged_truncated ||
|
||||
(Number(
|
||||
(activeRunSheet?.summary as any)?.unchanged ||
|
||||
runDetail?.summary?.unchanged ||
|
||||
(activeRunSheet?.summary as any)?.unchanged ??
|
||||
runDetail?.summary?.unchanged ??
|
||||
0,
|
||||
) >
|
||||
Number(
|
||||
|
|
@ -3899,6 +3974,7 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
className={`pt-list-table-wrap bs-sheet-table bs-cmp-result-table${
|
||||
diffsLoading ? " is-loading" : ""
|
||||
}`}
|
||||
aria-busy={diffsLoading}
|
||||
>
|
||||
<table className="data-table pt-list-table bs-cmp-diff-table">
|
||||
<thead>
|
||||
|
|
@ -4004,6 +4080,22 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
})()}
|
||||
</thead>
|
||||
<tbody>
|
||||
{diffsLoading || diffsError ? (
|
||||
<tr>
|
||||
<td colSpan={resultEmptyColSpan}>
|
||||
<div className={`bs-cmp-result-state${diffsError ? " is-error" : ""}`}
|
||||
role={diffsError ? "alert" : "status"}>
|
||||
<strong>{t(diffsError ? "bizCompare.resultLoadFailed" : "bizCompare.resultLoading")}</strong>
|
||||
{diffsError ? <>
|
||||
<span>{diffsError}</span>
|
||||
<Button size="sm" variant="secondary" onPress={() => setDiffsRetry((n) => n + 1)}>
|
||||
{t("bizCompare.retry")}
|
||||
</Button>
|
||||
</> : null}
|
||||
</div>
|
||||
</td>
|
||||
</tr>
|
||||
) : null}
|
||||
{pagedDiffs.map((d, i) => {
|
||||
const pre = pickSideRow(
|
||||
d.mapped_before as Record<string, unknown> | null | undefined,
|
||||
|
|
@ -4081,13 +4173,13 @@ export function BizComparePage({ pageMode = "all" }: { pageMode?: BizComparePage
|
|||
</tr>
|
||||
);
|
||||
})}
|
||||
{runDetail && !diffsLoading && !pagedDiffs.length ? (
|
||||
{runDetail && !diffsLoading && !diffsError && !pagedDiffs.length ? (
|
||||
<tr>
|
||||
<td colSpan={resultEmptyColSpan}>
|
||||
<div className="pt-list-empty">
|
||||
{kindFilter === "unchanged"
|
||||
? Number(runDetail?.summary?.unchanged || 0) > 0 &&
|
||||
!Number(runDetail?.summary?.unchanged_listed || 0)
|
||||
{hasResultSearch ? t("bizCompare.searchEmpty") : kindFilter === "unchanged"
|
||||
? Number(activeRunSheet?.summary?.unchanged || 0) > 0 &&
|
||||
!Number(activeRunSheet?.summary?.unchanged_listed || 0)
|
||||
? t("bizCompare.unchangedNotStored")
|
||||
: t("bizCompare.resultEmpty")
|
||||
: activeSheetPending
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@
|
|||
type Entry = { at: number; data: unknown };
|
||||
|
||||
const store = new Map<string, Entry>();
|
||||
const inFlight = new Map<string, Promise<unknown>>();
|
||||
const DEFAULT_TTL_MS = 45_000;
|
||||
|
||||
export async function cutoverCachedGet<T>(
|
||||
|
|
@ -21,9 +22,17 @@ export async function cutoverCachedGet<T>(
|
|||
return hit.data as T;
|
||||
}
|
||||
}
|
||||
const data = await fetcher();
|
||||
store.set(key, { at: Date.now(), data });
|
||||
return data;
|
||||
const existing = inFlight.get(key);
|
||||
if (!opts?.force && existing) return existing as Promise<T>;
|
||||
const request = Promise.resolve().then(fetcher).then((data) => {
|
||||
// An invalidation or forced refresh may supersede this request.
|
||||
if (inFlight.get(key) === request) store.set(key, { at: Date.now(), data });
|
||||
return data;
|
||||
}).finally(() => {
|
||||
if (inFlight.get(key) === request) inFlight.delete(key);
|
||||
});
|
||||
inFlight.set(key, request);
|
||||
return request;
|
||||
}
|
||||
|
||||
/** Return stale immediately (if any), refresh in background and notify. */
|
||||
|
|
@ -34,27 +43,28 @@ export async function cutoverCachedGetSWR<T>(
|
|||
): Promise<T> {
|
||||
const hit = store.get(key);
|
||||
if (hit) {
|
||||
void fetcher()
|
||||
void cutoverCachedGet(key, fetcher, { ttlMs: 0 })
|
||||
.then((data) => {
|
||||
store.set(key, { at: Date.now(), data });
|
||||
onFresh?.(data);
|
||||
if (store.get(key)?.data === data) onFresh?.(data);
|
||||
})
|
||||
.catch(() => {
|
||||
/* keep stale */
|
||||
});
|
||||
return hit.data as T;
|
||||
}
|
||||
const data = await fetcher();
|
||||
store.set(key, { at: Date.now(), data });
|
||||
return data;
|
||||
return cutoverCachedGet(key, fetcher);
|
||||
}
|
||||
|
||||
export function invalidateCutoverCache(prefix = ""): void {
|
||||
if (!prefix) {
|
||||
store.clear();
|
||||
inFlight.clear();
|
||||
return;
|
||||
}
|
||||
for (const k of [...store.keys()]) {
|
||||
if (k.startsWith(prefix)) store.delete(k);
|
||||
}
|
||||
for (const k of [...inFlight.keys()]) {
|
||||
if (k.startsWith(prefix)) inFlight.delete(k);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
133
web/tests/bizCompare.browser.mjs
Normal file
133
web/tests/bizCompare.browser.mjs
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
// Run against a local Vite server. All API responses are fixtures; no device or DB is touched.
|
||||
// NETX_PLAYWRIGHT_MODULE can point at a bundled Playwright package.
|
||||
import assert from "node:assert/strict";
|
||||
import { createRequire } from "node:module";
|
||||
import { mkdir } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const { chromium } = require(process.env.NETX_PLAYWRIGHT_MODULE || "playwright");
|
||||
const base = process.env.NETX_TEST_URL || "http://127.0.0.1:5179";
|
||||
const output = path.resolve(process.env.NETX_TEST_OUTPUT || "../docs/reviews/assets");
|
||||
await mkdir(output, { recursive: true });
|
||||
const browser = await chromium.launch({ headless: true, channel: process.env.NETX_TEST_BROWSER || "chrome" });
|
||||
const page = await browser.newPage({ viewport: { width: 1440, height: 1000 } });
|
||||
const errors = [];
|
||||
page.on("pageerror", (error) => errors.push(error.message));
|
||||
const fields = ["neighbor", "vrf", "remote_as", "state"];
|
||||
const sheet = {
|
||||
sheet_id: "peer", title: "BGP 邻居", metric_id: "bgp_peer",
|
||||
key_fields: ["neighbor", "vrf"], iface_fields: [], compare_fields: ["remote_as", "state"],
|
||||
status: "done", summary: { changed: 1, removed: 1, added: 1, unchanged: 2, unchanged_listed: 2 },
|
||||
};
|
||||
const side = (label, ne_name, time) => ({ label, ne_name, started_at: time, ne_ip: "10.0.0.1" });
|
||||
const run = {
|
||||
id: "run-1", status: "success", job_id: "job-1", config_snapshot_version: 1, created_at: "2026-10-10T10:08:00Z",
|
||||
before: side("before", "PE-BEFORE", "2026-10-10T09:00:00Z"),
|
||||
after: side("after", "PE-AFTER", "2026-10-10T10:00:00Z"),
|
||||
sheets: [sheet],
|
||||
summary: { ...sheet.summary, before_count: 4, after_count: 4, pass_rate: 50, duration_ms: 1250,
|
||||
sheet_cards: [{ ...sheet, ...sheet.summary, before_count: 4, after_count: 4, pass_rate: 50 }] },
|
||||
};
|
||||
const job = { id: "job-1", name: "割接前后 · 业务验收", template_id: "tpl", mode: "manual", status: "ready",
|
||||
before_task_id: "t1", after_task_id: "t2", before_batch_id: "b", after_batch_id: "a" };
|
||||
const row = (neighbor, state, remote_as = "64512") => ({ neighbor, vrf: "core", state, remote_as });
|
||||
const diffs = [
|
||||
{ kind: "changed", key: { neighbor: "10.2.1.1", vrf: "core" },
|
||||
before: row("10.2.1.1", "Established"), after: row("10.2.1.1", "Idle"),
|
||||
changes: { state: { before: "Established", after: "Idle" } } },
|
||||
{ kind: "removed", key: { neighbor: "10.2.1.2", vrf: "core" }, before: row("10.2.1.2", "Established"), after: null, changes: {} },
|
||||
{ kind: "added", key: { neighbor: "10.2.1.3", vrf: "core" }, before: null, after: row("10.2.1.3", "Established"), changes: {} },
|
||||
{ kind: "unchanged", key: { neighbor: "10.2.1.4", vrf: "core" }, before: row("10.2.1.4", "Established"), after: row("10.2.1.4", "Established"), changes: {} },
|
||||
];
|
||||
let failDiffs = false;
|
||||
let active = false;
|
||||
let legacy = false;
|
||||
let pollCounts = { runs: 0, detail: 0 };
|
||||
await page.route("**/v1/**", async (route) => {
|
||||
const url = new URL(route.request().url());
|
||||
const p = url.pathname;
|
||||
let data = {};
|
||||
if (p === "/v1/auth/me") data = { user: { id: "test", username: "test", role: "admin", is_active: true } };
|
||||
else if (p.endsWith("/compare/jobs")) data = { items: [job, { ...job, id: "job-slow", name: "慢任务" }] };
|
||||
else if (p.endsWith("/compare/templates")) data = { items: [{ id: "tpl", name: "割接验收模板", metrics: [sheet] }] };
|
||||
else if (p.endsWith("/compare/metrics")) data = { items: [{ metric_id: "bgp_peer", fields: fields.map((name) => ({ name, display_name: name, dtype: "string", is_key: sheet.key_fields.includes(name) })) }] };
|
||||
else if (p.endsWith("/tasks")) data = { items: [{ id: "t1", ne_name: "PE-BEFORE", ne_ip: "10.0.0.1" }, { id: "t2", ne_name: "PE-AFTER", ne_ip: "10.0.0.2" }] };
|
||||
else if (p.endsWith("/batches")) data = { items: [{ id: p.includes("t1") ? "b" : "a", status: "success", row_count: 4 }] };
|
||||
else if (p.endsWith("/runs") && p.includes("/jobs/")) {
|
||||
pollCounts.runs++;
|
||||
if (p.includes("job-slow")) await new Promise((resolve) => setTimeout(resolve, 900));
|
||||
data = { items: [{ ...run, status: active ? "running" : "success" }] };
|
||||
} else if (p.endsWith("/diffs")) {
|
||||
if (failDiffs) { await route.fulfill({ status: 503, contentType: "application/json", body: JSON.stringify({ detail: "测试:连接暂时中断" }) }); return; }
|
||||
const kind = url.searchParams.get("kind") || "diff";
|
||||
const kw = url.searchParams.get("kw") || "";
|
||||
const items = diffs.filter((d) => (kind === "all" || (kind === "diff" ? ["removed", "changed"].includes(d.kind) : d.kind === kind)) && JSON.stringify(d).includes(kw));
|
||||
data = { items, total: items.length, page: 1, page_size: 100, source: "stored", truncated: false };
|
||||
} else if (p.endsWith("/runs/run-1")) {
|
||||
pollCounts.detail++;
|
||||
data = { ...run, config_snapshot_version: legacy ? null : 1, status: active ? "running" : "success" };
|
||||
} else if (p.endsWith("/mappings")) data = { items: [] };
|
||||
else data = { items: [] };
|
||||
await route.fulfill({ status: 200, contentType: "application/json", body: JSON.stringify(data) });
|
||||
});
|
||||
|
||||
try {
|
||||
await page.goto(`${base}/network/cutover/biz-compare`);
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.getByRole("button", { name: "比对结果", exact: true }).click();
|
||||
await page.locator(".bs-cmp-diff-table tbody tr.bs-cmp-row").first().waitFor();
|
||||
assert.equal(await page.locator(".bs-cmp-diff-table tbody tr.bs-cmp-row").count(), 2);
|
||||
assert.equal(await page.locator(".bs-cmp-board__config").textContent(), "执行配置已保存");
|
||||
await page.screenshot({ path: path.join(output, "biz-compare-desktop.png"), fullPage: true });
|
||||
|
||||
const search = page.locator(".bs-cmp-filter-bar input");
|
||||
failDiffs = true;
|
||||
await search.fill("故障");
|
||||
await page.getByText("对比明细加载失败", { exact: true }).waitFor();
|
||||
assert.equal(await page.locator(".bs-cmp-row").count(), 0);
|
||||
await page.screenshot({ path: path.join(output, "biz-compare-error.png"), fullPage: true });
|
||||
failDiffs = false;
|
||||
await page.getByRole("button", { name: "重试", exact: true }).click();
|
||||
await page.getByText("当前筛选条件下没有匹配结果,可清空筛选或切换结果类型。", { exact: true }).waitFor();
|
||||
await page.getByRole("button", { name: "清空筛选", exact: true }).click();
|
||||
await page.locator(".bs-cmp-row").first().waitFor();
|
||||
|
||||
await page.setViewportSize({ width: 768, height: 1000 });
|
||||
await page.screenshot({ path: path.join(output, "biz-compare-narrow.png"), fullPage: true });
|
||||
const fits = await page.locator(".bs-cmp-strip").evaluate((element) => element.scrollWidth <= element.clientWidth + 1);
|
||||
assert.ok(fits, "filter toolbar must fit its available width");
|
||||
await page.setViewportSize({ width: 390, height: 1000 });
|
||||
assert.ok(await page.locator(".bs-cmp-strip").evaluate((element) => element.scrollWidth <= element.clientWidth + 1));
|
||||
assert.ok(await page.locator(".bs-cmp-board__toolbar").evaluate((element) => element.scrollWidth <= element.clientWidth + 1));
|
||||
await page.setViewportSize({ width: 768, height: 1000 });
|
||||
|
||||
await page.getByRole("button", { name: "关闭", exact: true }).click();
|
||||
await page.getByRole("row").filter({ hasText: "慢任务" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.getByRole("button", { name: "关闭", exact: true }).click();
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.getByRole("button", { name: "比对结果", exact: true }).click();
|
||||
await page.locator(".bs-cmp-row").first().waitFor();
|
||||
await page.waitForTimeout(1100);
|
||||
assert.equal(await page.locator(".bs-cmp-row").count(), 2, "slow old job must not change the current tab");
|
||||
|
||||
legacy = true;
|
||||
await page.reload();
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.getByRole("button", { name: "比对结果", exact: true }).click();
|
||||
await page.getByText("旧记录:未保存执行配置", { exact: true }).waitFor();
|
||||
await page.screenshot({ path: path.join(output, "biz-compare-legacy.png"), fullPage: true });
|
||||
|
||||
active = true;
|
||||
await page.reload();
|
||||
await page.getByRole("row").filter({ hasText: "割接前后 · 业务验收" }).getByRole("button", { name: "详情", exact: true }).click();
|
||||
await page.locator(".bs-cmp-runs").waitFor();
|
||||
pollCounts = { runs: 0, detail: 0 };
|
||||
await page.waitForTimeout(5200);
|
||||
assert.ok(pollCounts.runs <= 3 && pollCounts.detail <= 3, JSON.stringify(pollCounts));
|
||||
assert.ok(pollCounts.runs >= 2 && pollCounts.detail >= 2, "active run should keep polling");
|
||||
assert.deepEqual(errors, []);
|
||||
console.log(JSON.stringify({ passed: ["result layout", "saved execution settings", "legacy settings warning", "error clears stale rows", "retry", "clear filters", "768px toolbar", "old job response ignored", "single poller", "no runtime errors"], pollCounts, output }));
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
72
web/tests/cutoverDataCache.test.mjs
Normal file
72
web/tests/cutoverDataCache.test.mjs
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { beforeEach, test } from "node:test";
|
||||
import { cutoverCachedGet, cutoverCachedGetSWR, invalidateCutoverCache } from "../src/pages/network/cutoverDataCache.ts";
|
||||
|
||||
const deferred = () => {
|
||||
let resolve, reject;
|
||||
const promise = new Promise((ok, fail) => { resolve = ok; reject = fail; });
|
||||
return { promise, resolve, reject };
|
||||
};
|
||||
|
||||
beforeEach(() => invalidateCutoverCache());
|
||||
|
||||
test("concurrent readers share one fetch", async () => {
|
||||
const pending = deferred();
|
||||
let calls = 0;
|
||||
const fetcher = () => { calls++; return pending.promise; };
|
||||
const first = cutoverCachedGet("batches", fetcher);
|
||||
const second = cutoverCachedGet("batches", fetcher);
|
||||
pending.resolve("fresh");
|
||||
assert.deepEqual(await Promise.all([first, second]), ["fresh", "fresh"]);
|
||||
assert.equal(calls, 1);
|
||||
});
|
||||
|
||||
test("invalidated pending requests cannot repopulate the cache", async () => {
|
||||
const pending = deferred();
|
||||
const old = cutoverCachedGet("batches", () => pending.promise);
|
||||
invalidateCutoverCache("batches");
|
||||
pending.resolve("stale");
|
||||
await old;
|
||||
assert.equal(await cutoverCachedGet("batches", async () => "fresh"), "fresh");
|
||||
});
|
||||
|
||||
test("forced refresh supersedes a slow older request", async () => {
|
||||
const pending = deferred();
|
||||
const old = cutoverCachedGet("lists", () => pending.promise);
|
||||
assert.equal(await cutoverCachedGet("lists", async () => "fresh", { force: true }), "fresh");
|
||||
pending.resolve("old");
|
||||
await old;
|
||||
assert.equal(await cutoverCachedGet("lists", async () => "unexpected"), "fresh");
|
||||
});
|
||||
|
||||
test("a failed fetch can be retried", async () => {
|
||||
await assert.rejects(cutoverCachedGet("lists", async () => { throw new Error("offline"); }));
|
||||
assert.equal(await cutoverCachedGet("lists", async () => "recovered"), "recovered");
|
||||
});
|
||||
|
||||
test("SWR readers share background refresh and each receives the fresh value", async () => {
|
||||
await cutoverCachedGet("lists", async () => "cached");
|
||||
const pending = deferred();
|
||||
let calls = 0;
|
||||
const notifications = [];
|
||||
const fetcher = () => { calls++; return pending.promise; };
|
||||
const first = await cutoverCachedGetSWR("lists", fetcher, (x) => notifications.push(x));
|
||||
const second = await cutoverCachedGetSWR("lists", fetcher, (x) => notifications.push(x));
|
||||
assert.equal(first, "cached");
|
||||
assert.equal(second, "cached");
|
||||
pending.resolve("fresh");
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
assert.equal(calls, 1);
|
||||
assert.deepEqual(notifications, ["fresh", "fresh"]);
|
||||
});
|
||||
|
||||
test("invalidated SWR requests do not notify with stale data", async () => {
|
||||
await cutoverCachedGet("lists", async () => "cached");
|
||||
const pending = deferred();
|
||||
const notifications = [];
|
||||
await cutoverCachedGetSWR("lists", () => pending.promise, (x) => notifications.push(x));
|
||||
invalidateCutoverCache("lists");
|
||||
pending.resolve("stale");
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
assert.deepEqual(notifications, []);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue