diff --git a/docs/03-tech-stack.md b/docs/03-tech-stack.md index c29b59c..b341a23 100644 --- a/docs/03-tech-stack.md +++ b/docs/03-tech-stack.md @@ -24,8 +24,8 @@ | 语言 | Go | 已定 | 面向长时间 RTSP 运行、并发控制和单一交付物。 | | 模型工件 | 固定输入尺寸的 ONNX Pose 模型 | 已定 | Go 不直接加载 `.pt`;模型由已验收 V1 权重导出并记录哈希。 | | 推理运行时 | `github.com/yalue/onnxruntime_go` v1.31.0 + ONNX Runtime CPU DLL | Spike 已验证 | Windows 上以显式 DLL 路径加载锁定的 1×3×640×640 Pose ONNX,已完成一次真实录像首帧推理;该包装器的 C API 为 26,V2 构建必须启用 CGO/MinGW,并在发布包锁定匹配 DLL 的来源与 SHA-256。 | -| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | Spike 已验证 | 已读取 848×480 本地录像首帧;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 | -| 图像预处理 | Go 实现的 BGR→RGB、CHW、640 letterbox | Spike 已验证,待一致性回归 | 可生成 ONNX 所需 1,228,800 个 float;当前是最小 nearest-neighbour Spike,不得直接当作生产预处理,T-303 必须与 V1 缩放、补边、NMS 和坐标还原逐项对齐。 | +| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | 回放已验证 | Go 回归命令已完整读取 317 帧本地录像;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 | +| 图像预处理与 Pose 解析 | Go 实现的 BGR→RGB、CHW、640 letterbox、NMS、17 点及坐标还原 | 合成契约与无报警回放已验证 | 采用 Ultralytics 的 114 补边、双线性缩放后转 uint8 的语义;Go 首帧输出两个人,与 V1 对应结果的 box 坐标差在已记录的 6 像素容差内。仍需经同意的真实正例录像完成事件级确认。 | | 演示 UI | `github.com/lxn/walk`(Windows 原生控件) | Spike 已验证 | 双 Tab 骨架可在 `CGO_ENABLED=0` 编译,避免 Fyne/OpenGL 首次编译的高复杂度;正式版仍须实现浅灰蓝底、白色卡片,红色仅表示确认摔倒。 | | 测试 | `go test ./...` + V1 回归录像 | 已定 | V2 行为必须与 V1 基线对照。 | @@ -49,6 +49,7 @@ | V1 单元测试 | `python -m pytest v1/tests -v` | | V2 单元测试 | `Set-Location v2; $env:CGO_ENABLED='1'; go test ./...` | | V2 ONNX/视频 Spike | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/spike --ffmpeg --video --width --height --onnx assets\best.onnx --ort-dll ` | +| V2 录像事件回归 | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/regression --ffmpeg --video --width --height --fps --onnx assets\best.onnx --ort-dll ` | | V2 Walk UI 编译 Spike | `Set-Location v2; $env:CGO_ENABLED='0'; go build ./cmd/ui-spike` | ## 演进决策 diff --git a/docs/04-architecture.md b/docs/04-architecture.md index 75b8408..d6a1e1c 100644 --- a/docs/04-architecture.md +++ b/docs/04-architecture.md @@ -26,7 +26,7 @@ Pose 推理适配器(person box + 17 keypoints) V1 的同一数据流既可接 RTSP,也可回放本地录像。V2 复用同一配置、事件字段和录像集,但将 Pose 推理替换为 ONNX 适配器;不得改变事件语义。 -T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块生成固定 `1×3×640×640` RGB/CHW tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。T-303 必须先补齐与 V1 一致的 letterbox、NMS、17 点解析、坐标还原、跟踪和状态机,才可以接入实时 UI。 +T-302/T-303 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块以 114 补边、双线性缩放与 RGB/CHW 生成固定 `1×3×640×640` tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,随后由 NMS、17 点解析、坐标还原、轻量跟踪、质量/几何证据、倒地策略和四态状态机生成不变的 `FallEvent`。Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。 ## 模块职责 @@ -48,15 +48,15 @@ T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,G | 应用装配 | `v1/app.py` | 装配配置、Pose、管线线程与窗口;`FrameWorker` 只发出已判定的 `FrameAnalysis` | 事件判定、渲染细节 | | 回归工具 | `v1/tests/` 与 `v1/scripts/` | 回放录像、断言事件和延迟 | 生产 UI | -V2 对应模块将在 T-303/T-304 建立,职责预先固定如下: +V2 对应模块的当前落实与 T-304 目标如下: | 模块 | V2 目标位置 | 职责 | 不负责 | | --- | --- | --- | --- | -| 解码器 | `v2/internal/video` | 启动受控 FFmpeg、提供最新 BGR 帧与显式来源状态 | Pose、事件判定、UI 绘制 | -| ONNX Pose | `v2/internal/pose` | 校验模型/DLL 清单,完成精确预处理、推理、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 | +| 回放解码器 | `v2/cmd/regression` | 用受控 FFmpeg 顺序读取 BGR 帧,供本机录像回归 | RTSP 重连、UI 绘制 | +| ONNX Pose | `v2/internal/pose` | 114 letterbox、RGB/CHW、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 | | 事件引擎 | `v2/internal/fall` | 复现 V1 的跟踪、证据、四态状态机和 `FallEvent` | 视频解码、声音、文件 | | Windows UI | `v2/internal/ui` | Walk 顶部“监控/设置”Tab、渲染最新帧和已计算状态 | 直接读 RTSP、执行 ONNX 或事件规则 | -| 应用装配 | `v2/cmd/silver-pose` | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 | +| 应用装配 | `v2/cmd/silver-pose`(T-304) | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 | ## UI 导航与配置生效生命周期 diff --git a/docs/current-state.md b/docs/current-state.md index d6e2ed5..7cb414d 100644 --- a/docs/current-state.md +++ b/docs/current-state.md @@ -9,13 +9,13 @@ - 已验证环境:Windows PowerShell;Python 3.8.10;Ultralytics 8.3.205;PyQt5 可导入。注意:当前 WSL/Linux 环境未安装 PyQt5,且其 ultralytics/torch 版本与该基线不兼容(模型前向不可用),故 GUI 可视化与真实模型端到端渲染只在 Windows 验证;本环境用 Qt-free 的 `view_model.py` 单元测试与假适配器管线冒烟覆盖视图逻辑。 - 旧生产基线:`demo/main.py`、`demo/fall_detection_gui.py`、`demo/detect_fall.py`、`demo/best.pt`。 - V1 代码:已建立安全配置、显式 Replay/Stream 视频源、Pose、轻量跟踪、质量/几何证据、倒地领域规则、按 ID 四态事件机及 `v1/pipeline.py` 事件管线;运行事件带非敏感 `config_version`,缺帧/低质量/断流会中断证据确认。新增 `v1/view_model.py`(Qt-free 监控视图状态与设置草稿三份隔离)、`v1/gui.py` 薄 PyQt5 双 Tab 外壳与 `v1/app.py` 装配(`FrameWorker` 只发出已判定的 `FrameAnalysis`,窗口只渲染)。新增 `v1/alerts.py`:按 `event_id` 去重,对首次 CONFIRMED 保存带标注截图、追加 JSONL 事件行,并经可注入 `AlertSink`(Windows 侧 `QtAlertSink` 提供声音与一次性弹窗)触发一次声音/弹窗。新增 `v1/camera.py`(RTSP URL 构建 + 有界连接测试抓帧)与 `config.py` 的结构化来源(host/端口/通道/账号/密码,凭证百分号编码)、`write_local_camera_source`;设置页新增摄像头连接分组、测试连接与预览,以及传输协议/连接超时/低延迟三项抓流调优(由 `config` 存值、`app` 开流前写入 `OPENCV_FFMPEG_CAPTURE_OPTIONS`)。真实海康流已用 `VideoSource(STREAM)` 连通(1920×1080)。 -- V2 代码:已创建 Go 模块和受控 Spike。`cmd/spike` 用 FFmpeg 读取本地 BGR 帧、将其预处理为 640 tensor,并用 `onnxruntime_go` 对锁定 ONNX 执行一次推理;`cmd/ui-spike` 是可编译的 Windows Walk 双 Tab 外壳。它们不含 NMS/关键点解析、跟踪、状态机、RTSP、报警或发布打包,不能作为客户演示版。 +- V2 代码:T-303 已落地 `internal/pose`(Ultralytics 114 letterbox、BGR→RGB/CHW、YOLO Pose NMS、17 点及坐标还原)与 `internal/fall`(中心点跟踪、质量/几何证据、策略、四态状态机和事件 ID);`cmd/regression` 使用 FFmpeg 顺序回放本机录像并将 ONNX Pose 接入事件引擎。`cmd/ui-spike` 仍只是可编译的 Windows Walk 双 Tab 外壳。RTSP 重连、UI 实时渲染、声音、弹窗、截图和发布打包尚未实现,不能作为客户演示版。 - 非代码设计工件:docs/ui/silver-pose-ui-ux-spec.md、docs/ui/2026-07-20-html-prototype-plan.md、docs/ui/silver-pose-v1-prototype.html 与 docs/ui/silver-pose-v2-prototype.html 已建立。v2 HTML 是符合正式浅色 Windows 规范的当前视觉参考:浅灰蓝底、白色卡片,红色只表示确认摔倒、其弹窗和事件证据;文件名中的 v2 只表示原型设计修订,不能理解为 Go V2 实现已开始。v1 HTML 保留为历史深色对照。两者均使用顶部双 Tab、设置草稿与状态交互,且画面、事件和时间都是模拟数据,不连接真实摄像头、模型或网络,也不改变 Phase 1 任务顺序。 -- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped,覆盖配置、视频源、Pose、跟踪、证据、领域规则、状态机、管线、视图模型、报警工件、摄像头/URL 构建、抓流参数拼串、每帧诊断、灵敏度放宽、截图标注、断流重连、事件汇总、灵敏度写回和最新帧丢弃。T-302 的 `CGO_ENABLED=1 go test ./...` 通过;Go Spike 对 848×480 本地录像首帧生成 1,228,800 元素输入并获得 470,400 元素 ONNX 输出;Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些只验证管线、视图与报警工件可运行,不表示摔倒识别准确率或 Go 行为等价。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。 +- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped。T-303 的 `CGO_ENABLED=1 go test ./...` 通过。对同一 317 帧本机录像,V1 回放得到 253 个有人帧、263 人次、0 个 CONFIRMED;Go 回放得到 255 个有人帧、265 人次、0 个 CONFIRMED。人次差异来自 OpenCV/FFmpeg 解码像素不同,二者的无报警事件结论一致;Go 合成正例契约和 V1 状态机均在 1.81 秒时产生一个 CONFIRMED。Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些不表示摔倒识别准确率或完整 Go 客户演示已验收。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。 - 模型:`demo/best.pt` 可加载为 YOLO Pose,类别 `person`,`kpt_shape=[17, 3]`;与 `D:\PythonP\fall_detection\best.pt` 哈希一致。 - 当前标准启动:`./init.ps1`。 - 当前标准验证:`python -m compileall -q demo`。 -- 当前 blocker:没有 T-303 的代码阻塞;但 Go 尚未验证与 V1 的预后处理、NMS、关键点、跟踪、状态机和事件延迟一致,不能把 T-302 的单帧推理表述为可演示的 V2。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。 +- 当前 blocker:T-303 缺少经同意的、可在本机回放的正例摔倒录像及其 V1 基线;因此尚不能证明同一真实正例录像在 Go 中也确认且延迟为 1–3 秒。Go/V1 无报警回放和纯领域正例契约均已通过,但不能替代录像级正例。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。 全局环境的 `pip check` 存在其他项目的包冲突,因此它不是 Silver Pose 的验收命令。`init.ps1` 只检查本项目实际导入的 OpenCV、NumPy、Ultralytics 与 PyQt5,并在命令非零退出时失败。 @@ -25,7 +25,7 @@ | --- | --- | --- | | `demo/` | 已有 | 旧 Python GUI、Pose 模型与样例视频;当前单帧规则不适合作为正式事件逻辑。 | | `v1/` | 已验收 | Python V1 正式实现、现场验收记录与发布工具。 | -| `v2/` | Spike 已完成 | Go 模块、ONNX/视频命令行 Spike 与 Windows Walk UI 外壳;尚不是完整 V2。 | +| `v2/` | T-303 部分完成 | Go ONNX 后处理、事件引擎和本机录像无报警回归已实现;尚不是完整 V2。 | | `docs/` | 已有 | Harness Coding 文档、任务和计划。 | | `tests/` | 不适用 | V1 测试将在 `v1/tests/` 下创建。 | | `artifacts/` | 待创建 | 本地事件截图和 JSONL,不提交。 | @@ -33,7 +33,6 @@ ## 任务状态 - 已完成:T-000、T-101~T-106(V1 地基与事件管线)、T-201(PyQt 双 Tab)、T-202(本地报警/截图/JSONL)、T-203(海康 RTSP 与断流恢复)、T-204(正反例录像与现场验收)、T-205(客户演示脚本与发布包)、T-206(结构化摄像头配置与连接测试)、T-207(抓流低延迟参数)、T-208(每帧诊断叠加)、T-209(放宽俯视灵敏度)、T-210(截图 ASCII 标注)、T-211(灵敏度设置页)、T-212(实时流最新帧)、T-213(截图中文标注)、T-301(ONNX 一致性)、T-302(Go 推理、视频/UI Spike)。 -- 正在进行:无。 - 正在进行:T-303(实现 Go 事件引擎与 V1 回归对比)。 ## 当前可运行内容 @@ -42,6 +41,11 @@ ./init.ps1 Set-Location demo python main.py + +# 本机录像 Go 回归(需本机 ONNX Runtime DLL、FFmpeg 与已锁定 ONNX 工件) +Set-Location ..\v2 +$env:CGO_ENABLED='1' +go run ./cmd/regression --help ``` ## 已知事实和限制 diff --git a/progress.md b/progress.md index b5e63fc..af6bc0f 100644 --- a/progress.md +++ b/progress.md @@ -487,3 +487,12 @@ - 阻塞:无代码阻塞。任务验收将明确区分“提交的合成契约回归 / 本机 demo 无报警回放”与“需本机正例录像才能证实的录像级确认事件、延迟”。 - 决策:不复制或提交客户录像;先忠实复现 V1 的纯领域合约,并让回归工具接收外部本机录像和基线 JSON。 - 下一步:先为 Go Pose 后处理与事件状态机编写失败测试。 + +## 【2026-07-22】T-303 实现与无报警录像回归(部分完成) + +- 状态:PARTIAL(代码与无报警回归完成;真实正例录像回归待本机素材) +- 变更:新增 `v2/internal/pose`(114 补边、双线性缩放后 uint8 量化、BGR→RGB/CHW、YOLOv8 Pose NMS、17 点与坐标还原)、`v2/internal/fall`(归一化中心点跟踪、姿态质量/几何证据、倒地策略、四态状态机、事件 ID)和 `v2/cmd/regression`(FFmpeg 回放→ONNX Pose→事件引擎)。`spike` 的预处理现委托给同一正式实现,ONNX Runtime session 可跨帧复用。 +- 验证:Go 单元测试覆盖 letterbox 补边/通道布局、NMS 坐标还原、输入长度、事件确认、缺帧中断和持续站立不报警;`CGO_ENABLED=1 go test ./...` 通过。本机 317 帧回放:V1 为 253 个有人帧、263 人次、0 个 CONFIRMED;Go 为 255 个有人帧、265 人次、0 个 CONFIRMED。二者无报警结论一致。合成正例以同一 V1 策略/状态机合约在 1.81 秒生成一个 CONFIRMED,Go 测试同样通过。 +- 阻塞:无客户或正例录像被提交;仓库也没有经同意的正例回放素材。因此不能把合成契约或无报警录像表述为真实正例的 Go/V1 等价,也不能把 T-303 标为 DONE。 +- 决策:保持 T-303 为 DOING;回归命令接受本机路径,不把录像、截图、事件 JSONL、RTSP 地址或凭证写进仓库。解码器差异造成同一无报警录像的人次计数轻微差异,事件而非逐帧检测人次是当前验收基线。 +- 下一步:提供或指定一段经同意、无敏感信息的本机正例录像及其 V1 输出后,运行同一命令比对确认事件与 1–3 秒延迟;通过后才可完成 T-303,继而领取 T-304。 diff --git a/v2/cmd/regression/main.go b/v2/cmd/regression/main.go new file mode 100644 index 0000000..634bfaa --- /dev/null +++ b/v2/cmd/regression/main.go @@ -0,0 +1,118 @@ +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "os" + "os/exec" + "time" + + "silverpose/v2/internal/fall" + "silverpose/v2/internal/pose" + "silverpose/v2/internal/spike" +) + +func main() { + ffmpegPath := flag.String("ffmpeg", "ffmpeg", "path to ffmpeg.exe") + videoPath := flag.String("video", "..\\demo\\1.mp4", "local non-sensitive validation video") + width := flag.Int("width", 848, "source video frame width") + height := flag.Int("height", 480, "source video frame height") + fps := flag.Float64("fps", 30, "replay timestamp frames per second") + maxFrames := flag.Int("max-frames", 0, "maximum frames to process; zero reads the complete video") + modelPath := flag.String("onnx", "assets\\best.onnx", "locked ONNX pose model path") + runtimeDLLPath := flag.String("ort-dll", "", "absolute onnxruntime.dll path") + flag.Parse() + + if *runtimeDLLPath == "" || *width <= 0 || *height <= 0 || *fps <= 0 || *maxFrames < 0 { + fail("--ort-dll is required; width, height and fps must be positive; max-frames cannot be negative") + } + engine, err := fall.NewEngine(fall.EngineConfig{ + KeypointConfidenceThreshold: 0.4, + SuspectWindowSeconds: 0.5, + ConfirmWindowSeconds: 1.8, + RecoveryWindowSeconds: 2.0, + CooldownSeconds: 10.0, + RequireRapidDrop: false, + RequireLowerBody: false, + HorizontalAngleThresholdDegrees: 45.0, + ConfigVersion: "v1-default-20260722", + SessionID: "video-regression", + }) + if err != nil { + fail("create event engine: %v", err) + } + runtime, err := spike.OpenRuntime(*modelPath, *runtimeDLLPath) + if err != nil { + fail("open ONNX Pose runtime: %v", err) + } + defer runtime.Close() + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + command := exec.CommandContext(ctx, *ffmpegPath, "-v", "error", "-i", *videoPath, "-f", "rawvideo", "-pix_fmt", "bgr24", "-") + stdout, err := command.StdoutPipe() + if err != nil { + fail("open FFmpeg output: %v", err) + } + if err := command.Start(); err != nil { + fail("start FFmpeg: %v", err) + } + + frameBytes := *width * *height * 3 + frame := make([]byte, frameBytes) + frameCount := 0 + personFrames := 0 + personCount := 0 + events := make([]fall.Event, 0) + limited := false + for *maxFrames == 0 || frameCount < *maxFrames { + _, err := io.ReadFull(stdout, frame) + if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) { + break + } + if err != nil { + fail("read FFmpeg BGR frame: %v", err) + } + input, transform, err := pose.PreprocessBGR(frame, *width, *height, 640) + if err != nil { + fail("preprocess frame %d: %v", frameCount, err) + } + output, err := runtime.Run(input) + if err != nil { + fail("run ONNX Pose for frame %d: %v", frameCount, err) + } + people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70) + if err != nil { + fail("parse ONNX Pose for frame %d: %v", frameCount, err) + } + result := engine.Process(fall.Frame{Timestamp: float64(frameCount) / *fps, Width: *width, Height: *height, Poses: people}) + if len(people) > 0 { + personFrames++ + personCount += len(people) + } + events = append(events, result.Events...) + frameCount++ + } + if *maxFrames > 0 && frameCount == *maxFrames { + limited = true + cancel() + } + if err := command.Wait(); err != nil { + if !limited { + fail("FFmpeg ended with an error: %v", err) + } + } + + fmt.Printf("REGRESSION_OK frames=%d person_frames=%d people=%d events=%d\n", frameCount, personFrames, personCount, len(events)) + for _, event := range events { + fmt.Printf("EVENT id=%s track=%s latency=%.6f state=%s\n", event.EventID, event.TrackID, event.LatencySeconds, event.State) + } +} + +func fail(format string, args ...any) { + fmt.Fprintf(os.Stderr, "regression: "+format+"\n", args...) + os.Exit(1) +} diff --git a/v2/cmd/spike/main.go b/v2/cmd/spike/main.go index b327a47..69e2dbe 100644 --- a/v2/cmd/spike/main.go +++ b/v2/cmd/spike/main.go @@ -8,6 +8,7 @@ import ( "os/exec" "time" + "silverpose/v2/internal/pose" "silverpose/v2/internal/spike" ) @@ -41,7 +42,7 @@ func main() { fail("decoded BGR frame has %d bytes, want %d; verify --width/--height", len(raw), expectedBytes) } - input, err := spike.LetterboxBGRToNCHW(raw, *width, *height, 640) + input, transform, err := pose.PreprocessBGR(raw, *width, *height, 640) if err != nil { fail("preprocess frame: %v", err) } @@ -49,6 +50,10 @@ func main() { if err != nil { fail("run ONNX Pose: %v", err) } + people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70) + if err != nil { + fail("parse ONNX Pose: %v", err) + } max := float32(0) for _, value := range output { @@ -56,7 +61,10 @@ func main() { max = value } } - fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d max=%.4f\n", *width, *height, len(input), len(output), max) + fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d people=%d max=%.4f\n", *width, *height, len(input), len(output), len(people), max) + for index, person := range people { + fmt.Printf("PERSON index=%d conf=%.6f box=%.3f,%.3f,%.3f,%.3f\n", index, person.Confidence, person.Box.Left, person.Box.Top, person.Box.Right, person.Box.Bottom) + } } func fail(format string, args ...any) { diff --git a/v2/internal/fall/evidence.go b/v2/internal/fall/evidence.go new file mode 100644 index 0000000..474930a --- /dev/null +++ b/v2/internal/fall/evidence.go @@ -0,0 +1,62 @@ +package fall + +import ( + "math" + + "silverpose/v2/internal/pose" +) + +var torsoJoints = [...]int{5, 6, 11, 12} +var lowerJoints = [...]int{13, 14, 15, 16} + +func assessPoseQuality(person pose.PersonPose, threshold float32, requireLowerBody bool) PoseQuality { + if threshold < 0 || threshold > 1 { + return PoseQuality{Reason: "invalid_threshold"} + } + visible := 0 + for _, keypoint := range person.Keypoints { + if keypoint.Confidence >= threshold { + visible++ + } + } + for _, index := range torsoJoints { + if person.Keypoints[index].Confidence < threshold { + return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible} + } + } + if requireLowerBody { + for _, index := range lowerJoints { + if person.Keypoints[index].Confidence < threshold { + return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible} + } + } + } + return PoseQuality{Accepted: true, Reason: "accepted", VisibleJointCount: visible} +} + +func extractEvidence(person pose.PersonPose, quality PoseQuality, previous *PoseEvidence, horizontalAngleThreshold float32) PoseEvidence { + if !quality.Accepted { + return PoseEvidence{Reason: quality.Reason} + } + shoulderX, shoulderY := midpoint(person, 5, 6) + hipX, hipY := midpoint(person, 11, 12) + vectorX := hipX - shoulderX + vectorY := hipY - shoulderY + torsoLength := float32(math.Hypot(float64(vectorX), float64(vectorY))) + if torsoLength == 0 { + return PoseEvidence{HipCenterY: hipY, HasHipCenterY: true, TorsoLength: 0, Reason: "degenerate_torso"} + } + cosine := math.Max(-1, math.Min(1, math.Abs(float64(vectorX))/float64(torsoLength))) + angle := float32(math.Acos(cosine) * 180 / math.Pi) + rapid := previous != nil && previous.Accepted && previous.HasHipCenterY && hipY-previous.HipCenterY >= 0.5*torsoLength + return PoseEvidence{ + Accepted: true, HorizontalPose: angle <= horizontalAngleThreshold, + RapidVerticalChange: rapid, HorizontalAngleDegree: angle, HasAngle: true, + HipCenterY: hipY, HasHipCenterY: true, TorsoLength: torsoLength, Reason: "accepted", + } +} + +func midpoint(person pose.PersonPose, first, second int) (float32, float32) { + return (person.Keypoints[first].X + person.Keypoints[second].X) / 2, + (person.Keypoints[first].Y + person.Keypoints[second].Y) / 2 +} diff --git a/v2/internal/fall/pipeline.go b/v2/internal/fall/pipeline.go new file mode 100644 index 0000000..850c500 --- /dev/null +++ b/v2/internal/fall/pipeline.go @@ -0,0 +1,94 @@ +package fall + +import "fmt" + +type Engine struct { + config EngineConfig + tracker *tracker + policy *policy + stateMachine *stateMachine + previousEvidence map[string]PoseEvidence + activeTrackIDs map[string]bool +} + +func NewEngine(config EngineConfig) (*Engine, error) { + if config.KeypointConfidenceThreshold < 0 || config.KeypointConfidenceThreshold > 1 { + return nil, fmt.Errorf("keypoint confidence threshold must be between 0 and 1") + } + if config.SuspectWindowSeconds < 0 { + return nil, fmt.Errorf("suspect window seconds must be non-negative") + } + if config.HorizontalAngleThresholdDegrees < 0 || config.HorizontalAngleThresholdDegrees > 90 { + return nil, fmt.Errorf("horizontal angle threshold degrees must be between 0 and 90") + } + tracker, err := newTracker(0.2, 2.0) + if err != nil { + return nil, err + } + stateMachine, err := newStateMachine(config) + if err != nil { + return nil, err + } + return &Engine{ + config: config, tracker: tracker, policy: newPolicy(config.SuspectWindowSeconds, config.RequireRapidDrop), + stateMachine: stateMachine, previousEvidence: make(map[string]PoseEvidence), activeTrackIDs: make(map[string]bool), + }, nil +} + +func (engine *Engine) StateOf(trackID string) State { + return engine.stateMachine.stateOf(trackID) +} + +func (engine *Engine) Process(frame Frame) FrameResult { + tracked, err := engine.tracker.update(frame.Poses, frame.Timestamp, frame.Width, frame.Height) + if err != nil { + return FrameResult{} + } + currentIDs := make(map[string]bool, len(tracked)) + for _, trackedPose := range tracked { + currentIDs[trackedPose.TrackID] = true + } + events := engine.rejectMissingTracks(currentIDs, frame.Timestamp) + people := make([]PersonAnalysis, 0, len(tracked)) + for _, trackedPose := range tracked { + quality := assessPoseQuality(trackedPose.Pose, engine.config.KeypointConfidenceThreshold, engine.config.RequireLowerBody) + var previous *PoseEvidence + if candidate, found := engine.previousEvidence[trackedPose.TrackID]; found { + previous = &candidate + } + poseEvidence := extractEvidence(trackedPose.Pose, quality, previous, engine.config.HorizontalAngleThresholdDegrees) + stateBefore := engine.stateMachine.stateOf(trackedPose.TrackID) + evidence := engine.policy.evaluate(trackedPose.TrackID, poseEvidence, frame.Timestamp, stateBefore) + newEvents, err := engine.stateMachine.update(trackedPose.TrackID, evidence, frame.Timestamp) + if err == nil { + events = append(events, newEvents...) + } + if poseEvidence.Accepted { + engine.previousEvidence[trackedPose.TrackID] = poseEvidence + } else { + delete(engine.previousEvidence, trackedPose.TrackID) + } + people = append(people, PersonAnalysis{ + TrackedPose: trackedPose, PoseEvidence: poseEvidence, Evidence: evidence, + State: engine.stateMachine.stateOf(trackedPose.TrackID), + }) + } + engine.activeTrackIDs = currentIDs + return FrameResult{People: people, Events: events} +} + +func (engine *Engine) rejectMissingTracks(currentIDs map[string]bool, now float64) []Event { + events := make([]Event, 0) + for trackID := range engine.activeTrackIDs { + if currentIDs[trackID] { + continue + } + delete(engine.previousEvidence, trackID) + engine.policy.evaluate(trackID, PoseEvidence{}, now, engine.stateMachine.stateOf(trackID)) + newEvents, err := engine.stateMachine.update(trackID, Evidence{}, now) + if err == nil { + events = append(events, newEvents...) + } + } + return events +} diff --git a/v2/internal/fall/pipeline_test.go b/v2/internal/fall/pipeline_test.go new file mode 100644 index 0000000..91981c3 --- /dev/null +++ b/v2/internal/fall/pipeline_test.go @@ -0,0 +1,105 @@ +package fall + +import ( + "math" + "testing" + + "silverpose/v2/internal/pose" +) + +func TestEngineConfirmsPersistentHorizontalPoseAtV1Delay(t *testing.T) { + engine, err := NewEngine(EngineConfig{ + KeypointConfidenceThreshold: 0.4, + SuspectWindowSeconds: 0.5, + ConfirmWindowSeconds: 1.8, + RecoveryWindowSeconds: 2.0, + CooldownSeconds: 10.0, + RequireRapidDrop: false, + RequireLowerBody: false, + HorizontalAngleThresholdDegrees: 45.0, + ConfigVersion: "cfg-v1-contract", + SessionID: "regression", + }) + if err != nil { + t.Fatalf("NewEngine returned an error: %v", err) + } + + engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}}) + engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}}) + result := engine.Process(Frame{Timestamp: 1.91, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}}) + + if len(result.Events) != 1 { + t.Fatalf("events = %+v, want one confirmation", result.Events) + } + event := result.Events[0] + if event.TrackID != "P-0001" || math.Abs(event.LatencySeconds-1.81) > 1e-9 || event.ConfigVersion != "cfg-v1-contract" { + t.Fatalf("event = %+v", event) + } + if got := engine.StateOf("P-0001"); got != Confirmed { + t.Fatalf("state = %s, want CONFIRMED", got) + } +} + +func TestEngineDoesNotConfirmWhenTrackIsMissing(t *testing.T) { + engine, err := NewEngine(testConfig()) + if err != nil { + t.Fatalf("NewEngine returned an error: %v", err) + } + + engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}}) + engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}}) + engine.Process(Frame{Timestamp: 0.2, Width: 180, Height: 180}) + result := engine.Process(Frame{Timestamp: 1.9, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}}) + + if len(result.Events) != 0 { + t.Fatalf("events = %+v, want none after a missing track", result.Events) + } +} + +func TestEngineDoesNotAlarmForUprightSequence(t *testing.T) { + engine, err := NewEngine(testConfig()) + if err != nil { + t.Fatalf("NewEngine returned an error: %v", err) + } + + for _, timestamp := range []float64{0.0, 0.6, 1.2, 1.8, 2.4} { + result := engine.Process(Frame{Timestamp: timestamp, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}}) + if len(result.Events) != 0 { + t.Fatalf("timestamp %v emitted %+v for upright pose", timestamp, result.Events) + } + } +} + +func testConfig() EngineConfig { + return EngineConfig{ + KeypointConfidenceThreshold: 0.4, + SuspectWindowSeconds: 0.5, + ConfirmWindowSeconds: 1.8, + RecoveryWindowSeconds: 2.0, + CooldownSeconds: 10.0, + RequireRapidDrop: false, + RequireLowerBody: false, + HorizontalAngleThresholdDegrees: 45.0, + ConfigVersion: "cfg-v1-contract", + SessionID: "regression", + } +} + +func testPose(horizontal bool) pose.PersonPose { + person := pose.PersonPose{Box: pose.Box{Left: 20, Top: 20, Right: 160, Bottom: 160}, Confidence: 0.9} + for index := range person.Keypoints { + person.Keypoints[index] = pose.Keypoint{X: float32(index), Y: float32(index), Confidence: 0.9} + } + if horizontal { + person.Keypoints[5] = pose.Keypoint{X: 20, Y: 80, Confidence: 0.9} + person.Keypoints[6] = pose.Keypoint{X: 30, Y: 80, Confidence: 0.9} + person.Keypoints[11] = pose.Keypoint{X: 70, Y: 100, Confidence: 0.9} + person.Keypoints[12] = pose.Keypoint{X: 80, Y: 100, Confidence: 0.9} + } else { + person.Keypoints[5] = pose.Keypoint{X: 30, Y: 10, Confidence: 0.9} + person.Keypoints[6] = pose.Keypoint{X: 40, Y: 10, Confidence: 0.9} + person.Keypoints[11] = pose.Keypoint{X: 30, Y: 30, Confidence: 0.9} + person.Keypoints[12] = pose.Keypoint{X: 40, Y: 30, Confidence: 0.9} + } + return person +} diff --git a/v2/internal/fall/policy.go b/v2/internal/fall/policy.go new file mode 100644 index 0000000..b2240f9 --- /dev/null +++ b/v2/internal/fall/policy.go @@ -0,0 +1,37 @@ +package fall + +type policy struct { + suspectWindowSeconds float64 + requireRapidDrop bool + rapidDropAt map[string]float64 +} + +func newPolicy(suspectWindowSeconds float64, requireRapidDrop bool) *policy { + return &policy{ + suspectWindowSeconds: suspectWindowSeconds, + requireRapidDrop: requireRapidDrop, + rapidDropAt: make(map[string]float64), + } +} + +func (policy *policy) evaluate(trackID string, poseEvidence PoseEvidence, now float64, state State) Evidence { + if !poseEvidence.Accepted { + delete(policy.rapidDropAt, trackID) + return Evidence{} + } + if poseEvidence.RapidVerticalChange { + policy.rapidDropAt[trackID] = now + } + candidate := false + if state == Suspect { + candidate = poseEvidence.HorizontalPose + } else if poseEvidence.HorizontalPose { + if !policy.requireRapidDrop { + candidate = true + } else if droppedAt, found := policy.rapidDropAt[trackID]; found { + candidate = now-droppedAt <= policy.suspectWindowSeconds + } + } + recovery := (state == Confirmed || state == Recovering) && !poseEvidence.HorizontalPose && !poseEvidence.RapidVerticalChange + return Evidence{Accepted: true, IsFallCandidate: candidate, IsRecoveryCandidate: recovery} +} diff --git a/v2/internal/fall/state.go b/v2/internal/fall/state.go new file mode 100644 index 0000000..160d725 --- /dev/null +++ b/v2/internal/fall/state.go @@ -0,0 +1,142 @@ +package fall + +import "fmt" + +type stateRecord struct { + state State + + suspectStartedAt float64 + hasSuspectStartedAt bool + confirmedAt float64 + hasConfirmedAt bool + recoveryStartedAt float64 + hasRecoveryStartedAt bool + lastUpdatedAt float64 + hasLastUpdatedAt bool +} + +type stateMachine struct { + confirmWindowSeconds float64 + recoveryWindowSeconds float64 + cooldownSeconds float64 + configVersion string + sessionID string + records map[string]*stateRecord + nextEventNumber int +} + +func newStateMachine(config EngineConfig) (*stateMachine, error) { + if config.ConfirmWindowSeconds < 1 || config.ConfirmWindowSeconds > 3 { + return nil, fmt.Errorf("confirm window seconds must be between 1 and 3") + } + if config.RecoveryWindowSeconds <= 0 { + return nil, fmt.Errorf("recovery window seconds must be positive") + } + if config.CooldownSeconds < 0 { + return nil, fmt.Errorf("cooldown seconds must be non-negative") + } + if config.ConfigVersion == "" { + return nil, fmt.Errorf("config version must be non-empty") + } + return &stateMachine{ + confirmWindowSeconds: config.ConfirmWindowSeconds, recoveryWindowSeconds: config.RecoveryWindowSeconds, + cooldownSeconds: config.CooldownSeconds, configVersion: config.ConfigVersion, sessionID: config.SessionID, + records: make(map[string]*stateRecord), nextEventNumber: 1, + }, nil +} + +func (machine *stateMachine) stateOf(trackID string) State { + if record, found := machine.records[trackID]; found { + return record.state + } + return Normal +} + +func (machine *stateMachine) update(trackID string, evidence Evidence, now float64) ([]Event, error) { + if trackID == "" { + return nil, fmt.Errorf("track ID must be non-empty") + } + record, found := machine.records[trackID] + if !found { + record = &stateRecord{state: Normal} + machine.records[trackID] = record + } + if record.hasLastUpdatedAt && now < record.lastUpdatedAt { + return nil, fmt.Errorf("timestamps must be monotonic per track") + } + record.lastUpdatedAt, record.hasLastUpdatedAt = now, true + + if !evidence.Accepted { + machine.rejectEvidence(record) + return nil, nil + } + switch record.state { + case Normal: + if evidence.IsFallCandidate { + record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true + } + return nil, nil + case Suspect: + if !evidence.IsFallCandidate { + machine.setNormal(record) + return nil, nil + } + if now-record.suspectStartedAt >= machine.confirmWindowSeconds { + record.state, record.confirmedAt, record.hasConfirmedAt = Confirmed, now, true + event := machine.newEvent(trackID, record.suspectStartedAt, now) + record.hasSuspectStartedAt = false + return []Event{event}, nil + } + return nil, nil + case Confirmed: + if evidence.IsRecoveryCandidate && now-record.confirmedAt >= machine.cooldownSeconds { + record.state, record.recoveryStartedAt, record.hasRecoveryStartedAt = Recovering, now, true + } + return nil, nil + case Recovering: + if evidence.IsFallCandidate { + record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true + record.hasRecoveryStartedAt = false + return nil, nil + } + if !evidence.IsRecoveryCandidate { + record.state, record.hasRecoveryStartedAt = Confirmed, false + return nil, nil + } + if now-record.recoveryStartedAt >= machine.recoveryWindowSeconds { + machine.setNormal(record) + } + return nil, nil + default: + return nil, fmt.Errorf("unknown fall state") + } +} + +func (machine *stateMachine) rejectEvidence(record *stateRecord) { + if record.state == Suspect { + machine.setNormal(record) + } else if record.state == Recovering { + record.state, record.hasRecoveryStartedAt = Confirmed, false + } +} + +func (machine *stateMachine) setNormal(record *stateRecord) { + record.state = Normal + record.hasSuspectStartedAt = false + record.hasConfirmedAt = false + record.hasRecoveryStartedAt = false +} + +func (machine *stateMachine) newEvent(trackID string, suspectedAt, confirmedAt float64) Event { + prefix := "FALL-" + if machine.sessionID != "" { + prefix += machine.sessionID + "-" + } + event := Event{ + EventID: fmt.Sprintf("%s%06d", prefix, machine.nextEventNumber), TrackID: trackID, + ConfigVersion: machine.configVersion, SuspectedAtMonotonic: suspectedAt, + ConfirmedAtMonotonic: confirmedAt, LatencySeconds: confirmedAt - suspectedAt, State: Confirmed, + } + machine.nextEventNumber++ + return event +} diff --git a/v2/internal/fall/tracker.go b/v2/internal/fall/tracker.go new file mode 100644 index 0000000..73fdb4e --- /dev/null +++ b/v2/internal/fall/tracker.go @@ -0,0 +1,93 @@ +package fall + +import ( + "fmt" + "math" + "sort" + + "silverpose/v2/internal/pose" +) + +type track struct { + centerX float32 + centerY float32 + lastSeenAt float64 +} + +type tracker struct { + maxMatchDistanceRatio float64 + maxAgeSeconds float64 + tracks map[string]track + nextTrackNumber int +} + +func newTracker(maxMatchDistanceRatio, maxAgeSeconds float64) (*tracker, error) { + if maxMatchDistanceRatio <= 0 || maxMatchDistanceRatio > 1 { + return nil, fmt.Errorf("max match distance ratio must be in (0, 1]") + } + if maxAgeSeconds <= 0 { + return nil, fmt.Errorf("max age seconds must be positive") + } + return &tracker{ + maxMatchDistanceRatio: maxMatchDistanceRatio, + maxAgeSeconds: maxAgeSeconds, + tracks: make(map[string]track), + nextTrackNumber: 1, + }, nil +} + +func (tracker *tracker) update(poses []pose.PersonPose, now float64, width, height int) ([]TrackedPose, error) { + if width <= 0 || height <= 0 { + return nil, fmt.Errorf("frame size must contain positive width and height") + } + tracker.expire(now) + availableIDs := make(map[string]bool, len(tracker.tracks)) + for trackID := range tracker.tracks { + availableIDs[trackID] = true + } + tracked := make([]TrackedPose, 0, len(poses)) + for _, person := range poses { + centerX := (person.Box.Left + person.Box.Right) / 2 + centerY := (person.Box.Top + person.Box.Bottom) / 2 + trackID := tracker.nearestAvailable(centerX, centerY, availableIDs, width, height) + if trackID == "" { + trackID = fmt.Sprintf("P-%04d", tracker.nextTrackNumber) + tracker.nextTrackNumber++ + } else { + delete(availableIDs, trackID) + } + tracker.tracks[trackID] = track{centerX: centerX, centerY: centerY, lastSeenAt: now} + tracked = append(tracked, TrackedPose{TrackID: trackID, Pose: person}) + } + return tracked, nil +} + +func (tracker *tracker) nearestAvailable(centerX, centerY float32, availableIDs map[string]bool, width, height int) string { + ids := make([]string, 0, len(availableIDs)) + for trackID := range availableIDs { + ids = append(ids, trackID) + } + sort.Strings(ids) + closestID := "" + closestDistance := math.Inf(1) + for _, trackID := range ids { + previous := tracker.tracks[trackID] + distance := math.Hypot( + float64(centerX-previous.centerX)/float64(width), + float64(centerY-previous.centerY)/float64(height), + ) + if distance <= tracker.maxMatchDistanceRatio && distance < closestDistance { + closestID = trackID + closestDistance = distance + } + } + return closestID +} + +func (tracker *tracker) expire(now float64) { + for trackID, current := range tracker.tracks { + if now-current.lastSeenAt > tracker.maxAgeSeconds { + delete(tracker.tracks, trackID) + } + } +} diff --git a/v2/internal/fall/types.go b/v2/internal/fall/types.go new file mode 100644 index 0000000..7d4ec5f --- /dev/null +++ b/v2/internal/fall/types.go @@ -0,0 +1,83 @@ +package fall + +import "silverpose/v2/internal/pose" + +type State string + +const ( + Normal State = "NORMAL" + Suspect State = "SUSPECT" + Confirmed State = "CONFIRMED" + Recovering State = "RECOVERING" +) + +type Evidence struct { + Accepted bool + IsFallCandidate bool + IsRecoveryCandidate bool +} + +type PoseQuality struct { + Accepted bool + Reason string + VisibleJointCount int +} + +type PoseEvidence struct { + Accepted bool + HorizontalPose bool + RapidVerticalChange bool + HorizontalAngleDegree float32 + HasAngle bool + HipCenterY float32 + HasHipCenterY bool + TorsoLength float32 + Reason string +} + +type Event struct { + EventID string + TrackID string + ConfigVersion string + SuspectedAtMonotonic float64 + ConfirmedAtMonotonic float64 + LatencySeconds float64 + State State +} + +type Frame struct { + Timestamp float64 + Width int + Height int + Poses []pose.PersonPose +} + +type TrackedPose struct { + TrackID string + Pose pose.PersonPose +} + +type PersonAnalysis struct { + TrackedPose TrackedPose + PoseEvidence PoseEvidence + Evidence Evidence + State State +} + +type FrameResult struct { + People []PersonAnalysis + Events []Event +} + +type EngineConfig struct { + KeypointConfidenceThreshold float32 + SuspectWindowSeconds float64 + ConfirmWindowSeconds float64 + RecoveryWindowSeconds float64 + CooldownSeconds float64 + RequireRapidDrop bool + RequireLowerBody bool + HorizontalAngleThresholdDegrees float32 + ConfigVersion string + SessionID string +} diff --git a/v2/internal/pose/postprocess.go b/v2/internal/pose/postprocess.go new file mode 100644 index 0000000..c306762 --- /dev/null +++ b/v2/internal/pose/postprocess.go @@ -0,0 +1,125 @@ +package pose + +import ( + "fmt" + "sort" +) + +// ParseYOLOv8Pose turns the fixed [1,56,8400] YOLO Pose output into source +// pixel coordinates, applying confidence filtering and class-agnostic NMS. +func ParseYOLOv8Pose(output []float32, transform LetterboxTransform, confidenceThreshold, iouThreshold float32) ([]PersonPose, error) { + if len(output) != YOLOPoseOutputValues { + return nil, fmt.Errorf("YOLO Pose output length = %d, want %d", len(output), YOLOPoseOutputValues) + } + if transform.Scale <= 0 { + return nil, fmt.Errorf("letterbox scale must be positive") + } + if confidenceThreshold < 0 || confidenceThreshold > 1 || iouThreshold < 0 || iouThreshold > 1 { + return nil, fmt.Errorf("confidence and IoU thresholds must be between 0 and 1") + } + + candidates := make([]PersonPose, 0) + for candidate := 0; candidate < YOLOPoseCandidateCount; candidate++ { + confidence := valueAt(output, 4, candidate) + if confidence < confidenceThreshold { + continue + } + centerX := valueAt(output, 0, candidate) + centerY := valueAt(output, 1, candidate) + width := valueAt(output, 2, candidate) + height := valueAt(output, 3, candidate) + person := PersonPose{ + Box: Box{ + Left: transform.restoreX(centerX - width/2), + Top: transform.restoreY(centerY - height/2), + Right: transform.restoreX(centerX + width/2), + Bottom: transform.restoreY(centerY + height/2), + }, + Confidence: confidence, + } + for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ { + channel := 5 + keypoint*3 + person.Keypoints[keypoint] = Keypoint{ + X: transform.restoreX(valueAt(output, channel, candidate)), + Y: transform.restoreY(valueAt(output, channel+1, candidate)), + Confidence: valueAt(output, channel+2, candidate), + } + } + candidates = append(candidates, person) + } + + sort.SliceStable(candidates, func(left, right int) bool { + return candidates[left].Confidence > candidates[right].Confidence + }) + selected := make([]PersonPose, 0, len(candidates)) + for _, candidate := range candidates { + overlaps := false + for _, accepted := range selected { + if intersectionOverUnion(candidate.Box, accepted.Box) > iouThreshold { + overlaps = true + break + } + } + if !overlaps { + selected = append(selected, candidate) + } + } + return selected, nil +} + +func valueAt(output []float32, channel, candidate int) float32 { + return output[channel*YOLOPoseCandidateCount+candidate] +} + +func (transform LetterboxTransform) restoreX(value float32) float32 { + return clampCoordinate((value-float32(transform.PadLeft))/transform.Scale, transform.OriginalWidth) +} + +func (transform LetterboxTransform) restoreY(value float32) float32 { + return clampCoordinate((value-float32(transform.PadTop))/transform.Scale, transform.OriginalHeight) +} + +func clampCoordinate(value float32, size int) float32 { + if size <= 0 { + return value + } + if value < 0 { + return 0 + } + maximum := float32(size) + if value > maximum { + return maximum + } + return value +} + +func intersectionOverUnion(left, right Box) float32 { + intersectionLeft := maxFloat(left.Left, right.Left) + intersectionTop := maxFloat(left.Top, right.Top) + intersectionRight := minFloat(left.Right, right.Right) + intersectionBottom := minFloat(left.Bottom, right.Bottom) + intersectionWidth := maxFloat(0, intersectionRight-intersectionLeft) + intersectionHeight := maxFloat(0, intersectionBottom-intersectionTop) + intersection := intersectionWidth * intersectionHeight + leftArea := maxFloat(0, left.Right-left.Left) * maxFloat(0, left.Bottom-left.Top) + rightArea := maxFloat(0, right.Right-right.Left) * maxFloat(0, right.Bottom-right.Top) + union := leftArea + rightArea - intersection + if union <= 0 { + return 0 + } + return intersection / union +} + +func minFloat(left, right float32) float32 { + if left < right { + return left + } + return right +} + +func maxFloat(left, right float32) float32 { + if left > right { + return left + } + return right +} diff --git a/v2/internal/pose/postprocess_test.go b/v2/internal/pose/postprocess_test.go new file mode 100644 index 0000000..542557f --- /dev/null +++ b/v2/internal/pose/postprocess_test.go @@ -0,0 +1,45 @@ +package pose + +import "testing" + +func TestParseYOLOv8PoseRestoresCoordinatesAndSuppressesOverlap(t *testing.T) { + output := make([]float32, YOLOPoseOutputValues) + putDetection(output, 0, 320, 320, 100, 200, 0.90, 330, 340) + putDetection(output, 1, 322, 321, 100, 200, 0.80, 332, 341) + + people, err := ParseYOLOv8Pose(output, LetterboxTransform{Scale: 1}, 0.25, 0.70) + if err != nil { + t.Fatalf("ParseYOLOv8Pose returned an error: %v", err) + } + if len(people) != 1 { + t.Fatalf("people = %d, want one NMS survivor", len(people)) + } + person := people[0] + if person.Box != (Box{Left: 270, Top: 220, Right: 370, Bottom: 420}) { + t.Fatalf("box = %+v", person.Box) + } + if point := person.Keypoints[0]; point.X != 330 || point.Y != 340 || point.Confidence != 0.9 { + t.Fatalf("first keypoint = %+v", point) + } +} + +func TestParseYOLOv8PoseRejectsUnexpectedOutputLength(t *testing.T) { + _, err := ParseYOLOv8Pose([]float32{0}, LetterboxTransform{Scale: 1}, 0.25, 0.70) + if err == nil { + t.Fatal("ParseYOLOv8Pose accepted a malformed output") + } +} + +func putDetection(output []float32, candidate int, centerX, centerY, width, height, confidence, keypointX, keypointY float32) { + output[0*YOLOPoseCandidateCount+candidate] = centerX + output[1*YOLOPoseCandidateCount+candidate] = centerY + output[2*YOLOPoseCandidateCount+candidate] = width + output[3*YOLOPoseCandidateCount+candidate] = height + output[4*YOLOPoseCandidateCount+candidate] = confidence + for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ { + base := 5 + keypoint*3 + output[(base+0)*YOLOPoseCandidateCount+candidate] = keypointX + output[(base+1)*YOLOPoseCandidateCount+candidate] = keypointY + output[(base+2)*YOLOPoseCandidateCount+candidate] = 0.9 + } +} diff --git a/v2/internal/pose/preprocess.go b/v2/internal/pose/preprocess.go new file mode 100644 index 0000000..de8ed17 --- /dev/null +++ b/v2/internal/pose/preprocess.go @@ -0,0 +1,87 @@ +package pose + +import ( + "fmt" + "math" +) + +const letterboxPadding = byte(114) + +// PreprocessBGR reproduces the fixed-shape Ultralytics letterbox contract: +// BGR source pixels are resized with bilinear interpolation, padded in 114 +// gray, converted to RGB/CHW and normalised to [0, 1]. +func PreprocessBGR(frame []byte, width, height, target int) ([]float32, LetterboxTransform, error) { + if width <= 0 || height <= 0 || target <= 0 { + return nil, LetterboxTransform{}, fmt.Errorf("frame width, height and target must be positive") + } + if len(frame) != width*height*3 { + return nil, LetterboxTransform{}, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3) + } + + scale := math.Min(float64(target)/float64(width), float64(target)/float64(height)) + resizedWidth := int(math.Round(float64(width) * scale)) + resizedHeight := int(math.Round(float64(height) * scale)) + padWidth := target - resizedWidth + padHeight := target - resizedHeight + padLeft := int(math.Round(float64(padWidth)/2.0 - 0.1)) + padTop := int(math.Round(float64(padHeight)/2.0 - 0.1)) + transform := LetterboxTransform{ + OriginalWidth: width, OriginalHeight: height, InputSize: target, + ResizedWidth: resizedWidth, ResizedHeight: resizedHeight, + PadLeft: padLeft, PadTop: padTop, Scale: float32(scale), + } + + plane := target * target + input := make([]float32, 3*plane) + padding := float32(letterboxPadding) / 255.0 + for index := range input { + input[index] = padding + } + for y := 0; y < resizedHeight; y++ { + sourceY := clampFloat((float64(y)+0.5)*float64(height)/float64(resizedHeight)-0.5, 0, float64(height-1)) + y0 := int(math.Floor(sourceY)) + y1 := minInt(y0+1, height-1) + yWeight := float32(sourceY - float64(y0)) + for x := 0; x < resizedWidth; x++ { + sourceX := clampFloat((float64(x)+0.5)*float64(width)/float64(resizedWidth)-0.5, 0, float64(width-1)) + x0 := int(math.Floor(sourceX)) + x1 := minInt(x0+1, width-1) + xWeight := float32(sourceX - float64(x0)) + leftTop := (y0*width + x0) * 3 + rightTop := (y0*width + x1) * 3 + leftBottom := (y1*width + x0) * 3 + rightBottom := (y1*width + x1) * 3 + destination := (padTop+y)*target + padLeft + x + for sourceChannel := 0; sourceChannel < 3; sourceChannel++ { + value := bilinear( + frame[leftTop+sourceChannel], frame[rightTop+sourceChannel], + frame[leftBottom+sourceChannel], frame[rightBottom+sourceChannel], + xWeight, yWeight, + ) + // BGR source maps to RGB tensor planes. + tensorChannel := 2 - sourceChannel + // cv2.resize writes uint8 pixels before Ultralytics converts the + // image to float; round here to retain that observable contract. + input[tensorChannel*plane+destination] = float32(math.Round(float64(value))) / 255.0 + } + } + } + return input, transform, nil +} + +func bilinear(topLeft, topRight, bottomLeft, bottomRight byte, xWeight, yWeight float32) float32 { + top := float32(topLeft)*(1-xWeight) + float32(topRight)*xWeight + bottom := float32(bottomLeft)*(1-xWeight) + float32(bottomRight)*xWeight + return top*(1-yWeight) + bottom*yWeight +} + +func clampFloat(value, minimum, maximum float64) float64 { + return math.Max(minimum, math.Min(maximum, value)) +} + +func minInt(left, right int) int { + if left < right { + return left + } + return right +} diff --git a/v2/internal/pose/preprocess_test.go b/v2/internal/pose/preprocess_test.go new file mode 100644 index 0000000..0b9d600 --- /dev/null +++ b/v2/internal/pose/preprocess_test.go @@ -0,0 +1,39 @@ +package pose + +import "testing" + +func TestPreprocessBGRUsesUltralyticsPaddingAndRGBCHW(t *testing.T) { + // A 2x1 BGR frame letterboxes into a 4x4 tensor with one row of 114-gray + // padding above and below. The rightmost source pixel is pure blue in BGR. + frame := []byte{0, 0, 0, 255, 0, 0} + + input, transform, err := PreprocessBGR(frame, 2, 1, 4) + if err != nil { + t.Fatalf("PreprocessBGR returned an error: %v", err) + } + if transform.ResizedWidth != 4 || transform.ResizedHeight != 2 || transform.PadTop != 1 { + t.Fatalf("unexpected transform: %+v", transform) + } + + const plane = 16 + padding := float32(114.0 / 255.0) + if got := input[2*plane+0]; got != padding { + t.Fatalf("top padding blue = %v, want %v", got, padding) + } + if got := input[0*plane+1*4+3]; got != 0 { + t.Fatalf("red channel = %v, want 0", got) + } + if got := input[1*plane+1*4+3]; got != 0 { + t.Fatalf("green channel = %v, want 0", got) + } + if got := input[2*plane+1*4+3]; got != 1 { + t.Fatalf("blue channel = %v, want 1", got) + } +} + +func TestPreprocessBGRRejectsTruncatedFrame(t *testing.T) { + _, _, err := PreprocessBGR([]byte{0}, 2, 1, 4) + if err == nil { + t.Fatal("PreprocessBGR accepted a truncated BGR frame") + } +} diff --git a/v2/internal/pose/types.go b/v2/internal/pose/types.go new file mode 100644 index 0000000..54e56d2 --- /dev/null +++ b/v2/internal/pose/types.go @@ -0,0 +1,39 @@ +package pose + +const ( + YOLOPoseKeypointCount = 17 + YOLOPoseCandidateCount = 8400 + YOLOPoseChannelCount = 56 + YOLOPoseOutputValues = YOLOPoseChannelCount * YOLOPoseCandidateCount +) + +type Keypoint struct { + X float32 + Y float32 + Confidence float32 +} + +type Box struct { + Left float32 + Top float32 + Right float32 + Bottom float32 +} + +type PersonPose struct { + Box Box + Confidence float32 + Keypoints [YOLOPoseKeypointCount]Keypoint +} + +// LetterboxTransform maps fixed-square YOLO coordinates back to source pixels. +type LetterboxTransform struct { + OriginalWidth int + OriginalHeight int + InputSize int + ResizedWidth int + ResizedHeight int + PadLeft int + PadTop int + Scale float32 +} diff --git a/v2/internal/spike/ort.go b/v2/internal/spike/ort.go index cad8e83..de3d1a5 100644 --- a/v2/internal/spike/ort.go +++ b/v2/internal/spike/ort.go @@ -12,6 +12,90 @@ const ( poseOutputCount = 1 * 56 * 8400 ) +// Runtime owns one reusable ONNX session. V2 video replay must keep the +// model session alive across frames; recreating it for every frame would make +// event-latency measurements meaningless. +type Runtime struct { + input *ort.Tensor[float32] + output *ort.Tensor[float32] + session *ort.AdvancedSession + initialized bool + closed bool +} + +func OpenRuntime(modelPath, runtimeDLLPath string) (*Runtime, error) { + ort.SetSharedLibraryPath(runtimeDLLPath) + if err := ort.InitializeEnvironment(); err != nil { + return nil, fmt.Errorf("initialize ONNX Runtime: %w", err) + } + runtime := &Runtime{initialized: true} + var err error + runtime.input, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 3, poseInputSize, poseInputSize)) + if err != nil { + runtime.Close() + return nil, fmt.Errorf("create pose input tensor: %w", err) + } + runtime.output, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400)) + if err != nil { + runtime.Close() + return nil, fmt.Errorf("create pose output tensor: %w", err) + } + runtime.session, err = ort.NewAdvancedSession( + modelPath, + []string{"images"}, + []string{"output0"}, + []ort.Value{runtime.input}, + []ort.Value{runtime.output}, + nil, + ) + if err != nil { + runtime.Close() + return nil, fmt.Errorf("open pose ONNX session: %w", err) + } + return runtime, nil +} + +func (runtime *Runtime) Run(input []float32) ([]float32, error) { + if runtime == nil || runtime.closed { + return nil, fmt.Errorf("ONNX Runtime session is closed") + } + if len(input) != poseInputCount { + return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount) + } + copy(runtime.input.GetData(), input) + if err := runtime.session.Run(); err != nil { + return nil, fmt.Errorf("run pose ONNX session: %w", err) + } + output := runtime.output.GetData() + if len(output) != poseOutputCount { + return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount) + } + return append([]float32(nil), output...), nil +} + +func (runtime *Runtime) Close() { + if runtime == nil || runtime.closed { + return + } + if runtime.session != nil { + _ = runtime.session.Destroy() + runtime.session = nil + } + if runtime.output != nil { + _ = runtime.output.Destroy() + runtime.output = nil + } + if runtime.input != nil { + _ = runtime.input.Destroy() + runtime.input = nil + } + if runtime.initialized { + _ = ort.DestroyEnvironment() + runtime.initialized = false + } + runtime.closed = true +} + // RunPose executes the locked YOLO pose ONNX graph once using the CPU runtime. // The caller supplies absolute model and DLL paths so no camera credential or // machine-specific path is retained in V2 configuration or source. @@ -19,45 +103,10 @@ func RunPose(modelPath, runtimeDLLPath string, input []float32) ([]float32, erro if len(input) != poseInputCount { return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount) } - - ort.SetSharedLibraryPath(runtimeDLLPath) - if err := ort.InitializeEnvironment(); err != nil { - return nil, fmt.Errorf("initialize ONNX Runtime: %w", err) - } - defer func() { _ = ort.DestroyEnvironment() }() - - inputTensor, err := ort.NewTensor(ort.NewShape(1, 3, poseInputSize, poseInputSize), input) + runtime, err := OpenRuntime(modelPath, runtimeDLLPath) if err != nil { - return nil, fmt.Errorf("create pose input tensor: %w", err) + return nil, err } - defer func() { _ = inputTensor.Destroy() }() - - outputTensor, err := ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400)) - if err != nil { - return nil, fmt.Errorf("create pose output tensor: %w", err) - } - defer func() { _ = outputTensor.Destroy() }() - - session, err := ort.NewAdvancedSession( - modelPath, - []string{"images"}, - []string{"output0"}, - []ort.Value{inputTensor}, - []ort.Value{outputTensor}, - nil, - ) - if err != nil { - return nil, fmt.Errorf("open pose ONNX session: %w", err) - } - defer func() { _ = session.Destroy() }() - - if err := session.Run(); err != nil { - return nil, fmt.Errorf("run pose ONNX session: %w", err) - } - - output := outputTensor.GetData() - if len(output) != poseOutputCount { - return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount) - } - return append([]float32(nil), output...), nil + defer runtime.Close() + return runtime.Run(input) } diff --git a/v2/internal/spike/ort_test.go b/v2/internal/spike/ort_test.go index 1c87741..3af43ba 100644 --- a/v2/internal/spike/ort_test.go +++ b/v2/internal/spike/ort_test.go @@ -14,3 +14,13 @@ func TestRunPoseRejectsWrongInputLengthBeforeLoadingRuntime(t *testing.T) { t.Fatalf("RunPose error = %q, want input length error", err) } } + +func TestOpenRuntimeReportsMissingDLL(t *testing.T) { + _, err := OpenRuntime("missing.onnx", "missing.dll") + if err == nil { + t.Fatal("OpenRuntime accepted a missing ONNX Runtime DLL") + } + if !strings.Contains(err.Error(), "initialize ONNX Runtime") { + t.Fatalf("OpenRuntime error = %q", err) + } +} diff --git a/v2/internal/spike/preprocess.go b/v2/internal/spike/preprocess.go index 77431dd..52853da 100644 --- a/v2/internal/spike/preprocess.go +++ b/v2/internal/spike/preprocess.go @@ -1,43 +1,14 @@ package spike -import "fmt" +import "silverpose/v2/internal/pose" // LetterboxBGRToNCHW converts one packed BGR frame into the RGB, CHW, // float32 tensor expected by the locked 640-pixel YOLO pose ONNX model. // -// It intentionally uses nearest-neighbour scaling for this Spike. T-303 -// must replace or validate this preprocessing against the V1 implementation -// before it becomes the V2 production preprocessing path. +// T-303 aligned the implementation with the fixed-shape Ultralytics +// letterbox used by the V1/ONNX baseline. New V2 code should import the pose +// package directly to retain the returned coordinate transform. func LetterboxBGRToNCHW(frame []byte, width, height, target int) ([]float32, error) { - if width <= 0 || height <= 0 || target <= 0 { - return nil, fmt.Errorf("frame width, height and target must be positive") - } - if len(frame) != width*height*3 { - return nil, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3) - } - - scaleWidth := target - scaleHeight := height * target / width - if scaleHeight > target { - scaleHeight = target - scaleWidth = width * target / height - } - padX := (target - scaleWidth) / 2 - padY := (target - scaleHeight) / 2 - plane := target * target - input := make([]float32, 3*plane) - - for y := 0; y < scaleHeight; y++ { - sourceY := y * height / scaleHeight - for x := 0; x < scaleWidth; x++ { - sourceX := x * width / scaleWidth - source := (sourceY*width + sourceX) * 3 - destination := (padY+y)*target + padX + x - input[destination] = float32(frame[source+2]) / 255.0 - input[plane+destination] = float32(frame[source+1]) / 255.0 - input[2*plane+destination] = float32(frame[source]) / 255.0 - } - } - - return input, nil + input, _, err := pose.PreprocessBGR(frame, width, height, target) + return input, err } diff --git a/v2/internal/spike/preprocess_test.go b/v2/internal/spike/preprocess_test.go index 255fdeb..12f1784 100644 --- a/v2/internal/spike/preprocess_test.go +++ b/v2/internal/spike/preprocess_test.go @@ -16,8 +16,9 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) { if got := len(input); got != 3*plane { t.Fatalf("input length = %d, want %d", got, 3*plane) } - if got := input[2*plane+159*640+320]; got != 0 { - t.Fatalf("top padding blue channel = %v, want 0", got) + padding := float32(114.0 / 255.0) + if got := input[2*plane+159*640+320]; got != padding { + t.Fatalf("top padding blue channel = %v, want %v", got, padding) } if got := input[0*plane+320*640+320]; got != 0 { t.Fatalf("red channel = %v, want 0", got) @@ -25,8 +26,8 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) { if got := input[1*plane+320*640+320]; got != 0 { t.Fatalf("green channel = %v, want 0", got) } - if got := input[2*plane+320*640+320]; got != 1 { - t.Fatalf("blue channel = %v, want 1", got) + if got := input[2*plane+320*640+320]; got != float32(128.0/255.0) { + t.Fatalf("blue channel = %v, want cv2 INTER_LINEAR value 128/255", got) } }