feat(v2): add fall event regression engine
This commit is contained in:
@@ -24,8 +24,8 @@
|
||||
| 语言 | Go | 已定 | 面向长时间 RTSP 运行、并发控制和单一交付物。 |
|
||||
| 模型工件 | 固定输入尺寸的 ONNX Pose 模型 | 已定 | Go 不直接加载 `.pt`;模型由已验收 V1 权重导出并记录哈希。 |
|
||||
| 推理运行时 | `github.com/yalue/onnxruntime_go` v1.31.0 + ONNX Runtime CPU DLL | Spike 已验证 | Windows 上以显式 DLL 路径加载锁定的 1×3×640×640 Pose ONNX,已完成一次真实录像首帧推理;该包装器的 C API 为 26,V2 构建必须启用 CGO/MinGW,并在发布包锁定匹配 DLL 的来源与 SHA-256。 |
|
||||
| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | Spike 已验证 | 已读取 848×480 本地录像首帧;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 |
|
||||
| 图像预处理 | Go 实现的 BGR→RGB、CHW、640 letterbox | Spike 已验证,待一致性回归 | 可生成 ONNX 所需 1,228,800 个 float;当前是最小 nearest-neighbour Spike,不得直接当作生产预处理,T-303 必须与 V1 缩放、补边、NMS 和坐标还原逐项对齐。 |
|
||||
| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | 回放已验证 | Go 回归命令已完整读取 317 帧本地录像;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 |
|
||||
| 图像预处理与 Pose 解析 | Go 实现的 BGR→RGB、CHW、640 letterbox、NMS、17 点及坐标还原 | 合成契约与无报警回放已验证 | 采用 Ultralytics 的 114 补边、双线性缩放后转 uint8 的语义;Go 首帧输出两个人,与 V1 对应结果的 box 坐标差在已记录的 6 像素容差内。仍需经同意的真实正例录像完成事件级确认。 |
|
||||
| 演示 UI | `github.com/lxn/walk`(Windows 原生控件) | Spike 已验证 | 双 Tab 骨架可在 `CGO_ENABLED=0` 编译,避免 Fyne/OpenGL 首次编译的高复杂度;正式版仍须实现浅灰蓝底、白色卡片,红色仅表示确认摔倒。 |
|
||||
| 测试 | `go test ./...` + V1 回归录像 | 已定 | V2 行为必须与 V1 基线对照。 |
|
||||
|
||||
@@ -49,6 +49,7 @@
|
||||
| V1 单元测试 | `python -m pytest v1/tests -v` |
|
||||
| V2 单元测试 | `Set-Location v2; $env:CGO_ENABLED='1'; go test ./...` |
|
||||
| V2 ONNX/视频 Spike | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/spike --ffmpeg <ffmpeg.exe> --video <local-video> --width <w> --height <h> --onnx assets\best.onnx --ort-dll <onnxruntime.dll>` |
|
||||
| V2 录像事件回归 | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/regression --ffmpeg <ffmpeg.exe> --video <local-video> --width <w> --height <h> --fps <fps> --onnx assets\best.onnx --ort-dll <onnxruntime.dll>` |
|
||||
| V2 Walk UI 编译 Spike | `Set-Location v2; $env:CGO_ENABLED='0'; go build ./cmd/ui-spike` |
|
||||
|
||||
## 演进决策
|
||||
|
||||
@@ -26,7 +26,7 @@ Pose 推理适配器(person box + 17 keypoints)
|
||||
|
||||
V1 的同一数据流既可接 RTSP,也可回放本地录像。V2 复用同一配置、事件字段和录像集,但将 Pose 推理替换为 ONNX 适配器;不得改变事件语义。
|
||||
|
||||
T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块生成固定 `1×3×640×640` RGB/CHW tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。T-303 必须先补齐与 V1 一致的 letterbox、NMS、17 点解析、坐标还原、跟踪和状态机,才可以接入实时 UI。
|
||||
T-302/T-303 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块以 114 补边、双线性缩放与 RGB/CHW 生成固定 `1×3×640×640` tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,随后由 NMS、17 点解析、坐标还原、轻量跟踪、质量/几何证据、倒地策略和四态状态机生成不变的 `FallEvent`。Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。
|
||||
|
||||
## 模块职责
|
||||
|
||||
@@ -48,15 +48,15 @@ T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,G
|
||||
| 应用装配 | `v1/app.py` | 装配配置、Pose、管线线程与窗口;`FrameWorker` 只发出已判定的 `FrameAnalysis` | 事件判定、渲染细节 |
|
||||
| 回归工具 | `v1/tests/` 与 `v1/scripts/` | 回放录像、断言事件和延迟 | 生产 UI |
|
||||
|
||||
V2 对应模块将在 T-303/T-304 建立,职责预先固定如下:
|
||||
V2 对应模块的当前落实与 T-304 目标如下:
|
||||
|
||||
| 模块 | V2 目标位置 | 职责 | 不负责 |
|
||||
| --- | --- | --- | --- |
|
||||
| 解码器 | `v2/internal/video` | 启动受控 FFmpeg、提供最新 BGR 帧与显式来源状态 | Pose、事件判定、UI 绘制 |
|
||||
| ONNX Pose | `v2/internal/pose` | 校验模型/DLL 清单,完成精确预处理、推理、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 |
|
||||
| 回放解码器 | `v2/cmd/regression` | 用受控 FFmpeg 顺序读取 BGR 帧,供本机录像回归 | RTSP 重连、UI 绘制 |
|
||||
| ONNX Pose | `v2/internal/pose` | 114 letterbox、RGB/CHW、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 |
|
||||
| 事件引擎 | `v2/internal/fall` | 复现 V1 的跟踪、证据、四态状态机和 `FallEvent` | 视频解码、声音、文件 |
|
||||
| Windows UI | `v2/internal/ui` | Walk 顶部“监控/设置”Tab、渲染最新帧和已计算状态 | 直接读 RTSP、执行 ONNX 或事件规则 |
|
||||
| 应用装配 | `v2/cmd/silver-pose` | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 |
|
||||
| 应用装配 | `v2/cmd/silver-pose`(T-304) | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 |
|
||||
|
||||
## UI 导航与配置生效生命周期
|
||||
|
||||
|
||||
@@ -9,13 +9,13 @@
|
||||
- 已验证环境:Windows PowerShell;Python 3.8.10;Ultralytics 8.3.205;PyQt5 可导入。注意:当前 WSL/Linux 环境未安装 PyQt5,且其 ultralytics/torch 版本与该基线不兼容(模型前向不可用),故 GUI 可视化与真实模型端到端渲染只在 Windows 验证;本环境用 Qt-free 的 `view_model.py` 单元测试与假适配器管线冒烟覆盖视图逻辑。
|
||||
- 旧生产基线:`demo/main.py`、`demo/fall_detection_gui.py`、`demo/detect_fall.py`、`demo/best.pt`。
|
||||
- V1 代码:已建立安全配置、显式 Replay/Stream 视频源、Pose、轻量跟踪、质量/几何证据、倒地领域规则、按 ID 四态事件机及 `v1/pipeline.py` 事件管线;运行事件带非敏感 `config_version`,缺帧/低质量/断流会中断证据确认。新增 `v1/view_model.py`(Qt-free 监控视图状态与设置草稿三份隔离)、`v1/gui.py` 薄 PyQt5 双 Tab 外壳与 `v1/app.py` 装配(`FrameWorker` 只发出已判定的 `FrameAnalysis`,窗口只渲染)。新增 `v1/alerts.py`:按 `event_id` 去重,对首次 CONFIRMED 保存带标注截图、追加 JSONL 事件行,并经可注入 `AlertSink`(Windows 侧 `QtAlertSink` 提供声音与一次性弹窗)触发一次声音/弹窗。新增 `v1/camera.py`(RTSP URL 构建 + 有界连接测试抓帧)与 `config.py` 的结构化来源(host/端口/通道/账号/密码,凭证百分号编码)、`write_local_camera_source`;设置页新增摄像头连接分组、测试连接与预览,以及传输协议/连接超时/低延迟三项抓流调优(由 `config` 存值、`app` 开流前写入 `OPENCV_FFMPEG_CAPTURE_OPTIONS`)。真实海康流已用 `VideoSource(STREAM)` 连通(1920×1080)。
|
||||
- V2 代码:已创建 Go 模块和受控 Spike。`cmd/spike` 用 FFmpeg 读取本地 BGR 帧、将其预处理为 640 tensor,并用 `onnxruntime_go` 对锁定 ONNX 执行一次推理;`cmd/ui-spike` 是可编译的 Windows Walk 双 Tab 外壳。它们不含 NMS/关键点解析、跟踪、状态机、RTSP、报警或发布打包,不能作为客户演示版。
|
||||
- V2 代码:T-303 已落地 `internal/pose`(Ultralytics 114 letterbox、BGR→RGB/CHW、YOLO Pose NMS、17 点及坐标还原)与 `internal/fall`(中心点跟踪、质量/几何证据、策略、四态状态机和事件 ID);`cmd/regression` 使用 FFmpeg 顺序回放本机录像并将 ONNX Pose 接入事件引擎。`cmd/ui-spike` 仍只是可编译的 Windows Walk 双 Tab 外壳。RTSP 重连、UI 实时渲染、声音、弹窗、截图和发布打包尚未实现,不能作为客户演示版。
|
||||
- 非代码设计工件:docs/ui/silver-pose-ui-ux-spec.md、docs/ui/2026-07-20-html-prototype-plan.md、docs/ui/silver-pose-v1-prototype.html 与 docs/ui/silver-pose-v2-prototype.html 已建立。v2 HTML 是符合正式浅色 Windows 规范的当前视觉参考:浅灰蓝底、白色卡片,红色只表示确认摔倒、其弹窗和事件证据;文件名中的 v2 只表示原型设计修订,不能理解为 Go V2 实现已开始。v1 HTML 保留为历史深色对照。两者均使用顶部双 Tab、设置草稿与状态交互,且画面、事件和时间都是模拟数据,不连接真实摄像头、模型或网络,也不改变 Phase 1 任务顺序。
|
||||
- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped,覆盖配置、视频源、Pose、跟踪、证据、领域规则、状态机、管线、视图模型、报警工件、摄像头/URL 构建、抓流参数拼串、每帧诊断、灵敏度放宽、截图标注、断流重连、事件汇总、灵敏度写回和最新帧丢弃。T-302 的 `CGO_ENABLED=1 go test ./...` 通过;Go Spike 对 848×480 本地录像首帧生成 1,228,800 元素输入并获得 470,400 元素 ONNX 输出;Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些只验证管线、视图与报警工件可运行,不表示摔倒识别准确率或 Go 行为等价。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。
|
||||
- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped。T-303 的 `CGO_ENABLED=1 go test ./...` 通过。对同一 317 帧本机录像,V1 回放得到 253 个有人帧、263 人次、0 个 CONFIRMED;Go 回放得到 255 个有人帧、265 人次、0 个 CONFIRMED。人次差异来自 OpenCV/FFmpeg 解码像素不同,二者的无报警事件结论一致;Go 合成正例契约和 V1 状态机均在 1.81 秒时产生一个 CONFIRMED。Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些不表示摔倒识别准确率或完整 Go 客户演示已验收。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。
|
||||
- 模型:`demo/best.pt` 可加载为 YOLO Pose,类别 `person`,`kpt_shape=[17, 3]`;与 `D:\PythonP\fall_detection\best.pt` 哈希一致。
|
||||
- 当前标准启动:`./init.ps1`。
|
||||
- 当前标准验证:`python -m compileall -q demo`。
|
||||
- 当前 blocker:没有 T-303 的代码阻塞;但 Go 尚未验证与 V1 的预后处理、NMS、关键点、跟踪、状态机和事件延迟一致,不能把 T-302 的单帧推理表述为可演示的 V2。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。
|
||||
- 当前 blocker:T-303 缺少经同意的、可在本机回放的正例摔倒录像及其 V1 基线;因此尚不能证明同一真实正例录像在 Go 中也确认且延迟为 1–3 秒。Go/V1 无报警回放和纯领域正例契约均已通过,但不能替代录像级正例。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。
|
||||
|
||||
全局环境的 `pip check` 存在其他项目的包冲突,因此它不是 Silver Pose 的验收命令。`init.ps1` 只检查本项目实际导入的 OpenCV、NumPy、Ultralytics 与 PyQt5,并在命令非零退出时失败。
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
| --- | --- | --- |
|
||||
| `demo/` | 已有 | 旧 Python GUI、Pose 模型与样例视频;当前单帧规则不适合作为正式事件逻辑。 |
|
||||
| `v1/` | 已验收 | Python V1 正式实现、现场验收记录与发布工具。 |
|
||||
| `v2/` | Spike 已完成 | Go 模块、ONNX/视频命令行 Spike 与 Windows Walk UI 外壳;尚不是完整 V2。 |
|
||||
| `v2/` | T-303 部分完成 | Go ONNX 后处理、事件引擎和本机录像无报警回归已实现;尚不是完整 V2。 |
|
||||
| `docs/` | 已有 | Harness Coding 文档、任务和计划。 |
|
||||
| `tests/` | 不适用 | V1 测试将在 `v1/tests/` 下创建。 |
|
||||
| `artifacts/` | 待创建 | 本地事件截图和 JSONL,不提交。 |
|
||||
@@ -33,7 +33,6 @@
|
||||
## 任务状态
|
||||
|
||||
- 已完成:T-000、T-101~T-106(V1 地基与事件管线)、T-201(PyQt 双 Tab)、T-202(本地报警/截图/JSONL)、T-203(海康 RTSP 与断流恢复)、T-204(正反例录像与现场验收)、T-205(客户演示脚本与发布包)、T-206(结构化摄像头配置与连接测试)、T-207(抓流低延迟参数)、T-208(每帧诊断叠加)、T-209(放宽俯视灵敏度)、T-210(截图 ASCII 标注)、T-211(灵敏度设置页)、T-212(实时流最新帧)、T-213(截图中文标注)、T-301(ONNX 一致性)、T-302(Go 推理、视频/UI Spike)。
|
||||
- 正在进行:无。
|
||||
- 正在进行:T-303(实现 Go 事件引擎与 V1 回归对比)。
|
||||
|
||||
## 当前可运行内容
|
||||
@@ -42,6 +41,11 @@
|
||||
./init.ps1
|
||||
Set-Location demo
|
||||
python main.py
|
||||
|
||||
# 本机录像 Go 回归(需本机 ONNX Runtime DLL、FFmpeg 与已锁定 ONNX 工件)
|
||||
Set-Location ..\v2
|
||||
$env:CGO_ENABLED='1'
|
||||
go run ./cmd/regression --help
|
||||
```
|
||||
|
||||
## 已知事实和限制
|
||||
|
||||
@@ -487,3 +487,12 @@
|
||||
- 阻塞:无代码阻塞。任务验收将明确区分“提交的合成契约回归 / 本机 demo 无报警回放”与“需本机正例录像才能证实的录像级确认事件、延迟”。
|
||||
- 决策:不复制或提交客户录像;先忠实复现 V1 的纯领域合约,并让回归工具接收外部本机录像和基线 JSON。
|
||||
- 下一步:先为 Go Pose 后处理与事件状态机编写失败测试。
|
||||
|
||||
## 【2026-07-22】T-303 实现与无报警录像回归(部分完成)
|
||||
|
||||
- 状态:PARTIAL(代码与无报警回归完成;真实正例录像回归待本机素材)
|
||||
- 变更:新增 `v2/internal/pose`(114 补边、双线性缩放后 uint8 量化、BGR→RGB/CHW、YOLOv8 Pose NMS、17 点与坐标还原)、`v2/internal/fall`(归一化中心点跟踪、姿态质量/几何证据、倒地策略、四态状态机、事件 ID)和 `v2/cmd/regression`(FFmpeg 回放→ONNX Pose→事件引擎)。`spike` 的预处理现委托给同一正式实现,ONNX Runtime session 可跨帧复用。
|
||||
- 验证:Go 单元测试覆盖 letterbox 补边/通道布局、NMS 坐标还原、输入长度、事件确认、缺帧中断和持续站立不报警;`CGO_ENABLED=1 go test ./...` 通过。本机 317 帧回放:V1 为 253 个有人帧、263 人次、0 个 CONFIRMED;Go 为 255 个有人帧、265 人次、0 个 CONFIRMED。二者无报警结论一致。合成正例以同一 V1 策略/状态机合约在 1.81 秒生成一个 CONFIRMED,Go 测试同样通过。
|
||||
- 阻塞:无客户或正例录像被提交;仓库也没有经同意的正例回放素材。因此不能把合成契约或无报警录像表述为真实正例的 Go/V1 等价,也不能把 T-303 标为 DONE。
|
||||
- 决策:保持 T-303 为 DOING;回归命令接受本机路径,不把录像、截图、事件 JSONL、RTSP 地址或凭证写进仓库。解码器差异造成同一无报警录像的人次计数轻微差异,事件而非逐帧检测人次是当前验收基线。
|
||||
- 下一步:提供或指定一段经同意、无敏感信息的本机正例录像及其 V1 输出后,运行同一命令比对确认事件与 1–3 秒延迟;通过后才可完成 T-303,继而领取 T-304。
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"time"
|
||||
|
||||
"silverpose/v2/internal/fall"
|
||||
"silverpose/v2/internal/pose"
|
||||
"silverpose/v2/internal/spike"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ffmpegPath := flag.String("ffmpeg", "ffmpeg", "path to ffmpeg.exe")
|
||||
videoPath := flag.String("video", "..\\demo\\1.mp4", "local non-sensitive validation video")
|
||||
width := flag.Int("width", 848, "source video frame width")
|
||||
height := flag.Int("height", 480, "source video frame height")
|
||||
fps := flag.Float64("fps", 30, "replay timestamp frames per second")
|
||||
maxFrames := flag.Int("max-frames", 0, "maximum frames to process; zero reads the complete video")
|
||||
modelPath := flag.String("onnx", "assets\\best.onnx", "locked ONNX pose model path")
|
||||
runtimeDLLPath := flag.String("ort-dll", "", "absolute onnxruntime.dll path")
|
||||
flag.Parse()
|
||||
|
||||
if *runtimeDLLPath == "" || *width <= 0 || *height <= 0 || *fps <= 0 || *maxFrames < 0 {
|
||||
fail("--ort-dll is required; width, height and fps must be positive; max-frames cannot be negative")
|
||||
}
|
||||
engine, err := fall.NewEngine(fall.EngineConfig{
|
||||
KeypointConfidenceThreshold: 0.4,
|
||||
SuspectWindowSeconds: 0.5,
|
||||
ConfirmWindowSeconds: 1.8,
|
||||
RecoveryWindowSeconds: 2.0,
|
||||
CooldownSeconds: 10.0,
|
||||
RequireRapidDrop: false,
|
||||
RequireLowerBody: false,
|
||||
HorizontalAngleThresholdDegrees: 45.0,
|
||||
ConfigVersion: "v1-default-20260722",
|
||||
SessionID: "video-regression",
|
||||
})
|
||||
if err != nil {
|
||||
fail("create event engine: %v", err)
|
||||
}
|
||||
runtime, err := spike.OpenRuntime(*modelPath, *runtimeDLLPath)
|
||||
if err != nil {
|
||||
fail("open ONNX Pose runtime: %v", err)
|
||||
}
|
||||
defer runtime.Close()
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
command := exec.CommandContext(ctx, *ffmpegPath, "-v", "error", "-i", *videoPath, "-f", "rawvideo", "-pix_fmt", "bgr24", "-")
|
||||
stdout, err := command.StdoutPipe()
|
||||
if err != nil {
|
||||
fail("open FFmpeg output: %v", err)
|
||||
}
|
||||
if err := command.Start(); err != nil {
|
||||
fail("start FFmpeg: %v", err)
|
||||
}
|
||||
|
||||
frameBytes := *width * *height * 3
|
||||
frame := make([]byte, frameBytes)
|
||||
frameCount := 0
|
||||
personFrames := 0
|
||||
personCount := 0
|
||||
events := make([]fall.Event, 0)
|
||||
limited := false
|
||||
for *maxFrames == 0 || frameCount < *maxFrames {
|
||||
_, err := io.ReadFull(stdout, frame)
|
||||
if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
fail("read FFmpeg BGR frame: %v", err)
|
||||
}
|
||||
input, transform, err := pose.PreprocessBGR(frame, *width, *height, 640)
|
||||
if err != nil {
|
||||
fail("preprocess frame %d: %v", frameCount, err)
|
||||
}
|
||||
output, err := runtime.Run(input)
|
||||
if err != nil {
|
||||
fail("run ONNX Pose for frame %d: %v", frameCount, err)
|
||||
}
|
||||
people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70)
|
||||
if err != nil {
|
||||
fail("parse ONNX Pose for frame %d: %v", frameCount, err)
|
||||
}
|
||||
result := engine.Process(fall.Frame{Timestamp: float64(frameCount) / *fps, Width: *width, Height: *height, Poses: people})
|
||||
if len(people) > 0 {
|
||||
personFrames++
|
||||
personCount += len(people)
|
||||
}
|
||||
events = append(events, result.Events...)
|
||||
frameCount++
|
||||
}
|
||||
if *maxFrames > 0 && frameCount == *maxFrames {
|
||||
limited = true
|
||||
cancel()
|
||||
}
|
||||
if err := command.Wait(); err != nil {
|
||||
if !limited {
|
||||
fail("FFmpeg ended with an error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("REGRESSION_OK frames=%d person_frames=%d people=%d events=%d\n", frameCount, personFrames, personCount, len(events))
|
||||
for _, event := range events {
|
||||
fmt.Printf("EVENT id=%s track=%s latency=%.6f state=%s\n", event.EventID, event.TrackID, event.LatencySeconds, event.State)
|
||||
}
|
||||
}
|
||||
|
||||
func fail(format string, args ...any) {
|
||||
fmt.Fprintf(os.Stderr, "regression: "+format+"\n", args...)
|
||||
os.Exit(1)
|
||||
}
|
||||
+10
-2
@@ -8,6 +8,7 @@ import (
|
||||
"os/exec"
|
||||
"time"
|
||||
|
||||
"silverpose/v2/internal/pose"
|
||||
"silverpose/v2/internal/spike"
|
||||
)
|
||||
|
||||
@@ -41,7 +42,7 @@ func main() {
|
||||
fail("decoded BGR frame has %d bytes, want %d; verify --width/--height", len(raw), expectedBytes)
|
||||
}
|
||||
|
||||
input, err := spike.LetterboxBGRToNCHW(raw, *width, *height, 640)
|
||||
input, transform, err := pose.PreprocessBGR(raw, *width, *height, 640)
|
||||
if err != nil {
|
||||
fail("preprocess frame: %v", err)
|
||||
}
|
||||
@@ -49,6 +50,10 @@ func main() {
|
||||
if err != nil {
|
||||
fail("run ONNX Pose: %v", err)
|
||||
}
|
||||
people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70)
|
||||
if err != nil {
|
||||
fail("parse ONNX Pose: %v", err)
|
||||
}
|
||||
|
||||
max := float32(0)
|
||||
for _, value := range output {
|
||||
@@ -56,7 +61,10 @@ func main() {
|
||||
max = value
|
||||
}
|
||||
}
|
||||
fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d max=%.4f\n", *width, *height, len(input), len(output), max)
|
||||
fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d people=%d max=%.4f\n", *width, *height, len(input), len(output), len(people), max)
|
||||
for index, person := range people {
|
||||
fmt.Printf("PERSON index=%d conf=%.6f box=%.3f,%.3f,%.3f,%.3f\n", index, person.Confidence, person.Box.Left, person.Box.Top, person.Box.Right, person.Box.Bottom)
|
||||
}
|
||||
}
|
||||
|
||||
func fail(format string, args ...any) {
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package fall
|
||||
|
||||
import (
|
||||
"math"
|
||||
|
||||
"silverpose/v2/internal/pose"
|
||||
)
|
||||
|
||||
var torsoJoints = [...]int{5, 6, 11, 12}
|
||||
var lowerJoints = [...]int{13, 14, 15, 16}
|
||||
|
||||
func assessPoseQuality(person pose.PersonPose, threshold float32, requireLowerBody bool) PoseQuality {
|
||||
if threshold < 0 || threshold > 1 {
|
||||
return PoseQuality{Reason: "invalid_threshold"}
|
||||
}
|
||||
visible := 0
|
||||
for _, keypoint := range person.Keypoints {
|
||||
if keypoint.Confidence >= threshold {
|
||||
visible++
|
||||
}
|
||||
}
|
||||
for _, index := range torsoJoints {
|
||||
if person.Keypoints[index].Confidence < threshold {
|
||||
return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible}
|
||||
}
|
||||
}
|
||||
if requireLowerBody {
|
||||
for _, index := range lowerJoints {
|
||||
if person.Keypoints[index].Confidence < threshold {
|
||||
return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible}
|
||||
}
|
||||
}
|
||||
}
|
||||
return PoseQuality{Accepted: true, Reason: "accepted", VisibleJointCount: visible}
|
||||
}
|
||||
|
||||
func extractEvidence(person pose.PersonPose, quality PoseQuality, previous *PoseEvidence, horizontalAngleThreshold float32) PoseEvidence {
|
||||
if !quality.Accepted {
|
||||
return PoseEvidence{Reason: quality.Reason}
|
||||
}
|
||||
shoulderX, shoulderY := midpoint(person, 5, 6)
|
||||
hipX, hipY := midpoint(person, 11, 12)
|
||||
vectorX := hipX - shoulderX
|
||||
vectorY := hipY - shoulderY
|
||||
torsoLength := float32(math.Hypot(float64(vectorX), float64(vectorY)))
|
||||
if torsoLength == 0 {
|
||||
return PoseEvidence{HipCenterY: hipY, HasHipCenterY: true, TorsoLength: 0, Reason: "degenerate_torso"}
|
||||
}
|
||||
cosine := math.Max(-1, math.Min(1, math.Abs(float64(vectorX))/float64(torsoLength)))
|
||||
angle := float32(math.Acos(cosine) * 180 / math.Pi)
|
||||
rapid := previous != nil && previous.Accepted && previous.HasHipCenterY && hipY-previous.HipCenterY >= 0.5*torsoLength
|
||||
return PoseEvidence{
|
||||
Accepted: true, HorizontalPose: angle <= horizontalAngleThreshold,
|
||||
RapidVerticalChange: rapid, HorizontalAngleDegree: angle, HasAngle: true,
|
||||
HipCenterY: hipY, HasHipCenterY: true, TorsoLength: torsoLength, Reason: "accepted",
|
||||
}
|
||||
}
|
||||
|
||||
func midpoint(person pose.PersonPose, first, second int) (float32, float32) {
|
||||
return (person.Keypoints[first].X + person.Keypoints[second].X) / 2,
|
||||
(person.Keypoints[first].Y + person.Keypoints[second].Y) / 2
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
package fall
|
||||
|
||||
import "fmt"
|
||||
|
||||
type Engine struct {
|
||||
config EngineConfig
|
||||
tracker *tracker
|
||||
policy *policy
|
||||
stateMachine *stateMachine
|
||||
previousEvidence map[string]PoseEvidence
|
||||
activeTrackIDs map[string]bool
|
||||
}
|
||||
|
||||
func NewEngine(config EngineConfig) (*Engine, error) {
|
||||
if config.KeypointConfidenceThreshold < 0 || config.KeypointConfidenceThreshold > 1 {
|
||||
return nil, fmt.Errorf("keypoint confidence threshold must be between 0 and 1")
|
||||
}
|
||||
if config.SuspectWindowSeconds < 0 {
|
||||
return nil, fmt.Errorf("suspect window seconds must be non-negative")
|
||||
}
|
||||
if config.HorizontalAngleThresholdDegrees < 0 || config.HorizontalAngleThresholdDegrees > 90 {
|
||||
return nil, fmt.Errorf("horizontal angle threshold degrees must be between 0 and 90")
|
||||
}
|
||||
tracker, err := newTracker(0.2, 2.0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stateMachine, err := newStateMachine(config)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Engine{
|
||||
config: config, tracker: tracker, policy: newPolicy(config.SuspectWindowSeconds, config.RequireRapidDrop),
|
||||
stateMachine: stateMachine, previousEvidence: make(map[string]PoseEvidence), activeTrackIDs: make(map[string]bool),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (engine *Engine) StateOf(trackID string) State {
|
||||
return engine.stateMachine.stateOf(trackID)
|
||||
}
|
||||
|
||||
func (engine *Engine) Process(frame Frame) FrameResult {
|
||||
tracked, err := engine.tracker.update(frame.Poses, frame.Timestamp, frame.Width, frame.Height)
|
||||
if err != nil {
|
||||
return FrameResult{}
|
||||
}
|
||||
currentIDs := make(map[string]bool, len(tracked))
|
||||
for _, trackedPose := range tracked {
|
||||
currentIDs[trackedPose.TrackID] = true
|
||||
}
|
||||
events := engine.rejectMissingTracks(currentIDs, frame.Timestamp)
|
||||
people := make([]PersonAnalysis, 0, len(tracked))
|
||||
for _, trackedPose := range tracked {
|
||||
quality := assessPoseQuality(trackedPose.Pose, engine.config.KeypointConfidenceThreshold, engine.config.RequireLowerBody)
|
||||
var previous *PoseEvidence
|
||||
if candidate, found := engine.previousEvidence[trackedPose.TrackID]; found {
|
||||
previous = &candidate
|
||||
}
|
||||
poseEvidence := extractEvidence(trackedPose.Pose, quality, previous, engine.config.HorizontalAngleThresholdDegrees)
|
||||
stateBefore := engine.stateMachine.stateOf(trackedPose.TrackID)
|
||||
evidence := engine.policy.evaluate(trackedPose.TrackID, poseEvidence, frame.Timestamp, stateBefore)
|
||||
newEvents, err := engine.stateMachine.update(trackedPose.TrackID, evidence, frame.Timestamp)
|
||||
if err == nil {
|
||||
events = append(events, newEvents...)
|
||||
}
|
||||
if poseEvidence.Accepted {
|
||||
engine.previousEvidence[trackedPose.TrackID] = poseEvidence
|
||||
} else {
|
||||
delete(engine.previousEvidence, trackedPose.TrackID)
|
||||
}
|
||||
people = append(people, PersonAnalysis{
|
||||
TrackedPose: trackedPose, PoseEvidence: poseEvidence, Evidence: evidence,
|
||||
State: engine.stateMachine.stateOf(trackedPose.TrackID),
|
||||
})
|
||||
}
|
||||
engine.activeTrackIDs = currentIDs
|
||||
return FrameResult{People: people, Events: events}
|
||||
}
|
||||
|
||||
func (engine *Engine) rejectMissingTracks(currentIDs map[string]bool, now float64) []Event {
|
||||
events := make([]Event, 0)
|
||||
for trackID := range engine.activeTrackIDs {
|
||||
if currentIDs[trackID] {
|
||||
continue
|
||||
}
|
||||
delete(engine.previousEvidence, trackID)
|
||||
engine.policy.evaluate(trackID, PoseEvidence{}, now, engine.stateMachine.stateOf(trackID))
|
||||
newEvents, err := engine.stateMachine.update(trackID, Evidence{}, now)
|
||||
if err == nil {
|
||||
events = append(events, newEvents...)
|
||||
}
|
||||
}
|
||||
return events
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
package fall
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"silverpose/v2/internal/pose"
|
||||
)
|
||||
|
||||
func TestEngineConfirmsPersistentHorizontalPoseAtV1Delay(t *testing.T) {
|
||||
engine, err := NewEngine(EngineConfig{
|
||||
KeypointConfidenceThreshold: 0.4,
|
||||
SuspectWindowSeconds: 0.5,
|
||||
ConfirmWindowSeconds: 1.8,
|
||||
RecoveryWindowSeconds: 2.0,
|
||||
CooldownSeconds: 10.0,
|
||||
RequireRapidDrop: false,
|
||||
RequireLowerBody: false,
|
||||
HorizontalAngleThresholdDegrees: 45.0,
|
||||
ConfigVersion: "cfg-v1-contract",
|
||||
SessionID: "regression",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine returned an error: %v", err)
|
||||
}
|
||||
|
||||
engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
|
||||
engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
|
||||
result := engine.Process(Frame{Timestamp: 1.91, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
|
||||
|
||||
if len(result.Events) != 1 {
|
||||
t.Fatalf("events = %+v, want one confirmation", result.Events)
|
||||
}
|
||||
event := result.Events[0]
|
||||
if event.TrackID != "P-0001" || math.Abs(event.LatencySeconds-1.81) > 1e-9 || event.ConfigVersion != "cfg-v1-contract" {
|
||||
t.Fatalf("event = %+v", event)
|
||||
}
|
||||
if got := engine.StateOf("P-0001"); got != Confirmed {
|
||||
t.Fatalf("state = %s, want CONFIRMED", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineDoesNotConfirmWhenTrackIsMissing(t *testing.T) {
|
||||
engine, err := NewEngine(testConfig())
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine returned an error: %v", err)
|
||||
}
|
||||
|
||||
engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
|
||||
engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
|
||||
engine.Process(Frame{Timestamp: 0.2, Width: 180, Height: 180})
|
||||
result := engine.Process(Frame{Timestamp: 1.9, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
|
||||
|
||||
if len(result.Events) != 0 {
|
||||
t.Fatalf("events = %+v, want none after a missing track", result.Events)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineDoesNotAlarmForUprightSequence(t *testing.T) {
|
||||
engine, err := NewEngine(testConfig())
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine returned an error: %v", err)
|
||||
}
|
||||
|
||||
for _, timestamp := range []float64{0.0, 0.6, 1.2, 1.8, 2.4} {
|
||||
result := engine.Process(Frame{Timestamp: timestamp, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
|
||||
if len(result.Events) != 0 {
|
||||
t.Fatalf("timestamp %v emitted %+v for upright pose", timestamp, result.Events)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func testConfig() EngineConfig {
|
||||
return EngineConfig{
|
||||
KeypointConfidenceThreshold: 0.4,
|
||||
SuspectWindowSeconds: 0.5,
|
||||
ConfirmWindowSeconds: 1.8,
|
||||
RecoveryWindowSeconds: 2.0,
|
||||
CooldownSeconds: 10.0,
|
||||
RequireRapidDrop: false,
|
||||
RequireLowerBody: false,
|
||||
HorizontalAngleThresholdDegrees: 45.0,
|
||||
ConfigVersion: "cfg-v1-contract",
|
||||
SessionID: "regression",
|
||||
}
|
||||
}
|
||||
|
||||
func testPose(horizontal bool) pose.PersonPose {
|
||||
person := pose.PersonPose{Box: pose.Box{Left: 20, Top: 20, Right: 160, Bottom: 160}, Confidence: 0.9}
|
||||
for index := range person.Keypoints {
|
||||
person.Keypoints[index] = pose.Keypoint{X: float32(index), Y: float32(index), Confidence: 0.9}
|
||||
}
|
||||
if horizontal {
|
||||
person.Keypoints[5] = pose.Keypoint{X: 20, Y: 80, Confidence: 0.9}
|
||||
person.Keypoints[6] = pose.Keypoint{X: 30, Y: 80, Confidence: 0.9}
|
||||
person.Keypoints[11] = pose.Keypoint{X: 70, Y: 100, Confidence: 0.9}
|
||||
person.Keypoints[12] = pose.Keypoint{X: 80, Y: 100, Confidence: 0.9}
|
||||
} else {
|
||||
person.Keypoints[5] = pose.Keypoint{X: 30, Y: 10, Confidence: 0.9}
|
||||
person.Keypoints[6] = pose.Keypoint{X: 40, Y: 10, Confidence: 0.9}
|
||||
person.Keypoints[11] = pose.Keypoint{X: 30, Y: 30, Confidence: 0.9}
|
||||
person.Keypoints[12] = pose.Keypoint{X: 40, Y: 30, Confidence: 0.9}
|
||||
}
|
||||
return person
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
package fall
|
||||
|
||||
type policy struct {
|
||||
suspectWindowSeconds float64
|
||||
requireRapidDrop bool
|
||||
rapidDropAt map[string]float64
|
||||
}
|
||||
|
||||
func newPolicy(suspectWindowSeconds float64, requireRapidDrop bool) *policy {
|
||||
return &policy{
|
||||
suspectWindowSeconds: suspectWindowSeconds,
|
||||
requireRapidDrop: requireRapidDrop,
|
||||
rapidDropAt: make(map[string]float64),
|
||||
}
|
||||
}
|
||||
|
||||
func (policy *policy) evaluate(trackID string, poseEvidence PoseEvidence, now float64, state State) Evidence {
|
||||
if !poseEvidence.Accepted {
|
||||
delete(policy.rapidDropAt, trackID)
|
||||
return Evidence{}
|
||||
}
|
||||
if poseEvidence.RapidVerticalChange {
|
||||
policy.rapidDropAt[trackID] = now
|
||||
}
|
||||
candidate := false
|
||||
if state == Suspect {
|
||||
candidate = poseEvidence.HorizontalPose
|
||||
} else if poseEvidence.HorizontalPose {
|
||||
if !policy.requireRapidDrop {
|
||||
candidate = true
|
||||
} else if droppedAt, found := policy.rapidDropAt[trackID]; found {
|
||||
candidate = now-droppedAt <= policy.suspectWindowSeconds
|
||||
}
|
||||
}
|
||||
recovery := (state == Confirmed || state == Recovering) && !poseEvidence.HorizontalPose && !poseEvidence.RapidVerticalChange
|
||||
return Evidence{Accepted: true, IsFallCandidate: candidate, IsRecoveryCandidate: recovery}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package fall
|
||||
|
||||
import "fmt"
|
||||
|
||||
type stateRecord struct {
|
||||
state State
|
||||
|
||||
suspectStartedAt float64
|
||||
hasSuspectStartedAt bool
|
||||
confirmedAt float64
|
||||
hasConfirmedAt bool
|
||||
recoveryStartedAt float64
|
||||
hasRecoveryStartedAt bool
|
||||
lastUpdatedAt float64
|
||||
hasLastUpdatedAt bool
|
||||
}
|
||||
|
||||
type stateMachine struct {
|
||||
confirmWindowSeconds float64
|
||||
recoveryWindowSeconds float64
|
||||
cooldownSeconds float64
|
||||
configVersion string
|
||||
sessionID string
|
||||
records map[string]*stateRecord
|
||||
nextEventNumber int
|
||||
}
|
||||
|
||||
func newStateMachine(config EngineConfig) (*stateMachine, error) {
|
||||
if config.ConfirmWindowSeconds < 1 || config.ConfirmWindowSeconds > 3 {
|
||||
return nil, fmt.Errorf("confirm window seconds must be between 1 and 3")
|
||||
}
|
||||
if config.RecoveryWindowSeconds <= 0 {
|
||||
return nil, fmt.Errorf("recovery window seconds must be positive")
|
||||
}
|
||||
if config.CooldownSeconds < 0 {
|
||||
return nil, fmt.Errorf("cooldown seconds must be non-negative")
|
||||
}
|
||||
if config.ConfigVersion == "" {
|
||||
return nil, fmt.Errorf("config version must be non-empty")
|
||||
}
|
||||
return &stateMachine{
|
||||
confirmWindowSeconds: config.ConfirmWindowSeconds, recoveryWindowSeconds: config.RecoveryWindowSeconds,
|
||||
cooldownSeconds: config.CooldownSeconds, configVersion: config.ConfigVersion, sessionID: config.SessionID,
|
||||
records: make(map[string]*stateRecord), nextEventNumber: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (machine *stateMachine) stateOf(trackID string) State {
|
||||
if record, found := machine.records[trackID]; found {
|
||||
return record.state
|
||||
}
|
||||
return Normal
|
||||
}
|
||||
|
||||
func (machine *stateMachine) update(trackID string, evidence Evidence, now float64) ([]Event, error) {
|
||||
if trackID == "" {
|
||||
return nil, fmt.Errorf("track ID must be non-empty")
|
||||
}
|
||||
record, found := machine.records[trackID]
|
||||
if !found {
|
||||
record = &stateRecord{state: Normal}
|
||||
machine.records[trackID] = record
|
||||
}
|
||||
if record.hasLastUpdatedAt && now < record.lastUpdatedAt {
|
||||
return nil, fmt.Errorf("timestamps must be monotonic per track")
|
||||
}
|
||||
record.lastUpdatedAt, record.hasLastUpdatedAt = now, true
|
||||
|
||||
if !evidence.Accepted {
|
||||
machine.rejectEvidence(record)
|
||||
return nil, nil
|
||||
}
|
||||
switch record.state {
|
||||
case Normal:
|
||||
if evidence.IsFallCandidate {
|
||||
record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true
|
||||
}
|
||||
return nil, nil
|
||||
case Suspect:
|
||||
if !evidence.IsFallCandidate {
|
||||
machine.setNormal(record)
|
||||
return nil, nil
|
||||
}
|
||||
if now-record.suspectStartedAt >= machine.confirmWindowSeconds {
|
||||
record.state, record.confirmedAt, record.hasConfirmedAt = Confirmed, now, true
|
||||
event := machine.newEvent(trackID, record.suspectStartedAt, now)
|
||||
record.hasSuspectStartedAt = false
|
||||
return []Event{event}, nil
|
||||
}
|
||||
return nil, nil
|
||||
case Confirmed:
|
||||
if evidence.IsRecoveryCandidate && now-record.confirmedAt >= machine.cooldownSeconds {
|
||||
record.state, record.recoveryStartedAt, record.hasRecoveryStartedAt = Recovering, now, true
|
||||
}
|
||||
return nil, nil
|
||||
case Recovering:
|
||||
if evidence.IsFallCandidate {
|
||||
record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true
|
||||
record.hasRecoveryStartedAt = false
|
||||
return nil, nil
|
||||
}
|
||||
if !evidence.IsRecoveryCandidate {
|
||||
record.state, record.hasRecoveryStartedAt = Confirmed, false
|
||||
return nil, nil
|
||||
}
|
||||
if now-record.recoveryStartedAt >= machine.recoveryWindowSeconds {
|
||||
machine.setNormal(record)
|
||||
}
|
||||
return nil, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("unknown fall state")
|
||||
}
|
||||
}
|
||||
|
||||
func (machine *stateMachine) rejectEvidence(record *stateRecord) {
|
||||
if record.state == Suspect {
|
||||
machine.setNormal(record)
|
||||
} else if record.state == Recovering {
|
||||
record.state, record.hasRecoveryStartedAt = Confirmed, false
|
||||
}
|
||||
}
|
||||
|
||||
func (machine *stateMachine) setNormal(record *stateRecord) {
|
||||
record.state = Normal
|
||||
record.hasSuspectStartedAt = false
|
||||
record.hasConfirmedAt = false
|
||||
record.hasRecoveryStartedAt = false
|
||||
}
|
||||
|
||||
func (machine *stateMachine) newEvent(trackID string, suspectedAt, confirmedAt float64) Event {
|
||||
prefix := "FALL-"
|
||||
if machine.sessionID != "" {
|
||||
prefix += machine.sessionID + "-"
|
||||
}
|
||||
event := Event{
|
||||
EventID: fmt.Sprintf("%s%06d", prefix, machine.nextEventNumber), TrackID: trackID,
|
||||
ConfigVersion: machine.configVersion, SuspectedAtMonotonic: suspectedAt,
|
||||
ConfirmedAtMonotonic: confirmedAt, LatencySeconds: confirmedAt - suspectedAt, State: Confirmed,
|
||||
}
|
||||
machine.nextEventNumber++
|
||||
return event
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package fall
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"sort"
|
||||
|
||||
"silverpose/v2/internal/pose"
|
||||
)
|
||||
|
||||
type track struct {
|
||||
centerX float32
|
||||
centerY float32
|
||||
lastSeenAt float64
|
||||
}
|
||||
|
||||
type tracker struct {
|
||||
maxMatchDistanceRatio float64
|
||||
maxAgeSeconds float64
|
||||
tracks map[string]track
|
||||
nextTrackNumber int
|
||||
}
|
||||
|
||||
func newTracker(maxMatchDistanceRatio, maxAgeSeconds float64) (*tracker, error) {
|
||||
if maxMatchDistanceRatio <= 0 || maxMatchDistanceRatio > 1 {
|
||||
return nil, fmt.Errorf("max match distance ratio must be in (0, 1]")
|
||||
}
|
||||
if maxAgeSeconds <= 0 {
|
||||
return nil, fmt.Errorf("max age seconds must be positive")
|
||||
}
|
||||
return &tracker{
|
||||
maxMatchDistanceRatio: maxMatchDistanceRatio,
|
||||
maxAgeSeconds: maxAgeSeconds,
|
||||
tracks: make(map[string]track),
|
||||
nextTrackNumber: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (tracker *tracker) update(poses []pose.PersonPose, now float64, width, height int) ([]TrackedPose, error) {
|
||||
if width <= 0 || height <= 0 {
|
||||
return nil, fmt.Errorf("frame size must contain positive width and height")
|
||||
}
|
||||
tracker.expire(now)
|
||||
availableIDs := make(map[string]bool, len(tracker.tracks))
|
||||
for trackID := range tracker.tracks {
|
||||
availableIDs[trackID] = true
|
||||
}
|
||||
tracked := make([]TrackedPose, 0, len(poses))
|
||||
for _, person := range poses {
|
||||
centerX := (person.Box.Left + person.Box.Right) / 2
|
||||
centerY := (person.Box.Top + person.Box.Bottom) / 2
|
||||
trackID := tracker.nearestAvailable(centerX, centerY, availableIDs, width, height)
|
||||
if trackID == "" {
|
||||
trackID = fmt.Sprintf("P-%04d", tracker.nextTrackNumber)
|
||||
tracker.nextTrackNumber++
|
||||
} else {
|
||||
delete(availableIDs, trackID)
|
||||
}
|
||||
tracker.tracks[trackID] = track{centerX: centerX, centerY: centerY, lastSeenAt: now}
|
||||
tracked = append(tracked, TrackedPose{TrackID: trackID, Pose: person})
|
||||
}
|
||||
return tracked, nil
|
||||
}
|
||||
|
||||
func (tracker *tracker) nearestAvailable(centerX, centerY float32, availableIDs map[string]bool, width, height int) string {
|
||||
ids := make([]string, 0, len(availableIDs))
|
||||
for trackID := range availableIDs {
|
||||
ids = append(ids, trackID)
|
||||
}
|
||||
sort.Strings(ids)
|
||||
closestID := ""
|
||||
closestDistance := math.Inf(1)
|
||||
for _, trackID := range ids {
|
||||
previous := tracker.tracks[trackID]
|
||||
distance := math.Hypot(
|
||||
float64(centerX-previous.centerX)/float64(width),
|
||||
float64(centerY-previous.centerY)/float64(height),
|
||||
)
|
||||
if distance <= tracker.maxMatchDistanceRatio && distance < closestDistance {
|
||||
closestID = trackID
|
||||
closestDistance = distance
|
||||
}
|
||||
}
|
||||
return closestID
|
||||
}
|
||||
|
||||
func (tracker *tracker) expire(now float64) {
|
||||
for trackID, current := range tracker.tracks {
|
||||
if now-current.lastSeenAt > tracker.maxAgeSeconds {
|
||||
delete(tracker.tracks, trackID)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package fall
|
||||
|
||||
import "silverpose/v2/internal/pose"
|
||||
|
||||
type State string
|
||||
|
||||
const (
|
||||
Normal State = "NORMAL"
|
||||
Suspect State = "SUSPECT"
|
||||
Confirmed State = "CONFIRMED"
|
||||
Recovering State = "RECOVERING"
|
||||
)
|
||||
|
||||
type Evidence struct {
|
||||
Accepted bool
|
||||
IsFallCandidate bool
|
||||
IsRecoveryCandidate bool
|
||||
}
|
||||
|
||||
type PoseQuality struct {
|
||||
Accepted bool
|
||||
Reason string
|
||||
VisibleJointCount int
|
||||
}
|
||||
|
||||
type PoseEvidence struct {
|
||||
Accepted bool
|
||||
HorizontalPose bool
|
||||
RapidVerticalChange bool
|
||||
HorizontalAngleDegree float32
|
||||
HasAngle bool
|
||||
HipCenterY float32
|
||||
HasHipCenterY bool
|
||||
TorsoLength float32
|
||||
Reason string
|
||||
}
|
||||
|
||||
type Event struct {
|
||||
EventID string
|
||||
TrackID string
|
||||
ConfigVersion string
|
||||
SuspectedAtMonotonic float64
|
||||
ConfirmedAtMonotonic float64
|
||||
LatencySeconds float64
|
||||
State State
|
||||
}
|
||||
|
||||
type Frame struct {
|
||||
Timestamp float64
|
||||
Width int
|
||||
Height int
|
||||
Poses []pose.PersonPose
|
||||
}
|
||||
|
||||
type TrackedPose struct {
|
||||
TrackID string
|
||||
Pose pose.PersonPose
|
||||
}
|
||||
|
||||
type PersonAnalysis struct {
|
||||
TrackedPose TrackedPose
|
||||
PoseEvidence PoseEvidence
|
||||
Evidence Evidence
|
||||
State State
|
||||
}
|
||||
|
||||
type FrameResult struct {
|
||||
People []PersonAnalysis
|
||||
Events []Event
|
||||
}
|
||||
|
||||
type EngineConfig struct {
|
||||
KeypointConfidenceThreshold float32
|
||||
SuspectWindowSeconds float64
|
||||
ConfirmWindowSeconds float64
|
||||
RecoveryWindowSeconds float64
|
||||
CooldownSeconds float64
|
||||
RequireRapidDrop bool
|
||||
RequireLowerBody bool
|
||||
HorizontalAngleThresholdDegrees float32
|
||||
ConfigVersion string
|
||||
SessionID string
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package pose
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// ParseYOLOv8Pose turns the fixed [1,56,8400] YOLO Pose output into source
|
||||
// pixel coordinates, applying confidence filtering and class-agnostic NMS.
|
||||
func ParseYOLOv8Pose(output []float32, transform LetterboxTransform, confidenceThreshold, iouThreshold float32) ([]PersonPose, error) {
|
||||
if len(output) != YOLOPoseOutputValues {
|
||||
return nil, fmt.Errorf("YOLO Pose output length = %d, want %d", len(output), YOLOPoseOutputValues)
|
||||
}
|
||||
if transform.Scale <= 0 {
|
||||
return nil, fmt.Errorf("letterbox scale must be positive")
|
||||
}
|
||||
if confidenceThreshold < 0 || confidenceThreshold > 1 || iouThreshold < 0 || iouThreshold > 1 {
|
||||
return nil, fmt.Errorf("confidence and IoU thresholds must be between 0 and 1")
|
||||
}
|
||||
|
||||
candidates := make([]PersonPose, 0)
|
||||
for candidate := 0; candidate < YOLOPoseCandidateCount; candidate++ {
|
||||
confidence := valueAt(output, 4, candidate)
|
||||
if confidence < confidenceThreshold {
|
||||
continue
|
||||
}
|
||||
centerX := valueAt(output, 0, candidate)
|
||||
centerY := valueAt(output, 1, candidate)
|
||||
width := valueAt(output, 2, candidate)
|
||||
height := valueAt(output, 3, candidate)
|
||||
person := PersonPose{
|
||||
Box: Box{
|
||||
Left: transform.restoreX(centerX - width/2),
|
||||
Top: transform.restoreY(centerY - height/2),
|
||||
Right: transform.restoreX(centerX + width/2),
|
||||
Bottom: transform.restoreY(centerY + height/2),
|
||||
},
|
||||
Confidence: confidence,
|
||||
}
|
||||
for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ {
|
||||
channel := 5 + keypoint*3
|
||||
person.Keypoints[keypoint] = Keypoint{
|
||||
X: transform.restoreX(valueAt(output, channel, candidate)),
|
||||
Y: transform.restoreY(valueAt(output, channel+1, candidate)),
|
||||
Confidence: valueAt(output, channel+2, candidate),
|
||||
}
|
||||
}
|
||||
candidates = append(candidates, person)
|
||||
}
|
||||
|
||||
sort.SliceStable(candidates, func(left, right int) bool {
|
||||
return candidates[left].Confidence > candidates[right].Confidence
|
||||
})
|
||||
selected := make([]PersonPose, 0, len(candidates))
|
||||
for _, candidate := range candidates {
|
||||
overlaps := false
|
||||
for _, accepted := range selected {
|
||||
if intersectionOverUnion(candidate.Box, accepted.Box) > iouThreshold {
|
||||
overlaps = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !overlaps {
|
||||
selected = append(selected, candidate)
|
||||
}
|
||||
}
|
||||
return selected, nil
|
||||
}
|
||||
|
||||
func valueAt(output []float32, channel, candidate int) float32 {
|
||||
return output[channel*YOLOPoseCandidateCount+candidate]
|
||||
}
|
||||
|
||||
func (transform LetterboxTransform) restoreX(value float32) float32 {
|
||||
return clampCoordinate((value-float32(transform.PadLeft))/transform.Scale, transform.OriginalWidth)
|
||||
}
|
||||
|
||||
func (transform LetterboxTransform) restoreY(value float32) float32 {
|
||||
return clampCoordinate((value-float32(transform.PadTop))/transform.Scale, transform.OriginalHeight)
|
||||
}
|
||||
|
||||
func clampCoordinate(value float32, size int) float32 {
|
||||
if size <= 0 {
|
||||
return value
|
||||
}
|
||||
if value < 0 {
|
||||
return 0
|
||||
}
|
||||
maximum := float32(size)
|
||||
if value > maximum {
|
||||
return maximum
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func intersectionOverUnion(left, right Box) float32 {
|
||||
intersectionLeft := maxFloat(left.Left, right.Left)
|
||||
intersectionTop := maxFloat(left.Top, right.Top)
|
||||
intersectionRight := minFloat(left.Right, right.Right)
|
||||
intersectionBottom := minFloat(left.Bottom, right.Bottom)
|
||||
intersectionWidth := maxFloat(0, intersectionRight-intersectionLeft)
|
||||
intersectionHeight := maxFloat(0, intersectionBottom-intersectionTop)
|
||||
intersection := intersectionWidth * intersectionHeight
|
||||
leftArea := maxFloat(0, left.Right-left.Left) * maxFloat(0, left.Bottom-left.Top)
|
||||
rightArea := maxFloat(0, right.Right-right.Left) * maxFloat(0, right.Bottom-right.Top)
|
||||
union := leftArea + rightArea - intersection
|
||||
if union <= 0 {
|
||||
return 0
|
||||
}
|
||||
return intersection / union
|
||||
}
|
||||
|
||||
func minFloat(left, right float32) float32 {
|
||||
if left < right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
|
||||
func maxFloat(left, right float32) float32 {
|
||||
if left > right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package pose
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestParseYOLOv8PoseRestoresCoordinatesAndSuppressesOverlap(t *testing.T) {
|
||||
output := make([]float32, YOLOPoseOutputValues)
|
||||
putDetection(output, 0, 320, 320, 100, 200, 0.90, 330, 340)
|
||||
putDetection(output, 1, 322, 321, 100, 200, 0.80, 332, 341)
|
||||
|
||||
people, err := ParseYOLOv8Pose(output, LetterboxTransform{Scale: 1}, 0.25, 0.70)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseYOLOv8Pose returned an error: %v", err)
|
||||
}
|
||||
if len(people) != 1 {
|
||||
t.Fatalf("people = %d, want one NMS survivor", len(people))
|
||||
}
|
||||
person := people[0]
|
||||
if person.Box != (Box{Left: 270, Top: 220, Right: 370, Bottom: 420}) {
|
||||
t.Fatalf("box = %+v", person.Box)
|
||||
}
|
||||
if point := person.Keypoints[0]; point.X != 330 || point.Y != 340 || point.Confidence != 0.9 {
|
||||
t.Fatalf("first keypoint = %+v", point)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseYOLOv8PoseRejectsUnexpectedOutputLength(t *testing.T) {
|
||||
_, err := ParseYOLOv8Pose([]float32{0}, LetterboxTransform{Scale: 1}, 0.25, 0.70)
|
||||
if err == nil {
|
||||
t.Fatal("ParseYOLOv8Pose accepted a malformed output")
|
||||
}
|
||||
}
|
||||
|
||||
func putDetection(output []float32, candidate int, centerX, centerY, width, height, confidence, keypointX, keypointY float32) {
|
||||
output[0*YOLOPoseCandidateCount+candidate] = centerX
|
||||
output[1*YOLOPoseCandidateCount+candidate] = centerY
|
||||
output[2*YOLOPoseCandidateCount+candidate] = width
|
||||
output[3*YOLOPoseCandidateCount+candidate] = height
|
||||
output[4*YOLOPoseCandidateCount+candidate] = confidence
|
||||
for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ {
|
||||
base := 5 + keypoint*3
|
||||
output[(base+0)*YOLOPoseCandidateCount+candidate] = keypointX
|
||||
output[(base+1)*YOLOPoseCandidateCount+candidate] = keypointY
|
||||
output[(base+2)*YOLOPoseCandidateCount+candidate] = 0.9
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package pose
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
)
|
||||
|
||||
const letterboxPadding = byte(114)
|
||||
|
||||
// PreprocessBGR reproduces the fixed-shape Ultralytics letterbox contract:
|
||||
// BGR source pixels are resized with bilinear interpolation, padded in 114
|
||||
// gray, converted to RGB/CHW and normalised to [0, 1].
|
||||
func PreprocessBGR(frame []byte, width, height, target int) ([]float32, LetterboxTransform, error) {
|
||||
if width <= 0 || height <= 0 || target <= 0 {
|
||||
return nil, LetterboxTransform{}, fmt.Errorf("frame width, height and target must be positive")
|
||||
}
|
||||
if len(frame) != width*height*3 {
|
||||
return nil, LetterboxTransform{}, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3)
|
||||
}
|
||||
|
||||
scale := math.Min(float64(target)/float64(width), float64(target)/float64(height))
|
||||
resizedWidth := int(math.Round(float64(width) * scale))
|
||||
resizedHeight := int(math.Round(float64(height) * scale))
|
||||
padWidth := target - resizedWidth
|
||||
padHeight := target - resizedHeight
|
||||
padLeft := int(math.Round(float64(padWidth)/2.0 - 0.1))
|
||||
padTop := int(math.Round(float64(padHeight)/2.0 - 0.1))
|
||||
transform := LetterboxTransform{
|
||||
OriginalWidth: width, OriginalHeight: height, InputSize: target,
|
||||
ResizedWidth: resizedWidth, ResizedHeight: resizedHeight,
|
||||
PadLeft: padLeft, PadTop: padTop, Scale: float32(scale),
|
||||
}
|
||||
|
||||
plane := target * target
|
||||
input := make([]float32, 3*plane)
|
||||
padding := float32(letterboxPadding) / 255.0
|
||||
for index := range input {
|
||||
input[index] = padding
|
||||
}
|
||||
for y := 0; y < resizedHeight; y++ {
|
||||
sourceY := clampFloat((float64(y)+0.5)*float64(height)/float64(resizedHeight)-0.5, 0, float64(height-1))
|
||||
y0 := int(math.Floor(sourceY))
|
||||
y1 := minInt(y0+1, height-1)
|
||||
yWeight := float32(sourceY - float64(y0))
|
||||
for x := 0; x < resizedWidth; x++ {
|
||||
sourceX := clampFloat((float64(x)+0.5)*float64(width)/float64(resizedWidth)-0.5, 0, float64(width-1))
|
||||
x0 := int(math.Floor(sourceX))
|
||||
x1 := minInt(x0+1, width-1)
|
||||
xWeight := float32(sourceX - float64(x0))
|
||||
leftTop := (y0*width + x0) * 3
|
||||
rightTop := (y0*width + x1) * 3
|
||||
leftBottom := (y1*width + x0) * 3
|
||||
rightBottom := (y1*width + x1) * 3
|
||||
destination := (padTop+y)*target + padLeft + x
|
||||
for sourceChannel := 0; sourceChannel < 3; sourceChannel++ {
|
||||
value := bilinear(
|
||||
frame[leftTop+sourceChannel], frame[rightTop+sourceChannel],
|
||||
frame[leftBottom+sourceChannel], frame[rightBottom+sourceChannel],
|
||||
xWeight, yWeight,
|
||||
)
|
||||
// BGR source maps to RGB tensor planes.
|
||||
tensorChannel := 2 - sourceChannel
|
||||
// cv2.resize writes uint8 pixels before Ultralytics converts the
|
||||
// image to float; round here to retain that observable contract.
|
||||
input[tensorChannel*plane+destination] = float32(math.Round(float64(value))) / 255.0
|
||||
}
|
||||
}
|
||||
}
|
||||
return input, transform, nil
|
||||
}
|
||||
|
||||
func bilinear(topLeft, topRight, bottomLeft, bottomRight byte, xWeight, yWeight float32) float32 {
|
||||
top := float32(topLeft)*(1-xWeight) + float32(topRight)*xWeight
|
||||
bottom := float32(bottomLeft)*(1-xWeight) + float32(bottomRight)*xWeight
|
||||
return top*(1-yWeight) + bottom*yWeight
|
||||
}
|
||||
|
||||
func clampFloat(value, minimum, maximum float64) float64 {
|
||||
return math.Max(minimum, math.Min(maximum, value))
|
||||
}
|
||||
|
||||
func minInt(left, right int) int {
|
||||
if left < right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package pose
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestPreprocessBGRUsesUltralyticsPaddingAndRGBCHW(t *testing.T) {
|
||||
// A 2x1 BGR frame letterboxes into a 4x4 tensor with one row of 114-gray
|
||||
// padding above and below. The rightmost source pixel is pure blue in BGR.
|
||||
frame := []byte{0, 0, 0, 255, 0, 0}
|
||||
|
||||
input, transform, err := PreprocessBGR(frame, 2, 1, 4)
|
||||
if err != nil {
|
||||
t.Fatalf("PreprocessBGR returned an error: %v", err)
|
||||
}
|
||||
if transform.ResizedWidth != 4 || transform.ResizedHeight != 2 || transform.PadTop != 1 {
|
||||
t.Fatalf("unexpected transform: %+v", transform)
|
||||
}
|
||||
|
||||
const plane = 16
|
||||
padding := float32(114.0 / 255.0)
|
||||
if got := input[2*plane+0]; got != padding {
|
||||
t.Fatalf("top padding blue = %v, want %v", got, padding)
|
||||
}
|
||||
if got := input[0*plane+1*4+3]; got != 0 {
|
||||
t.Fatalf("red channel = %v, want 0", got)
|
||||
}
|
||||
if got := input[1*plane+1*4+3]; got != 0 {
|
||||
t.Fatalf("green channel = %v, want 0", got)
|
||||
}
|
||||
if got := input[2*plane+1*4+3]; got != 1 {
|
||||
t.Fatalf("blue channel = %v, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreprocessBGRRejectsTruncatedFrame(t *testing.T) {
|
||||
_, _, err := PreprocessBGR([]byte{0}, 2, 1, 4)
|
||||
if err == nil {
|
||||
t.Fatal("PreprocessBGR accepted a truncated BGR frame")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package pose
|
||||
|
||||
const (
|
||||
YOLOPoseKeypointCount = 17
|
||||
YOLOPoseCandidateCount = 8400
|
||||
YOLOPoseChannelCount = 56
|
||||
YOLOPoseOutputValues = YOLOPoseChannelCount * YOLOPoseCandidateCount
|
||||
)
|
||||
|
||||
type Keypoint struct {
|
||||
X float32
|
||||
Y float32
|
||||
Confidence float32
|
||||
}
|
||||
|
||||
type Box struct {
|
||||
Left float32
|
||||
Top float32
|
||||
Right float32
|
||||
Bottom float32
|
||||
}
|
||||
|
||||
type PersonPose struct {
|
||||
Box Box
|
||||
Confidence float32
|
||||
Keypoints [YOLOPoseKeypointCount]Keypoint
|
||||
}
|
||||
|
||||
// LetterboxTransform maps fixed-square YOLO coordinates back to source pixels.
|
||||
type LetterboxTransform struct {
|
||||
OriginalWidth int
|
||||
OriginalHeight int
|
||||
InputSize int
|
||||
ResizedWidth int
|
||||
ResizedHeight int
|
||||
PadLeft int
|
||||
PadTop int
|
||||
Scale float32
|
||||
}
|
||||
+88
-39
@@ -12,6 +12,90 @@ const (
|
||||
poseOutputCount = 1 * 56 * 8400
|
||||
)
|
||||
|
||||
// Runtime owns one reusable ONNX session. V2 video replay must keep the
|
||||
// model session alive across frames; recreating it for every frame would make
|
||||
// event-latency measurements meaningless.
|
||||
type Runtime struct {
|
||||
input *ort.Tensor[float32]
|
||||
output *ort.Tensor[float32]
|
||||
session *ort.AdvancedSession
|
||||
initialized bool
|
||||
closed bool
|
||||
}
|
||||
|
||||
func OpenRuntime(modelPath, runtimeDLLPath string) (*Runtime, error) {
|
||||
ort.SetSharedLibraryPath(runtimeDLLPath)
|
||||
if err := ort.InitializeEnvironment(); err != nil {
|
||||
return nil, fmt.Errorf("initialize ONNX Runtime: %w", err)
|
||||
}
|
||||
runtime := &Runtime{initialized: true}
|
||||
var err error
|
||||
runtime.input, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 3, poseInputSize, poseInputSize))
|
||||
if err != nil {
|
||||
runtime.Close()
|
||||
return nil, fmt.Errorf("create pose input tensor: %w", err)
|
||||
}
|
||||
runtime.output, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400))
|
||||
if err != nil {
|
||||
runtime.Close()
|
||||
return nil, fmt.Errorf("create pose output tensor: %w", err)
|
||||
}
|
||||
runtime.session, err = ort.NewAdvancedSession(
|
||||
modelPath,
|
||||
[]string{"images"},
|
||||
[]string{"output0"},
|
||||
[]ort.Value{runtime.input},
|
||||
[]ort.Value{runtime.output},
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
runtime.Close()
|
||||
return nil, fmt.Errorf("open pose ONNX session: %w", err)
|
||||
}
|
||||
return runtime, nil
|
||||
}
|
||||
|
||||
func (runtime *Runtime) Run(input []float32) ([]float32, error) {
|
||||
if runtime == nil || runtime.closed {
|
||||
return nil, fmt.Errorf("ONNX Runtime session is closed")
|
||||
}
|
||||
if len(input) != poseInputCount {
|
||||
return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount)
|
||||
}
|
||||
copy(runtime.input.GetData(), input)
|
||||
if err := runtime.session.Run(); err != nil {
|
||||
return nil, fmt.Errorf("run pose ONNX session: %w", err)
|
||||
}
|
||||
output := runtime.output.GetData()
|
||||
if len(output) != poseOutputCount {
|
||||
return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount)
|
||||
}
|
||||
return append([]float32(nil), output...), nil
|
||||
}
|
||||
|
||||
func (runtime *Runtime) Close() {
|
||||
if runtime == nil || runtime.closed {
|
||||
return
|
||||
}
|
||||
if runtime.session != nil {
|
||||
_ = runtime.session.Destroy()
|
||||
runtime.session = nil
|
||||
}
|
||||
if runtime.output != nil {
|
||||
_ = runtime.output.Destroy()
|
||||
runtime.output = nil
|
||||
}
|
||||
if runtime.input != nil {
|
||||
_ = runtime.input.Destroy()
|
||||
runtime.input = nil
|
||||
}
|
||||
if runtime.initialized {
|
||||
_ = ort.DestroyEnvironment()
|
||||
runtime.initialized = false
|
||||
}
|
||||
runtime.closed = true
|
||||
}
|
||||
|
||||
// RunPose executes the locked YOLO pose ONNX graph once using the CPU runtime.
|
||||
// The caller supplies absolute model and DLL paths so no camera credential or
|
||||
// machine-specific path is retained in V2 configuration or source.
|
||||
@@ -19,45 +103,10 @@ func RunPose(modelPath, runtimeDLLPath string, input []float32) ([]float32, erro
|
||||
if len(input) != poseInputCount {
|
||||
return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount)
|
||||
}
|
||||
|
||||
ort.SetSharedLibraryPath(runtimeDLLPath)
|
||||
if err := ort.InitializeEnvironment(); err != nil {
|
||||
return nil, fmt.Errorf("initialize ONNX Runtime: %w", err)
|
||||
}
|
||||
defer func() { _ = ort.DestroyEnvironment() }()
|
||||
|
||||
inputTensor, err := ort.NewTensor(ort.NewShape(1, 3, poseInputSize, poseInputSize), input)
|
||||
runtime, err := OpenRuntime(modelPath, runtimeDLLPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create pose input tensor: %w", err)
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = inputTensor.Destroy() }()
|
||||
|
||||
outputTensor, err := ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create pose output tensor: %w", err)
|
||||
}
|
||||
defer func() { _ = outputTensor.Destroy() }()
|
||||
|
||||
session, err := ort.NewAdvancedSession(
|
||||
modelPath,
|
||||
[]string{"images"},
|
||||
[]string{"output0"},
|
||||
[]ort.Value{inputTensor},
|
||||
[]ort.Value{outputTensor},
|
||||
nil,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("open pose ONNX session: %w", err)
|
||||
}
|
||||
defer func() { _ = session.Destroy() }()
|
||||
|
||||
if err := session.Run(); err != nil {
|
||||
return nil, fmt.Errorf("run pose ONNX session: %w", err)
|
||||
}
|
||||
|
||||
output := outputTensor.GetData()
|
||||
if len(output) != poseOutputCount {
|
||||
return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount)
|
||||
}
|
||||
return append([]float32(nil), output...), nil
|
||||
defer runtime.Close()
|
||||
return runtime.Run(input)
|
||||
}
|
||||
|
||||
@@ -14,3 +14,13 @@ func TestRunPoseRejectsWrongInputLengthBeforeLoadingRuntime(t *testing.T) {
|
||||
t.Fatalf("RunPose error = %q, want input length error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenRuntimeReportsMissingDLL(t *testing.T) {
|
||||
_, err := OpenRuntime("missing.onnx", "missing.dll")
|
||||
if err == nil {
|
||||
t.Fatal("OpenRuntime accepted a missing ONNX Runtime DLL")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "initialize ONNX Runtime") {
|
||||
t.Fatalf("OpenRuntime error = %q", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,43 +1,14 @@
|
||||
package spike
|
||||
|
||||
import "fmt"
|
||||
import "silverpose/v2/internal/pose"
|
||||
|
||||
// LetterboxBGRToNCHW converts one packed BGR frame into the RGB, CHW,
|
||||
// float32 tensor expected by the locked 640-pixel YOLO pose ONNX model.
|
||||
//
|
||||
// It intentionally uses nearest-neighbour scaling for this Spike. T-303
|
||||
// must replace or validate this preprocessing against the V1 implementation
|
||||
// before it becomes the V2 production preprocessing path.
|
||||
// T-303 aligned the implementation with the fixed-shape Ultralytics
|
||||
// letterbox used by the V1/ONNX baseline. New V2 code should import the pose
|
||||
// package directly to retain the returned coordinate transform.
|
||||
func LetterboxBGRToNCHW(frame []byte, width, height, target int) ([]float32, error) {
|
||||
if width <= 0 || height <= 0 || target <= 0 {
|
||||
return nil, fmt.Errorf("frame width, height and target must be positive")
|
||||
}
|
||||
if len(frame) != width*height*3 {
|
||||
return nil, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3)
|
||||
}
|
||||
|
||||
scaleWidth := target
|
||||
scaleHeight := height * target / width
|
||||
if scaleHeight > target {
|
||||
scaleHeight = target
|
||||
scaleWidth = width * target / height
|
||||
}
|
||||
padX := (target - scaleWidth) / 2
|
||||
padY := (target - scaleHeight) / 2
|
||||
plane := target * target
|
||||
input := make([]float32, 3*plane)
|
||||
|
||||
for y := 0; y < scaleHeight; y++ {
|
||||
sourceY := y * height / scaleHeight
|
||||
for x := 0; x < scaleWidth; x++ {
|
||||
sourceX := x * width / scaleWidth
|
||||
source := (sourceY*width + sourceX) * 3
|
||||
destination := (padY+y)*target + padX + x
|
||||
input[destination] = float32(frame[source+2]) / 255.0
|
||||
input[plane+destination] = float32(frame[source+1]) / 255.0
|
||||
input[2*plane+destination] = float32(frame[source]) / 255.0
|
||||
}
|
||||
}
|
||||
|
||||
return input, nil
|
||||
input, _, err := pose.PreprocessBGR(frame, width, height, target)
|
||||
return input, err
|
||||
}
|
||||
|
||||
@@ -16,8 +16,9 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) {
|
||||
if got := len(input); got != 3*plane {
|
||||
t.Fatalf("input length = %d, want %d", got, 3*plane)
|
||||
}
|
||||
if got := input[2*plane+159*640+320]; got != 0 {
|
||||
t.Fatalf("top padding blue channel = %v, want 0", got)
|
||||
padding := float32(114.0 / 255.0)
|
||||
if got := input[2*plane+159*640+320]; got != padding {
|
||||
t.Fatalf("top padding blue channel = %v, want %v", got, padding)
|
||||
}
|
||||
if got := input[0*plane+320*640+320]; got != 0 {
|
||||
t.Fatalf("red channel = %v, want 0", got)
|
||||
@@ -25,8 +26,8 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) {
|
||||
if got := input[1*plane+320*640+320]; got != 0 {
|
||||
t.Fatalf("green channel = %v, want 0", got)
|
||||
}
|
||||
if got := input[2*plane+320*640+320]; got != 1 {
|
||||
t.Fatalf("blue channel = %v, want 1", got)
|
||||
if got := input[2*plane+320*640+320]; got != float32(128.0/255.0) {
|
||||
t.Fatalf("blue channel = %v, want cv2 INTER_LINEAR value 128/255", got)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user