feat(v2): add fall event regression engine

This commit is contained in:
ila
2026-07-22 16:18:16 +08:00
parent 258fa49259
commit 1298369054
22 changed files with 1214 additions and 92 deletions
+3 -2
View File
@@ -24,8 +24,8 @@
| 语言 | Go | 已定 | 面向长时间 RTSP 运行、并发控制和单一交付物。 |
| 模型工件 | 固定输入尺寸的 ONNX Pose 模型 | 已定 | Go 不直接加载 `.pt`;模型由已验收 V1 权重导出并记录哈希。 |
| 推理运行时 | `github.com/yalue/onnxruntime_go` v1.31.0 + ONNX Runtime CPU DLL | Spike 已验证 | Windows 上以显式 DLL 路径加载锁定的 1×3×640×640 Pose ONNX,已完成一次真实录像首帧推理;该包装器的 C API 为 26,V2 构建必须启用 CGO/MinGW,并在发布包锁定匹配 DLL 的来源与 SHA-256。 |
| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | Spike 已验证 | 已读取 848×480 本地录像首帧;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 |
| 图像预处理 | Go 实现的 BGR→RGB、CHW、640 letterbox | Spike 已验证,待一致性回归 | 可生成 ONNX 所需 1,228,800 个 float;当前是最小 nearest-neighbour Spike,不得直接当作生产预处理,T-303 必须与 V1 缩放、补边、NMS 和坐标还原逐项对齐。 |
| 视频输入 | 受控的 `ffmpeg.exe` 子进程输出 `bgr24` 原始帧 | 回放已验证 | Go 回归命令已完整读取 317 帧本地录像;避免未配置的 GoCV/OpenCV 开发绑定。T-304 必须随应用锁定 FFmpeg/FFprobe 的版本、哈希与 RTSP 重连行为。 |
| 图像预处理与 Pose 解析 | Go 实现的 BGR→RGB、CHW、640 letterbox、NMS、17 点及坐标还原 | 合成契约与无报警回放已验证 | 采用 Ultralytics 的 114 补边、双线性缩放后转 uint8 的语义;Go 首帧输出两个人,与 V1 对应结果的 box 坐标差在已记录的 6 像素容差内。仍需经同意的真实正例录像完成事件级确认。 |
| 演示 UI | `github.com/lxn/walk`(Windows 原生控件) | Spike 已验证 | 双 Tab 骨架可在 `CGO_ENABLED=0` 编译,避免 Fyne/OpenGL 首次编译的高复杂度;正式版仍须实现浅灰蓝底、白色卡片,红色仅表示确认摔倒。 |
| 测试 | `go test ./...` + V1 回归录像 | 已定 | V2 行为必须与 V1 基线对照。 |
@@ -49,6 +49,7 @@
| V1 单元测试 | `python -m pytest v1/tests -v` |
| V2 单元测试 | `Set-Location v2; $env:CGO_ENABLED='1'; go test ./...` |
| V2 ONNX/视频 Spike | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/spike --ffmpeg <ffmpeg.exe> --video <local-video> --width <w> --height <h> --onnx assets\best.onnx --ort-dll <onnxruntime.dll>` |
| V2 录像事件回归 | `Set-Location v2; $env:CGO_ENABLED='1'; go run ./cmd/regression --ffmpeg <ffmpeg.exe> --video <local-video> --width <w> --height <h> --fps <fps> --onnx assets\best.onnx --ort-dll <onnxruntime.dll>` |
| V2 Walk UI 编译 Spike | `Set-Location v2; $env:CGO_ENABLED='0'; go build ./cmd/ui-spike` |
## 演进决策
+5 -5
View File
@@ -26,7 +26,7 @@ Pose 推理适配器(person box + 17 keypoints)
V1 的同一数据流既可接 RTSP,也可回放本地录像。V2 复用同一配置、事件字段和录像集,但将 Pose 推理替换为 ONNX 适配器;不得改变事件语义。
T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块生成固定 `1×3×640×640` RGB/CHW tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。T-303 必须先补齐与 V1 一致的 letterbox、NMS、17 点解析、坐标还原、跟踪和状态机,才可以接入实时 UI。
T-302/T-303 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,Go 预处理模块以 114 补边、双线性缩放与 RGB/CHW 生成固定 `1×3×640×640` tensor,`onnxruntime_go` 以显式 DLL 路径执行 Pose ONNX,随后由 NMS、17 点解析、坐标还原、轻量跟踪、质量/几何证据、倒地策略和四态状态机生成不变的 `FallEvent`。Windows Walk UI 只接收已完成渲染的数据。解码、推理和事件计算都不得在 UI 消息线程运行;UI 只显示最新完成帧和事件状态。
## 模块职责
@@ -48,15 +48,15 @@ T-302 的实现路线为:受控 `ffmpeg.exe` 负责解码为 BGR 原始帧,G
| 应用装配 | `v1/app.py` | 装配配置、Pose、管线线程与窗口;`FrameWorker` 只发出已判定的 `FrameAnalysis` | 事件判定、渲染细节 |
| 回归工具 | `v1/tests/` 与 `v1/scripts/` | 回放录像、断言事件和延迟 | 生产 UI |
V2 对应模块将在 T-303/T-304 建立,职责预先固定如下:
V2 对应模块的当前落实与 T-304 目标如下:
| 模块 | V2 目标位置 | 职责 | 不负责 |
| --- | --- | --- | --- |
| 解码器 | `v2/internal/video` | 启动受控 FFmpeg、提供最新 BGR 帧与显式来源状态 | Pose、事件判定、UI 绘制 |
| ONNX Pose | `v2/internal/pose` | 校验模型/DLL 清单,完成精确预处理、推理、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 |
| 回放解码器 | `v2/cmd/regression` | 用受控 FFmpeg 顺序读取 BGR 帧,供本机录像回归 | RTSP 重连、UI 绘制 |
| ONNX Pose | `v2/internal/pose` | 114 letterbox、RGB/CHW、NMS、关键点及坐标还原 | 人员 ID、摔倒结论 |
| 事件引擎 | `v2/internal/fall` | 复现 V1 的跟踪、证据、四态状态机和 `FallEvent` | 视频解码、声音、文件 |
| Windows UI | `v2/internal/ui` | Walk 顶部“监控/设置”Tab、渲染最新帧和已计算状态 | 直接读 RTSP、执行 ONNX 或事件规则 |
| 应用装配 | `v2/cmd/silver-pose` | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 |
| 应用装配 | `v2/cmd/silver-pose`(T-304) | 管理 worker 生命周期、取消、最新帧投递和依赖注入 | 重写领域规则 |
## UI 导航与配置生效生命周期
+9 -5
View File
@@ -9,13 +9,13 @@
- 已验证环境:Windows PowerShell;Python 3.8.10;Ultralytics 8.3.205;PyQt5 可导入。注意:当前 WSL/Linux 环境未安装 PyQt5,且其 ultralytics/torch 版本与该基线不兼容(模型前向不可用),故 GUI 可视化与真实模型端到端渲染只在 Windows 验证;本环境用 Qt-free 的 `view_model.py` 单元测试与假适配器管线冒烟覆盖视图逻辑。
- 旧生产基线:`demo/main.py`、`demo/fall_detection_gui.py`、`demo/detect_fall.py`、`demo/best.pt`。
- V1 代码:已建立安全配置、显式 Replay/Stream 视频源、Pose、轻量跟踪、质量/几何证据、倒地领域规则、按 ID 四态事件机及 `v1/pipeline.py` 事件管线;运行事件带非敏感 `config_version`,缺帧/低质量/断流会中断证据确认。新增 `v1/view_model.py`(Qt-free 监控视图状态与设置草稿三份隔离)、`v1/gui.py` 薄 PyQt5 双 Tab 外壳与 `v1/app.py` 装配(`FrameWorker` 只发出已判定的 `FrameAnalysis`,窗口只渲染)。新增 `v1/alerts.py`:按 `event_id` 去重,对首次 CONFIRMED 保存带标注截图、追加 JSONL 事件行,并经可注入 `AlertSink`(Windows 侧 `QtAlertSink` 提供声音与一次性弹窗)触发一次声音/弹窗。新增 `v1/camera.py`(RTSP URL 构建 + 有界连接测试抓帧)与 `config.py` 的结构化来源(host/端口/通道/账号/密码,凭证百分号编码)、`write_local_camera_source`;设置页新增摄像头连接分组、测试连接与预览,以及传输协议/连接超时/低延迟三项抓流调优(由 `config` 存值、`app` 开流前写入 `OPENCV_FFMPEG_CAPTURE_OPTIONS`)。真实海康流已用 `VideoSource(STREAM)` 连通(1920×1080)。
- V2 代码:已创建 Go 模块和受控 Spike。`cmd/spike` 用 FFmpeg 读取本地 BGR 帧、将其预处理为 640 tensor,并用 `onnxruntime_go` 对锁定 ONNX 执行一次推理;`cmd/ui-spike` 是可编译的 Windows Walk 双 Tab 外壳。它们不含 NMS/关键点解析、跟踪、状态机、RTSP、报警或发布打包,不能作为客户演示版。
- V2 代码:T-303 已落地 `internal/pose`(Ultralytics 114 letterbox、BGR→RGB/CHW、YOLO Pose NMS、17 点及坐标还原)与 `internal/fall`(中心点跟踪、质量/几何证据、策略、四态状态机和事件 ID);`cmd/regression` 使用 FFmpeg 顺序回放本机录像并将 ONNX Pose 接入事件引擎。`cmd/ui-spike` 仍只是可编译的 Windows Walk 双 Tab 外壳。RTSP 重连、UI 实时渲染、声音、弹窗、截图和发布打包尚未实现,不能作为客户演示版。
- 非代码设计工件:docs/ui/silver-pose-ui-ux-spec.md、docs/ui/2026-07-20-html-prototype-plan.md、docs/ui/silver-pose-v1-prototype.html 与 docs/ui/silver-pose-v2-prototype.html 已建立。v2 HTML 是符合正式浅色 Windows 规范的当前视觉参考:浅灰蓝底、白色卡片,红色只表示确认摔倒、其弹窗和事件证据;文件名中的 v2 只表示原型设计修订,不能理解为 Go V2 实现已开始。v1 HTML 保留为历史深色对照。两者均使用顶部双 Tab、设置草稿与状态交互,且画面、事件和时间都是模拟数据,不连接真实摄像头、模型或网络,也不改变 Phase 1 任务顺序。
- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped,覆盖配置、视频源、Pose、跟踪、证据、领域规则、状态机、管线、视图模型、报警工件、摄像头/URL 构建、抓流参数拼串、每帧诊断、灵敏度放宽、截图标注、断流重连、事件汇总、灵敏度写回和最新帧丢弃。T-302 的 `CGO_ENABLED=1 go test ./...` 通过;Go Spike 对 848×480 本地录像首帧生成 1,228,800 元素输入并获得 470,400 元素 ONNX 输出;Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些只验证管线、视图与报警工件可运行,不表示摔倒识别准确率或 Go 行为等价。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。
- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`./init.ps1` 当前运行 V1 测试为 88 passed、1 skipped。T-303 的 `CGO_ENABLED=1 go test ./...` 通过。对同一 317 帧本机录像,V1 回放得到 253 个有人帧、263 人次、0 个 CONFIRMED;Go 回放得到 255 个有人帧、265 人次、0 个 CONFIRMED。人次差异来自 OpenCV/FFmpeg 解码像素不同,二者的无报警事件结论一致;Go 合成正例契约和 V1 状态机均在 1.81 秒时产生一个 CONFIRMED。Walk UI Spike 以 `CGO_ENABLED=0 go build ./cmd/ui-spike` 编译通过。这些不表示摔倒识别准确率或完整 Go 客户演示已验收。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。
- 模型:`demo/best.pt` 可加载为 YOLO Pose,类别 `person`,`kpt_shape=[17, 3]`;与 `D:\PythonP\fall_detection\best.pt` 哈希一致。
- 当前标准启动:`./init.ps1`。
- 当前标准验证:`python -m compileall -q demo`。
- 当前 blocker:没有 T-303 的代码阻塞;但 Go 尚未验证与 V1 的预后处理、NMS、关键点、跟踪、状态机和事件延迟一致,不能把 T-302 的单帧推理表述为可演示的 V2。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。
- 当前 blocker:T-303 缺少经同意的、可在本机回放的正例摔倒录像及其 V1 基线;因此尚不能证明同一真实正例录像在 Go 中也确认且延迟为 1–3 秒。Go/V1 无报警回放和纯领域正例契约均已通过,但不能替代录像级正例。ONNX Runtime DLL、FFmpeg/FFprobe 仍须在 T-304 以版本、来源和哈希纳入发布包;现场录像和事件工件保持本机不提交。
全局环境的 `pip check` 存在其他项目的包冲突,因此它不是 Silver Pose 的验收命令。`init.ps1` 只检查本项目实际导入的 OpenCV、NumPy、Ultralytics 与 PyQt5,并在命令非零退出时失败。
@@ -25,7 +25,7 @@
| --- | --- | --- |
| `demo/` | 已有 | 旧 Python GUI、Pose 模型与样例视频;当前单帧规则不适合作为正式事件逻辑。 |
| `v1/` | 已验收 | Python V1 正式实现、现场验收记录与发布工具。 |
| `v2/` | Spike 已完成 | Go 模块、ONNX/视频命令行 Spike 与 Windows Walk UI 外壳;尚不是完整 V2。 |
| `v2/` | T-303 部分完成 | Go ONNX 后处理、事件引擎和本机录像无报警回归已实现;尚不是完整 V2。 |
| `docs/` | 已有 | Harness Coding 文档、任务和计划。 |
| `tests/` | 不适用 | V1 测试将在 `v1/tests/` 下创建。 |
| `artifacts/` | 待创建 | 本地事件截图和 JSONL,不提交。 |
@@ -33,7 +33,6 @@
## 任务状态
- 已完成:T-000、T-101~T-106(V1 地基与事件管线)、T-201(PyQt 双 Tab)、T-202(本地报警/截图/JSONL)、T-203(海康 RTSP 与断流恢复)、T-204(正反例录像与现场验收)、T-205(客户演示脚本与发布包)、T-206(结构化摄像头配置与连接测试)、T-207(抓流低延迟参数)、T-208(每帧诊断叠加)、T-209(放宽俯视灵敏度)、T-210(截图 ASCII 标注)、T-211(灵敏度设置页)、T-212(实时流最新帧)、T-213(截图中文标注)、T-301(ONNX 一致性)、T-302(Go 推理、视频/UI Spike)。
- 正在进行:无。
- 正在进行:T-303(实现 Go 事件引擎与 V1 回归对比)。
## 当前可运行内容
@@ -42,6 +41,11 @@
./init.ps1
Set-Location demo
python main.py
# 本机录像 Go 回归(需本机 ONNX Runtime DLL、FFmpeg 与已锁定 ONNX 工件)
Set-Location ..\v2
$env:CGO_ENABLED='1'
go run ./cmd/regression --help
```
## 已知事实和限制
+9
View File
@@ -487,3 +487,12 @@
- 阻塞:无代码阻塞。任务验收将明确区分“提交的合成契约回归 / 本机 demo 无报警回放”与“需本机正例录像才能证实的录像级确认事件、延迟”。
- 决策:不复制或提交客户录像;先忠实复现 V1 的纯领域合约,并让回归工具接收外部本机录像和基线 JSON。
- 下一步:先为 Go Pose 后处理与事件状态机编写失败测试。
## 【2026-07-22】T-303 实现与无报警录像回归(部分完成)
- 状态:PARTIAL(代码与无报警回归完成;真实正例录像回归待本机素材)
- 变更:新增 `v2/internal/pose`(114 补边、双线性缩放后 uint8 量化、BGR→RGB/CHW、YOLOv8 Pose NMS、17 点与坐标还原)、`v2/internal/fall`(归一化中心点跟踪、姿态质量/几何证据、倒地策略、四态状态机、事件 ID)和 `v2/cmd/regression`(FFmpeg 回放→ONNX Pose→事件引擎)。`spike` 的预处理现委托给同一正式实现,ONNX Runtime session 可跨帧复用。
- 验证:Go 单元测试覆盖 letterbox 补边/通道布局、NMS 坐标还原、输入长度、事件确认、缺帧中断和持续站立不报警;`CGO_ENABLED=1 go test ./...` 通过。本机 317 帧回放:V1 为 253 个有人帧、263 人次、0 个 CONFIRMED;Go 为 255 个有人帧、265 人次、0 个 CONFIRMED。二者无报警结论一致。合成正例以同一 V1 策略/状态机合约在 1.81 秒生成一个 CONFIRMED,Go 测试同样通过。
- 阻塞:无客户或正例录像被提交;仓库也没有经同意的正例回放素材。因此不能把合成契约或无报警录像表述为真实正例的 Go/V1 等价,也不能把 T-303 标为 DONE。
- 决策:保持 T-303 为 DOING;回归命令接受本机路径,不把录像、截图、事件 JSONL、RTSP 地址或凭证写进仓库。解码器差异造成同一无报警录像的人次计数轻微差异,事件而非逐帧检测人次是当前验收基线。
- 下一步:提供或指定一段经同意、无敏感信息的本机正例录像及其 V1 输出后,运行同一命令比对确认事件与 1–3 秒延迟;通过后才可完成 T-303,继而领取 T-304。
+118
View File
@@ -0,0 +1,118 @@
package main
import (
"context"
"errors"
"flag"
"fmt"
"io"
"os"
"os/exec"
"time"
"silverpose/v2/internal/fall"
"silverpose/v2/internal/pose"
"silverpose/v2/internal/spike"
)
func main() {
ffmpegPath := flag.String("ffmpeg", "ffmpeg", "path to ffmpeg.exe")
videoPath := flag.String("video", "..\\demo\\1.mp4", "local non-sensitive validation video")
width := flag.Int("width", 848, "source video frame width")
height := flag.Int("height", 480, "source video frame height")
fps := flag.Float64("fps", 30, "replay timestamp frames per second")
maxFrames := flag.Int("max-frames", 0, "maximum frames to process; zero reads the complete video")
modelPath := flag.String("onnx", "assets\\best.onnx", "locked ONNX pose model path")
runtimeDLLPath := flag.String("ort-dll", "", "absolute onnxruntime.dll path")
flag.Parse()
if *runtimeDLLPath == "" || *width <= 0 || *height <= 0 || *fps <= 0 || *maxFrames < 0 {
fail("--ort-dll is required; width, height and fps must be positive; max-frames cannot be negative")
}
engine, err := fall.NewEngine(fall.EngineConfig{
KeypointConfidenceThreshold: 0.4,
SuspectWindowSeconds: 0.5,
ConfirmWindowSeconds: 1.8,
RecoveryWindowSeconds: 2.0,
CooldownSeconds: 10.0,
RequireRapidDrop: false,
RequireLowerBody: false,
HorizontalAngleThresholdDegrees: 45.0,
ConfigVersion: "v1-default-20260722",
SessionID: "video-regression",
})
if err != nil {
fail("create event engine: %v", err)
}
runtime, err := spike.OpenRuntime(*modelPath, *runtimeDLLPath)
if err != nil {
fail("open ONNX Pose runtime: %v", err)
}
defer runtime.Close()
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
command := exec.CommandContext(ctx, *ffmpegPath, "-v", "error", "-i", *videoPath, "-f", "rawvideo", "-pix_fmt", "bgr24", "-")
stdout, err := command.StdoutPipe()
if err != nil {
fail("open FFmpeg output: %v", err)
}
if err := command.Start(); err != nil {
fail("start FFmpeg: %v", err)
}
frameBytes := *width * *height * 3
frame := make([]byte, frameBytes)
frameCount := 0
personFrames := 0
personCount := 0
events := make([]fall.Event, 0)
limited := false
for *maxFrames == 0 || frameCount < *maxFrames {
_, err := io.ReadFull(stdout, frame)
if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
break
}
if err != nil {
fail("read FFmpeg BGR frame: %v", err)
}
input, transform, err := pose.PreprocessBGR(frame, *width, *height, 640)
if err != nil {
fail("preprocess frame %d: %v", frameCount, err)
}
output, err := runtime.Run(input)
if err != nil {
fail("run ONNX Pose for frame %d: %v", frameCount, err)
}
people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70)
if err != nil {
fail("parse ONNX Pose for frame %d: %v", frameCount, err)
}
result := engine.Process(fall.Frame{Timestamp: float64(frameCount) / *fps, Width: *width, Height: *height, Poses: people})
if len(people) > 0 {
personFrames++
personCount += len(people)
}
events = append(events, result.Events...)
frameCount++
}
if *maxFrames > 0 && frameCount == *maxFrames {
limited = true
cancel()
}
if err := command.Wait(); err != nil {
if !limited {
fail("FFmpeg ended with an error: %v", err)
}
}
fmt.Printf("REGRESSION_OK frames=%d person_frames=%d people=%d events=%d\n", frameCount, personFrames, personCount, len(events))
for _, event := range events {
fmt.Printf("EVENT id=%s track=%s latency=%.6f state=%s\n", event.EventID, event.TrackID, event.LatencySeconds, event.State)
}
}
func fail(format string, args ...any) {
fmt.Fprintf(os.Stderr, "regression: "+format+"\n", args...)
os.Exit(1)
}
+10 -2
View File
@@ -8,6 +8,7 @@ import (
"os/exec"
"time"
"silverpose/v2/internal/pose"
"silverpose/v2/internal/spike"
)
@@ -41,7 +42,7 @@ func main() {
fail("decoded BGR frame has %d bytes, want %d; verify --width/--height", len(raw), expectedBytes)
}
input, err := spike.LetterboxBGRToNCHW(raw, *width, *height, 640)
input, transform, err := pose.PreprocessBGR(raw, *width, *height, 640)
if err != nil {
fail("preprocess frame: %v", err)
}
@@ -49,6 +50,10 @@ func main() {
if err != nil {
fail("run ONNX Pose: %v", err)
}
people, err := pose.ParseYOLOv8Pose(output, transform, 0.25, 0.70)
if err != nil {
fail("parse ONNX Pose: %v", err)
}
max := float32(0)
for _, value := range output {
@@ -56,7 +61,10 @@ func main() {
max = value
}
}
fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d max=%.4f\n", *width, *height, len(input), len(output), max)
fmt.Printf("SPIKE_OK frame=%dx%d input=%d output=%d people=%d max=%.4f\n", *width, *height, len(input), len(output), len(people), max)
for index, person := range people {
fmt.Printf("PERSON index=%d conf=%.6f box=%.3f,%.3f,%.3f,%.3f\n", index, person.Confidence, person.Box.Left, person.Box.Top, person.Box.Right, person.Box.Bottom)
}
}
func fail(format string, args ...any) {
+62
View File
@@ -0,0 +1,62 @@
package fall
import (
"math"
"silverpose/v2/internal/pose"
)
var torsoJoints = [...]int{5, 6, 11, 12}
var lowerJoints = [...]int{13, 14, 15, 16}
func assessPoseQuality(person pose.PersonPose, threshold float32, requireLowerBody bool) PoseQuality {
if threshold < 0 || threshold > 1 {
return PoseQuality{Reason: "invalid_threshold"}
}
visible := 0
for _, keypoint := range person.Keypoints {
if keypoint.Confidence >= threshold {
visible++
}
}
for _, index := range torsoJoints {
if person.Keypoints[index].Confidence < threshold {
return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible}
}
}
if requireLowerBody {
for _, index := range lowerJoints {
if person.Keypoints[index].Confidence < threshold {
return PoseQuality{Reason: "required_joint_low_confidence", VisibleJointCount: visible}
}
}
}
return PoseQuality{Accepted: true, Reason: "accepted", VisibleJointCount: visible}
}
func extractEvidence(person pose.PersonPose, quality PoseQuality, previous *PoseEvidence, horizontalAngleThreshold float32) PoseEvidence {
if !quality.Accepted {
return PoseEvidence{Reason: quality.Reason}
}
shoulderX, shoulderY := midpoint(person, 5, 6)
hipX, hipY := midpoint(person, 11, 12)
vectorX := hipX - shoulderX
vectorY := hipY - shoulderY
torsoLength := float32(math.Hypot(float64(vectorX), float64(vectorY)))
if torsoLength == 0 {
return PoseEvidence{HipCenterY: hipY, HasHipCenterY: true, TorsoLength: 0, Reason: "degenerate_torso"}
}
cosine := math.Max(-1, math.Min(1, math.Abs(float64(vectorX))/float64(torsoLength)))
angle := float32(math.Acos(cosine) * 180 / math.Pi)
rapid := previous != nil && previous.Accepted && previous.HasHipCenterY && hipY-previous.HipCenterY >= 0.5*torsoLength
return PoseEvidence{
Accepted: true, HorizontalPose: angle <= horizontalAngleThreshold,
RapidVerticalChange: rapid, HorizontalAngleDegree: angle, HasAngle: true,
HipCenterY: hipY, HasHipCenterY: true, TorsoLength: torsoLength, Reason: "accepted",
}
}
func midpoint(person pose.PersonPose, first, second int) (float32, float32) {
return (person.Keypoints[first].X + person.Keypoints[second].X) / 2,
(person.Keypoints[first].Y + person.Keypoints[second].Y) / 2
}
+94
View File
@@ -0,0 +1,94 @@
package fall
import "fmt"
type Engine struct {
config EngineConfig
tracker *tracker
policy *policy
stateMachine *stateMachine
previousEvidence map[string]PoseEvidence
activeTrackIDs map[string]bool
}
func NewEngine(config EngineConfig) (*Engine, error) {
if config.KeypointConfidenceThreshold < 0 || config.KeypointConfidenceThreshold > 1 {
return nil, fmt.Errorf("keypoint confidence threshold must be between 0 and 1")
}
if config.SuspectWindowSeconds < 0 {
return nil, fmt.Errorf("suspect window seconds must be non-negative")
}
if config.HorizontalAngleThresholdDegrees < 0 || config.HorizontalAngleThresholdDegrees > 90 {
return nil, fmt.Errorf("horizontal angle threshold degrees must be between 0 and 90")
}
tracker, err := newTracker(0.2, 2.0)
if err != nil {
return nil, err
}
stateMachine, err := newStateMachine(config)
if err != nil {
return nil, err
}
return &Engine{
config: config, tracker: tracker, policy: newPolicy(config.SuspectWindowSeconds, config.RequireRapidDrop),
stateMachine: stateMachine, previousEvidence: make(map[string]PoseEvidence), activeTrackIDs: make(map[string]bool),
}, nil
}
func (engine *Engine) StateOf(trackID string) State {
return engine.stateMachine.stateOf(trackID)
}
func (engine *Engine) Process(frame Frame) FrameResult {
tracked, err := engine.tracker.update(frame.Poses, frame.Timestamp, frame.Width, frame.Height)
if err != nil {
return FrameResult{}
}
currentIDs := make(map[string]bool, len(tracked))
for _, trackedPose := range tracked {
currentIDs[trackedPose.TrackID] = true
}
events := engine.rejectMissingTracks(currentIDs, frame.Timestamp)
people := make([]PersonAnalysis, 0, len(tracked))
for _, trackedPose := range tracked {
quality := assessPoseQuality(trackedPose.Pose, engine.config.KeypointConfidenceThreshold, engine.config.RequireLowerBody)
var previous *PoseEvidence
if candidate, found := engine.previousEvidence[trackedPose.TrackID]; found {
previous = &candidate
}
poseEvidence := extractEvidence(trackedPose.Pose, quality, previous, engine.config.HorizontalAngleThresholdDegrees)
stateBefore := engine.stateMachine.stateOf(trackedPose.TrackID)
evidence := engine.policy.evaluate(trackedPose.TrackID, poseEvidence, frame.Timestamp, stateBefore)
newEvents, err := engine.stateMachine.update(trackedPose.TrackID, evidence, frame.Timestamp)
if err == nil {
events = append(events, newEvents...)
}
if poseEvidence.Accepted {
engine.previousEvidence[trackedPose.TrackID] = poseEvidence
} else {
delete(engine.previousEvidence, trackedPose.TrackID)
}
people = append(people, PersonAnalysis{
TrackedPose: trackedPose, PoseEvidence: poseEvidence, Evidence: evidence,
State: engine.stateMachine.stateOf(trackedPose.TrackID),
})
}
engine.activeTrackIDs = currentIDs
return FrameResult{People: people, Events: events}
}
func (engine *Engine) rejectMissingTracks(currentIDs map[string]bool, now float64) []Event {
events := make([]Event, 0)
for trackID := range engine.activeTrackIDs {
if currentIDs[trackID] {
continue
}
delete(engine.previousEvidence, trackID)
engine.policy.evaluate(trackID, PoseEvidence{}, now, engine.stateMachine.stateOf(trackID))
newEvents, err := engine.stateMachine.update(trackID, Evidence{}, now)
if err == nil {
events = append(events, newEvents...)
}
}
return events
}
+105
View File
@@ -0,0 +1,105 @@
package fall
import (
"math"
"testing"
"silverpose/v2/internal/pose"
)
func TestEngineConfirmsPersistentHorizontalPoseAtV1Delay(t *testing.T) {
engine, err := NewEngine(EngineConfig{
KeypointConfidenceThreshold: 0.4,
SuspectWindowSeconds: 0.5,
ConfirmWindowSeconds: 1.8,
RecoveryWindowSeconds: 2.0,
CooldownSeconds: 10.0,
RequireRapidDrop: false,
RequireLowerBody: false,
HorizontalAngleThresholdDegrees: 45.0,
ConfigVersion: "cfg-v1-contract",
SessionID: "regression",
})
if err != nil {
t.Fatalf("NewEngine returned an error: %v", err)
}
engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
result := engine.Process(Frame{Timestamp: 1.91, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
if len(result.Events) != 1 {
t.Fatalf("events = %+v, want one confirmation", result.Events)
}
event := result.Events[0]
if event.TrackID != "P-0001" || math.Abs(event.LatencySeconds-1.81) > 1e-9 || event.ConfigVersion != "cfg-v1-contract" {
t.Fatalf("event = %+v", event)
}
if got := engine.StateOf("P-0001"); got != Confirmed {
t.Fatalf("state = %s, want CONFIRMED", got)
}
}
func TestEngineDoesNotConfirmWhenTrackIsMissing(t *testing.T) {
engine, err := NewEngine(testConfig())
if err != nil {
t.Fatalf("NewEngine returned an error: %v", err)
}
engine.Process(Frame{Timestamp: 0.0, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
engine.Process(Frame{Timestamp: 0.1, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
engine.Process(Frame{Timestamp: 0.2, Width: 180, Height: 180})
result := engine.Process(Frame{Timestamp: 1.9, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(true)}})
if len(result.Events) != 0 {
t.Fatalf("events = %+v, want none after a missing track", result.Events)
}
}
func TestEngineDoesNotAlarmForUprightSequence(t *testing.T) {
engine, err := NewEngine(testConfig())
if err != nil {
t.Fatalf("NewEngine returned an error: %v", err)
}
for _, timestamp := range []float64{0.0, 0.6, 1.2, 1.8, 2.4} {
result := engine.Process(Frame{Timestamp: timestamp, Width: 180, Height: 180, Poses: []pose.PersonPose{testPose(false)}})
if len(result.Events) != 0 {
t.Fatalf("timestamp %v emitted %+v for upright pose", timestamp, result.Events)
}
}
}
func testConfig() EngineConfig {
return EngineConfig{
KeypointConfidenceThreshold: 0.4,
SuspectWindowSeconds: 0.5,
ConfirmWindowSeconds: 1.8,
RecoveryWindowSeconds: 2.0,
CooldownSeconds: 10.0,
RequireRapidDrop: false,
RequireLowerBody: false,
HorizontalAngleThresholdDegrees: 45.0,
ConfigVersion: "cfg-v1-contract",
SessionID: "regression",
}
}
func testPose(horizontal bool) pose.PersonPose {
person := pose.PersonPose{Box: pose.Box{Left: 20, Top: 20, Right: 160, Bottom: 160}, Confidence: 0.9}
for index := range person.Keypoints {
person.Keypoints[index] = pose.Keypoint{X: float32(index), Y: float32(index), Confidence: 0.9}
}
if horizontal {
person.Keypoints[5] = pose.Keypoint{X: 20, Y: 80, Confidence: 0.9}
person.Keypoints[6] = pose.Keypoint{X: 30, Y: 80, Confidence: 0.9}
person.Keypoints[11] = pose.Keypoint{X: 70, Y: 100, Confidence: 0.9}
person.Keypoints[12] = pose.Keypoint{X: 80, Y: 100, Confidence: 0.9}
} else {
person.Keypoints[5] = pose.Keypoint{X: 30, Y: 10, Confidence: 0.9}
person.Keypoints[6] = pose.Keypoint{X: 40, Y: 10, Confidence: 0.9}
person.Keypoints[11] = pose.Keypoint{X: 30, Y: 30, Confidence: 0.9}
person.Keypoints[12] = pose.Keypoint{X: 40, Y: 30, Confidence: 0.9}
}
return person
}
+37
View File
@@ -0,0 +1,37 @@
package fall
type policy struct {
suspectWindowSeconds float64
requireRapidDrop bool
rapidDropAt map[string]float64
}
func newPolicy(suspectWindowSeconds float64, requireRapidDrop bool) *policy {
return &policy{
suspectWindowSeconds: suspectWindowSeconds,
requireRapidDrop: requireRapidDrop,
rapidDropAt: make(map[string]float64),
}
}
func (policy *policy) evaluate(trackID string, poseEvidence PoseEvidence, now float64, state State) Evidence {
if !poseEvidence.Accepted {
delete(policy.rapidDropAt, trackID)
return Evidence{}
}
if poseEvidence.RapidVerticalChange {
policy.rapidDropAt[trackID] = now
}
candidate := false
if state == Suspect {
candidate = poseEvidence.HorizontalPose
} else if poseEvidence.HorizontalPose {
if !policy.requireRapidDrop {
candidate = true
} else if droppedAt, found := policy.rapidDropAt[trackID]; found {
candidate = now-droppedAt <= policy.suspectWindowSeconds
}
}
recovery := (state == Confirmed || state == Recovering) && !poseEvidence.HorizontalPose && !poseEvidence.RapidVerticalChange
return Evidence{Accepted: true, IsFallCandidate: candidate, IsRecoveryCandidate: recovery}
}
+142
View File
@@ -0,0 +1,142 @@
package fall
import "fmt"
type stateRecord struct {
state State
suspectStartedAt float64
hasSuspectStartedAt bool
confirmedAt float64
hasConfirmedAt bool
recoveryStartedAt float64
hasRecoveryStartedAt bool
lastUpdatedAt float64
hasLastUpdatedAt bool
}
type stateMachine struct {
confirmWindowSeconds float64
recoveryWindowSeconds float64
cooldownSeconds float64
configVersion string
sessionID string
records map[string]*stateRecord
nextEventNumber int
}
func newStateMachine(config EngineConfig) (*stateMachine, error) {
if config.ConfirmWindowSeconds < 1 || config.ConfirmWindowSeconds > 3 {
return nil, fmt.Errorf("confirm window seconds must be between 1 and 3")
}
if config.RecoveryWindowSeconds <= 0 {
return nil, fmt.Errorf("recovery window seconds must be positive")
}
if config.CooldownSeconds < 0 {
return nil, fmt.Errorf("cooldown seconds must be non-negative")
}
if config.ConfigVersion == "" {
return nil, fmt.Errorf("config version must be non-empty")
}
return &stateMachine{
confirmWindowSeconds: config.ConfirmWindowSeconds, recoveryWindowSeconds: config.RecoveryWindowSeconds,
cooldownSeconds: config.CooldownSeconds, configVersion: config.ConfigVersion, sessionID: config.SessionID,
records: make(map[string]*stateRecord), nextEventNumber: 1,
}, nil
}
func (machine *stateMachine) stateOf(trackID string) State {
if record, found := machine.records[trackID]; found {
return record.state
}
return Normal
}
func (machine *stateMachine) update(trackID string, evidence Evidence, now float64) ([]Event, error) {
if trackID == "" {
return nil, fmt.Errorf("track ID must be non-empty")
}
record, found := machine.records[trackID]
if !found {
record = &stateRecord{state: Normal}
machine.records[trackID] = record
}
if record.hasLastUpdatedAt && now < record.lastUpdatedAt {
return nil, fmt.Errorf("timestamps must be monotonic per track")
}
record.lastUpdatedAt, record.hasLastUpdatedAt = now, true
if !evidence.Accepted {
machine.rejectEvidence(record)
return nil, nil
}
switch record.state {
case Normal:
if evidence.IsFallCandidate {
record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true
}
return nil, nil
case Suspect:
if !evidence.IsFallCandidate {
machine.setNormal(record)
return nil, nil
}
if now-record.suspectStartedAt >= machine.confirmWindowSeconds {
record.state, record.confirmedAt, record.hasConfirmedAt = Confirmed, now, true
event := machine.newEvent(trackID, record.suspectStartedAt, now)
record.hasSuspectStartedAt = false
return []Event{event}, nil
}
return nil, nil
case Confirmed:
if evidence.IsRecoveryCandidate && now-record.confirmedAt >= machine.cooldownSeconds {
record.state, record.recoveryStartedAt, record.hasRecoveryStartedAt = Recovering, now, true
}
return nil, nil
case Recovering:
if evidence.IsFallCandidate {
record.state, record.suspectStartedAt, record.hasSuspectStartedAt = Suspect, now, true
record.hasRecoveryStartedAt = false
return nil, nil
}
if !evidence.IsRecoveryCandidate {
record.state, record.hasRecoveryStartedAt = Confirmed, false
return nil, nil
}
if now-record.recoveryStartedAt >= machine.recoveryWindowSeconds {
machine.setNormal(record)
}
return nil, nil
default:
return nil, fmt.Errorf("unknown fall state")
}
}
func (machine *stateMachine) rejectEvidence(record *stateRecord) {
if record.state == Suspect {
machine.setNormal(record)
} else if record.state == Recovering {
record.state, record.hasRecoveryStartedAt = Confirmed, false
}
}
func (machine *stateMachine) setNormal(record *stateRecord) {
record.state = Normal
record.hasSuspectStartedAt = false
record.hasConfirmedAt = false
record.hasRecoveryStartedAt = false
}
func (machine *stateMachine) newEvent(trackID string, suspectedAt, confirmedAt float64) Event {
prefix := "FALL-"
if machine.sessionID != "" {
prefix += machine.sessionID + "-"
}
event := Event{
EventID: fmt.Sprintf("%s%06d", prefix, machine.nextEventNumber), TrackID: trackID,
ConfigVersion: machine.configVersion, SuspectedAtMonotonic: suspectedAt,
ConfirmedAtMonotonic: confirmedAt, LatencySeconds: confirmedAt - suspectedAt, State: Confirmed,
}
machine.nextEventNumber++
return event
}
+93
View File
@@ -0,0 +1,93 @@
package fall
import (
"fmt"
"math"
"sort"
"silverpose/v2/internal/pose"
)
type track struct {
centerX float32
centerY float32
lastSeenAt float64
}
type tracker struct {
maxMatchDistanceRatio float64
maxAgeSeconds float64
tracks map[string]track
nextTrackNumber int
}
func newTracker(maxMatchDistanceRatio, maxAgeSeconds float64) (*tracker, error) {
if maxMatchDistanceRatio <= 0 || maxMatchDistanceRatio > 1 {
return nil, fmt.Errorf("max match distance ratio must be in (0, 1]")
}
if maxAgeSeconds <= 0 {
return nil, fmt.Errorf("max age seconds must be positive")
}
return &tracker{
maxMatchDistanceRatio: maxMatchDistanceRatio,
maxAgeSeconds: maxAgeSeconds,
tracks: make(map[string]track),
nextTrackNumber: 1,
}, nil
}
func (tracker *tracker) update(poses []pose.PersonPose, now float64, width, height int) ([]TrackedPose, error) {
if width <= 0 || height <= 0 {
return nil, fmt.Errorf("frame size must contain positive width and height")
}
tracker.expire(now)
availableIDs := make(map[string]bool, len(tracker.tracks))
for trackID := range tracker.tracks {
availableIDs[trackID] = true
}
tracked := make([]TrackedPose, 0, len(poses))
for _, person := range poses {
centerX := (person.Box.Left + person.Box.Right) / 2
centerY := (person.Box.Top + person.Box.Bottom) / 2
trackID := tracker.nearestAvailable(centerX, centerY, availableIDs, width, height)
if trackID == "" {
trackID = fmt.Sprintf("P-%04d", tracker.nextTrackNumber)
tracker.nextTrackNumber++
} else {
delete(availableIDs, trackID)
}
tracker.tracks[trackID] = track{centerX: centerX, centerY: centerY, lastSeenAt: now}
tracked = append(tracked, TrackedPose{TrackID: trackID, Pose: person})
}
return tracked, nil
}
func (tracker *tracker) nearestAvailable(centerX, centerY float32, availableIDs map[string]bool, width, height int) string {
ids := make([]string, 0, len(availableIDs))
for trackID := range availableIDs {
ids = append(ids, trackID)
}
sort.Strings(ids)
closestID := ""
closestDistance := math.Inf(1)
for _, trackID := range ids {
previous := tracker.tracks[trackID]
distance := math.Hypot(
float64(centerX-previous.centerX)/float64(width),
float64(centerY-previous.centerY)/float64(height),
)
if distance <= tracker.maxMatchDistanceRatio && distance < closestDistance {
closestID = trackID
closestDistance = distance
}
}
return closestID
}
func (tracker *tracker) expire(now float64) {
for trackID, current := range tracker.tracks {
if now-current.lastSeenAt > tracker.maxAgeSeconds {
delete(tracker.tracks, trackID)
}
}
}
+83
View File
@@ -0,0 +1,83 @@
package fall
import "silverpose/v2/internal/pose"
type State string
const (
Normal State = "NORMAL"
Suspect State = "SUSPECT"
Confirmed State = "CONFIRMED"
Recovering State = "RECOVERING"
)
type Evidence struct {
Accepted bool
IsFallCandidate bool
IsRecoveryCandidate bool
}
type PoseQuality struct {
Accepted bool
Reason string
VisibleJointCount int
}
type PoseEvidence struct {
Accepted bool
HorizontalPose bool
RapidVerticalChange bool
HorizontalAngleDegree float32
HasAngle bool
HipCenterY float32
HasHipCenterY bool
TorsoLength float32
Reason string
}
type Event struct {
EventID string
TrackID string
ConfigVersion string
SuspectedAtMonotonic float64
ConfirmedAtMonotonic float64
LatencySeconds float64
State State
}
type Frame struct {
Timestamp float64
Width int
Height int
Poses []pose.PersonPose
}
type TrackedPose struct {
TrackID string
Pose pose.PersonPose
}
type PersonAnalysis struct {
TrackedPose TrackedPose
PoseEvidence PoseEvidence
Evidence Evidence
State State
}
type FrameResult struct {
People []PersonAnalysis
Events []Event
}
type EngineConfig struct {
KeypointConfidenceThreshold float32
SuspectWindowSeconds float64
ConfirmWindowSeconds float64
RecoveryWindowSeconds float64
CooldownSeconds float64
RequireRapidDrop bool
RequireLowerBody bool
HorizontalAngleThresholdDegrees float32
ConfigVersion string
SessionID string
}
+125
View File
@@ -0,0 +1,125 @@
package pose
import (
"fmt"
"sort"
)
// ParseYOLOv8Pose turns the fixed [1,56,8400] YOLO Pose output into source
// pixel coordinates, applying confidence filtering and class-agnostic NMS.
func ParseYOLOv8Pose(output []float32, transform LetterboxTransform, confidenceThreshold, iouThreshold float32) ([]PersonPose, error) {
if len(output) != YOLOPoseOutputValues {
return nil, fmt.Errorf("YOLO Pose output length = %d, want %d", len(output), YOLOPoseOutputValues)
}
if transform.Scale <= 0 {
return nil, fmt.Errorf("letterbox scale must be positive")
}
if confidenceThreshold < 0 || confidenceThreshold > 1 || iouThreshold < 0 || iouThreshold > 1 {
return nil, fmt.Errorf("confidence and IoU thresholds must be between 0 and 1")
}
candidates := make([]PersonPose, 0)
for candidate := 0; candidate < YOLOPoseCandidateCount; candidate++ {
confidence := valueAt(output, 4, candidate)
if confidence < confidenceThreshold {
continue
}
centerX := valueAt(output, 0, candidate)
centerY := valueAt(output, 1, candidate)
width := valueAt(output, 2, candidate)
height := valueAt(output, 3, candidate)
person := PersonPose{
Box: Box{
Left: transform.restoreX(centerX - width/2),
Top: transform.restoreY(centerY - height/2),
Right: transform.restoreX(centerX + width/2),
Bottom: transform.restoreY(centerY + height/2),
},
Confidence: confidence,
}
for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ {
channel := 5 + keypoint*3
person.Keypoints[keypoint] = Keypoint{
X: transform.restoreX(valueAt(output, channel, candidate)),
Y: transform.restoreY(valueAt(output, channel+1, candidate)),
Confidence: valueAt(output, channel+2, candidate),
}
}
candidates = append(candidates, person)
}
sort.SliceStable(candidates, func(left, right int) bool {
return candidates[left].Confidence > candidates[right].Confidence
})
selected := make([]PersonPose, 0, len(candidates))
for _, candidate := range candidates {
overlaps := false
for _, accepted := range selected {
if intersectionOverUnion(candidate.Box, accepted.Box) > iouThreshold {
overlaps = true
break
}
}
if !overlaps {
selected = append(selected, candidate)
}
}
return selected, nil
}
func valueAt(output []float32, channel, candidate int) float32 {
return output[channel*YOLOPoseCandidateCount+candidate]
}
func (transform LetterboxTransform) restoreX(value float32) float32 {
return clampCoordinate((value-float32(transform.PadLeft))/transform.Scale, transform.OriginalWidth)
}
func (transform LetterboxTransform) restoreY(value float32) float32 {
return clampCoordinate((value-float32(transform.PadTop))/transform.Scale, transform.OriginalHeight)
}
func clampCoordinate(value float32, size int) float32 {
if size <= 0 {
return value
}
if value < 0 {
return 0
}
maximum := float32(size)
if value > maximum {
return maximum
}
return value
}
func intersectionOverUnion(left, right Box) float32 {
intersectionLeft := maxFloat(left.Left, right.Left)
intersectionTop := maxFloat(left.Top, right.Top)
intersectionRight := minFloat(left.Right, right.Right)
intersectionBottom := minFloat(left.Bottom, right.Bottom)
intersectionWidth := maxFloat(0, intersectionRight-intersectionLeft)
intersectionHeight := maxFloat(0, intersectionBottom-intersectionTop)
intersection := intersectionWidth * intersectionHeight
leftArea := maxFloat(0, left.Right-left.Left) * maxFloat(0, left.Bottom-left.Top)
rightArea := maxFloat(0, right.Right-right.Left) * maxFloat(0, right.Bottom-right.Top)
union := leftArea + rightArea - intersection
if union <= 0 {
return 0
}
return intersection / union
}
func minFloat(left, right float32) float32 {
if left < right {
return left
}
return right
}
func maxFloat(left, right float32) float32 {
if left > right {
return left
}
return right
}
+45
View File
@@ -0,0 +1,45 @@
package pose
import "testing"
func TestParseYOLOv8PoseRestoresCoordinatesAndSuppressesOverlap(t *testing.T) {
output := make([]float32, YOLOPoseOutputValues)
putDetection(output, 0, 320, 320, 100, 200, 0.90, 330, 340)
putDetection(output, 1, 322, 321, 100, 200, 0.80, 332, 341)
people, err := ParseYOLOv8Pose(output, LetterboxTransform{Scale: 1}, 0.25, 0.70)
if err != nil {
t.Fatalf("ParseYOLOv8Pose returned an error: %v", err)
}
if len(people) != 1 {
t.Fatalf("people = %d, want one NMS survivor", len(people))
}
person := people[0]
if person.Box != (Box{Left: 270, Top: 220, Right: 370, Bottom: 420}) {
t.Fatalf("box = %+v", person.Box)
}
if point := person.Keypoints[0]; point.X != 330 || point.Y != 340 || point.Confidence != 0.9 {
t.Fatalf("first keypoint = %+v", point)
}
}
func TestParseYOLOv8PoseRejectsUnexpectedOutputLength(t *testing.T) {
_, err := ParseYOLOv8Pose([]float32{0}, LetterboxTransform{Scale: 1}, 0.25, 0.70)
if err == nil {
t.Fatal("ParseYOLOv8Pose accepted a malformed output")
}
}
func putDetection(output []float32, candidate int, centerX, centerY, width, height, confidence, keypointX, keypointY float32) {
output[0*YOLOPoseCandidateCount+candidate] = centerX
output[1*YOLOPoseCandidateCount+candidate] = centerY
output[2*YOLOPoseCandidateCount+candidate] = width
output[3*YOLOPoseCandidateCount+candidate] = height
output[4*YOLOPoseCandidateCount+candidate] = confidence
for keypoint := 0; keypoint < YOLOPoseKeypointCount; keypoint++ {
base := 5 + keypoint*3
output[(base+0)*YOLOPoseCandidateCount+candidate] = keypointX
output[(base+1)*YOLOPoseCandidateCount+candidate] = keypointY
output[(base+2)*YOLOPoseCandidateCount+candidate] = 0.9
}
}
+87
View File
@@ -0,0 +1,87 @@
package pose
import (
"fmt"
"math"
)
const letterboxPadding = byte(114)
// PreprocessBGR reproduces the fixed-shape Ultralytics letterbox contract:
// BGR source pixels are resized with bilinear interpolation, padded in 114
// gray, converted to RGB/CHW and normalised to [0, 1].
func PreprocessBGR(frame []byte, width, height, target int) ([]float32, LetterboxTransform, error) {
if width <= 0 || height <= 0 || target <= 0 {
return nil, LetterboxTransform{}, fmt.Errorf("frame width, height and target must be positive")
}
if len(frame) != width*height*3 {
return nil, LetterboxTransform{}, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3)
}
scale := math.Min(float64(target)/float64(width), float64(target)/float64(height))
resizedWidth := int(math.Round(float64(width) * scale))
resizedHeight := int(math.Round(float64(height) * scale))
padWidth := target - resizedWidth
padHeight := target - resizedHeight
padLeft := int(math.Round(float64(padWidth)/2.0 - 0.1))
padTop := int(math.Round(float64(padHeight)/2.0 - 0.1))
transform := LetterboxTransform{
OriginalWidth: width, OriginalHeight: height, InputSize: target,
ResizedWidth: resizedWidth, ResizedHeight: resizedHeight,
PadLeft: padLeft, PadTop: padTop, Scale: float32(scale),
}
plane := target * target
input := make([]float32, 3*plane)
padding := float32(letterboxPadding) / 255.0
for index := range input {
input[index] = padding
}
for y := 0; y < resizedHeight; y++ {
sourceY := clampFloat((float64(y)+0.5)*float64(height)/float64(resizedHeight)-0.5, 0, float64(height-1))
y0 := int(math.Floor(sourceY))
y1 := minInt(y0+1, height-1)
yWeight := float32(sourceY - float64(y0))
for x := 0; x < resizedWidth; x++ {
sourceX := clampFloat((float64(x)+0.5)*float64(width)/float64(resizedWidth)-0.5, 0, float64(width-1))
x0 := int(math.Floor(sourceX))
x1 := minInt(x0+1, width-1)
xWeight := float32(sourceX - float64(x0))
leftTop := (y0*width + x0) * 3
rightTop := (y0*width + x1) * 3
leftBottom := (y1*width + x0) * 3
rightBottom := (y1*width + x1) * 3
destination := (padTop+y)*target + padLeft + x
for sourceChannel := 0; sourceChannel < 3; sourceChannel++ {
value := bilinear(
frame[leftTop+sourceChannel], frame[rightTop+sourceChannel],
frame[leftBottom+sourceChannel], frame[rightBottom+sourceChannel],
xWeight, yWeight,
)
// BGR source maps to RGB tensor planes.
tensorChannel := 2 - sourceChannel
// cv2.resize writes uint8 pixels before Ultralytics converts the
// image to float; round here to retain that observable contract.
input[tensorChannel*plane+destination] = float32(math.Round(float64(value))) / 255.0
}
}
}
return input, transform, nil
}
func bilinear(topLeft, topRight, bottomLeft, bottomRight byte, xWeight, yWeight float32) float32 {
top := float32(topLeft)*(1-xWeight) + float32(topRight)*xWeight
bottom := float32(bottomLeft)*(1-xWeight) + float32(bottomRight)*xWeight
return top*(1-yWeight) + bottom*yWeight
}
func clampFloat(value, minimum, maximum float64) float64 {
return math.Max(minimum, math.Min(maximum, value))
}
func minInt(left, right int) int {
if left < right {
return left
}
return right
}
+39
View File
@@ -0,0 +1,39 @@
package pose
import "testing"
func TestPreprocessBGRUsesUltralyticsPaddingAndRGBCHW(t *testing.T) {
// A 2x1 BGR frame letterboxes into a 4x4 tensor with one row of 114-gray
// padding above and below. The rightmost source pixel is pure blue in BGR.
frame := []byte{0, 0, 0, 255, 0, 0}
input, transform, err := PreprocessBGR(frame, 2, 1, 4)
if err != nil {
t.Fatalf("PreprocessBGR returned an error: %v", err)
}
if transform.ResizedWidth != 4 || transform.ResizedHeight != 2 || transform.PadTop != 1 {
t.Fatalf("unexpected transform: %+v", transform)
}
const plane = 16
padding := float32(114.0 / 255.0)
if got := input[2*plane+0]; got != padding {
t.Fatalf("top padding blue = %v, want %v", got, padding)
}
if got := input[0*plane+1*4+3]; got != 0 {
t.Fatalf("red channel = %v, want 0", got)
}
if got := input[1*plane+1*4+3]; got != 0 {
t.Fatalf("green channel = %v, want 0", got)
}
if got := input[2*plane+1*4+3]; got != 1 {
t.Fatalf("blue channel = %v, want 1", got)
}
}
func TestPreprocessBGRRejectsTruncatedFrame(t *testing.T) {
_, _, err := PreprocessBGR([]byte{0}, 2, 1, 4)
if err == nil {
t.Fatal("PreprocessBGR accepted a truncated BGR frame")
}
}
+39
View File
@@ -0,0 +1,39 @@
package pose
const (
YOLOPoseKeypointCount = 17
YOLOPoseCandidateCount = 8400
YOLOPoseChannelCount = 56
YOLOPoseOutputValues = YOLOPoseChannelCount * YOLOPoseCandidateCount
)
type Keypoint struct {
X float32
Y float32
Confidence float32
}
type Box struct {
Left float32
Top float32
Right float32
Bottom float32
}
type PersonPose struct {
Box Box
Confidence float32
Keypoints [YOLOPoseKeypointCount]Keypoint
}
// LetterboxTransform maps fixed-square YOLO coordinates back to source pixels.
type LetterboxTransform struct {
OriginalWidth int
OriginalHeight int
InputSize int
ResizedWidth int
ResizedHeight int
PadLeft int
PadTop int
Scale float32
}
+88 -39
View File
@@ -12,6 +12,90 @@ const (
poseOutputCount = 1 * 56 * 8400
)
// Runtime owns one reusable ONNX session. V2 video replay must keep the
// model session alive across frames; recreating it for every frame would make
// event-latency measurements meaningless.
type Runtime struct {
input *ort.Tensor[float32]
output *ort.Tensor[float32]
session *ort.AdvancedSession
initialized bool
closed bool
}
func OpenRuntime(modelPath, runtimeDLLPath string) (*Runtime, error) {
ort.SetSharedLibraryPath(runtimeDLLPath)
if err := ort.InitializeEnvironment(); err != nil {
return nil, fmt.Errorf("initialize ONNX Runtime: %w", err)
}
runtime := &Runtime{initialized: true}
var err error
runtime.input, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 3, poseInputSize, poseInputSize))
if err != nil {
runtime.Close()
return nil, fmt.Errorf("create pose input tensor: %w", err)
}
runtime.output, err = ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400))
if err != nil {
runtime.Close()
return nil, fmt.Errorf("create pose output tensor: %w", err)
}
runtime.session, err = ort.NewAdvancedSession(
modelPath,
[]string{"images"},
[]string{"output0"},
[]ort.Value{runtime.input},
[]ort.Value{runtime.output},
nil,
)
if err != nil {
runtime.Close()
return nil, fmt.Errorf("open pose ONNX session: %w", err)
}
return runtime, nil
}
func (runtime *Runtime) Run(input []float32) ([]float32, error) {
if runtime == nil || runtime.closed {
return nil, fmt.Errorf("ONNX Runtime session is closed")
}
if len(input) != poseInputCount {
return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount)
}
copy(runtime.input.GetData(), input)
if err := runtime.session.Run(); err != nil {
return nil, fmt.Errorf("run pose ONNX session: %w", err)
}
output := runtime.output.GetData()
if len(output) != poseOutputCount {
return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount)
}
return append([]float32(nil), output...), nil
}
func (runtime *Runtime) Close() {
if runtime == nil || runtime.closed {
return
}
if runtime.session != nil {
_ = runtime.session.Destroy()
runtime.session = nil
}
if runtime.output != nil {
_ = runtime.output.Destroy()
runtime.output = nil
}
if runtime.input != nil {
_ = runtime.input.Destroy()
runtime.input = nil
}
if runtime.initialized {
_ = ort.DestroyEnvironment()
runtime.initialized = false
}
runtime.closed = true
}
// RunPose executes the locked YOLO pose ONNX graph once using the CPU runtime.
// The caller supplies absolute model and DLL paths so no camera credential or
// machine-specific path is retained in V2 configuration or source.
@@ -19,45 +103,10 @@ func RunPose(modelPath, runtimeDLLPath string, input []float32) ([]float32, erro
if len(input) != poseInputCount {
return nil, fmt.Errorf("pose input length = %d, want %d", len(input), poseInputCount)
}
ort.SetSharedLibraryPath(runtimeDLLPath)
if err := ort.InitializeEnvironment(); err != nil {
return nil, fmt.Errorf("initialize ONNX Runtime: %w", err)
}
defer func() { _ = ort.DestroyEnvironment() }()
inputTensor, err := ort.NewTensor(ort.NewShape(1, 3, poseInputSize, poseInputSize), input)
runtime, err := OpenRuntime(modelPath, runtimeDLLPath)
if err != nil {
return nil, fmt.Errorf("create pose input tensor: %w", err)
return nil, err
}
defer func() { _ = inputTensor.Destroy() }()
outputTensor, err := ort.NewEmptyTensor[float32](ort.NewShape(1, 56, 8400))
if err != nil {
return nil, fmt.Errorf("create pose output tensor: %w", err)
}
defer func() { _ = outputTensor.Destroy() }()
session, err := ort.NewAdvancedSession(
modelPath,
[]string{"images"},
[]string{"output0"},
[]ort.Value{inputTensor},
[]ort.Value{outputTensor},
nil,
)
if err != nil {
return nil, fmt.Errorf("open pose ONNX session: %w", err)
}
defer func() { _ = session.Destroy() }()
if err := session.Run(); err != nil {
return nil, fmt.Errorf("run pose ONNX session: %w", err)
}
output := outputTensor.GetData()
if len(output) != poseOutputCount {
return nil, fmt.Errorf("pose output length = %d, want %d", len(output), poseOutputCount)
}
return append([]float32(nil), output...), nil
defer runtime.Close()
return runtime.Run(input)
}
+10
View File
@@ -14,3 +14,13 @@ func TestRunPoseRejectsWrongInputLengthBeforeLoadingRuntime(t *testing.T) {
t.Fatalf("RunPose error = %q, want input length error", err)
}
}
func TestOpenRuntimeReportsMissingDLL(t *testing.T) {
_, err := OpenRuntime("missing.onnx", "missing.dll")
if err == nil {
t.Fatal("OpenRuntime accepted a missing ONNX Runtime DLL")
}
if !strings.Contains(err.Error(), "initialize ONNX Runtime") {
t.Fatalf("OpenRuntime error = %q", err)
}
}
+6 -35
View File
@@ -1,43 +1,14 @@
package spike
import "fmt"
import "silverpose/v2/internal/pose"
// LetterboxBGRToNCHW converts one packed BGR frame into the RGB, CHW,
// float32 tensor expected by the locked 640-pixel YOLO pose ONNX model.
//
// It intentionally uses nearest-neighbour scaling for this Spike. T-303
// must replace or validate this preprocessing against the V1 implementation
// before it becomes the V2 production preprocessing path.
// T-303 aligned the implementation with the fixed-shape Ultralytics
// letterbox used by the V1/ONNX baseline. New V2 code should import the pose
// package directly to retain the returned coordinate transform.
func LetterboxBGRToNCHW(frame []byte, width, height, target int) ([]float32, error) {
if width <= 0 || height <= 0 || target <= 0 {
return nil, fmt.Errorf("frame width, height and target must be positive")
}
if len(frame) != width*height*3 {
return nil, fmt.Errorf("BGR frame length = %d, want %d", len(frame), width*height*3)
}
scaleWidth := target
scaleHeight := height * target / width
if scaleHeight > target {
scaleHeight = target
scaleWidth = width * target / height
}
padX := (target - scaleWidth) / 2
padY := (target - scaleHeight) / 2
plane := target * target
input := make([]float32, 3*plane)
for y := 0; y < scaleHeight; y++ {
sourceY := y * height / scaleHeight
for x := 0; x < scaleWidth; x++ {
sourceX := x * width / scaleWidth
source := (sourceY*width + sourceX) * 3
destination := (padY+y)*target + padX + x
input[destination] = float32(frame[source+2]) / 255.0
input[plane+destination] = float32(frame[source+1]) / 255.0
input[2*plane+destination] = float32(frame[source]) / 255.0
}
}
return input, nil
input, _, err := pose.PreprocessBGR(frame, width, height, target)
return input, err
}
+5 -4
View File
@@ -16,8 +16,9 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) {
if got := len(input); got != 3*plane {
t.Fatalf("input length = %d, want %d", got, 3*plane)
}
if got := input[2*plane+159*640+320]; got != 0 {
t.Fatalf("top padding blue channel = %v, want 0", got)
padding := float32(114.0 / 255.0)
if got := input[2*plane+159*640+320]; got != padding {
t.Fatalf("top padding blue channel = %v, want %v", got, padding)
}
if got := input[0*plane+320*640+320]; got != 0 {
t.Fatalf("red channel = %v, want 0", got)
@@ -25,8 +26,8 @@ func TestLetterboxBGRToNCHWPlacesPixelInScaledImage(t *testing.T) {
if got := input[1*plane+320*640+320]; got != 0 {
t.Fatalf("green channel = %v, want 0", got)
}
if got := input[2*plane+320*640+320]; got != 1 {
t.Fatalf("blue channel = %v, want 1", got)
if got := input[2*plane+320*640+320]; got != float32(128.0/255.0) {
t.Fatalf("blue channel = %v, want cv2 INTER_LINEAR value 128/255", got)
}
}