feat(v1): latest-frame-only reader thread for low-latency RTSP

Adds _FrameGrabber which decodes in a background thread and keeps only the
newest frame; VideoSource drop_stale (STREAM only) returns the latest frame
so a slow consumer never builds a backlog. app enables it for streams.
REPLAY is unchanged. 82 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ila
2026-07-22 00:04:00 +08:00
co-authored by Claude Opus 4.8
parent b32c36fc44
commit 6e69518742
6 changed files with 164 additions and 5 deletions
+1 -1
View File
@@ -36,7 +36,7 @@
| T-209 | 放宽俯视场景摔倒判定灵敏度(可配置) | T-208 | 判定灵敏度参数化:`require_rapid_drop`(默认关)、`require_lower_body`(默认关)、`horizontal_angle_threshold_degrees`(默认放宽);默认下持续水平即可进入疑似,仅靠确认窗防误报;参数进 config 与 config_version,逻辑可测。 | DONE | | T-209 | 放宽俯视场景摔倒判定灵敏度(可配置) | T-208 | 判定灵敏度参数化:`require_rapid_drop`(默认关)、`require_lower_body`(默认关)、`horizontal_angle_threshold_degrees`(默认放宽);默认下持续水平即可进入疑似,仅靠确认窗防误报;参数进 config 与 config_version,逻辑可测。 | DONE |
| T-210 | 修复截图标注乱码(cv2 ASCII 文字) | T-202 | 截图标注文字改为纯 ASCII(`cv2.putText` 只支持 ASCII),去掉非 ASCII 分隔符导致的乱码;实时画面标签不变;ASCII 标签逻辑可测。 | DONE | | T-210 | 修复截图标注乱码(cv2 ASCII 文字) | T-202 | 截图标注文字改为纯 ASCII(`cv2.putText` 只支持 ASCII),去掉非 ASCII 分隔符导致的乱码;实时画面标签不变;ASCII 标签逻辑可测。 | DONE |
| T-211 | 检测灵敏度参数进设置页 | T-209 | 设置页可编辑 `require_rapid_drop`/`require_lower_body`/`horizontal_angle_threshold_degrees`/`confirm_window_seconds`(下拉/数字/勾选),保存写回未跟踪 `config.local.json`、下次启动生效;写回逻辑可测。 | DONE | | T-211 | 检测灵敏度参数进设置页 | T-209 | 设置页可编辑 `require_rapid_drop`/`require_lower_body`/`horizontal_angle_threshold_degrees`/`confirm_window_seconds`(下拉/数字/勾选),保存写回未跟踪 `config.local.json`、下次启动生效;写回逻辑可测。 | DONE |
| T-212 | 实时流只处理最新帧降延迟 | T-203 | STREAM 后台读帧只保留最新帧、丢弃积压,实时延迟不随推理速度增长;REPLAY 逐帧不丢;最新帧保留逻辑可测。 | DOING | | T-212 | 实时流只处理最新帧降延迟 | T-203 | STREAM 后台读帧只保留最新帧、丢弃积压,实时延迟不随推理速度增长;REPLAY 逐帧不丢;最新帧保留逻辑可测。 | DONE |
| T-213 | 截图中文标注(Pillow) | T-210 | CONFIRMED 截图可用中文标注(如“确认摔倒 P-0001”);缺 Pillow/字体时回退 ASCII 不崩溃;标签与回退逻辑可测。 | TODO | | T-213 | 截图中文标注(Pillow) | T-210 | CONFIRMED 截图可用中文标注(如“确认摔倒 P-0001”);缺 Pillow/字体时回退 ASCII 不崩溃;标签与回退逻辑可测。 | TODO |
## Phase 3 · Go V2 迁移门槛与实现 ## Phase 3 · Go V2 迁移门槛与实现
+1 -1
View File
@@ -11,7 +11,7 @@
- V1 代码:已建立安全配置、显式 Replay/Stream 视频源、Pose、轻量跟踪、质量/几何证据、倒地领域规则、按 ID 四态事件机及 `v1/pipeline.py` 事件管线;运行事件带非敏感 `config_version`,缺帧/低质量/断流会中断证据确认。新增 `v1/view_model.py`(Qt-free 监控视图状态与设置草稿三份隔离)、`v1/gui.py` 薄 PyQt5 双 Tab 外壳与 `v1/app.py` 装配(`FrameWorker` 只发出已判定的 `FrameAnalysis`,窗口只渲染)。新增 `v1/alerts.py`:按 `event_id` 去重,对首次 CONFIRMED 保存带标注截图、追加 JSONL 事件行,并经可注入 `AlertSink`(Windows 侧 `QtAlertSink` 提供声音与一次性弹窗)触发一次声音/弹窗。新增 `v1/camera.py`(RTSP URL 构建 + 有界连接测试抓帧)与 `config.py` 的结构化来源(host/端口/通道/账号/密码,凭证百分号编码)、`write_local_camera_source`;设置页新增摄像头连接分组、测试连接与预览,以及传输协议/连接超时/低延迟三项抓流调优(由 `config` 存值、`app` 开流前写入 `OPENCV_FFMPEG_CAPTURE_OPTIONS`)。真实海康流已用 `VideoSource(STREAM)` 连通(1920×1080)。 - V1 代码:已建立安全配置、显式 Replay/Stream 视频源、Pose、轻量跟踪、质量/几何证据、倒地领域规则、按 ID 四态事件机及 `v1/pipeline.py` 事件管线;运行事件带非敏感 `config_version`,缺帧/低质量/断流会中断证据确认。新增 `v1/view_model.py`(Qt-free 监控视图状态与设置草稿三份隔离)、`v1/gui.py` 薄 PyQt5 双 Tab 外壳与 `v1/app.py` 装配(`FrameWorker` 只发出已判定的 `FrameAnalysis`,窗口只渲染)。新增 `v1/alerts.py`:按 `event_id` 去重,对首次 CONFIRMED 保存带标注截图、追加 JSONL 事件行,并经可注入 `AlertSink`(Windows 侧 `QtAlertSink` 提供声音与一次性弹窗)触发一次声音/弹窗。新增 `v1/camera.py`(RTSP URL 构建 + 有界连接测试抓帧)与 `config.py` 的结构化来源(host/端口/通道/账号/密码,凭证百分号编码)、`write_local_camera_source`;设置页新增摄像头连接分组、测试连接与预览,以及传输协议/连接超时/低延迟三项抓流调优(由 `config` 存值、`app` 开流前写入 `OPENCV_FFMPEG_CAPTURE_OPTIONS`)。真实海康流已用 `VideoSource(STREAM)` 连通(1920×1080)。
- V2 代码:`v2/` 目录存在但尚无实现。 - V2 代码:`v2/` 目录存在但尚无实现。
- 非代码设计工件:docs/ui/silver-pose-ui-ux-spec.md、docs/ui/2026-07-20-html-prototype-plan.md、docs/ui/silver-pose-v1-prototype.html 与 docs/ui/silver-pose-v2-prototype.html 已建立。v2 HTML 是符合正式浅色 Windows 规范的当前视觉参考:浅灰蓝底、白色卡片,红色只表示确认摔倒、其弹窗和事件证据;文件名中的 v2 只表示原型设计修订,不能理解为 Go V2 实现已开始。v1 HTML 保留为历史深色对照。两者均使用顶部双 Tab、设置草稿与状态交互,且画面、事件和时间都是模拟数据,不连接真实摄像头、模型或网络,也不改变 Phase 1 任务顺序。 - 非代码设计工件:docs/ui/silver-pose-ui-ux-spec.md、docs/ui/2026-07-20-html-prototype-plan.md、docs/ui/silver-pose-v1-prototype.html 与 docs/ui/silver-pose-v2-prototype.html 已建立。v2 HTML 是符合正式浅色 Windows 规范的当前视觉参考:浅灰蓝底、白色卡片,红色只表示确认摔倒、其弹窗和事件证据;文件名中的 v2 只表示原型设计修订,不能理解为 Go V2 实现已开始。v1 HTML 保留为历史深色对照。两者均使用顶部双 Tab、设置草稿与状态交互,且画面、事件和时间都是模拟数据,不连接真实摄像头、模型或网络,也不改变 Phase 1 任务顺序。
- 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`python -m pytest v1/tests -v` 当前有 79 项配置、视频源、Pose、跟踪、证据、领域规则、状态机、管线、视图模型、报警工件、摄像头/URL 构建、抓流参数拼串、每帧诊断、灵敏度放宽、截图 ASCII 标注、断流重连、事件汇总和灵敏度写回测试并已通过(含来源模式显式声明、模型置信度经适配器生效、事件号跨轮唯一、结构化凭证不入 config_version 且公开示例无凭证的守卫)。`demo/1.mp4` 的首两帧回放时间戳已验证为 0.000000 与 0.033333 秒;T-106 的真实模型/录像 smoke 在首帧得到 2 名已分析人员、第二帧得到 0 名人员且未创建事件;T-201 的视图冒烟以真实录像解码 + 真实管线 + 确定性假 Pose 适配器驱动 `build_monitor_view`,得到稳定 ID、box、14 段骨架、17/17 关键点与 NORMAL/success,断流帧 0 人且不显示摔倒标签;T-202 的报警冒烟用 `demo/1.mp4` 首帧(848×480)落盘一张可被 `cv2.imread` 读回的标注截图(480×848×3,约 330 KB)与一行 JSONL(相对截图路径、含 config_version/source_id、无 rtsp),重复派发返回 0。这些只验证管线、视图与报警工件可运行,不表示摔倒识别准确率。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。 - 测试:`python -m compileall -q v1 demo` 已通过(含 `gui.py`、`app.py`、`alerts.py`、`camera.py` 语法);`python -m pytest v1/tests -v` 当前有 82 项配置、视频源、Pose、跟踪、证据、领域规则、状态机、管线、视图模型、报警工件、摄像头/URL 构建、抓流参数拼串、每帧诊断、灵敏度放宽、截图 ASCII 标注、断流重连、事件汇总、灵敏度写回和最新帧丢弃测试并已通过(含来源模式显式声明、模型置信度经适配器生效、事件号跨轮唯一、结构化凭证不入 config_version 且公开示例无凭证的守卫)。`demo/1.mp4` 的首两帧回放时间戳已验证为 0.000000 与 0.033333 秒;T-106 的真实模型/录像 smoke 在首帧得到 2 名已分析人员、第二帧得到 0 名人员且未创建事件;T-201 的视图冒烟以真实录像解码 + 真实管线 + 确定性假 Pose 适配器驱动 `build_monitor_view`,得到稳定 ID、box、14 段骨架、17/17 关键点与 NORMAL/success,断流帧 0 人且不显示摔倒标签;T-202 的报警冒烟用 `demo/1.mp4` 首帧(848×480)落盘一张可被 `cv2.imread` 读回的标注截图(480×848×3,约 330 KB)与一行 JSONL(相对截图路径、含 config_version/source_id、无 rtsp),重复派发返回 0。这些只验证管线、视图与报警工件可运行,不表示摔倒识别准确率。`init.ps1` 会检查运行时依赖、编译旧基线并运行 V1 测试,但不会安装软件包。
- 模型:`demo/best.pt` 可加载为 YOLO Pose,类别 `person`,`kpt_shape=[17, 3]`;与 `D:\PythonP\fall_detection\best.pt` 哈希一致。 - 模型:`demo/best.pt` 可加载为 YOLO Pose,类别 `person`,`kpt_shape=[17, 3]`;与 `D:\PythonP\fall_detection\best.pt` 哈希一致。
- 当前标准启动:`./init.ps1`。 - 当前标准启动:`./init.ps1`。
- 当前标准验证:`python -m compileall -q demo`。 - 当前标准验证:`python -m compileall -q demo`。
+9
View File
@@ -388,3 +388,12 @@
- 阻塞:真实线程整合需 Windows 冒烟;最新帧保留核心 `_pump` 逻辑本环境可确定性测。 - 阻塞:真实线程整合需 Windows 冒烟;最新帧保留核心 `_pump` 逻辑本环境可确定性测。
- 决策:读帧解码放后台线程、消费端只取最新帧、丢弃积压,使延迟不随推理速度增长且被钳在一次推理量级;REPLAY 保持逐帧不丢;STREAM 用收帧时单调时钟。 - 决策:读帧解码放后台线程、消费端只取最新帧、丢弃积压,使延迟不随推理速度增长且被钳在一次推理量级;REPLAY 保持逐帧不丢;STREAM 用收帧时单调时钟。
- 下一步:写 `_FrameGrabber._pump` 只留最新帧的失败测试,再实现并在 app 对 STREAM 开启。 - 下一步:写 `_FrameGrabber._pump` 只留最新帧的失败测试,再实现并在 app 对 STREAM 开启。
## 【2026-07-22】T-212 实时流只处理最新帧降延迟(完成)
- 状态:DONE
- 变更:`video_source.py` 新增 `_FrameGrabber`(daemon 线程,持续 `cap.read()` 只保留最新帧、丢弃积压,读失败置 failed,读异常不崩线程)。`VideoSource` 增 `drop_stale`(仅 STREAM 生效):懒开流后启动 grabber,`read()` 返回最新帧(无帧时 RETRYING/connecting,failed 则释放+有界退避重连),`close()` 停 grabber。`app.py` 对 STREAM 开启 `drop_stale=True`。REPLAY 路径不变(逐帧不丢)。
- 验证:新增测试——`_FrameGrabber._pump` 连续三帧后 `take_latest` 只留最新(丢弃前两帧)、读失败置 failed;STREAM+drop_stale 集成(真实线程)在轮询内返回 CONNECTED 帧。`python3 -m pytest v1/tests -q` 为 82 passed;`python3 -m compileall -q v1` 退出码 0。
- 阻塞:真实 RTSP 上的端到端延迟改善需 Windows 目视确认;丢弃最新帧核心逻辑与集成本环境已测。
- 决策:解码放后台线程、消费端只取最新帧,使延迟被钳在一次推理量级、不随推理速度增长;STREAM 用收帧时单调时钟,丢帧不破坏摔倒计时(每处理帧取真实到达时间)。
- 下一步:T-213 截图中文标注(Pillow,缺库/字体回退 ASCII)。
+3 -1
View File
@@ -38,7 +38,9 @@ class FrameWorker(QtCore.QThread):
# OpenCV reads this env var when it opens the stream, so set it first. # OpenCV reads this env var when it opens the stream, so set it first.
os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = self._config.ffmpeg_capture_options os.environ["OPENCV_FFMPEG_CAPTURE_OPTIONS"] = self._config.ffmpeg_capture_options
mode = SourceMode.STREAM if self._config.source_mode == "stream" else SourceMode.REPLAY mode = SourceMode.STREAM if self._config.source_mode == "stream" else SourceMode.REPLAY
source = VideoSource(self._config.source_url, mode=mode) source = VideoSource(
self._config.source_url, mode=mode, drop_stale=mode is SourceMode.STREAM
)
pipeline = FallPipeline.from_config(self._config, self._pose_adapter) pipeline = FallPipeline.from_config(self._config, self._pose_adapter)
try: try:
while not self._stop: while not self._stop:
+69 -1
View File
@@ -1,8 +1,10 @@
import time
import cv2 import cv2
import numpy as np import numpy as np
import pytest import pytest
from v1.video_source import SourceMode, SourceStatus, VideoSource from v1.video_source import SourceMode, SourceStatus, VideoSource, _FrameGrabber
def _write_sample_video(path): def _write_sample_video(path):
@@ -135,6 +137,72 @@ def test_stream_recovers_after_mid_stream_drop():
assert recovered.image is not None assert recovered.image is not None
class _ScriptedCapture:
def __init__(self, frames, fail_after=False):
self._frames = list(frames)
self._fail_after = fail_after
def isOpened(self):
return True
def read(self):
if self._frames:
return True, self._frames.pop(0)
return False, None
def get(self, _property_id):
return 0.0
def release(self):
pass
def test_frame_grabber_keeps_only_the_latest_frame():
frame_a = np.full((4, 4, 3), 1, dtype=np.uint8)
frame_b = np.full((4, 4, 3), 2, dtype=np.uint8)
frame_c = np.full((4, 4, 3), 3, dtype=np.uint8)
grabber = _FrameGrabber(_ScriptedCapture([frame_a, frame_b, frame_c]))
assert grabber._pump() is True
assert grabber._pump() is True
assert grabber._pump() is True
image, failed = grabber.take_latest()
assert failed is False
assert image is frame_c # older frames dropped
def test_frame_grabber_marks_failed_on_read_failure():
grabber = _FrameGrabber(_ScriptedCapture([np.zeros((4, 4, 3), dtype=np.uint8)]))
assert grabber._pump() is True
assert grabber._pump() is False
_, failed = grabber.take_latest()
assert failed is True
def test_stream_with_drop_stale_returns_a_connected_frame():
source = VideoSource(
"rtsp://demo",
mode=SourceMode.STREAM,
drop_stale=True,
capture_factory=lambda _s: _OpenCapture(),
)
try:
packet = None
for _ in range(40):
packet = source.read()
if packet.status is SourceStatus.CONNECTED and packet.image is not None:
break
time.sleep(0.02)
assert packet is not None
assert packet.status is SourceStatus.CONNECTED
assert packet.image is not None
finally:
source.close()
class _NegativeFirstTimestampCapture: class _NegativeFirstTimestampCapture:
def __init__(self): def __init__(self):
self._read_count = 0 self._read_count = 0
+81 -1
View File
@@ -1,5 +1,6 @@
"""OpenCV frame acquisition with explicit replay and reconnect states.""" """OpenCV frame acquisition with explicit replay and reconnect states."""
import threading
import time import time
from dataclasses import dataclass from dataclasses import dataclass
from enum import Enum from enum import Enum
@@ -10,6 +11,48 @@ import cv2
import numpy as np import numpy as np
class _FrameGrabber(threading.Thread):
"""Continuously read a capture, keeping only the newest frame.
Decoding runs here so a slow consumer never builds a backlog: it always gets
the latest decoded frame and older frames are dropped. This bounds live RTSP
latency to roughly one inference regardless of stream vs processing rate.
"""
def __init__(self, capture: Any) -> None:
super().__init__(daemon=True)
self._capture = capture
self._lock = threading.Lock()
self._latest: Optional[np.ndarray] = None
self._failed = False
self._stopped = threading.Event()
def _pump(self) -> bool:
try:
success, image = self._capture.read()
except Exception: # noqa: BLE001 - a released capture must not crash the thread
success, image = False, None
if not success or image is None:
with self._lock:
self._failed = True
return False
with self._lock:
self._latest = image
return True
def run(self) -> None:
while not self._stopped.is_set():
if not self._pump():
break
def take_latest(self):
with self._lock:
return self._latest, self._failed
def stop(self) -> None:
self._stopped.set()
class SourceStatus(str, Enum): class SourceStatus(str, Enum):
CONNECTED = "connected" CONNECTED = "connected"
RETRYING = "retrying" RETRYING = "retrying"
@@ -44,6 +87,7 @@ class VideoSource:
retry_initial_seconds: float = 1.0, retry_initial_seconds: float = 1.0,
retry_max_seconds: float = 16.0, retry_max_seconds: float = 16.0,
capture_factory: Optional[CaptureFactory] = None, capture_factory: Optional[CaptureFactory] = None,
drop_stale: bool = False,
) -> None: ) -> None:
if retry_initial_seconds <= 0: if retry_initial_seconds <= 0:
raise ValueError("retry_initial_seconds must be positive") raise ValueError("retry_initial_seconds must be positive")
@@ -53,6 +97,8 @@ class VideoSource:
raise ValueError("mode must be a SourceMode") raise ValueError("mode must be a SourceMode")
self._source = str(source) self._source = str(source)
self._mode = mode self._mode = mode
self._drop_stale = bool(drop_stale) and mode is SourceMode.STREAM
self._grabber: Optional[_FrameGrabber] = None
self._retry_initial_seconds = retry_initial_seconds self._retry_initial_seconds = retry_initial_seconds
self._retry_max_seconds = retry_max_seconds self._retry_max_seconds = retry_max_seconds
self._capture_factory = capture_factory or cv2.VideoCapture self._capture_factory = capture_factory or cv2.VideoCapture
@@ -69,7 +115,7 @@ class VideoSource:
return self._status return self._status
def close(self) -> None: def close(self) -> None:
self._release_capture() self._stop_grabber()
self._closed = True self._closed = True
self._status = SourceStatus.CLOSED self._status = SourceStatus.CLOSED
@@ -84,6 +130,9 @@ class VideoSource:
if self._closed: if self._closed:
return self._packet(timestamp, SourceStatus.CLOSED, "source is closed") return self._packet(timestamp, SourceStatus.CLOSED, "source is closed")
if self._drop_stale:
return self._read_latest(timestamp)
if self._mode is SourceMode.REPLAY and self._status is SourceStatus.EOF: if self._mode is SourceMode.REPLAY and self._status is SourceStatus.EOF:
return self._packet(timestamp, SourceStatus.EOF, self._last_error) return self._packet(timestamp, SourceStatus.EOF, self._last_error)
@@ -109,6 +158,37 @@ class VideoSource:
frame_timestamp = self._frame_timestamp(timestamp) frame_timestamp = self._frame_timestamp(timestamp)
return FramePacket(image=image, timestamp_monotonic=frame_timestamp, status=SourceStatus.CONNECTED) return FramePacket(image=image, timestamp_monotonic=frame_timestamp, status=SourceStatus.CONNECTED)
def _read_latest(self, timestamp: float) -> FramePacket:
"""STREAM path: return the newest grabbed frame, dropping any backlog."""
if self._capture is None:
if self._status is SourceStatus.RETRYING and timestamp < self._next_retry_at:
return self._packet(timestamp, SourceStatus.RETRYING, self._last_error)
if not self._open_capture(timestamp):
return self._packet(timestamp, self._status, self._last_error)
self._grabber = _FrameGrabber(self._capture)
self._grabber.start()
image, failed = self._grabber.take_latest()
if failed:
self._stop_grabber()
self._schedule_retry(timestamp, "stream read failed; retry scheduled")
return self._packet(timestamp, SourceStatus.RETRYING, self._last_error)
if image is None:
return self._packet(timestamp, SourceStatus.RETRYING, "connecting")
self._status = SourceStatus.CONNECTED
self._last_error = None
self._retry_delay_seconds = self._retry_initial_seconds
self._last_timestamp = timestamp
return FramePacket(image=image, timestamp_monotonic=timestamp, status=SourceStatus.CONNECTED)
def _stop_grabber(self) -> None:
if self._grabber is not None:
self._grabber.stop()
self._grabber = None
self._release_capture()
def _open_capture(self, now: float) -> bool: def _open_capture(self, now: float) -> bool:
capture = self._capture_factory(self._source) capture = self._capture_factory(self._source)
if capture is None or not capture.isOpened(): if capture is None or not capture.isOpened():