feat(store): add PostgreSQL foundation [T-009]
Harness governance / validate (push) Has been cancelled
Harness governance / validate (pull_request) Has been cancelled

This commit is contained in:
QiuSW
2026-08-07 17:52:50 +08:00
parent 04231d006a
commit 2ff7ff5618
23 changed files with 1593 additions and 24 deletions
+27 -4
View File
@@ -1,6 +1,6 @@
# Sense M1 骨架
# Sense M1/M2 接入骨架
本目录是 YoVision Sense 的 M1 接入骨架。数据库保存期望态,ONVIF 和 MediaMTX 通过端口隔离;默认关闭真实 ONVIF,显式设置 `SENSE_ONVIF_MODE=standard` 后才启用标准 SOAP/WS-Security 适配器。T-006 的真实样机结论仅覆盖已批准的精确海康基线,不能据此宣称多品牌兼容。
本目录是 YoVision Sense 的 M1/M2 接入骨架。数据库保存期望态,ONVIF 和 MediaMTX 通过端口隔离;M1 默认使用 SQLite,T-009 增加 PostgreSQL 双 schema 生产路径。默认关闭真实 ONVIF,显式设置 `SENSE_ONVIF_MODE=standard` 后才启用标准 SOAP/WS-Security 适配器。T-006 的真实样机结论仅覆盖已批准的精确海康基线,不能据此宣称多品牌兼容。
## 常用命令
@@ -14,7 +14,7 @@ go build -o bin/sense-api.exe ./cmd/sense-api
go run ./cmd/sense-api
```
Unix 将构建产物改为 `bin/sense-api`。服务默认监听 `127.0.0.1:8080`,SQLite 默认写入 `Sense/data/sense.db`,MediaMTX 控制 API 默认是 `http://127.0.0.1:9997`。当前只有 `/healthz` 与 `/readyz`,设备管理公共 API 尚未冻结。
Unix 将构建产物改为 `bin/sense-api`。服务默认监听 `127.0.0.1:8080`,SQLite 默认写入 `Sense/data/sense.db`,MediaMTX 控制 API 默认是 `http://127.0.0.1:9997`。当前运行代码只有 `/healthz` 与 `/readyz`;Sense Control API v1 虽已冻结,但 HTTP handler 和认证尚未实现。
常用环境变量:
@@ -22,7 +22,8 @@ Unix 将构建产物改为 `bin/sense-api`。服务默认监听 `127.0.0.1:8080`
| --- | --- | --- |
| `SENSE_HTTP_ADDR` | `127.0.0.1:8080` | HTTP 监听地址 |
| `SENSE_ALLOW_NON_LOOPBACK` | `false` | 显式允许监听非回环地址;只应在可信网络及外部认证/防火墙就绪后开启 |
| `SENSE_DB_DSN` | `file:data/sense.db` | SQLite DSN;凭据不得放入该值 |
| `SENSE_DB_DRIVER` | `sqlite` | `sqlite` 或 `postgres`;生产切换必须显式选择 `postgres` |
| `SENSE_DB_DSN` | `file:data/sense.db` | 所选 driver 的私有 DSN;不得写入日志、文档或仓库,PostgreSQL 密码优先由 `PGPASSFILE`/环境密钥提供 |
| `SENSE_MEDIAMTX_URL` | `http://127.0.0.1:9997` | MediaMTX 控制 API;不得包含 userinfo |
| `SENSE_RECONCILE_INTERVAL` | `5s` | 对账周期 |
| `SENSE_PROBE_INTERVAL` | `10s` | path 探活周期 |
@@ -44,6 +45,28 @@ SENSE_CREDENTIAL_<KEY>_RTSP_PASSWORD
MediaMTX `v1.19.3` 应作为独立二进制启动并只在可信网络开放 API。获取与 SHA-256 校验值见 `docs/03-tech-stack.md`。生成客户端使用固定版本工具和 vendored 官方 OpenAPI;`internal/mtx/generated/client.gen.go` 不可手改。
## T-009 PostgreSQL 17.10
初始化 SQL 位于 `deploy/postgres/`,由高权限部署步骤按文件名前缀执行;Sense 进程不会自动创建角色、schema 或 Bell 对象。`bell_app` 拥有 Site/配额和 `bell.site_quota_v1`,`sense_app` 只能读取该视图,不能读取或写入 Bell 源表。应用登录角色和密码由部署环境创建,不进入仓库。
Windows 本机集成测试从仓库根目录执行:
```powershell
./scripts/test_postgres.ps1 -PgRoot D:\pgsql17
```
脚本要求冻结的 PostgreSQL `17.10`,使用 `initdb` 创建临时 trust 集群并只监听随机回环端口,重放 migration、执行权限断言和 PostgreSQL repository 测试后停止并清理。它会核对现有 `5432` listener 前后未变化,不读取或修改 `D:\pgsql17\data`。生产安装和恢复边界见 `deploy/postgres/README.md`。
选择 PostgreSQL 运行前,管理员必须已经安装 migration,并私下设置无密码回显的连接环境:
```powershell
$env:SENSE_DB_DRIVER = 'postgres'
$env:SENSE_DB_DSN = '由部署环境私下设置'
go run ./cmd/sense-api
```
PostgreSQL 启动会检查 Sense migration 版本及当前角色的跨 schema 权限;权限过宽、配额视图不可读或 schema 未安装时 readiness 初始化失败。默认 SQLite 路径和 `cmd/sense-lab` 保持不变,T-009 不搬迁现有 SQLite 数据。
Windows 本地准备 MediaMTX(从仓库根目录执行):
```powershell
+1 -1
View File
@@ -38,7 +38,7 @@ func run(logger *slog.Logger) error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
repository, err := store.OpenSQLite(ctx, cfg.DatabaseDSN)
repository, err := store.OpenRepository(ctx, cfg.DatabaseDriver, cfg.DatabaseDSN)
if err != nil {
return err
}
+4
View File
@@ -5,6 +5,7 @@ go 1.26.0
toolchain go1.26.5
require (
github.com/jackc/pgx/v5 v5.10.0
github.com/oapi-codegen/runtime v1.6.0
modernc.org/sqlite v1.54.0
)
@@ -17,6 +18,9 @@ require (
github.com/go-openapi/jsonpointer v0.23.1 // indirect
github.com/go-openapi/swag/jsonname v0.26.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/oapi-codegen/oapi-codegen/v2 v2.8.0 // indirect
+9
View File
@@ -52,6 +52,14 @@ github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -109,6 +117,7 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/vmware-labs/yaml-jsonpath v0.3.2 h1:/5QKeCBGdsInyDCyVNLbXyilb61MXGi9NP674f9Hobk=
+13
View File
@@ -13,6 +13,7 @@ import (
const (
defaultHTTPAddress = "127.0.0.1:8080"
defaultDatabaseDriver = "sqlite"
defaultDatabaseDSN = "file:data/sense.db"
defaultMediaMTXURL = "http://127.0.0.1:9997"
defaultReconcilePeriod = 5 * time.Second
@@ -23,6 +24,7 @@ const (
type Config struct {
HTTPAddress string
AllowNonLoopback bool
DatabaseDriver string
DatabaseDSN string
MediaMTXURL string
ReconcileInterval time.Duration
@@ -58,6 +60,7 @@ func Load() (Config, error) {
cfg := Config{
HTTPAddress: stringEnv("SENSE_HTTP_ADDR", defaultHTTPAddress),
AllowNonLoopback: allow,
DatabaseDriver: stringEnv("SENSE_DB_DRIVER", defaultDatabaseDriver),
DatabaseDSN: stringEnv("SENSE_DB_DSN", defaultDatabaseDSN),
MediaMTXURL: stringEnv("SENSE_MEDIAMTX_URL", defaultMediaMTXURL),
ReconcileInterval: reconcilePeriod,
@@ -86,6 +89,16 @@ func (c Config) Validate() error {
if c.DatabaseDSN == "" {
return fmt.Errorf("SENSE_DB_DSN must not be empty")
}
databaseDriver := strings.ToLower(strings.TrimSpace(c.DatabaseDriver))
if databaseDriver == "" {
databaseDriver = defaultDatabaseDriver
}
if databaseDriver != "sqlite" && databaseDriver != "postgres" {
return fmt.Errorf("SENSE_DB_DRIVER must be sqlite or postgres")
}
if databaseDriver == "postgres" && strings.HasPrefix(c.DatabaseDSN, "file:") {
return fmt.Errorf("postgres SENSE_DB_DRIVER requires an explicit PostgreSQL SENSE_DB_DSN")
}
mediaURL, err := url.Parse(c.MediaMTXURL)
if err != nil || mediaURL.Scheme == "" || mediaURL.Host == "" {
return fmt.Errorf("invalid SENSE_MEDIAMTX_URL")
+23
View File
@@ -70,3 +70,26 @@ func TestLoadRTSPStripQueryOptIn(t *testing.T) {
t.Fatal("explicit RTSP query stripping was not loaded")
}
}
func TestValidateDatabaseDriver(t *testing.T) {
t.Parallel()
cfg := Config{
HTTPAddress: "127.0.0.1:8080",
DatabaseDriver: "postgres",
DatabaseDSN: "file:test.db",
MediaMTXURL: "http://127.0.0.1:9997",
ReconcileInterval: 1,
ProbeInterval: 1,
}
if err := cfg.Validate(); err == nil {
t.Fatal("postgres driver must reject the SQLite default DSN")
}
cfg.DatabaseDSN = "postgres://sense-runtime@127.0.0.1/yovision?sslmode=disable"
if err := cfg.Validate(); err != nil {
t.Fatalf("valid PostgreSQL selection failed: %v", err)
}
cfg.DatabaseDriver = "mysql"
if err := cfg.Validate(); err == nil {
t.Fatal("unknown database driver must be rejected")
}
}
+523
View File
@@ -0,0 +1,523 @@
package store
import (
"context"
"database/sql"
"errors"
"fmt"
"sort"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/stdlib"
"yovision/sense/internal/device"
)
// Postgres persists Sense state in the sense schema and consumes only Bell's
// versioned quota view. Migrations are deliberately installed out of process.
type Postgres struct {
db *sql.DB
}
func OpenPostgres(ctx context.Context, dsn string) (*Postgres, error) {
configuration, err := pgx.ParseConfig(dsn)
if err != nil {
return nil, errors.New("invalid postgres DSN")
}
if configuration.RuntimeParams == nil {
configuration.RuntimeParams = make(map[string]string)
}
configuration.RuntimeParams["application_name"] = "yovision-sense"
db := stdlib.OpenDB(*configuration)
db.SetMaxOpenConns(16)
db.SetMaxIdleConns(4)
db.SetConnMaxLifetime(30 * time.Minute)
if err := db.PingContext(ctx); err != nil {
db.Close()
return nil, errors.New("connect postgres database")
}
store := &Postgres{db: db}
if err := store.verifySchemaAndPrivileges(ctx); err != nil {
db.Close()
return nil, err
}
return store, nil
}
func (s *Postgres) Close() error {
return s.db.Close()
}
func (s *Postgres) verifySchemaAndPrivileges(ctx context.Context) error {
var version sql.NullInt64
if err := s.db.QueryRowContext(ctx,
`SELECT MAX(version) FROM sense.schema_migrations`).Scan(&version); err != nil || !version.Valid || version.Int64 < 1 {
return errors.New("postgres sense schema migration v1 is required")
}
var canReadView, canWriteView, canReadSource, canWriteSource bool
if err := s.db.QueryRowContext(ctx, `SELECT
has_table_privilege(current_user, 'bell.site_quota_v1', 'SELECT'),
has_table_privilege(current_user, 'bell.site_quota_v1', 'INSERT,UPDATE,DELETE'),
has_table_privilege(current_user, 'bell.sites', 'SELECT'),
has_table_privilege(current_user, 'bell.sites', 'INSERT,UPDATE,DELETE')`).
Scan(&canReadView, &canWriteView, &canReadSource, &canWriteSource); err != nil {
return errors.New("verify postgres quota privileges")
}
if !canReadView || canWriteView || canReadSource || canWriteSource {
return errors.New("postgres role violates Bell quota privilege boundary")
}
return nil
}
func (s *Postgres) CreateDevice(ctx context.Context, value device.Device) error {
if value.Generation == 0 {
value.Generation = 1
}
if value.ActualState == "" {
value.ActualState = device.ActualPending
}
if err := value.Validate(); err != nil {
return fmt.Errorf("validate device: %w", err)
}
now := time.Now().UTC()
if value.CreatedAt.IsZero() {
value.CreatedAt = now
}
value.UpdatedAt = now
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return errors.New("begin postgres create device")
}
defer tx.Rollback()
var quotaVersion any
if value.ConsumesVideoChannel() {
version, quotaErr := checkPostgresVideoQuota(ctx, tx, value.TenantID, value.SiteID, now)
if quotaErr != nil {
return quotaErr
}
quotaVersion = version
}
_, err = tx.ExecContext(ctx, `INSERT INTO sense.devices(
id, tenant_id, site_id, serial_number, name, modality,
desired_state, actual_state, endpoint_ref, credential_ref,
path_name, generation, quota_source_version, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)`,
value.ID, value.TenantID, value.SiteID, value.SerialNumber, value.Name,
value.Modality, value.DesiredState, value.ActualState, value.EndpointRef,
value.CredentialRef, value.PathName, value.Generation, quotaVersion,
value.CreatedAt, value.UpdatedAt)
if err != nil {
return errors.New("insert postgres device")
}
for _, capability := range sortedCapabilities(value.Capabilities) {
if _, err := tx.ExecContext(ctx,
`INSERT INTO sense.device_capabilities(device_id, capability) VALUES ($1, $2)`,
value.ID, capability); err != nil {
return errors.New("insert postgres device capability")
}
}
if _, err := tx.ExecContext(ctx, `INSERT INTO sense.reconcile_state(device_id, updated_at)
VALUES ($1, $2)`, value.ID, now); err != nil {
return errors.New("insert postgres reconcile state")
}
if err := tx.Commit(); err != nil {
return errors.New("commit postgres create device")
}
return nil
}
func checkPostgresVideoQuota(
ctx context.Context,
tx *sql.Tx,
tenantID, siteID string,
now time.Time,
) (int64, error) {
// A transaction-scoped lock shared by all Sense instances makes count +
// write atomic per logical site without locking Bell-owned rows.
if _, err := tx.ExecContext(ctx,
`SELECT pg_advisory_xact_lock(hashtext($1), hashtext($2))`, tenantID, siteID); err != nil {
return 0, errors.New("lock postgres site quota admission")
}
var limit int
var sourceVersion int64
var sourceUpdatedAt time.Time
err := tx.QueryRowContext(ctx, `SELECT max_video_channels, source_version, source_updated_at
FROM bell.site_quota_v1 WHERE tenant_id = $1 AND site_id = $2`, tenantID, siteID).
Scan(&limit, &sourceVersion, &sourceUpdatedAt)
if errors.Is(err, sql.ErrNoRows) {
return 0, projectionUnavailable()
}
if err != nil {
return 0, projectionUnavailable()
}
if limit < 1 || limit > device.MaximumVideoChannels || sourceVersion < 1 || sourceUpdatedAt.IsZero() {
return 0, projectionInvalid()
}
var previous sql.NullInt64
err = tx.QueryRowContext(ctx, `SELECT source_version
FROM sense.site_quota_projection_state WHERE tenant_id = $1 AND site_id = $2`,
tenantID, siteID).Scan(&previous)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return 0, errors.New("read postgres quota projection state")
}
if previous.Valid && sourceVersion < previous.Int64 {
return 0, projectionInvalid()
}
if _, err := tx.ExecContext(ctx, `INSERT INTO sense.site_quota_projection_state(
tenant_id, site_id, source_version, synced_at
) VALUES ($1, $2, $3, $4)
ON CONFLICT (tenant_id, site_id) DO UPDATE SET
source_version = EXCLUDED.source_version,
synced_at = EXCLUDED.synced_at`, tenantID, siteID, sourceVersion, now); err != nil {
return 0, errors.New("record postgres quota projection state")
}
var current int
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*)
FROM sense.devices d
JOIN sense.device_capabilities c ON c.device_id = d.id
WHERE d.tenant_id = $1 AND d.site_id = $2
AND d.desired_state = 'enabled'
AND c.capability = 'video_capture'`, tenantID, siteID).Scan(&current); err != nil {
return 0, errors.New("count postgres site video channels")
}
if current >= limit {
return 0, &device.QuotaExceededError{TenantID: tenantID, SiteID: siteID, Limit: limit}
}
return sourceVersion, nil
}
func (s *Postgres) SetDesiredState(ctx context.Context, id string, desired device.DesiredState) error {
if desired != device.DesiredEnabled && desired != device.DesiredDisabled {
return fmt.Errorf("invalid desired state %q", desired)
}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return errors.New("begin postgres desired-state update")
}
defer tx.Rollback()
var tenantID, siteID, endpointRef, pathName string
var current device.DesiredState
err = tx.QueryRowContext(ctx, `SELECT tenant_id, site_id, desired_state, endpoint_ref, path_name
FROM sense.devices WHERE id = $1 FOR UPDATE`, id).
Scan(&tenantID, &siteID, &current, &endpointRef, &pathName)
if errors.Is(err, sql.ErrNoRows) {
return ErrNotFound
}
if err != nil {
return errors.New("read postgres device desired state")
}
if current == desired {
return tx.Commit()
}
var quotaVersion any
if desired == device.DesiredEnabled {
var hasVideo bool
if err := tx.QueryRowContext(ctx, `SELECT EXISTS(
SELECT 1 FROM sense.device_capabilities
WHERE device_id = $1 AND capability = 'video_capture'
)`, id).Scan(&hasVideo); err != nil {
return errors.New("read postgres video capability")
}
if hasVideo {
if strings.TrimSpace(endpointRef) == "" || strings.TrimSpace(pathName) == "" {
return errors.New("enabled video devices require endpoint ref and path name")
}
version, quotaErr := checkPostgresVideoQuota(ctx, tx, tenantID, siteID, time.Now().UTC())
if quotaErr != nil {
return quotaErr
}
quotaVersion = version
}
}
result, err := tx.ExecContext(ctx, `UPDATE sense.devices SET
desired_state = $1, actual_state = 'pending', generation = generation + 1,
quota_source_version = COALESCE($2, quota_source_version), updated_at = $3
WHERE id = $4`, desired, quotaVersion, time.Now().UTC(), id)
if err != nil {
return errors.New("update postgres desired state")
}
if affected, _ := result.RowsAffected(); affected != 1 {
return ErrNotFound
}
if _, err := tx.ExecContext(ctx, `UPDATE sense.reconcile_state SET
failure_count = 0, next_attempt_at = NULL, last_error_code = NULL, updated_at = $1
WHERE device_id = $2`, time.Now().UTC(), id); err != nil {
return errors.New("reset postgres reconcile state")
}
if err := tx.Commit(); err != nil {
return errors.New("commit postgres desired-state update")
}
return nil
}
func (s *Postgres) GetDevice(ctx context.Context, id string) (device.Device, error) {
value, err := scanPostgresDevice(s.db.QueryRowContext(ctx, postgresDeviceSelect+` WHERE d.id = $1`, id))
if errors.Is(err, sql.ErrNoRows) {
return device.Device{}, ErrNotFound
}
if err != nil {
return device.Device{}, errors.New("get postgres device")
}
value.Capabilities, err = s.capabilities(ctx, value.ID)
if err != nil {
return device.Device{}, err
}
return value, nil
}
func (s *Postgres) ListDueReconcile(ctx context.Context, now time.Time, limit int) ([]ReconcileCandidate, error) {
if limit <= 0 {
return nil, nil
}
rows, err := s.db.QueryContext(ctx, `SELECT `+postgresDeviceColumns+`, r.failure_count, r.next_attempt_at
FROM sense.devices d
JOIN sense.reconcile_state r ON r.device_id = d.id
WHERE d.desired_state = 'enabled'
AND EXISTS (SELECT 1 FROM sense.device_capabilities c
WHERE c.device_id = d.id AND c.capability = 'video_capture')
AND (r.observed_generation < d.generation OR r.failure_count > 0)
AND (r.next_attempt_at IS NULL OR r.next_attempt_at <= $1)
ORDER BY d.updated_at, d.id LIMIT $2`, now, limit)
if err != nil {
return nil, errors.New("list postgres due reconcile devices")
}
defer rows.Close()
values := make([]ReconcileCandidate, 0)
for rows.Next() {
var candidate ReconcileCandidate
var nextAttempt sql.NullTime
if err := rows.Scan(
&candidate.Device.ID, &candidate.Device.TenantID, &candidate.Device.SiteID,
&candidate.Device.SerialNumber, &candidate.Device.Name, &candidate.Device.Modality,
&candidate.Device.DesiredState, &candidate.Device.ActualState,
&candidate.Device.EndpointRef, &candidate.Device.CredentialRef,
&candidate.Device.PathName, &candidate.Device.Generation,
&candidate.Device.CreatedAt, &candidate.Device.UpdatedAt,
&candidate.FailureCount, &nextAttempt,
); err != nil {
return nil, errors.New("scan postgres due reconcile device")
}
if nextAttempt.Valid {
value := nextAttempt.Time
candidate.NextAttempt = &value
}
values = append(values, candidate)
}
if err := rows.Err(); err != nil {
return nil, errors.New("iterate postgres due reconcile devices")
}
for index := range values {
values[index].Device.Capabilities, err = s.capabilities(ctx, values[index].Device.ID)
if err != nil {
return nil, err
}
}
return values, nil
}
func (s *Postgres) ListEnabledVideoDevices(ctx context.Context, limit int) ([]device.Device, error) {
if limit <= 0 {
return nil, nil
}
rows, err := s.db.QueryContext(ctx, postgresDeviceSelect+`
WHERE d.desired_state = 'enabled'
AND EXISTS (SELECT 1 FROM sense.device_capabilities c
WHERE c.device_id = d.id AND c.capability = 'video_capture')
ORDER BY d.id LIMIT $1`, limit)
if err != nil {
return nil, errors.New("list postgres enabled video devices")
}
defer rows.Close()
values := make([]device.Device, 0)
for rows.Next() {
value, scanErr := scanPostgresDevice(rows)
if scanErr != nil {
return nil, errors.New("scan postgres enabled video device")
}
values = append(values, value)
}
if err := rows.Err(); err != nil {
return nil, errors.New("iterate postgres enabled video devices")
}
for index := range values {
values[index].Capabilities, err = s.capabilities(ctx, values[index].ID)
if err != nil {
return nil, err
}
}
return values, nil
}
func (s *Postgres) MarkReconciled(ctx context.Context, id string, generation int64, now time.Time) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return errors.New("begin postgres reconciled update")
}
defer tx.Rollback()
result, err := tx.ExecContext(ctx, `UPDATE sense.reconcile_state SET
failure_count = 0, next_attempt_at = NULL, last_error_code = NULL,
observed_generation = $1, updated_at = $2 WHERE device_id = $3`, generation, now, id)
if err != nil {
return errors.New("mark postgres device reconciled")
}
if affected, _ := result.RowsAffected(); affected != 1 {
return ErrNotFound
}
if _, err := tx.ExecContext(ctx, `UPDATE sense.devices
SET actual_state = 'pending', updated_at = $1 WHERE id = $2`, now, id); err != nil {
return errors.New("mark postgres reconciled device pending")
}
if err := tx.Commit(); err != nil {
return errors.New("commit postgres reconciled update")
}
return nil
}
func (s *Postgres) MarkReconcileFailure(
ctx context.Context,
id string,
failureCount int,
nextAttempt time.Time,
errorCode string,
now time.Time,
) error {
tx, err := s.db.BeginTx(ctx, nil)
if err != nil {
return errors.New("begin postgres reconcile failure update")
}
defer tx.Rollback()
result, err := tx.ExecContext(ctx, `UPDATE sense.reconcile_state SET
failure_count = $1, next_attempt_at = $2, last_error_code = $3, updated_at = $4
WHERE device_id = $5`, failureCount, nextAttempt, errorCode, now, id)
if err != nil {
return errors.New("mark postgres reconcile failure")
}
if affected, _ := result.RowsAffected(); affected != 1 {
return ErrNotFound
}
if _, err := tx.ExecContext(ctx, `UPDATE sense.devices
SET actual_state = 'failed', updated_at = $1 WHERE id = $2`, now, id); err != nil {
return errors.New("mark postgres failed device state")
}
if err := tx.Commit(); err != nil {
return errors.New("commit postgres reconcile failure")
}
return nil
}
func (s *Postgres) UpdateActualState(ctx context.Context, id string, state device.ActualState, now time.Time) error {
if state != device.ActualPending && state != device.ActualOnline &&
state != device.ActualOffline && state != device.ActualFailed {
return fmt.Errorf("invalid actual state %q", state)
}
result, err := s.db.ExecContext(ctx, `UPDATE sense.devices
SET actual_state = $1, updated_at = $2 WHERE id = $3`, state, now, id)
if err != nil {
return errors.New("update postgres actual state")
}
if affected, _ := result.RowsAffected(); affected != 1 {
return ErrNotFound
}
return nil
}
func (s *Postgres) RequestReconcile(ctx context.Context, id string, now time.Time) error {
result, err := s.db.ExecContext(ctx, `UPDATE sense.reconcile_state
SET observed_generation = 0, updated_at = $1 WHERE device_id = $2`, now, id)
if err != nil {
return errors.New("request postgres device reconciliation")
}
if affected, _ := result.RowsAffected(); affected != 1 {
return ErrNotFound
}
return nil
}
func (s *Postgres) ConvergenceSnapshot(ctx context.Context) (ConvergenceSnapshot, error) {
rows, err := s.db.QueryContext(ctx, `SELECT
d.id, d.path_name, d.desired_state, d.actual_state, d.generation,
r.observed_generation, r.failure_count, r.next_attempt_at, r.last_error_code
FROM sense.devices d
JOIN sense.reconcile_state r ON r.device_id = d.id
WHERE d.desired_state = 'enabled'
AND EXISTS (SELECT 1 FROM sense.device_capabilities c
WHERE c.device_id = d.id AND c.capability = 'video_capture')
ORDER BY d.id`)
if err != nil {
return ConvergenceSnapshot{}, errors.New("query postgres convergence snapshot")
}
defer rows.Close()
snapshot := ConvergenceSnapshot{Devices: make([]DeviceConvergence, 0)}
for rows.Next() {
var value DeviceConvergence
var nextAttempt sql.NullTime
var lastError sql.NullString
if err := rows.Scan(
&value.ID, &value.PathName, &value.DesiredState, &value.ActualState,
&value.Generation, &value.ObservedGeneration, &value.FailureCount,
&nextAttempt, &lastError,
); err != nil {
return ConvergenceSnapshot{}, errors.New("scan postgres convergence snapshot")
}
if nextAttempt.Valid {
point := nextAttempt.Time
value.NextAttemptAt = &point
}
if lastError.Valid {
value.LastErrorCode = lastError.String
}
value.Converged = value.ObservedGeneration == value.Generation &&
value.FailureCount == 0 && value.ActualState == device.ActualOnline
if !value.Converged {
snapshot.Unconverged++
}
snapshot.Devices = append(snapshot.Devices, value)
}
if err := rows.Err(); err != nil {
return ConvergenceSnapshot{}, errors.New("iterate postgres convergence snapshot")
}
snapshot.Total = len(snapshot.Devices)
return snapshot, nil
}
const postgresDeviceColumns = `d.id, d.tenant_id, d.site_id, d.serial_number, d.name, d.modality,
d.desired_state, d.actual_state, d.endpoint_ref, d.credential_ref,
d.path_name, d.generation, d.created_at, d.updated_at`
const postgresDeviceSelect = `SELECT ` + postgresDeviceColumns + ` FROM sense.devices d`
func scanPostgresDevice(row scanner) (device.Device, error) {
var value device.Device
err := row.Scan(
&value.ID, &value.TenantID, &value.SiteID, &value.SerialNumber,
&value.Name, &value.Modality, &value.DesiredState, &value.ActualState,
&value.EndpointRef, &value.CredentialRef, &value.PathName,
&value.Generation, &value.CreatedAt, &value.UpdatedAt,
)
return value, err
}
func (s *Postgres) capabilities(ctx context.Context, id string) ([]device.Capability, error) {
rows, err := s.db.QueryContext(ctx, `SELECT capability
FROM sense.device_capabilities WHERE device_id = $1 ORDER BY capability`, id)
if err != nil {
return nil, errors.New("list postgres device capabilities")
}
defer rows.Close()
values := make([]device.Capability, 0)
for rows.Next() {
var value device.Capability
if err := rows.Scan(&value); err != nil {
return nil, errors.New("scan postgres device capability")
}
values = append(values, value)
}
if err := rows.Err(); err != nil {
return nil, errors.New("iterate postgres device capabilities")
}
sort.Slice(values, func(i, j int) bool { return values[i] < values[j] })
return values, nil
}
+353
View File
@@ -0,0 +1,353 @@
package store
import (
"context"
"database/sql"
"errors"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
"yovision/sense/internal/device"
)
const (
postgresTestDSNEnv = "YOVISION_TEST_POSTGRES_DSN"
postgresTestAdminDSNEnv = "YOVISION_TEST_POSTGRES_ADMIN_DSN"
)
func TestPostgresInvalidDSNDoesNotLeakInput(t *testing.T) {
secret := "do-not-echo-this-value"
_, err := OpenPostgres(context.Background(), "postgres://sense:"+secret+"@%zz")
if err == nil {
t.Fatal("invalid PostgreSQL DSN was accepted")
}
if strings.Contains(err.Error(), secret) {
t.Fatal("PostgreSQL configuration error leaked DSN input")
}
}
func TestOpenRepositoryRejectsUnknownDriver(t *testing.T) {
_, err := OpenRepository(context.Background(), "mysql", "unused")
if err == nil {
t.Fatal("unknown repository driver was accepted")
}
}
func TestPostgresDefaultAndMaximumQuota(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
insertBellSite(t, admin, "tenant-default", "site-default", 0)
for index := 1; index <= device.DefaultVideoChannels; index++ {
if err := store.CreateDevice(ctx, videoDevice(index, "tenant-default", "site-default")); err != nil {
t.Fatalf("create default channel %d: %v", index, err)
}
}
radar := device.Device{
ID: "radar-default", TenantID: "tenant-default", SiteID: "site-default",
SerialNumber: "radar-default", Name: "Radar", Modality: device.ModalityRadar,
Capabilities: []device.Capability{device.CapabilityTelemetry},
DesiredState: device.DesiredEnabled, ActualState: device.ActualPending,
}
if err := store.CreateDevice(ctx, radar); err != nil {
t.Fatalf("non-video device must not consume video quota: %v", err)
}
err := store.CreateDevice(ctx, videoDevice(17, "tenant-default", "site-default"))
var quotaError *device.QuotaExceededError
if !errors.As(err, &quotaError) || quotaError.Limit != 16 {
t.Fatalf("expected default quota error, got %v", err)
}
insertBellSite(t, admin, "tenant-max", "site-max", 128)
for index := 1; index <= device.MaximumVideoChannels; index++ {
if err := store.CreateDevice(ctx, videoDevice(index+1000, "tenant-max", "site-max")); err != nil {
t.Fatalf("create maximum channel %d: %v", index, err)
}
}
err = store.CreateDevice(ctx, videoDevice(1129, "tenant-max", "site-max"))
if !errors.As(err, &quotaError) || quotaError.Limit != 128 {
t.Fatalf("expected maximum quota error, got %v", err)
}
}
func TestPostgresConcurrentAdmissionCannotExceedQuota(t *testing.T) {
store, admin := openPostgresTestStore(t)
insertBellSite(t, admin, "tenant", "site", 1)
ctx := context.Background()
start := make(chan struct{})
errorsFound := make(chan error, 2)
var wait sync.WaitGroup
for index := 1; index <= 2; index++ {
wait.Add(1)
go func(index int) {
defer wait.Done()
<-start
errorsFound <- store.CreateDevice(ctx, videoDevice(index, "tenant", "site"))
}(index)
}
close(start)
wait.Wait()
close(errorsFound)
var succeeded, rejected int
for err := range errorsFound {
if err == nil {
succeeded++
continue
}
var quotaError *device.QuotaExceededError
if errors.As(err, &quotaError) {
rejected++
continue
}
t.Fatalf("unexpected concurrent admission error: %v", err)
}
if succeeded != 1 || rejected != 1 {
t.Fatalf("expected one success and one quota rejection, got success=%d rejected=%d", succeeded, rejected)
}
}
func TestPostgresProjectionFailureAndRollbackFailClosed(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
err := store.CreateDevice(ctx, videoDevice(1, "missing-tenant", "missing-site"))
if !errors.Is(err, ErrQuotaProjectionUnavailable) {
t.Fatalf("missing projection must fail closed, got %v", err)
}
insertBellSite(t, admin, "tenant", "site", 2)
if _, err := store.db.ExecContext(ctx, `INSERT INTO sense.site_quota_projection_state(
tenant_id, site_id, source_version, synced_at
) VALUES ('tenant', 'site', 99, clock_timestamp())`); err != nil {
t.Fatal(err)
}
err = store.CreateDevice(ctx, videoDevice(2, "tenant", "site"))
if !errors.Is(err, ErrQuotaProjectionInvalid) {
t.Fatalf("projection version rollback must fail closed, got %v", err)
}
var devices int
if err := store.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM sense.devices`).Scan(&devices); err != nil {
t.Fatal(err)
}
if devices != 0 {
t.Fatalf("failed admissions must not persist devices, got %d", devices)
}
}
func TestPostgresLowerQuotaDoesNotDisableAndEnableStillChecks(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
insertBellSite(t, admin, "tenant", "site", 2)
for index := 1; index <= 2; index++ {
if err := store.CreateDevice(ctx, videoDevice(index, "tenant", "site")); err != nil {
t.Fatal(err)
}
}
disabled := videoDevice(3, "tenant", "site")
disabled.DesiredState = device.DesiredDisabled
if err := store.CreateDevice(ctx, disabled); err != nil {
t.Fatal(err)
}
if _, err := admin.ExecContext(ctx, `UPDATE bell.sites SET max_video_channels = 1
WHERE tenant_id = 'tenant' AND id = 'site'`); err != nil {
t.Fatal(err)
}
for index := 1; index <= 2; index++ {
value, err := store.GetDevice(ctx, fmt.Sprintf("camera-%03d", index))
if err != nil {
t.Fatal(err)
}
if value.DesiredState != device.DesiredEnabled {
t.Fatalf("quota decrease disabled existing device %d", index)
}
}
err := store.SetDesiredState(ctx, "camera-003", device.DesiredEnabled)
var quotaError *device.QuotaExceededError
if !errors.As(err, &quotaError) || quotaError.Limit != 1 {
t.Fatalf("new enable after quota decrease must be rejected, got %v", err)
}
}
func TestPostgresTenantIsolationAndConvergenceParity(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
insertBellSite(t, admin, "tenant-a", "site", 1)
insertBellSite(t, admin, "tenant-b", "site", 1)
left := videoDevice(1, "tenant-a", "site")
right := videoDevice(2, "tenant-b", "site")
right.SerialNumber = left.SerialNumber
if err := store.CreateDevice(ctx, left); err != nil {
t.Fatal(err)
}
if err := store.CreateDevice(ctx, right); err != nil {
t.Fatal(err)
}
now := time.Date(2026, 8, 7, 0, 0, 0, 0, time.UTC)
if err := store.MarkReconciled(ctx, left.ID, 1, now); err != nil {
t.Fatal(err)
}
if err := store.UpdateActualState(ctx, left.ID, device.ActualOnline, now); err != nil {
t.Fatal(err)
}
snapshot, err := store.ConvergenceSnapshot(ctx)
if err != nil {
t.Fatal(err)
}
if snapshot.Total != 2 || snapshot.Unconverged != 1 {
t.Fatalf("unexpected convergence snapshot: %+v", snapshot)
}
if err := store.RequestReconcile(ctx, left.ID, now.Add(time.Second)); err != nil {
t.Fatal(err)
}
snapshot, err = store.ConvergenceSnapshot(ctx)
if err != nil {
t.Fatal(err)
}
if snapshot.Unconverged != 2 {
t.Fatalf("runtime reconcile request was not persisted: %+v", snapshot)
}
}
func TestPostgresRecordsQuotaSourceVersion(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
insertBellSite(t, admin, "tenant", "site", 2)
if err := store.CreateDevice(ctx, videoDevice(1, "tenant", "site")); err != nil {
t.Fatal(err)
}
var firstVersion int64
if err := store.db.QueryRowContext(ctx, `SELECT quota_source_version
FROM sense.devices WHERE id = 'camera-001'`).Scan(&firstVersion); err != nil {
t.Fatal(err)
}
if _, err := admin.ExecContext(ctx, `UPDATE bell.sites SET name = name
WHERE tenant_id = 'tenant' AND id = 'site'`); err != nil {
t.Fatal(err)
}
second := videoDevice(2, "tenant", "site")
if err := store.CreateDevice(ctx, second); err != nil {
t.Fatal(err)
}
var secondVersion int64
if err := store.db.QueryRowContext(ctx, `SELECT quota_source_version
FROM sense.devices WHERE id = 'camera-002'`).Scan(&secondVersion); err != nil {
t.Fatal(err)
}
if firstVersion != 1 || secondVersion != 2 {
t.Fatalf("expected recorded projection versions 1 and 2, got %d and %d", firstVersion, secondVersion)
}
}
func TestPostgresReconcilePortParity(t *testing.T) {
store, admin := openPostgresTestStore(t)
ctx := context.Background()
insertBellSite(t, admin, "tenant", "site", 1)
if err := store.CreateDevice(ctx, videoDevice(1, "tenant", "site")); err != nil {
t.Fatal(err)
}
now := time.Now().UTC()
due, err := store.ListDueReconcile(ctx, now.Add(time.Second), 10)
if err != nil || len(due) != 1 || len(due[0].Device.Capabilities) != 2 {
t.Fatalf("unexpected due reconcile list: values=%+v error=%v", due, err)
}
enabled, err := store.ListEnabledVideoDevices(ctx, 10)
if err != nil || len(enabled) != 1 || enabled[0].ID != "camera-001" {
t.Fatalf("unexpected enabled video list: values=%+v error=%v", enabled, err)
}
if err := store.SetDesiredState(ctx, "camera-001", device.DesiredEnabled); err != nil {
t.Fatal(err)
}
value, err := store.GetDevice(ctx, "camera-001")
if err != nil || value.Generation != 1 {
t.Fatalf("idempotent desired state changed generation: value=%+v error=%v", value, err)
}
nextAttempt := now.Add(time.Minute)
if err := store.MarkReconcileFailure(ctx, "camera-001", 1, nextAttempt, "test_failure", now); err != nil {
t.Fatal(err)
}
due, err = store.ListDueReconcile(ctx, now.Add(30*time.Second), 10)
if err != nil || len(due) != 0 {
t.Fatalf("backoff device became due early: values=%+v error=%v", due, err)
}
due, err = store.ListDueReconcile(ctx, now.Add(2*time.Minute), 10)
if err != nil || len(due) != 1 || due[0].FailureCount != 1 {
t.Fatalf("backoff device did not become due: values=%+v error=%v", due, err)
}
value, err = store.GetDevice(ctx, "camera-001")
if err != nil || value.ActualState != device.ActualFailed {
t.Fatalf("reconcile failure did not persist actual state: value=%+v error=%v", value, err)
}
}
func TestPostgresOpenRejectsOverprivilegedRuntimeRole(t *testing.T) {
_, admin := openPostgresTestStore(t)
ctx := context.Background()
if _, err := admin.ExecContext(ctx,
`GRANT UPDATE ON bell.site_quota_v1 TO yovision_t009_sense`); err != nil {
t.Fatal(err)
}
defer func() {
_, _ = admin.ExecContext(context.Background(),
`REVOKE UPDATE ON bell.site_quota_v1 FROM yovision_t009_sense`)
}()
value, err := OpenPostgres(ctx, os.Getenv(postgresTestDSNEnv))
if value != nil {
_ = value.Close()
t.Fatal("overprivileged runtime role was accepted")
}
if err == nil || !strings.Contains(err.Error(), "privilege boundary") {
t.Fatalf("expected privilege-boundary error, got %v", err)
}
}
func openPostgresTestStore(t *testing.T) (*Postgres, *sql.DB) {
t.Helper()
dsn := os.Getenv(postgresTestDSNEnv)
adminDSN := os.Getenv(postgresTestAdminDSNEnv)
if dsn == "" || adminDSN == "" {
t.Skip("PostgreSQL integration DSNs are not configured")
}
admin, err := sql.Open("pgx", adminDSN)
if err != nil {
t.Fatal(err)
}
if err := admin.PingContext(context.Background()); err != nil {
admin.Close()
t.Fatal("connect PostgreSQL test administrator")
}
if _, err := admin.ExecContext(context.Background(), `TRUNCATE
sense.device_capabilities,
sense.reconcile_state,
sense.devices,
sense.site_quota_projection_state,
bell.sites CASCADE`); err != nil {
admin.Close()
t.Fatal(err)
}
store, err := OpenPostgres(context.Background(), dsn)
if err != nil {
admin.Close()
t.Fatal(err)
}
t.Cleanup(func() {
_ = store.Close()
_ = admin.Close()
})
return store, admin
}
func insertBellSite(t *testing.T, admin *sql.DB, tenantID, siteID string, quota int) {
t.Helper()
query := `INSERT INTO bell.sites(tenant_id, id, name, max_video_channels)
VALUES ($1, $2, $3, $4)`
arguments := []any{tenantID, siteID, "Test Site", quota}
if quota == 0 {
query = `INSERT INTO bell.sites(tenant_id, id, name) VALUES ($1, $2, $3)`
arguments = arguments[:3]
}
if _, err := admin.ExecContext(context.Background(), query, arguments...); err != nil {
t.Fatal(err)
}
}
+63
View File
@@ -0,0 +1,63 @@
package store
import (
"context"
"errors"
"fmt"
"strings"
"time"
"yovision/sense/internal/device"
)
const (
DriverSQLite = "sqlite"
DriverPostgres = "postgres"
)
var (
ErrQuotaProjectionUnavailable = errors.New("quota projection unavailable")
ErrQuotaProjectionInvalid = errors.New("quota projection invalid")
)
// Repository is the storage boundary used by the Sense process. SQLite stays
// available for M1 development; PostgreSQL implements the M2 production path.
type Repository interface {
Close() error
CreateDevice(context.Context, device.Device) error
SetDesiredState(context.Context, string, device.DesiredState) error
GetDevice(context.Context, string) (device.Device, error)
ListDueReconcile(context.Context, time.Time, int) ([]ReconcileCandidate, error)
ListEnabledVideoDevices(context.Context, int) ([]device.Device, error)
MarkReconciled(context.Context, string, int64, time.Time) error
MarkReconcileFailure(context.Context, string, int, time.Time, string, time.Time) error
UpdateActualState(context.Context, string, device.ActualState, time.Time) error
RequestReconcile(context.Context, string, time.Time) error
ConvergenceSnapshot(context.Context) (ConvergenceSnapshot, error)
}
func OpenRepository(ctx context.Context, driver, dsn string) (Repository, error) {
switch strings.ToLower(strings.TrimSpace(driver)) {
case "", DriverSQLite:
return OpenSQLite(ctx, dsn)
case DriverPostgres:
return OpenPostgres(ctx, dsn)
default:
return nil, fmt.Errorf("unsupported database driver %q", driver)
}
}
type quotaProjectionError struct {
kind error
}
func (e *quotaProjectionError) Error() string { return e.kind.Error() }
func (e *quotaProjectionError) Unwrap() error { return e.kind }
func projectionUnavailable() error {
return &quotaProjectionError{kind: ErrQuotaProjectionUnavailable}
}
func projectionInvalid() error {
return &quotaProjectionError{kind: ErrQuotaProjectionInvalid}
}