mirror of
https://github.com/go-sigma/sigma.git
synced 2026-10-28 14:34:34 +08:00
docs(todo): move non-HA backlog above completed list
Promote the password recovery flow to its own backlog section and renumber the completed items so the remaining non-HA work is grouped together.
This commit is contained in:
Tosone committed
1 parent
747da40b7d
commit
47dc5db656
1 file changed
+107
-107
@@ -74,115 +74,9 @@ Remaining:
|
||||
|
||||
---
|
||||
|
||||
## Completed
|
||||
|
||||
### 5. Database Connection Pool Configuration
|
||||
|
||||
**Result**: [`pkg/dal/dal.go`](file:///Users/tosone/code/github/sigma/pkg/dal/dal.go)
|
||||
|
||||
- MySQL and PostgreSQL now configure `SetMaxOpenConns`, `SetMaxIdleConns`, `SetConnMaxLifetime`, and `SetConnMaxIdleTime` instead of using unlimited driver defaults
|
||||
- All pool parameters are exposed in `config.yaml` so each environment can tune them
|
||||
|
||||
### 6. Full Aggregation Queries on the Push Hot Path
|
||||
|
||||
**Result**: [`pkg/dal/repository/registry/artifact.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/registry/artifact.go)
|
||||
|
||||
- Repository/namespace size is updated atomically inside the push transaction with `UPDATE repository SET size = size + ?` (O(1) instead of O(N))
|
||||
- Quota checks use the incremental counters instead of real-time `SELECT SUM(...)`
|
||||
|
||||
### 7. Database Hot Spots in the Analytics Backend
|
||||
|
||||
**Result**: [`pkg/infra/counter/`](file:///Users/tosone/code/github/sigma/pkg/infra/counter/), [`pkg/background/daemon/analytics/flush.go`](file:///Users/tosone/code/github/sigma/pkg/background/daemon/analytics/flush.go), [`pkg/service/analytics/analytics.go`](file:///Users/tosone/code/github/sigma/pkg/service/analytics/analytics.go)
|
||||
|
||||
- The `database` counter backend has been removed; only `inmemory` and `redis` are registered
|
||||
- `counterBackend` defaults to `inmemory` in `conf/config.yaml`; the inmemory counter aggregates by hour with mutex-protected maps and flushes dirty hours to the DB in `Flush`
|
||||
- Inmemory and redis backends both flush during graceful shutdown via `graceful.RunAtShutdown`
|
||||
- Production deployments can switch to `redis` for shared multi-replica aggregation
|
||||
|
||||
**Follow-ups (small)**:
|
||||
|
||||
- [ ] Restore the current-hour counters from the database on startup so a rolling restart does not lose the in-flight hour
|
||||
- [ ] Wire the `analytics.reconcileCron` config value to a reconcile job (defined in config but not yet scheduled)
|
||||
|
||||
### 8. Insufficient Cache System
|
||||
|
||||
**Result**: [`pkg/infra/cache/`](file:///Users/tosone/code/github/sigma/pkg/infra/cache/), [`pkg/dal/repository/namespace/namespace_cache.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/namespace/namespace_cache.go), [`pkg/dal/repository/registry/repository_cache.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/registry/repository_cache.go), [`pkg/app/cache_prewarm.go`](file:///Users/tosone/code/github/sigma/pkg/app/cache_prewarm.go)
|
||||
|
||||
- `Get` merges concurrent fetches for the same missed key with `singleflight`; in-memory and Redis caches both support TTL
|
||||
- Short-lived negative caching prevents deterministic misses (e.g. `gorm.ErrRecordNotFound`) from hitting the DB
|
||||
- `cache_requests_total` and `cache_fetch_duration_seconds` expose low-cardinality metrics by backend, prefix, and result
|
||||
- Namespace/repository metadata is cached through DI-injected DAL cached decorators; transaction-scoped `New...Repository(tx)` returns the raw implementation to avoid caching uncommitted data
|
||||
- Create/update/delete/size paths invalidate cache after successful writes; startup prewarming covers recently updated namespaces/repositories up to `cache.prewarm.limit`
|
||||
- Multi-replica production deployments should use Redis cache so invalidation is shared; single-node dev can use inmemory
|
||||
|
||||
### 9. Distributed Lock Reliability
|
||||
|
||||
**Result**: [`pkg/infra/lock/`](file:///Users/tosone/code/github/sigma/pkg/infra/lock/)
|
||||
|
||||
- Redis locks use `github.com/go-redsync/redsync/v4` with a renewal watchdog on `AcquireWithRenew`
|
||||
- Lock retry uses exponential backoff through the redsync retry delay function
|
||||
- Metrics cover lock wait time, hold time, and renewal results
|
||||
- Multi-node Redlock/Raft quorum was evaluated and intentionally not introduced; re-evaluate only if stronger multi-Redis consistency is required
|
||||
|
||||
### 10. Missing Streaming and Multipart Uploads in Storage
|
||||
|
||||
**Result**: [`pkg/storage/storage.go`](file:///Users/tosone/code/github/sigma/pkg/storage/storage.go), drivers in [`pkg/storage/`](file:///Users/tosone/code/github/sigma/pkg/storage/), [`pkg/service/distribution/upload/upload.go`](file:///Users/tosone/code/github/sigma/pkg/service/distribution/upload/upload.go)
|
||||
|
||||
- The `StorageDriver` interface now exposes a multipart abstraction: `CreateUploadID`, `UploadPart`, `CommitUpload`, `AbortUpload`
|
||||
- All four drivers (filesystem, S3, COS, OSS) implement multipart upload; the blob upload path uses `UploadPart`/`CommitUpload` with part-number tracking
|
||||
- Multipart copy constants (`MultipartCopyThresholdSize=64MB`, chunk size, max concurrency) support large object copies
|
||||
|
||||
**Follow-ups (small)**:
|
||||
|
||||
- [ ] Expose upload progress tracking through a `StreamingWriter`/progress callback API (OCI `PATCH` already provides resumable semantics)
|
||||
|
||||
### 11. Incomplete Graceful Shutdown and Startup Checks
|
||||
|
||||
**Result**: [`pkg/graceful/graceful.go`](file:///Users/tosone/code/github/sigma/pkg/graceful/graceful.go), [`pkg/app/readiness.go`](file:///Users/tosone/code/github/sigma/pkg/app/readiness.go), [`pkg/app/drain.go`](file:///Users/tosone/code/github/sigma/pkg/app/drain.go)
|
||||
|
||||
- One shared 30-second `context.WithTimeout` budget for both `httpServer.Shutdown(ctx)` and `graceful.Shutdown(ctx)`
|
||||
- `Shutdown(ctx)` is idempotent (`sync.Once`), panic-safe (`defer recover()` with structured slog fields), and returns `context.DeadlineExceeded` on timeout
|
||||
- `/readyz` checks DB ping, Redis ping, and storage `Ping` (implemented by all four drivers via probe object write/delete); startup validation retries dependencies with 2-second backoff up to 60 seconds
|
||||
- Pre-stop draining: `SetDraining(true)` makes `/readyz` return 503, waits `drainDelay` (default 5s), then shuts down; manifests add `lifecycle.preStop` as a fallback and `terminationGracePeriodSeconds: 45`
|
||||
- Tests follow project conventions (`require`, `t.Context()`, idempotency and panic-recovery cases)
|
||||
|
||||
### 12. Outdated Kubernetes Deployment Configuration
|
||||
|
||||
**Result**: [`deploy/sigma/templates/`](file:///Users/tosone/code/github/sigma/deploy/sigma/templates/)
|
||||
|
||||
- HPA upgraded to `autoscaling/v2`; PodDisruptionBudget added
|
||||
- Default requests/limits configured for server, worker, distribution, and web
|
||||
- Go components use `/readyz` for readinessProbe (dependency checks) and `/healthz` for livenessProbe (process liveness); `terminationGracePeriodSeconds` added
|
||||
|
||||
### 13. Code Architecture Optimization
|
||||
|
||||
**Result**: [`pkg/dal/dal.go`](file:///Users/tosone/code/github/sigma/pkg/dal/dal.go), [`pkg/service/repositories/repositories.go`](file:///Users/tosone/code/github/sigma/pkg/service/repositories/repositories.go), [`pkg/service/distribution/manifest/manifest.go`](file:///Users/tosone/code/github/sigma/pkg/service/distribution/manifest/manifest.go)
|
||||
|
||||
- Global `DB *gorm.DB` removed; database connections are injected through dig (`pkg/dal/dig.go`)
|
||||
- Commented-out dead code removed
|
||||
- Audit moved to Gin middleware recording request context (user, path, method) instead of hand-written audit calls in services/repositories
|
||||
- Auto-create orchestration (namespace/repository, namespace member, webhook) moved from the registry DAL into the repository and manifest services (commit `29b101e9`); the DAL repository now only handles repository table CRUD
|
||||
|
||||
### 14. Unit Test Coverage Priorities
|
||||
|
||||
**Result**: coverage added across the runtime-critical paths:
|
||||
|
||||
- ✅ P0: `pkg/service/distribution/manifest` — push/pull flows, digest validation, tag handling, delete, error mapping, referrers (commit `a7abbb63`)
|
||||
- ✅ P0: `pkg/service/distribution/blob` / `upload` — blob reads, uploads, mounts, cleanup, storage errors
|
||||
- ✅ P0: `pkg/server/handlers/distribution/base`, `blob`, `manifest`, `upload` — HTTP methods, headers, status codes, range behavior, distribution-spec errors (commit `72107f6f`)
|
||||
- ✅ P0: `pkg/service/repositories`, `namespaces`, `users` — permissions, quotas, member changes, login flows, state updates
|
||||
- ✅ P0: `pkg/service/builders`, `coderepos`, `daemons` — task creation, queue payloads, status transitions
|
||||
- ✅ P1: `pkg/server/handlers/webhooks` + `pkg/service/webhooks` — validation, authorization, event enqueueing, log deletion/resend
|
||||
- ✅ P1: `pkg/dal/repository/namespace` and `registry` — complex queries, soft delete, count updates, size reconciliation
|
||||
- ✅ P2: `pkg/background/daemon/gc`, `pkg/background/daemon/scan/vulngrype`, `pkg/infra/workq` (+redis) — background executor and queue adapters
|
||||
|
||||
**Leftovers**: keep `analytics`, `validators`, `systems`, `tokens`, `password`, `pkg/storage`, and `pkg/telemetry` tests current when their adapters change; extend integration-style tests for `pkg/background/buildrunner/docker` and any new storage driver behavior.
|
||||
|
||||
---
|
||||
|
||||
## Backlog (non-HA)
|
||||
|
||||
### 15. Password Recovery Flow Is Not Usable End-to-End
|
||||
### 5. Password Recovery Flow Is Not Usable End-to-End
|
||||
|
||||
**Current state**: the backend is already wired up — `GET /recover-password` and
|
||||
`PUT /recover-password-reset/:code`, `UserService.RecoverPassword` /
|
||||
@@ -197,3 +91,109 @@ receive or use it.
|
||||
- [ ] Deliver the recovery code to the user (email sender + `smtp`/`mail` config)
|
||||
- [ ] Add recover-password and reset-password pages to the web UI
|
||||
- [ ] Give recovery codes a TTL and rate-limit/reset them on use
|
||||
|
||||
---
|
||||
|
||||
## Completed
|
||||
|
||||
### 6. Database Connection Pool Configuration
|
||||
|
||||
**Result**: [`pkg/dal/dal.go`](file:///Users/tosone/code/github/sigma/pkg/dal/dal.go)
|
||||
|
||||
- MySQL and PostgreSQL now configure `SetMaxOpenConns`, `SetMaxIdleConns`, `SetConnMaxLifetime`, and `SetConnMaxIdleTime` instead of using unlimited driver defaults
|
||||
- All pool parameters are exposed in `config.yaml` so each environment can tune them
|
||||
|
||||
### 7. Full Aggregation Queries on the Push Hot Path
|
||||
|
||||
**Result**: [`pkg/dal/repository/registry/artifact.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/registry/artifact.go)
|
||||
|
||||
- Repository/namespace size is updated atomically inside the push transaction with `UPDATE repository SET size = size + ?` (O(1) instead of O(N))
|
||||
- Quota checks use the incremental counters instead of real-time `SELECT SUM(...)`
|
||||
|
||||
### 8. Database Hot Spots in the Analytics Backend
|
||||
|
||||
**Result**: [`pkg/infra/counter/`](file:///Users/tosone/code/github/sigma/pkg/infra/counter/), [`pkg/background/daemon/analytics/flush.go`](file:///Users/tosone/code/github/sigma/pkg/background/daemon/analytics/flush.go), [`pkg/service/analytics/analytics.go`](file:///Users/tosone/code/github/sigma/pkg/service/analytics/analytics.go)
|
||||
|
||||
- The `database` counter backend has been removed; only `inmemory` and `redis` are registered
|
||||
- `counterBackend` defaults to `inmemory` in `conf/config.yaml`; the inmemory counter aggregates by hour with mutex-protected maps and flushes dirty hours to the DB in `Flush`
|
||||
- Inmemory and redis backends both flush during graceful shutdown via `graceful.RunAtShutdown`
|
||||
- Production deployments can switch to `redis` for shared multi-replica aggregation
|
||||
|
||||
**Follow-ups (small)**:
|
||||
|
||||
- [ ] Restore the current-hour counters from the database on startup so a rolling restart does not lose the in-flight hour
|
||||
- [ ] Wire the `analytics.reconcileCron` config value to a reconcile job (defined in config but not yet scheduled)
|
||||
|
||||
### 9. Insufficient Cache System
|
||||
|
||||
**Result**: [`pkg/infra/cache/`](file:///Users/tosone/code/github/sigma/pkg/infra/cache/), [`pkg/dal/repository/namespace/namespace_cache.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/namespace/namespace_cache.go), [`pkg/dal/repository/registry/repository_cache.go`](file:///Users/tosone/code/github/sigma/pkg/dal/repository/registry/repository_cache.go), [`pkg/app/cache_prewarm.go`](file:///Users/tosone/code/github/sigma/pkg/app/cache_prewarm.go)
|
||||
|
||||
- `Get` merges concurrent fetches for the same missed key with `singleflight`; in-memory and Redis caches both support TTL
|
||||
- Short-lived negative caching prevents deterministic misses (e.g. `gorm.ErrRecordNotFound`) from hitting the DB
|
||||
- `cache_requests_total` and `cache_fetch_duration_seconds` expose low-cardinality metrics by backend, prefix, and result
|
||||
- Namespace/repository metadata is cached through DI-injected DAL cached decorators; transaction-scoped `New...Repository(tx)` returns the raw implementation to avoid caching uncommitted data
|
||||
- Create/update/delete/size paths invalidate cache after successful writes; startup prewarming covers recently updated namespaces/repositories up to `cache.prewarm.limit`
|
||||
- Multi-replica production deployments should use Redis cache so invalidation is shared; single-node dev can use inmemory
|
||||
|
||||
### 10. Distributed Lock Reliability
|
||||
|
||||
**Result**: [`pkg/infra/lock/`](file:///Users/tosone/code/github/sigma/pkg/infra/lock/)
|
||||
|
||||
- Redis locks use `github.com/go-redsync/redsync/v4` with a renewal watchdog on `AcquireWithRenew`
|
||||
- Lock retry uses exponential backoff through the redsync retry delay function
|
||||
- Metrics cover lock wait time, hold time, and renewal results
|
||||
- Multi-node Redlock/Raft quorum was evaluated and intentionally not introduced; re-evaluate only if stronger multi-Redis consistency is required
|
||||
|
||||
### 11. Missing Streaming and Multipart Uploads in Storage
|
||||
|
||||
**Result**: [`pkg/storage/storage.go`](file:///Users/tosone/code/github/sigma/pkg/storage/storage.go), drivers in [`pkg/storage/`](file:///Users/tosone/code/github/sigma/pkg/storage/), [`pkg/service/distribution/upload/upload.go`](file:///Users/tosone/code/github/sigma/pkg/service/distribution/upload/upload.go)
|
||||
|
||||
- The `StorageDriver` interface now exposes a multipart abstraction: `CreateUploadID`, `UploadPart`, `CommitUpload`, `AbortUpload`
|
||||
- All four drivers (filesystem, S3, COS, OSS) implement multipart upload; the blob upload path uses `UploadPart`/`CommitUpload` with part-number tracking
|
||||
- Multipart copy constants (`MultipartCopyThresholdSize=64MB`, chunk size, max concurrency) support large object copies
|
||||
|
||||
**Follow-ups (small)**:
|
||||
|
||||
- [ ] Expose upload progress tracking through a `StreamingWriter`/progress callback API (OCI `PATCH` already provides resumable semantics)
|
||||
|
||||
### 12. Incomplete Graceful Shutdown and Startup Checks
|
||||
|
||||
**Result**: [`pkg/graceful/graceful.go`](file:///Users/tosone/code/github/sigma/pkg/graceful/graceful.go), [`pkg/app/readiness.go`](file:///Users/tosone/code/github/sigma/pkg/app/readiness.go), [`pkg/app/drain.go`](file:///Users/tosone/code/github/sigma/pkg/app/drain.go)
|
||||
|
||||
- One shared 30-second `context.WithTimeout` budget for both `httpServer.Shutdown(ctx)` and `graceful.Shutdown(ctx)`
|
||||
- `Shutdown(ctx)` is idempotent (`sync.Once`), panic-safe (`defer recover()` with structured slog fields), and returns `context.DeadlineExceeded` on timeout
|
||||
- `/readyz` checks DB ping, Redis ping, and storage `Ping` (implemented by all four drivers via probe object write/delete); startup validation retries dependencies with 2-second backoff up to 60 seconds
|
||||
- Pre-stop draining: `SetDraining(true)` makes `/readyz` return 503, waits `drainDelay` (default 5s), then shuts down; manifests add `lifecycle.preStop` as a fallback and `terminationGracePeriodSeconds: 45`
|
||||
- Tests follow project conventions (`require`, `t.Context()`, idempotency and panic-recovery cases)
|
||||
|
||||
### 13. Outdated Kubernetes Deployment Configuration
|
||||
|
||||
**Result**: [`deploy/sigma/templates/`](file:///Users/tosone/code/github/sigma/deploy/sigma/templates/)
|
||||
|
||||
- HPA upgraded to `autoscaling/v2`; PodDisruptionBudget added
|
||||
- Default requests/limits configured for server, worker, distribution, and web
|
||||
- Go components use `/readyz` for readinessProbe (dependency checks) and `/healthz` for livenessProbe (process liveness); `terminationGracePeriodSeconds` added
|
||||
|
||||
### 14. Code Architecture Optimization
|
||||
|
||||
**Result**: [`pkg/dal/dal.go`](file:///Users/tosone/code/github/sigma/pkg/dal/dal.go), [`pkg/service/repositories/repositories.go`](file:///Users/tosone/code/github/sigma/pkg/service/repositories/repositories.go), [`pkg/service/distribution/manifest/manifest.go`](file:///Users/tosone/code/github/sigma/pkg/service/distribution/manifest/manifest.go)
|
||||
|
||||
- Global `DB *gorm.DB` removed; database connections are injected through dig (`pkg/dal/dig.go`)
|
||||
- Commented-out dead code removed
|
||||
- Audit moved to Gin middleware recording request context (user, path, method) instead of hand-written audit calls in services/repositories
|
||||
- Auto-create orchestration (namespace/repository, namespace member, webhook) moved from the registry DAL into the repository and manifest services (commit `29b101e9`); the DAL repository now only handles repository table CRUD
|
||||
|
||||
### 15. Unit Test Coverage Priorities
|
||||
|
||||
**Result**: coverage added across the runtime-critical paths:
|
||||
|
||||
- ✅ P0: `pkg/service/distribution/manifest` — push/pull flows, digest validation, tag handling, delete, error mapping, referrers (commit `a7abbb63`)
|
||||
- ✅ P0: `pkg/service/distribution/blob` / `upload` — blob reads, uploads, mounts, cleanup, storage errors
|
||||
- ✅ P0: `pkg/server/handlers/distribution/base`, `blob`, `manifest`, `upload` — HTTP methods, headers, status codes, range behavior, distribution-spec errors (commit `72107f6f`)
|
||||
- ✅ P0: `pkg/service/repositories`, `namespaces`, `users` — permissions, quotas, member changes, login flows, state updates
|
||||
- ✅ P0: `pkg/service/builders`, `coderepos`, `daemons` — task creation, queue payloads, status transitions
|
||||
- ✅ P1: `pkg/server/handlers/webhooks` + `pkg/service/webhooks` — validation, authorization, event enqueueing, log deletion/resend
|
||||
- ✅ P1: `pkg/dal/repository/namespace` and `registry` — complex queries, soft delete, count updates, size reconciliation
|
||||
- ✅ P2: `pkg/background/daemon/gc`, `pkg/background/daemon/scan/vulngrype`, `pkg/infra/workq` (+redis) — background executor and queue adapters
|
||||
|
||||
**Leftovers**: keep `analytics`, `validators`, `systems`, `tokens`, `password`, `pkg/storage`, and `pkg/telemetry` tests current when their adapters change; extend integration-style tests for `pkg/background/buildrunner/docker` and any new storage driver behavior.
|
||||
Reference in new issue
Block a user