Ship production optimization M1-M12.5A: queues, metrics, HTTP observability, and vector search reliability.
Keep similar-ai from tripping the global circuit on a lone URL 502, clamp Qdrant search to 100, and add Server-Timing plus slow-request logging. Studio shared props, Academy S3 exists caching, heat chunking, and Redis/scheduler hygiene stay in this rollout.
This commit is contained in:
@@ -0,0 +1,180 @@
|
||||
# M8 — Redis Orphan Cleanup & Presence Compaction
|
||||
|
||||
```text
|
||||
STATUS: COMPLETE (tooling + tests; no production UNLINK/SREM executed)
|
||||
WHEN: 2026-08-23 ~18:55 Europe/Ljubljana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Production before snapshot (read-only)
|
||||
|
||||
| Metric | Value |
|
||||
| ------ | ----- |
|
||||
| used_memory | **883.41 MB** (926,320,256) |
|
||||
| used_memory_rss | **176.88 MB** |
|
||||
| used_memory_peak | 917.11 MB |
|
||||
| maxmemory | 2.00 GB |
|
||||
| fragmentation_ratio | **0.20** (RSS ≪ used → swapped) |
|
||||
| evicted_keys | 0 |
|
||||
| connected_clients | 28 |
|
||||
| blocked_clients | 0 |
|
||||
| ops/sec | 92 |
|
||||
| keys | 4838 (4100 with TTL) |
|
||||
| REDIS_PREFIX | `skinbase-database-` |
|
||||
| HORIZON_PREFIX | `skinbase_horizon:` |
|
||||
|
||||
| Queue | llen | delayed | reserved |
|
||||
| ----- | ---: | ------: | -------: |
|
||||
| default | 0 | 0 | 0 |
|
||||
| search | 0 | 0 | 0 |
|
||||
| mail | 0 | 0 | 0 |
|
||||
| collections | **32632** | 0 | 0 |
|
||||
| forum-moderation | **553090** | 0 | 0 |
|
||||
| forum-security | **5538** | 0 | 0 |
|
||||
|
||||
Presence `SCARD` **2,480,634**. Sample 200 members: **200 stale / 0 live** (ratio 1.0).
|
||||
|
||||
Legacy SCAN: `skinbasenova_horizon:` 662 keys ~7 MB; `skinbasenova-database-` 11 keys ~23 MB.
|
||||
|
||||
---
|
||||
|
||||
## 2. Producer containment
|
||||
|
||||
45s later: collections 32632, forum-moderation 553090, forum-security 5538 — **unchanged**.
|
||||
|
||||
Config: collections/forum_ai/forum_sec **dispatch_enabled=false**.
|
||||
|
||||
Newest forum payloads ~18:47 then **no growth**. **Proceed.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Queue class samples (LINDEX/LRANGE ends only, not full 553k)
|
||||
|
||||
| Queue | Sample | Classes | Oldest | Newest | avg bytes |
|
||||
| ----- | -----: | ------- | ------ | ------ | --------: |
|
||||
| forum-moderation | 204 | **100% AnalyzeForumPostJob** | 2026-04-30 | 2026-08-23 18:47 | 987 |
|
||||
| forum-security | 204 | FirewallActivityMonitor + BotActivityMonitor | 2026-04-30 | 2026-08-23 18:47 | 993 |
|
||||
| collections | 204 | Duplicate / Health / Recommendation (~equal) | 2026-04-30 | 2026-08-23 17:43 | 1054 |
|
||||
|
||||
No reserved/delayed.
|
||||
|
||||
---
|
||||
|
||||
## 4. forum-moderation
|
||||
|
||||
**UNLINK waiting + notify** after dry-run.
|
||||
|
||||
Months-old `AnalyzeForumPostJob` has no moderation value. Command samples classes via LINDEX stride (no 553k LRANGE). Refuses if dispatch_enabled, reserved/delayed, or unexpected class (unless `--force`).
|
||||
|
||||
---
|
||||
|
||||
## 5. forum-security
|
||||
|
||||
**UNLINK.** Windowed monitors from April–August are expired. Same guards.
|
||||
|
||||
---
|
||||
|
||||
## 6. collections
|
||||
|
||||
**A. SAFE TO DISCARD AND REBUILD.** Jobs hold `collectionId` only; `handle()` loads current DB. Old queue is a 4-month backlog of the same refresh. Do not auto-delete; operator `--execute collections` when ready. Re-enable `COLLECTIONS_V5_DISPATCH_ENABLED` only after a Horizon consumer exists.
|
||||
|
||||
---
|
||||
|
||||
## 7. Presence stale ratio
|
||||
|
||||
Sample **100% stale** (200/200). Estimated ~2.48M ghost members, ~272 MB.
|
||||
|
||||
---
|
||||
|
||||
## 8–9. Presence pruning
|
||||
|
||||
`PresenceIndexPruner`: SSCAN, pipeline EXISTS, Lua `EXISTS record==0 then SREM`. Never SMEMBERS, never DEL index.
|
||||
|
||||
Race: Lua is atomic vs SETEX. If a visitor is rewritten after SREM, they SADD again on next track.
|
||||
|
||||
Schedule: **hourlyAt(33)** if `ONLINE_VISITOR_INDEX_PRUNE_ENABLED=true` (default **false**). Avoids :02 snapshot, :15 rank, :21 leaderboards, :25 nova, nightly rec 02:00. First catch-up: raise `--max-batches` manually (200 batches × 500 ≈ 100k members/run → ~25 hours at hourly default).
|
||||
|
||||
---
|
||||
|
||||
## 10–11. Legacy prefix
|
||||
|
||||
Current process uses `skinbase-database-` / `skinbase_horizon:`. Obsolete `skinbasenova-*` ~30 MB. Command uses an **unprefixed** Predis client, UNLINKs only `skinbasenova-database-` and `skinbasenova_horizon:` keys, never current prefixes.
|
||||
|
||||
---
|
||||
|
||||
## 12. Expected reclaim (used_memory, RSS later)
|
||||
|
||||
| Stage | ~used_memory |
|
||||
| ----- | -----------: |
|
||||
| forum-security | 5.5 MB |
|
||||
| forum-moderation | **557 MB** |
|
||||
| legacy prefix | 25–30 MB |
|
||||
| presence (after full prune) | **~272 MB** |
|
||||
| collections (if later) | 35 MB |
|
||||
| **Total if all stages** | **~870 MB** toward a small Redis |
|
||||
|
||||
RSS may stay ~177 MB until OS reclaims. No MEMORY PURGE.
|
||||
|
||||
---
|
||||
|
||||
## 13. Operator commands (not run here)
|
||||
|
||||
```bash
|
||||
# A audit
|
||||
php artisan tinker --execute="echo json_encode(['used'=>Redis::info('memory')['used_memory_human'],'mod'=>Redis::llen('queues:forum-moderation'),'sec'=>Redis::llen('queues:forum-security'),'col'=>Redis::llen('queues:collections'),'scard'=>Redis::scard('skinbase:presence:online:index')]);"
|
||||
|
||||
# B forum-security
|
||||
php artisan skinbase:redis-cleanup-orphans forum-security --dry-run
|
||||
php artisan skinbase:redis-cleanup-orphans forum-security --execute
|
||||
|
||||
# C forum-moderation
|
||||
php artisan skinbase:redis-cleanup-orphans forum-moderation --dry-run
|
||||
php artisan skinbase:redis-cleanup-orphans forum-moderation --execute
|
||||
|
||||
# D legacy
|
||||
php artisan skinbase:redis-cleanup-legacy-prefix --dry-run
|
||||
php artisan skinbase:redis-cleanup-legacy-prefix --execute --max-keys=500
|
||||
|
||||
# E presence (repeat)
|
||||
php artisan skinbase:prune-online-visitor-index --dry-run --max-batches=5
|
||||
php artisan skinbase:prune-online-visitor-index --execute --max-batches=200
|
||||
# then set ONLINE_VISITOR_INDEX_PRUNE_ENABLED=true and config:cache
|
||||
|
||||
# F collections (optional later)
|
||||
php artisan skinbase:redis-cleanup-orphans collections --dry-run
|
||||
php artisan skinbase:redis-cleanup-orphans collections --execute
|
||||
```
|
||||
|
||||
Abort: skip `--execute`. UNLINK is async; cannot restore payloads. Presence SREM is idempotent; live visitors re-SADD on next request.
|
||||
|
||||
---
|
||||
|
||||
## 14. Files
|
||||
|
||||
```text
|
||||
app/Support/Redis/OrphanQueueCleanup.php
|
||||
app/Services/Traffic/PresenceIndexPruner.php
|
||||
app/Console/Commands/RedisCleanupOrphansCommand.php
|
||||
app/Console/Commands/PruneOnlineVisitorIndexCommand.php
|
||||
app/Console/Commands/RedisCleanupLegacyPrefixCommand.php
|
||||
config/traffic.php
|
||||
routes/console.php
|
||||
.env.example
|
||||
tests/Feature/Redis/OrphanQueueCleanupTest.php
|
||||
tests/Unit/Traffic/PresenceIndexPrunerTest.php
|
||||
tests/Unit/Redis/LegacyPrefixFilterTest.php
|
||||
docs/optimization-m8-redis-orphan-cleanup.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. Risks
|
||||
|
||||
- Sampled classes (400 LINDEX) might miss a rare job type; `--force` required then.
|
||||
- `queues:*:notify` UNLINK with the list is required for Laravel blocking pop leftovers.
|
||||
- Presence first pass is slow; enable scheduler only after a few manual batches.
|
||||
- Predis fatals: do **not** LRANGE forum-moderation in tinker/Horizon UI until UNLINK.
|
||||
- Collections UNLINK loses queued refresh intent; DB is source of truth.
|
||||
|
||||
No FLUSH, no DEL of huge keys from HTTP, no Horizon workers, no Redis restart.
|
||||
Reference in New Issue
Block a user