klevze
8a80aae21e
Keep similar-ai from tripping the global circuit on a lone URL 502, clamp Qdrant search to 100, and add Server-Timing plus slow-request logging. Studio shared props, Academy S3 exists caching, heat chunking, and Redis/scheduler hygiene stay in this rollout.
181 lines
6.2 KiB
Markdown
181 lines
6.2 KiB
Markdown
# M8 — Redis Orphan Cleanup & Presence Compaction
|
||
|
||
```text
|
||
STATUS: COMPLETE (tooling + tests; no production UNLINK/SREM executed)
|
||
WHEN: 2026-08-23 ~18:55 Europe/Ljubljana
|
||
```
|
||
|
||
---
|
||
|
||
## 1. Production before snapshot (read-only)
|
||
|
||
| Metric | Value |
|
||
| ------ | ----- |
|
||
| used_memory | **883.41 MB** (926,320,256) |
|
||
| used_memory_rss | **176.88 MB** |
|
||
| used_memory_peak | 917.11 MB |
|
||
| maxmemory | 2.00 GB |
|
||
| fragmentation_ratio | **0.20** (RSS ≪ used → swapped) |
|
||
| evicted_keys | 0 |
|
||
| connected_clients | 28 |
|
||
| blocked_clients | 0 |
|
||
| ops/sec | 92 |
|
||
| keys | 4838 (4100 with TTL) |
|
||
| REDIS_PREFIX | `skinbase-database-` |
|
||
| HORIZON_PREFIX | `skinbase_horizon:` |
|
||
|
||
| Queue | llen | delayed | reserved |
|
||
| ----- | ---: | ------: | -------: |
|
||
| default | 0 | 0 | 0 |
|
||
| search | 0 | 0 | 0 |
|
||
| mail | 0 | 0 | 0 |
|
||
| collections | **32632** | 0 | 0 |
|
||
| forum-moderation | **553090** | 0 | 0 |
|
||
| forum-security | **5538** | 0 | 0 |
|
||
|
||
Presence `SCARD` **2,480,634**. Sample 200 members: **200 stale / 0 live** (ratio 1.0).
|
||
|
||
Legacy SCAN: `skinbasenova_horizon:` 662 keys ~7 MB; `skinbasenova-database-` 11 keys ~23 MB.
|
||
|
||
---
|
||
|
||
## 2. Producer containment
|
||
|
||
45s later: collections 32632, forum-moderation 553090, forum-security 5538 — **unchanged**.
|
||
|
||
Config: collections/forum_ai/forum_sec **dispatch_enabled=false**.
|
||
|
||
Newest forum payloads ~18:47 then **no growth**. **Proceed.**
|
||
|
||
---
|
||
|
||
## 3. Queue class samples (LINDEX/LRANGE ends only, not full 553k)
|
||
|
||
| Queue | Sample | Classes | Oldest | Newest | avg bytes |
|
||
| ----- | -----: | ------- | ------ | ------ | --------: |
|
||
| forum-moderation | 204 | **100% AnalyzeForumPostJob** | 2026-04-30 | 2026-08-23 18:47 | 987 |
|
||
| forum-security | 204 | FirewallActivityMonitor + BotActivityMonitor | 2026-04-30 | 2026-08-23 18:47 | 993 |
|
||
| collections | 204 | Duplicate / Health / Recommendation (~equal) | 2026-04-30 | 2026-08-23 17:43 | 1054 |
|
||
|
||
No reserved/delayed.
|
||
|
||
---
|
||
|
||
## 4. forum-moderation
|
||
|
||
**UNLINK waiting + notify** after dry-run.
|
||
|
||
Months-old `AnalyzeForumPostJob` has no moderation value. Command samples classes via LINDEX stride (no 553k LRANGE). Refuses if dispatch_enabled, reserved/delayed, or unexpected class (unless `--force`).
|
||
|
||
---
|
||
|
||
## 5. forum-security
|
||
|
||
**UNLINK.** Windowed monitors from April–August are expired. Same guards.
|
||
|
||
---
|
||
|
||
## 6. collections
|
||
|
||
**A. SAFE TO DISCARD AND REBUILD.** Jobs hold `collectionId` only; `handle()` loads current DB. Old queue is a 4-month backlog of the same refresh. Do not auto-delete; operator `--execute collections` when ready. Re-enable `COLLECTIONS_V5_DISPATCH_ENABLED` only after a Horizon consumer exists.
|
||
|
||
---
|
||
|
||
## 7. Presence stale ratio
|
||
|
||
Sample **100% stale** (200/200). Estimated ~2.48M ghost members, ~272 MB.
|
||
|
||
---
|
||
|
||
## 8–9. Presence pruning
|
||
|
||
`PresenceIndexPruner`: SSCAN, pipeline EXISTS, Lua `EXISTS record==0 then SREM`. Never SMEMBERS, never DEL index.
|
||
|
||
Race: Lua is atomic vs SETEX. If a visitor is rewritten after SREM, they SADD again on next track.
|
||
|
||
Schedule: **hourlyAt(33)** if `ONLINE_VISITOR_INDEX_PRUNE_ENABLED=true` (default **false**). Avoids :02 snapshot, :15 rank, :21 leaderboards, :25 nova, nightly rec 02:00. First catch-up: raise `--max-batches` manually (200 batches × 500 ≈ 100k members/run → ~25 hours at hourly default).
|
||
|
||
---
|
||
|
||
## 10–11. Legacy prefix
|
||
|
||
Current process uses `skinbase-database-` / `skinbase_horizon:`. Obsolete `skinbasenova-*` ~30 MB. Command uses an **unprefixed** Predis client, UNLINKs only `skinbasenova-database-` and `skinbasenova_horizon:` keys, never current prefixes.
|
||
|
||
---
|
||
|
||
## 12. Expected reclaim (used_memory, RSS later)
|
||
|
||
| Stage | ~used_memory |
|
||
| ----- | -----------: |
|
||
| forum-security | 5.5 MB |
|
||
| forum-moderation | **557 MB** |
|
||
| legacy prefix | 25–30 MB |
|
||
| presence (after full prune) | **~272 MB** |
|
||
| collections (if later) | 35 MB |
|
||
| **Total if all stages** | **~870 MB** toward a small Redis |
|
||
|
||
RSS may stay ~177 MB until OS reclaims. No MEMORY PURGE.
|
||
|
||
---
|
||
|
||
## 13. Operator commands (not run here)
|
||
|
||
```bash
|
||
# A audit
|
||
php artisan tinker --execute="echo json_encode(['used'=>Redis::info('memory')['used_memory_human'],'mod'=>Redis::llen('queues:forum-moderation'),'sec'=>Redis::llen('queues:forum-security'),'col'=>Redis::llen('queues:collections'),'scard'=>Redis::scard('skinbase:presence:online:index')]);"
|
||
|
||
# B forum-security
|
||
php artisan skinbase:redis-cleanup-orphans forum-security --dry-run
|
||
php artisan skinbase:redis-cleanup-orphans forum-security --execute
|
||
|
||
# C forum-moderation
|
||
php artisan skinbase:redis-cleanup-orphans forum-moderation --dry-run
|
||
php artisan skinbase:redis-cleanup-orphans forum-moderation --execute
|
||
|
||
# D legacy
|
||
php artisan skinbase:redis-cleanup-legacy-prefix --dry-run
|
||
php artisan skinbase:redis-cleanup-legacy-prefix --execute --max-keys=500
|
||
|
||
# E presence (repeat)
|
||
php artisan skinbase:prune-online-visitor-index --dry-run --max-batches=5
|
||
php artisan skinbase:prune-online-visitor-index --execute --max-batches=200
|
||
# then set ONLINE_VISITOR_INDEX_PRUNE_ENABLED=true and config:cache
|
||
|
||
# F collections (optional later)
|
||
php artisan skinbase:redis-cleanup-orphans collections --dry-run
|
||
php artisan skinbase:redis-cleanup-orphans collections --execute
|
||
```
|
||
|
||
Abort: skip `--execute`. UNLINK is async; cannot restore payloads. Presence SREM is idempotent; live visitors re-SADD on next request.
|
||
|
||
---
|
||
|
||
## 14. Files
|
||
|
||
```text
|
||
app/Support/Redis/OrphanQueueCleanup.php
|
||
app/Services/Traffic/PresenceIndexPruner.php
|
||
app/Console/Commands/RedisCleanupOrphansCommand.php
|
||
app/Console/Commands/PruneOnlineVisitorIndexCommand.php
|
||
app/Console/Commands/RedisCleanupLegacyPrefixCommand.php
|
||
config/traffic.php
|
||
routes/console.php
|
||
.env.example
|
||
tests/Feature/Redis/OrphanQueueCleanupTest.php
|
||
tests/Unit/Traffic/PresenceIndexPrunerTest.php
|
||
tests/Unit/Redis/LegacyPrefixFilterTest.php
|
||
docs/optimization-m8-redis-orphan-cleanup.md
|
||
```
|
||
|
||
---
|
||
|
||
## 15. Risks
|
||
|
||
- Sampled classes (400 LINDEX) might miss a rare job type; `--force` required then.
|
||
- `queues:*:notify` UNLINK with the list is required for Laravel blocking pop leftovers.
|
||
- Presence first pass is slow; enable scheduler only after a few manual batches.
|
||
- Predis fatals: do **not** LRANGE forum-moderation in tinker/Horizon UI until UNLINK.
|
||
- Collections UNLINK loses queued refresh intent; DB is source of truth.
|
||
|
||
No FLUSH, no DEL of huge keys from HTTP, no Horizon workers, no Redis restart.
|