SkinbaseNova/docs/optimization-m7-default-queue-backlog.md

# M7 — Default Queue Backlog & Redis/PHP Memory Failures

```text
STATUS: COMPLETE (investigation + local smallest fixes; production unchanged)
WHEN: 2026-08-23 17:49 Europe/Ljubljana
```

No Horizon restart, Redis flush, job retry, memory_limit change, or worker-count change on server3.

---

## Verdict

```text
Default backlog class:     C + D (producer > consumer; ranking/rec jobs block index jobs)
Default depth:             1208 → 1213 in 60s (net +5/min, growing)
Oldest default job:        ~2.5 h
Search queue:              0 (idle workers exist on supervisor-default)
collections:               32,632 jobs, oldest 2026-04-30, NO Horizon consumer
forum-moderation:          552,888 AnalyzeForumPostJob, ~557 MB list, NO consumer
forum-security:            5,533 jobs, no consumer
presence index:            SET 2,480,292 members, ~272 MB, TTL -1
Predis 256 MB fatals:      loading those huge Redis replies over HTTP/FPM
Horizon workers:           do NOT increase
search+default supervisor: KEEP together; put Index* jobs on search
```

---

## 1. `queues:default` contents (metadata only)

| Class | Count | Avg payload |
| ----- | ----: | ----------: |
| `RankBuildScopeListsJob` | **588** | 1004 B |
| `IndexUserJob` | **395** | 954 B |
| `RecComputeSimilarByTagsJob` | **169** | 1212 B |
| `IndexArtworkJob` | 40 | 918 B |
| `RecalculateRisingNovaCardsJob` | 10 | 987 B |
| `RebuildTrendingNovaCardsJob` | 3 | 977 B |
| `RankBuildListsJob` | 2 | 931 B |
| `RecBuildItemPairsFromFavouritesJob` | 1 | 1102 B |

- Oldest pushedAt: **2026-08-23 15:19** (~2.5 h)
- Newest: 17:48 (~75 s)
- Delayed: 1 `RankBuildScopeListsJob`
- Reserved: 0
- Attempts: 1206×0, 2×1
- Max payload: **1212 B** — not oversized job payloads

Net 60s: 1208 → 1213 (**accumulating**).

---

## 2. Throughput

Horizon metrics repo empty from this user (prefix split `skinbase_horizon` vs `skinbasenova_horizon`).

Evidence: backlog growing +5 net/min while 5 default-supervisor processes exist. Ranking jobs `timeout=300s`; rec tag jobs `timeout=600s`. A few long jobs starve `IndexUserJob` (30s, user-facing search).

---

## 3. Producers

| Producer | Job | Queue | Cadence | Volume |
| -------- | --- | ----- | ------- | ------ |
| `RankBuildListsJob` hourlyAt(15) | `RankBuildScopeListsJob` per category/content type | default | hourly fan-out | **~588/hour** still sitting |
| `UserStatsService::reindex` | `IndexUserJob` | **default** (bug; Scout config is `search`) | per stats change | 395 queued |
| Artwork indexers | `IndexArtworkJob` | **default** (same bug) | publish/trending/tags | 40 queued |
| Scheduler rec nightly | `RecComputeSimilarByTagsJob` batches | default (`RECOMMENDATIONS_QUEUE`) | nightly + leftover | 169 |
| Nova card crons | Rising/Trending rebuild | default | 15 min / hourly | 13 |
| `collections:dispatch-maintenance` hourlyAt(43) | Health/rec/dup | **collections** | hourly ~110 | **32k unconsumed** |
| Forum plugin | `AnalyzeForumPostJob` | **forum-moderation** | continuous | **552,888 unconsumed** |

---

## 4. Isolation

Move **IndexArtworkJob / IndexUserJob** to `search` (already on supervisor-default, currently idle). Same worker budget, better latency. Do **not** add processes.

Do **not** merge mail. Rec stays on default until ranking unique + M1 afterArtworkId is healthy.

**collections** and **forum-moderation** have **no Horizon supervisor** — isolation without a consumer is a dump. Do not add workers until RAM is freed (M6). Stop/limit dispatch first.

---

## 5. Predis 256 MB fatals (13 Aug, 18 Aug)

FPM error log (Skinbase pool, HTTP):

```text
Allowed memory size of 268435456 bytes exhausted
(tried to allocate 67108872 bytes)
vendor/predis/predis/src/Connection/StreamConnection.php:212
```

Two Redis values large enough to explode a 256 MB FPM worker when Predis reads the reply:

| Key | Type | Size | How it is read |
| --- | ---- | ---- | -------------- |
| `queues:forum-moderation` | list | **552,888 jobs ≈ 557 MB** | Horizon UI / any `LRANGE` of the queue |
| `skinbase:presence:online:index` | set | **2,480,292 members ≈ 272 MB** | `OnlineVisitorRepository::readIndexMembers()` → **`SMEMBERS`** then `all()` on moderation traffic page |

Index members have **TTL -1**; per-visitor keys TTL 300s. Expired visitors are never removed unless `all()` finishes — it cannot, so the set grows forever.

**Do not raise memory_limit.** Stop loading the full set/list in one Redis command.

Legacy prefix `skinbasenova-database-queues:*` still holds leftover lists (another ~11 MB default + forum copies).

---

## 6. Payload size

Default-queue jobs are **~1 KB**. Not the memory issue.

---

## 7. Largest Redis keys/prefixes (SCAN, names redacted)

| Group | Approx |
| ----- | -----: |
| `queues:forum-moderation` | 557 MB |
| `skinbase:presence:online:index` | 272 MB |
| `queues:collections` | 35 MB |
| `skinbasenova-database-queues:default` (legacy prefix) | 11 MB |
| `queues:forum-security` | 5.5 MB |
| Horizon recent/pending (legacy prefix) | ~3 MB |

Keyspace ~3.7k keys but a few lists/sets dominate `used_memory` (~879 MB).

---

## 8. search vs default supervisor

**Keep sharing supervisor-default.** Search workers are idle because index jobs were on `default`. After Index* → `search`, auto-balance can drain indexing without extra RAM.

---

## 9. Backlog classification

**Default: C+D** — hourly ranking fan-out + 10-minute rec jobs exceed 5 workers; index jobs wait behind them. Not a retry storm (attempts mostly 0).

**collections / forum-*: F** — orphan queues, growing for months (collections from 2026-04-30; forum still enqueuing today).

**failed_jobs:** 133 rows, **all today 14:37–14:56** (Horizon SIGTERM 14:36). Classes: RecCompute* `Typed property $afterArtworkId must not be accessed before initialization` on release `20260823-12*` — production rec job constructor mismatch. Enhance jobs: curl 127.0.0.1:8095 refused. **Do not retry.**

---

## 10. User-facing latency

`IndexUserJob` / `IndexArtworkJob` wait behind ranking/rec on default. Oldest index jobs share the 2.5 h window. Emails/notifications not on default (mail/notify queues empty). Uploads/vision queue empty.

---

## 11. Redis latency vs swap

ops/sec ~42, blocked clients 0, evicted 0. Slowlog parse inconclusive. Swapped Redis **will** hurt when someone reads a 557 MB list (that *is* the 256 MB fatal). Queue pop of 1 KB jobs is small.

---

## 12. Smallest fix (local; not deployed)

1. `IndexArtworkJob` + `IndexUserJob` → `scout.queue.queue` (`search`) + `ShouldBeUnique`.
2. `RankBuildScopeListsJob` `ShouldBeUnique` per scope so hourly fan-out cannot stack 588 copies.
3. Presence index: **SSCAN + cap 2000**, never `SMEMBERS` of 2.4M.

**Not in this deploy:** delete 552k forum jobs, add Horizon supervisors, raise PHP memory, raise Horizon processes.

**After RAM (M6):** optional `supervisor-collections` / forum workers, or stop those dispatchers.

---

## 13. Tests

`tests/Feature/Queue/DefaultQueueAssignmentTest.php`

---

## 14. Deploy

Ship the three job/presence changes. Reload Horizon so new job `onQueue` applies to **new** dispatches (existing 395 IndexUserJob stay on default until drained). No FPM memory_limit change. No Redis delete.

Rollback: revert those files; Horizon reload.

Risk: unique ranking skips a scope if a stale unique lock remains (`uniqueFor=360s`). Presence admin UI shows at most 2000 index members until a later prune job (write) is designed.

Production not modified.