Ship production optimization M1-M12.5A: queues, metrics, HTTP observability, and vector search reliability.
Keep similar-ai from tripping the global circuit on a lone URL 502, clamp Qdrant search to 100, and add Server-Timing plus slow-request logging. Studio shared props, Academy S3 exists caching, heat chunking, and Redis/scheduler hygiene stay in this rollout.
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# M2 — Mail Queue Consumption
|
||||
|
||||
## Root cause
|
||||
|
||||
Application mail jobs are pushed to Redis queue **`mail`**. Production Horizon supervisors only consume:
|
||||
|
||||
```text
|
||||
search, default
|
||||
broadcasts, notifications
|
||||
```
|
||||
|
||||
There is **no** `queue:work` process for `mail`. The three items have `attempts=0` — they were never reserved.
|
||||
|
||||
Isolation is **intentional** in code (`onQueue('mail')`) and in `docs/QUEUE.md`. Horizon simply never subscribed.
|
||||
|
||||
## The three queued items (no PII)
|
||||
|
||||
Inspected with `LRANGE`/`LINDEX` via Laravel Redis; only class metadata:
|
||||
|
||||
| # | displayName | maxTries | attempts |
|
||||
| - | ----------- | -------: | -------: |
|
||||
| 0 | `App\Jobs\SendVerificationEmailJob` | 5 | 0 |
|
||||
| 1 | `App\Mail\EmailChangedSecurityAlertMail` | 3 | 0 |
|
||||
| 2 | `App\Mail\EmailChangeVerificationCodeMail` | 3 | 0 |
|
||||
|
||||
`pushedAt` unix times: 1777665006, 1777697989, 1784800925 (older registration/email-change mail sitting until a worker exists).
|
||||
|
||||
## Local vs production
|
||||
|
||||
Same gap on both: Horizon defaults omitted `mail`. Production `config('horizon.defaults.supervisor-*.queue')` confirmed `["search","default"]` and `["broadcasts","notifications"]`.
|
||||
|
||||
Legacy `deploy/supervisor/skinbase-queue.conf` **does** include `mail`, but production runs Horizon, not that unit.
|
||||
|
||||
## Fix
|
||||
|
||||
Added Horizon `supervisor-mail`:
|
||||
|
||||
```text
|
||||
queue: mail
|
||||
connection: redis
|
||||
tries: 5 (matches SendVerificationEmailJob; rate-limit uses release())
|
||||
timeout: 90
|
||||
maxProcesses: 1 local / 2 production
|
||||
```
|
||||
|
||||
Kept isolation so SMTP does not block rec/search workers and so `--tries=1` on default workers cannot fail mail jobs on `release()`.
|
||||
|
||||
## Files changed
|
||||
|
||||
```text
|
||||
config/horizon.php
|
||||
tests/Unit/HorizonMailQueueTest.php
|
||||
docs/QUEUE.md
|
||||
docs/optimization-m2-mail-queue.md
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
1. Deploy release with `config/horizon.php`.
|
||||
2. `php artisan config:cache` / `optimize` as usual.
|
||||
3. Restart Horizon (Supervisor `skinbase-horizon` / `queue:restart`).
|
||||
4. Confirm `php artisan horizon:status` and a worker with `--queue=mail`.
|
||||
|
||||
No `.env` change required. No Redis delete.
|
||||
|
||||
## Should the 3 messages be processed after deploy?
|
||||
|
||||
**Yes, automatically** once `supervisor-mail` starts — do not delete the list.
|
||||
|
||||
Risks:
|
||||
|
||||
- Recipients may receive **late** verification/security emails.
|
||||
- Tokens/codes may already be expired; user sees a failed verify, not a security hole from us sending mail.
|
||||
- `SendVerificationEmailJob` may still send `RegistrationVerificationMail` (also `mail` queue) — second job is fine once workers run.
|
||||
- Do **not** manually `queue:retry` or `lpop` before Horizon is up.
|
||||
|
||||
## Tests
|
||||
|
||||
`php artisan test --filter=HorizonMailQueue`
|
||||
Reference in New Issue
Block a user