Ship production optimization M1-M12.5A: queues, metrics, HTTP observability, and vector search reliability.
Keep similar-ai from tripping the global circuit on a lone URL 502, clamp Qdrant search to 100, and add Server-Timing plus slow-request logging. Studio shared props, Academy S3 exists caching, heat chunking, and Redis/scheduler hygiene stay in this rollout.
This commit is contained in:
@@ -0,0 +1,303 @@
|
||||
# M12 — Production Performance Observability & Slow Request Logging
|
||||
|
||||
```text
|
||||
STATUS: COMPLETE (audit + local artifacts; production nginx/FPM/env NOT modified)
|
||||
WHEN: 2026-08-24 16:46 CEST
|
||||
PRODUCTION: read-only audit only
|
||||
```
|
||||
|
||||
This milestone is **observability**. No application behavior changes except:
|
||||
|
||||
- Server-Timing now wraps the full HTTP kernel (prepend) and can add `ssr;dur=` when Inertia SSR runs
|
||||
- Slow-request logging is **off until** `HTTP_SLOW_REQUEST_LOG_ENABLED=true`
|
||||
|
||||
---
|
||||
|
||||
## 1. Current nginx access logging
|
||||
|
||||
| Item | Production |
|
||||
| ---- | ---------- |
|
||||
| nginx | **1.26.3** |
|
||||
| `log_format` | **none** (default `combined`) |
|
||||
| http default | `access_log /var/log/nginx/access.log;` |
|
||||
| Skinbase vhost | `access_log /var/log/nginx/skinbase.org-access.log;` (no format → combined) |
|
||||
| `access_log off` | static `css/js/images`, `/photo/*.jpg` |
|
||||
| Query strings | **yes** in combined `$request` |
|
||||
| Timing vars | **absent** |
|
||||
| SHA256 `nginx.conf` | `d1720f312b054a2007ffa731158c8bf4c9bd64bba35ba3971551f28cbabd2bd5` |
|
||||
| SHA256 `skinbase.org.conf` | `4e7e4aa29cb406ccedf3e42613997683a92f1c8e2786bd087adab6012dd4b26f` |
|
||||
|
||||
Multiple vhosts have **separate** logs. `http {}` includes `/etc/nginx/conf.d/*.conf`.
|
||||
|
||||
**Cloudflare** sits in front (`server: cloudflare`, `cf-cache-status: DYNAMIC`, `00-cloudflare-realip.conf`). Origin `request_time` is nginx-on-server3, not browser RTT.
|
||||
|
||||
---
|
||||
|
||||
## 2. Logrotate
|
||||
|
||||
`/etc/logrotate.d/nginx`:
|
||||
|
||||
```
|
||||
/var/log/nginx/*.log {
|
||||
daily
|
||||
rotate 14
|
||||
compress
|
||||
delaycompress
|
||||
create 0640 www-data adm
|
||||
postrotate: invoke-rc.d nginx rotate
|
||||
}
|
||||
```
|
||||
|
||||
A new `/var/log/nginx/skinbase-performance.log` **is already covered**. **No extra logrotate rule.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Current FPM slowlog
|
||||
|
||||
Pool `/etc/php/8.4/fpm/pool.d/skinbase.conf` (SHA256 `a0211876…`):
|
||||
|
||||
| Setting | Value |
|
||||
| ------- | ----- |
|
||||
| `pm.max_children` | 14 (**do not change**) |
|
||||
| `request_terminate_timeout` | 120s |
|
||||
| `request_slowlog_timeout` | **10s** (already on) |
|
||||
| `slowlog` | `/var/log/php8.4-fpm-skinbase-slow.log` (~23 MB) |
|
||||
| `catch_workers_output` | yes |
|
||||
| `pm.status_path` | `/fpm-status` localhost only |
|
||||
|
||||
**FPM slowlog recommendation: NO CHANGE.** 10s is appropriate for rare stacks. Do not lower to 2s in the same deploy as nginx JSON logging.
|
||||
|
||||
`/fpm-status` and `/fpm-ping` are `allow 127.0.0.1; deny all`. No public status. No `stub_status`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Existing Server-Timing
|
||||
|
||||
Live header (server3 curl `/`):
|
||||
|
||||
```
|
||||
server-timing: app;desc="Laravel";dur=96.9
|
||||
cache-control: max-age=60, public, s-maxage=300, stale-while-revalidate=600
|
||||
```
|
||||
|
||||
M11 middleware was **web-appended**, so it missed earlier kernel/session work (TTFB ~0.45 s vs Laravel ~110 ms). M12 **prepends** it globally so `app;dur` is closer to full Laravel time. Streamed/download: skip if `headers_sent()`. No SQL/Redis/cookies in the header.
|
||||
|
||||
---
|
||||
|
||||
## 5. Proposed performance log format
|
||||
|
||||
File: `/var/log/nginx/skinbase-performance.log`
|
||||
Format name: `skinbase_perf` `escape=json`
|
||||
Prepared: `deploy/nginx/conf.d/skinbase-perf-log-format.conf`
|
||||
Vhost extra line: `deploy/nginx/skinbase-vhost-performance-log.snippet`
|
||||
|
||||
Existing combined log **stays**.
|
||||
|
||||
Static locations with `access_log off` stay off (not forced through PHP).
|
||||
|
||||
`$upstream_*` is `"-"` for static; analyzer treats as null. Fastcgi PHP has numeric upstream times. `$upstream_cache_status` is typically empty (no nginx proxy_cache for Laravel) — still logged as `cache`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Privacy / security
|
||||
|
||||
**Logged:** time, host, method, `$uri` (path only), status, timings, bytes, content-type, cache token.
|
||||
|
||||
**Not logged:** IP, cookies, Authorization, query string, POST body, user id, session.
|
||||
|
||||
---
|
||||
|
||||
## 7. Expected log volume
|
||||
|
||||
Combined Skinbase access log: ~17 MB so far today, ~31 MB yesterday uncompressed; rotated daily 14 days.
|
||||
|
||||
JSON lines ~300–400 B. Static already excluded. Estimate **~20–40 MB/day** uncompressed, **~300–600 MB** retained (14d) before gzip of older copies (~2–4 MB/day gz). Acceptable; no field trim required.
|
||||
|
||||
Permissions: nginx `0640 www-data adm`. Do not chmod 777. Analyzer may need `sudo`.
|
||||
|
||||
---
|
||||
|
||||
## 8–9. Laravel slow-request logger + threshold
|
||||
|
||||
- Middleware: `LogSlowHttpRequest` (global prepend; **no-op when disabled**)
|
||||
- Channel: `slow-http` → `storage/logs/slow-http.log` daily, **14 days**
|
||||
- Fields: time, method, route_name, route_uri (**template**), status, duration_ms, peak_memory_mb, authenticated bool, inertia bool
|
||||
- **Threshold: 750 ms**
|
||||
|
||||
Rationale: homepage Laravel ~100 ms; guest TTFB 400–500 ms; Academy still 0.6–1.1 s. 750 ms captures Academy/profile/SSR outliers without logging every homepage. If volume is high after 24 h, raise to 1000 ms — do not sample first.
|
||||
|
||||
Disable: `HTTP_SLOW_REQUEST_LOG_ENABLED=false` (default in `.env.example`).
|
||||
|
||||
---
|
||||
|
||||
## 10. SSR timing
|
||||
|
||||
`TimedInertiaSsrGateway` wraps Inertia `HttpGateway` (no vendor patch, SSR not disabled).
|
||||
|
||||
When SSR runs, Server-Timing becomes:
|
||||
|
||||
```
|
||||
app;desc="Laravel";dur=...
|
||||
ssr;desc="Inertia SSR";dur=...
|
||||
```
|
||||
|
||||
`app` includes SSR (nested). Blade home has no `ssr` metric.
|
||||
|
||||
---
|
||||
|
||||
## 11. FPM slowlog
|
||||
|
||||
**NO CHANGE** (already 10s). Optional later: 2s in a **separate** change, not bundled with nginx JSON.
|
||||
|
||||
---
|
||||
|
||||
## 12–14. Analyzer, URI rules, percentiles
|
||||
|
||||
`scripts/analyze-http-performance.php` + `HttpPerformanceLogAnalyzer`
|
||||
|
||||
**Host filter (M12.1):** `--host=skinbase.org` exact match only. Without `--host`, rows aggregate on `host + normalized URI` (hosts are never merged).
|
||||
|
||||
**URI normalization (analysis only):** numeric segments → `{id}`; `/@name` → `/@{user}`; UUID → `{uuid}`; long hex → `{hash}`; `/art/{id}/{slug}` vs `/art/{id}/similar`.
|
||||
|
||||
**Percentile:** nearest-rank, rank = `ceil(p * n)`, 1-indexed. Not an average.
|
||||
|
||||
**Upstream multi-value:** sum numeric comma-separated parts (`0.100, 0.220` → 0.32). `request_time` is authoritative for request latency.
|
||||
|
||||
**499:** counted separately; not treated as 5xx.
|
||||
|
||||
---
|
||||
|
||||
## 15–16. Tests
|
||||
|
||||
```text
|
||||
pest tests/Unit/Http/HttpPerformanceLogAnalyzerTest.php PASS (4)
|
||||
pest tests/Feature/Http/SlowHttpRequestLoggingTest.php PASS (5)
|
||||
```
|
||||
|
||||
Fixtures cover PHP, static `-`, multi-upstream, 200/302/404/499/500, malformed JSON, `/download/artwork/{id}`, `/@{user}`.
|
||||
|
||||
---
|
||||
|
||||
## 17. Files changed
|
||||
|
||||
- `config/http_observability.php`
|
||||
- `config/logging.php`
|
||||
- `.env.example`
|
||||
- `phpunit.xml`
|
||||
- `bootstrap/app.php`
|
||||
- `app/Providers/AppServiceProvider.php`
|
||||
- `app/Http/Middleware/AddServerTiming.php`
|
||||
- `app/Http/Middleware/LogSlowHttpRequest.php`
|
||||
- `app/Support/Http/HttpUriNormalizer.php`
|
||||
- `app/Support/Http/HttpPerformanceLogAnalyzer.php`
|
||||
- `app/Support/Http/TimedInertiaSsrGateway.php`
|
||||
- `scripts/analyze-http-performance.php`
|
||||
- `deploy/nginx/conf.d/skinbase-perf-log-format.conf`
|
||||
- `deploy/nginx/skinbase-vhost-performance-log.snippet`
|
||||
- tests + fixtures
|
||||
- this document
|
||||
|
||||
---
|
||||
|
||||
## 18. Nginx files prepared (not installed)
|
||||
|
||||
See `deploy/nginx/conf.d/skinbase-perf-log-format.conf` and `skinbase-vhost-performance-log.snippet`.
|
||||
|
||||
---
|
||||
|
||||
## 19. Env
|
||||
|
||||
Production recommended:
|
||||
|
||||
```
|
||||
HTTP_SLOW_REQUEST_LOG_ENABLED=true
|
||||
HTTP_SLOW_REQUEST_MS=750
|
||||
HTTP_SLOW_REQUEST_LOG_DAYS=14
|
||||
```
|
||||
|
||||
Local default: **false**.
|
||||
|
||||
---
|
||||
|
||||
## 20. Application deploy (A)
|
||||
|
||||
1. `bash ./sync.sh` (do not `git pull` on production).
|
||||
2. Add the three env vars to production `.env`.
|
||||
3. Config refresh as **skinbase**, only if the host uses config cache:
|
||||
|
||||
`sudo -u skinbase php artisan config:cache`
|
||||
|
||||
4. Do not run artisan as root. Do not `cache:clear` globally unless required.
|
||||
5. PHP-FPM reload only if the deploy process already does (code opcache). No Horizon/Redis/MySQL restart.
|
||||
|
||||
---
|
||||
|
||||
## 21. Nginx deploy (B) — separate from app
|
||||
|
||||
1. Backup: copy `nginx.conf` and `sites-enabled/skinbase.org.conf`.
|
||||
2. Install `skinbase-perf-log-format.conf` into `/etc/nginx/conf.d/`.
|
||||
3. Add **one** extra `access_log` line in the HTTPS server (keep combined).
|
||||
4. `sudo nginx -t` — **stop if fail**.
|
||||
5. `sudo systemctl reload nginx` (**not restart**).
|
||||
6. One `curl` to `/`.
|
||||
7. Confirm a JSON line in `skinbase-performance.log` and a combined line in `skinbase.org-access.log`.
|
||||
8. Confirm no query string / no IP in JSON.
|
||||
9. `sudo -u skinbase php scripts/analyze-http-performance.php --file=/var/log/nginx/skinbase-performance.log --since=1h --top=10`
|
||||
(may need sudo to read `0640 www-data adm`).
|
||||
|
||||
---
|
||||
|
||||
## 22. Optional FPM (C)
|
||||
|
||||
**Skip.** Already logging stacks at 10s.
|
||||
|
||||
---
|
||||
|
||||
## 23. Immediate verification
|
||||
|
||||
```bash
|
||||
curl -sS -D - -o /dev/null --max-time 20 -H 'Accept-Encoding: br' https://skinbase.org/ \
|
||||
| grep -iE 'HTTP/|server-timing|cache-control'
|
||||
|
||||
sudo tail -n 1 /var/log/nginx/skinbase-performance.log
|
||||
sudo tail -n 1 /var/log/nginx/skinbase.org-access.log
|
||||
sudo -u skinbase tail -n 1 /opt/www/virtual/SkinbaseNova/storage/logs/slow-http-$(date +%F).log
|
||||
```
|
||||
|
||||
Expect `Server-Timing` with `app;dur=` (and `ssr;dur=` on Inertia pages). Home may be **below** 750 ms and **not** appear in slow-http.
|
||||
|
||||
---
|
||||
|
||||
## 24. 24h analysis
|
||||
|
||||
```bash
|
||||
sudo php /opt/www/virtual/SkinbaseNova/scripts/analyze-http-performance.php \
|
||||
--file=/var/log/nginx/skinbase-performance.log --since=24h --top=20
|
||||
```
|
||||
|
||||
Then `--status=5xx`, `--status=499`, `--uri=/academy`, `--slow-ms=1000`.
|
||||
|
||||
Do **not** start M13 optimizations from a handful of curls.
|
||||
|
||||
---
|
||||
|
||||
## 25. Rollback
|
||||
|
||||
- App: `HTTP_SLOW_REQUEST_LOG_ENABLED=false` then config cache as skinbase. No code revert required for logging.
|
||||
- Nginx: remove the extra `access_log` line and/or conf.d file, `nginx -t`, `reload`. Combined log untouched.
|
||||
- SSR timing: revert Gateway bind only if a defect appears (SSR behavior unchanged).
|
||||
|
||||
---
|
||||
|
||||
## 26. Risks
|
||||
|
||||
- JSON `access_log` disk (~40 MB/day). Rotation already exists.
|
||||
- Slow-http volume if many pages >750 ms (raise threshold).
|
||||
- `app;dur` will **increase** vs M11’s ~110 ms because timing now includes earlier middleware (this is more accurate, not a regression).
|
||||
- Cloudflare still hides client RTT.
|
||||
|
||||
---
|
||||
|
||||
## 27. NO CHANGE
|
||||
|
||||
FPM `pm.max_children`, Horizon, MySQL, Redis, snapshot indexes, SSR on/off, Telescope, Debugbar, SQL/Redis tracing, forum/collections queues, load tests, nginx restart, public status pages.
|
||||
Reference in New Issue
Block a user