SkinbaseNova/docs/skinbase-production-baseline.md

# Skinbase.org — M0 Production Baseline & Runtime Audit

```text
STATUS: COMPLETE
```

Primary origin runtime unknowns were resolved via **read-only `ssh server3`**. Some items remain UNKNOWN (MySQL `slow.log` not readable by this user; root crontab not readable without sudo; `performance_schema` denied to `skinbase@localhost`; `iostat` not installed).

**Nothing was modified on server3.** No restarts, no cache clears, no SQL writes, no deploys.

Evidence classes used below:

```text
LOCAL CODE EVIDENCE
PRODUCTION SERVER EVIDENCE
PUBLIC CLOUDFLARE/HTTP EVIDENCE
```

---

## Executive Summary

```text
Production OS:                 Debian 13 (trixie), kernel 6.12.96+deb13-cloud-amd64, x86_64
CPU:                           AMD EPYC, 8 cores / 8 threads (1 socket)
RAM:                           23 GiB; ~8.6 GiB available; SWAP 2.0 GiB **almost fully used**
Storage:                       400G virtio disk `/dev/sda1` ext4 ~185G/394G (49%); rotational=0 (SSD/NVMe-like)
nginx:                         1.26.3; HTTP/2 on; gzip+brotli in nginx.conf; vhost skinbase.org.conf
PHP:                           8.4.24 (FPM + CLI)
PHP-FPM:                       pool `skinbase`, dynamic, max_children=14, memory_limit=256M, terminate=120s
MySQL:                         8.4.11-11 local; DB `skinbase` 2758 MB; buffer pool 10 GB
Redis:                         8.10.1 local 127.0.0.1:6379; 878 MB used / 2 GB max; allkeys-lru
Meilisearch:                   local systemd; health available; RSS ~401 MB
Queue manager:                 Supervisor → `php artisan horizon` (NOT standalone queue:work)
Cache store:                   **redis** (PRODUCTION; not repo default database)
Session store:                 **redis**
CDN:                           Cloudflare + cdn.skinbase.org RGW/S3
Download acceleration:         **OFF** (`download_accel_enabled=false`; no X-Accel location in vhost)
Sitemap static serving:        **ACTIVE** (nginx try_files + shared files). Index file stale (May 13).
Debug mode:                    OFF
Debugbar production:           NOT PRESENT (packages-dev=0, no debugbar routes)
```

```text
PRODUCTION_APP_PATH=/opt/www/virtual/SkinbaseNova
  → symlink → /opt/www/virtual/SkinbaseNova.releases/current
  → /opt/www/virtual/SkinbaseNova.releases/releases/20260801-172803-5af95f65-dirty

Local commit:       f52879edbb19ecc5d3e807bd4872016f3071dad7 (develop)
Production commit:  5af95f65 (release name 5af95f65-dirty, 2026-08-01)
Same revision:      no
Production git:     not a git checkout (no .git in release)
```

```text
Active P0 findings:
  QUEUE-001 (90s vs 900s): MITIGATED — Horizon default workers timeout=960
  QUEUE rec-job failures:  ACTIVE P0 — RecComputeSimilar* fail every night MaxAttemptsExceeded
  SITEMAP-001 live-build:  NOT the current incident — static files exist
  SITEMAP index stale:     ACTIVE P1/P0-SEO — sitemap.xml mtime 2026-05-13 while shards refresh daily
  STAT-001:                NOT currently P0 at 2666 views/24h; design still sync writes (P1)
  DB-001:                  indexes PRESENT; residual UNKNOWN without slow.log. Snapshot table 1.85 GB is the dominant store.

Active P1 findings:        swap exhaustion; mail queue unconsumed (LLEN=3); upload 20M vs app 50M; Sentry traces 100%; FPM 10s slowlog volume; CF HTML DYNAMIC; asset TTL 30d not 1y immutable

Major current bottleneck:  MySQL row-read volume + 1.85 GB hourly snapshots; nightly rec jobs failing; memory pressure (swap full)

Largest remaining unknown: contents of /var/log/mysql/slow.log (permission)

M1 readiness:              YES for planning. Do not implement in M0.
```

---

## 1. Repository State (LOCAL)

```text
branch: develop
HEAD:   f52879edbb19ecc5d3e807bd4872016f3071dad7
commit: f52879ed Current state with latest updates
tree:   clean at start of this SSH pass (ahead of origin/develop by 1)
```

No reset/stash/clean on local or production.

---

## 2. Production Host (PRODUCTION SERVER)

```text
hostname:     server3
FQDN:         server3.klevze.si
OS:           Debian GNU/Linux 13 (trixie) 13.6
kernel:       6.12.96+deb13-cloud-amd64
arch:         x86_64
uptime:       24 days, 16:58 (sampled 2026-08-23 13:43 CEST)
```

---

## 3. Hardware Baseline (PRODUCTION SERVER)

```text
CPU model:     AMD EPYC Processor (with IBPB)
sockets:       1
cores:         8
threads:       8 (1 thread/core)
RAM:           23 GiB
swap:          2.0 GiB, ~2.0 GiB used, 2.7 MiB free
disk:          /dev/sda 400G → sda1 399.9G on /
filesystem:    ~394G, 185G used, 49%
rotational:    0  → SSD/cloud SSD (not HDD)
```

`dmidecode` not used (would need sudo).

---

## 4. Current System Load (PRODUCTION SERVER)

Short snapshot only (limitation: not peak-hour).

```text
load average:  0.85, 2.47, 2.88
vmstat 5s:     idle ~76–83%, wa ~0–1%, r=3–6
CPU notables:  mysqld ~20%, containerd/dockerd ~45% each (other workloads on same host), redis ~5%
RAM:           14 GiB used + 7 GiB cache; 8.6 GiB available
swap:          fully used
```

Largest RSS:

| Process | RSS | Note |
| ------- | --: | ---- |
| mysqld | ~7.7 GiB | 32.7% |
| python uvicorn × several | up to 1.3 GiB | vision/ML stack |
| clamd | ~1.1 GiB | |
| meilisearch | ~397–401 MiB | |
| redis | ~164 MiB | |
| php-fpm skinbase | ~95–100 MiB each | 6 workers sampled |
| node ssr.js | ~122 MiB | |

Classification: **WATCH / CONCERNING** (swap exhausted; shared host with Docker/ML/Qdrant/Gitea/Crowdsec). Not a Skinbase-only box.

---

## 5. Process Inventory (PRODUCTION SERVER)

| Component | Reality |
| --------- | ------- |
| Horizon | **YES** — Supervisor `skinbase-horizon`, pid running ~1d |
| standalone `queue:work` | **NO** |
| Supervisor | **YES** — horizon + ssr |
| systemd queue workers | **NO** |
| Inertia SSR | **YES** — `node bootstrap/ssr/ssr.js` as www-data |
| Reverb | **YES** — systemd `skinbase-reverb.service`, 127.0.0.1:8080 |
| Meilisearch | **YES** — local systemd |
| Redis | **local** 127.0.0.1:6379 |
| MySQL | **local** `/usr/sbin/mysqld` |
| PHP-FPM | php8.4, pool `skinbase` + unused pool `www` |
| nginx | master + 8 workers, www-data |
| Docker/ML | uvicorn:8000, qdrant, gitea — **co-tenant** |

---

## 6. nginx (PRODUCTION SERVER)

```text
version: nginx/1.26.3
vhost:   /etc/nginx/sites-enabled/skinbase.org.conf
root:    /opt/www/virtual/SkinbaseNova/public
listen:  80 → 301 HTTPS; 443 ssl; http2 on
PHP:     unix:/run/php/php8.4-fpm-skinbase.sock
client_max_body_size: 24m
gzip:    on (nginx.conf)
brotli:  on (nginx.conf)
real IP: /etc/nginx/conf.d/00-cloudflare-realip.conf ACTIVE
search:  location = /search + conf.d/13-skinbase-search-protection.conf (20r/m, abusive query map)
HTTP/3:  not in this vhost (Cloudflare alt-svc h3 is edge-only)
```

Snippet vs live:

| Snippet | Status |
| ------- | ------ |
| sitemaps.conf (`max-age=21600`, try_files) | **ACTIVE** (inlined in vhost) |
| static-cache.conf 1y immutable `/build/assets` | **NOT ACTIVE** — generic `expires 30d` for css/js/images |
| download-accel.conf | **NOT ACTIVE** |
| search-rate-limit.conf (repo) | **EQUIVALENT ACTIVE** as `13-skinbase-search-protection.conf` |
| upstream-error-pages.conf | **NOT seen** in vhost |

**Bug/risk:** sitemap locations use `try_files $uri @php` but **no `location @php`** exists in the vhost. Missing files will not fall through to Laravel as the snippet comments claim.

No `nginx -T` via sudo (password required). Readable vhost was enough.

---

## 7–9. HTTP / CDN (PUBLIC CLOUDFLARE/HTTP + PRODUCTION)

Unchanged from edge probes, now correlated with origin:

- Homepage Cache-Control matches Laravel guest headers; CF **DYNAMIC**
- Build assets CF **HIT**, origin `expires 30d`
- CDN webp `max-age=31536000` HIT + Polish
- Brotli **CONFIRMED** at edge; origin brotli **on**
- HTTP/2 **CONFIRMED** at origin (`http2 on`); this Windows curl was HTTP/1.1 to CF
- HTTP/3 **offered by CF** (`alt-svc`), not configured on origin vhost

---

## 10–12. PHP / OPcache / FPM (PRODUCTION SERVER)

```text
PHP:              8.4.24
FPM master:       /etc/php/8.4/fpm/php-fpm.conf
pool:             [skinbase] user=skinbase
listen:           /run/php/php8.4-fpm-skinbase.sock
pm:               dynamic
max_children:     14
start_servers:    4
min/max spare:    2 / 6
max_requests:     500
request_terminate_timeout: 120s
slowlog:          10s → /var/log/php8.4-fpm-skinbase-slow.log (23 MB, 139556 lines)
memory_limit:     256M (pool)
max_execution:    60s (pool)
upload/post:      20M / 24M (pool)  ← app allows 50M images / 200M archives
CLI upload_max:   2M (irrelevant to FPM)
extensions:       opcache, redis, pcntl, posix, intl, mbstring, curl, gd, pdo_mysql
imagick:          NOT in php -m
opcache (FPM php.ini): memory 256MB, interned 16, max_files 20000, validate_timestamps=1, revalidate_freq=2, jit=off
opcache runtime hit rate: UNKNOWN (no status scrape of /fpm-status from remote; localhost-only)
```

Workers sampled: **6** skinbase processes, **avg RSS ~95 MB**, total ~570 MB.

Theoretical ceiling: `14 × 256 MB = 3.6 GB` if every worker hit `memory_limit`. Typical: `14 × 95 MB ≈ 1.3 GB`.

Available RAM 8.6 GiB **but swap already full** — do not raise max_children in M0.

---

## 13–15. Laravel Production Configuration (PRODUCTION SERVER)

`php artisan about` + tinker `config()` (no `.env` dump):

```text
APP_ENV=production
APP_DEBUG=false
CACHE_STORE=redis
SESSION_DRIVER=redis
QUEUE_CONNECTION=redis
SCOUT_DRIVER=meilisearch
FILESYSTEM_DISK=local
UPLOAD_QUEUE_DERIVATIVES=false
DOWNLOAD_ACCEL_ENABLED=false
DOWNLOAD_ACCEL_PATH=/internal/originals   (path set, flag off, nginx location missing)
SITEMAPS_BUILD_ON_REQUEST=true
SITEMAPS_FALLBACK_TO_LIVE_BUILD=true
SITEMAPS_PREGENERATED_ENABLED=true
SITEMAPS_PREFER_PUBLISHED_RELEASE=true
SITEMAPS_STATIC_PUBLISH_ENABLED=true
REDIS_CLIENT=predis
homepage.cache_store=homepage
homepage.guest_payload_ttl_seconds=1800
recommendations.queue=default
vision.queue=default
discovery.queue=default
broadcasting=reverb
horizon.path=horizon
Sentry enabled, sample rate errors 100%, performance 100%
```

Laravel caches: **config, events, routes, views ALL CACHED**.

Debugbar/Telescope/Clockwork routes: **none**. Composer `packages-dev=0` → **--no-dev install**.

---

## 16–22. MySQL (PRODUCTION SERVER)

```text
location:     local mysqld
version:      8.4.11-11
database:     skinbase
size:         2758.2 MB (data 1230.4 + indexes 1527.8)
buffer pool:  10 GB, 10 instances
max_conn:     120 (Max_used 25)
slow_query_log: ON, long_query_time=0.5s, file /var/log/mysql/slow.log (not readable here)
Slow_queries: 11342 in Uptime 97330s (~27h) ≈ 0.12/s
Threads:      connected 3, running 2
InnoDB hit:   1 - 103791/18420885623 ≈ 99.999%
Rows read:    9.70e9 in ~27h  (high — job/scan load)
tmp tables:   455567 created, 21 on disk
```

Largest tables (information_schema estimates):

| Table | ~rows | total MB |
| ----- | -----: | -------: |
| artwork_metric_snapshots_hourly | 8.62M | **1855.5** |
| artwork_downloads | 579k | 167.3 |
| forum_bot_logs | 267k | 141.8 |
| artworks | 50k | 94.2 |
| user_activities | 240k | 69.3 |
| artwork_comments | 172k | 62.2 |
| artwork_view_events | 161k | 25.4 |
| rank_artwork_scores | 50k | 22.4 |
| rec_item_pairs | 74k | 16.0 |
| artwork_stats | 49k | 15.2 |

`cache`/`sessions` tables nearly empty (drivers are Redis). `jobs` table empty (Redis queues).

### Indexes (batch1) — PRODUCTION

| Index | Status |
| ----- | ------ |
| artworks.idx_public_approved_published_id | **PRESENT** |
| artworks.idx_public_approved_user_id | **PRESENT** |
| artworks FULLTEXT title+description | **PRESENT** |
| snapshots.idx_bucket_artwork | **PRESENT** |
| rank_artwork_scores idx_mv_trending/new_hot/best | **PRESENT** |
| tags.artworks_count + idx_tags_artworks_count | **PRESENT** |

### DB-001 re-evaluation

```text
Classification: UNKNOWN as “still 78%”, NOT “unindexed public scans”
```

April indexes **are deployed**. Dominant storage is hourly snapshots ≈ 50k artworks × 24h × 7d. Heat/ranking jobs that join this table can still dominate `Innodb_rows_read` even with `idx_bucket_artwork`. Cannot confirm fingerprints without `slow.log` / performance_schema (denied).

Do not add more indexes in M0.

---

## 23–25. Redis / Cache / Sessions (PRODUCTION SERVER)

```text
redis_version: 8.10.1
used:          877.55 MB (peak 917 MB)
maxmemory:     2.00 G, policy allkeys-lru
evicted_keys:  0
clients:       20
ops/sec:       44 (instant)
keyspace hits/misses: 1.40M / 1.30M  (~52% hit)
db0: 1932 keys; db1: 6641; db2: 55176 (all with TTL)
```

| System | Redis in production? |
| ------ | -------------------- |
| Cache default | **YES** |
| Homepage store `homepage` | configured (failover redis→database) |
| Sessions | **YES** |
| Queues / Horizon | **YES** |
| Presence | code uses Redis (not separately counted) |
| Stats deltas | flush command scheduled; view path still MySQL sync |

CACHE-001 from the code audit (**database cache default**) is **NOT ACTIVE in production**.

---

## 26–32. Horizon / Queues / QUEUE-001 (PRODUCTION SERVER)

Horizon **running**. Supervisors:

| Supervisor | queues | timeout | maxProcesses |
| ---------- | ------ | ------: | -----------: |
| supervisor-default | search, default | **960** | 5 |
| supervisor-messaging | broadcasts, notifications | **90** | 3 |

`recommendations.queue` / `vision.queue` / `discovery.queue` = **`default`** → consumed by 960s workers.

Standalone supervisor `queue:work --timeout=90` from `deploy/supervisor/skinbase-queue.conf` is **NOT the production worker**.

Failed jobs: **329** rows (`queue:failed` listed many; table_rows estimate 159). Dominant:

```text
RecComputeSimilarByBehaviorJob  ~02:18 daily  MaxAttemptsExceededException
RecComputeSimilarHybridJob      ~02:31 daily  MaxAttemptsExceededException
```

Horizon worker `tries=1`, job `$timeout=900`, worker timeout 960. Failures are **not** explained by the old 90s worker. Likely 900s job timeout, 128 MB worker memory, or lock/overlap. **Do not retry in M0.**

Queue LLEN (Redis): all listed queues 0 except **`queues:mail` = 3**. Horizon does **not** listen to `mail`. Mail jobs can stall.

```text
QUEUE-001 (90 vs 900): MITIGATED
Nightly rec job failures: ACTIVE P0 (related, different mechanism)
mail queue unconsumed: P1
```

---

## 31–32. Scheduler (PRODUCTION SERVER)

`php artisan schedule:list` matches `routes/console.php` (generate sitemaps 10:30/22:30, rec jobs 02:00–02:30, etc.).

**How `schedule:run` is invoked:** user crontab empty; sudo crontab **not readable**; no systemd timer named laravel/schedule. **Empirically running** (sitemap shards 10:30 today; rec jobs fail 02:18/02:31 daily). Likely root crontab. Do not change.

02:00–05:00 window (code + evidence it fires): rec jobs, ranking/heat, analytics, prune snapshots, sitemap validate — plus nightly rec **failures**.

---

## 33. Meilisearch (PRODUCTION SERVER)

```text
process: /usr/local/bin/meilisearch --config-file-path /etc/meilisearch.toml
health:  {"status":"available"}
version/stats: 401 without key (not printed)
RSS:     ~401 MB
```

No reindex.

---

## 34–36. Storage / Downloads / Sitemaps

- `FILESYSTEM_DISK=local`; CDN for public derivatives.
- Download accel **off**; originals would stream via PHP if used.
- Sitemaps live under shared `.../shared/public/sitemaps/` (20M, 27 xml files).
- **Shards + academy + users + forum-threads mtime 2026-08-23 10:30–10:31**
- **`sitemap.xml` mtime 2026-05-13 21:39, 1906 bytes** — generate/publish does **not** update the index file.
- May index omits academy-* families that now exist on disk.

SITEMAP-001 live-build-on-missing: files exist, so crawlers are not building XML in PHP for the index. `@php` named location missing anyway.

```text
SITEMAP-001 original (PHP live-build stampede): NOT ACTIVE right now
Stale sitemap index vs daily shards: ACTIVE (SEO/ops)
```

---

## 37–38. Views / Downloads

```text
artwork_view_events last 1h:   72
artwork_view_events last 24h:  2666
artwork_downloads last 24h:    4718
view_events table:             ~161k rows, 25.4 MB
downloads table:               ~579k rows, 167 MB
```

STAT-001 architecture (sync INSERT+UPDATE, `defer=false`) is **still the code on this release**. At **~0.03 views/s** it is not the current DB bottleneck. Classify **P1** (architecture / growth), not active P0 load.

Downloads were not fetched (would write).

---

## 39–47. Latency / Traffic / Resources

Origin access-log QPS not aggregated (would include IPs — skipped). Edge n=1 TTFBs from earlier remain **PUBLIC** evidence, not origin APM.

FPM slowlog: 139k historical lines, last file mtime Aug 23 05:05, timeout 10s. Scripts are `index.php` only (no URI dump here).

Resource baseline: see §4. **Swap full = CONCERNING.**

---

## 48. Lighthouse / CWV

Not re-run. Prior lab file is skinbase.top 2026-03-23 — do not use as this baseline.

---

## 49. Observability

- Sentry **on**, 100% error **and** performance sample (P1 cost).
- Netdata on host.
- Horizon dashboard path `horizon` (auth not verified; do not probe).
- `/fpm-status` localhost only.
- `/stats/` basic-auth (not accessed).
- MySQL slow.log exists but unreadable to this SSH user.

---

## 50. Re-evaluated P0 Findings

| ID | Verdict | Why |
| -- | ------- | --- |
| QUEUE-001 90s worker vs 900s job | **MITIGATED** | Horizon default timeout **960**; rec queues aliased to `default` |
| RecCompute* nightly MaxAttemptsExceeded | **ACTIVE P0** | every night 02:18/02:31 on `redis@default`; 329 failed jobs |
| STAT-001 sync view writes | **P1** at current 2666/day | still sync in config/code; not the load leader |
| SITEMAP-001 request-time build | **NOT ACTIVE** | static files present; `@php` missing |
| Stale sitemap.xml (May vs Aug shards) | **ACTIVE P1** (SEO) | generate writes children, not index |
| DB-001 unindexed aggregates | **MITIGATED for missing indexes** | batch1 **PRESENT**. Residual scan cost **UNKNOWN**; snapshots 1.85 GB |

---

## 51. Re-evaluated P1 Findings

- Production **Redis** cache/session (repo default database is wrong for prod).
- Upload 20M/24M FPM vs app 50M/200M.
- `UPLOAD_QUEUE_DERIVATIVES=false` — sync image work on publish.
- `mail` Redis queue not consumed by Horizon.
- Sentry performance sampling 100%.
- CF HTML DYNAMIC despite public Cache-Control.
- Asset TTL 30d not 1y immutable.
- Swap 2G full; shared ML/Docker host.
- FPM slowlog 10s historically large.
- Local git **newer** than production release (f52879ed vs 5af95f65).
- Session cookies on `/search` and 404 (code-consistent).
- Duplicate security headers (app middleware + nginx snippet).

---

## 52. Baseline Metric Table

| Metric | Value | Source |
| ------ | ----- | ------ |
| DB size | 2758 MB | information_schema |
| Snapshots table | 1855 MB / ~8.6M rows | information_schema |
| Artworks | ~50k / 94 MB | information_schema |
| Views 24h | 2666 | count on 161k table |
| Downloads 24h | 4718 | count |
| Slow_queries / 27h | 11342 | SHOW STATUS |
| InnoDB pool hit | ~100% | STATUS |
| Redis used | 878 MB / 2 GB | INFO |
| Redis hit ratio | ~52% | INFO |
| FPM workers | 6 / max 14, ~95 MB RSS | ps |
| Horizon | running, timeout 960/90 | ps + artisan |
| Failed jobs | 329 | DB |
| Homepage TTFB (edge) | 146 ms n=1 | PUBLIC |
| Production release | 20260801-172803-5af95f65-dirty | filesystem |

---

## 53. Production Risk Matrix

| Risk | Evidence | Impact | Confidence |
| ---- | -------- | ------ | ---------- |
| Nightly rec jobs never succeed | failed_jobs every day | stale similar-art | high |
| Hourly snapshot table 1.85 GB | information_schema | heat/rank I/O | high |
| Swap full + co-tenant ML | free/ps | latency spikes | high |
| Stale sitemap index | mtime May 13 vs shards today | crawl/SEO | high |
| mail queue not consumed | LLEN=3, Horizon queues | delayed mail | high |
| Missing @php location | nginx vhost | missing shard → error not Laravel | high |
| Cannot read slow.log | permissions | DB-001 residual | high |

---

## 54. Optimization Targets (do not implement now)

1. Rec job failures (timeout/memory/overlap) — after measuring why MaxAttemptsExceeded.
2. Snapshot retention / heat query EXPLAIN once slow.log is readable.
3. Publish `sitemap.xml` index with shards.
4. Consume `mail` or stop using that queue name.
5. Download X-Accel if originals still hit FPM.
6. Align upload limits.
7. Memory/swap / co-tenancy.
8. Sentry sample rates.
9. CF HTML cache policy (product decision).

---

## 55. M1–M4 Readiness

```text
M1 planning: READY (production drivers known; P0 recast)
M1 implementation: not this milestone
Blockers for DB-001 closeout: mysql slow.log or GRANTs on performance_schema
```

---

## 56. Unknowns

- Root crontab contents (scheduler invocation path)
- MySQL slow.log fingerprints **now**
- OPcache hit rate (FPM status localhost-only)
- Meilisearch document counts (auth)
- Peak QPS / access-log (IPs omitted on purpose)
- Why rec jobs MaxAttemptsExceeded (timeout vs memory vs exception) — do not dump payloads
- HTTP/3 to origin (not configured)
- iostat (command missing)

---

## 57. Repository Changes

```text
Created earlier:
- scripts/collect-production-baseline.sh

Updated:
- docs/skinbase-production-baseline.md

Modified application:
- none

Database changes:
- none

Server configuration changes:
- none

Service restarts:
- none

Cache clears:
- none

Queue operations:
- none (failed jobs listed only, not retried)
```