diff --git a/docs/optimization-plan.md b/docs/optimization-plan.md new file mode 100644 index 00000000..758fed81 --- /dev/null +++ b/docs/optimization-plan.md @@ -0,0 +1,1212 @@ +Seveda. Spodaj je celoten pregled, kaj smo pri **SkinbaseNova performance/legacy cleanup** že uredili, kaj je trenutno v teku in kaj še ostaja v planu. + +# 1. M1 — Recommendation nightly jobs + +Uredili smo background recommendation jobs: + +* popravljena `retry_after` logika, +* uvedeni batchi, +* ohranjena kompatibilnost serialized jobov, +* preprečeno podvajanje/dolgotrajno visenje queue jobov. + +Status: + +```text +M1 ✅ COMPLETE +``` + +# 2. M2 — Mail queue + +Mail pošiljanje je bilo prestavljeno oziroma urejeno prek queue sistema, da mail requesti ne blokirajo HTTP requestov. + +```text +M2 ✅ COMPLETE +``` + +# 3. M3 — Sitemap optimizacija + +Urejeno generiranje sitemapov, da niso nepotrebno dragi oziroma blokirajoči. + +```text +M3 ✅ COMPLETE +``` + +# 4. M4 — Recommendation read path + +Optimiziran read path za recommendations: + +* manj nepotrebnih queryjev, +* boljša uporaba že pripravljenih podatkov, +* brez spremembe recommendation algoritma. + +```text +M4 ✅ COMPLETE +``` + +# 5. M5 — Metrics cleanup + +Uredili smo pruning starih metrics podatkov: + +```text +retention = 30 dni +``` + +Brez agresivnega zmanjševanja zgodovine. + +```text +M5 ✅ COMPLETE +``` + +# 6. M6 — Memory / PHP-FPM audit + +Pregledali smo: + +* PHP-FPM procese, +* memory usage, +* worker sizing, +* server obremenitev. + +Pomembno: **nismo povečevali workerjev ali spreminjali FPM sizinga**, ker ni bilo dokaza, da je to pravi bottleneck. + +```text +M6 ✅ COMPLETE +``` + +# 7. M7 — Queue isolation / dedup / presence + +Urejeno: + +* boljša izolacija queue jobov, +* deduplication, +* omejeno online/presence delo, +* preprečevanje nepotrebnih Redis operacij. + +```text +M7 ✅ COMPLETE +``` + +# 8. M8 — Redis cleanup + +Uredili smo bounded Redis cleanup. + +Nismo uporabljali nevarnih stvari kot: + +```text +SMEMBERS na ogromnih setih +DEL velikih keyjev +Redis restart +flushall +``` + +```text +M8 ✅ COMPLETE +``` + +# 9. M9 — Scheduler / cron + +Uredili smo production scheduler in cron ownership. + +```text +M9 ✅ COMPLETE +``` + +# 10. M10 — Database cleanup + +Uredili večino DB problemov in query/index cleanup. + +Pomembna izjema: + +```text +dangerous snapshot migration ❌ NE SME biti izvedena +``` + +Ker bi odstranila uporaben production index: + +```text +idx_bucket_artwork(bucket_hour, artwork_id) +``` + +Ta index smo namenoma ohranili. + +```text +M10 ✅ COMPLETE +razen blokirane nevarne migracije +``` + +# 11. M11 — Endpoint performance + +Prvi sistematični endpoint performance pregled. + +Od tu naprej smo začeli meriti realne HTTP stroške namesto ugibanja. + +```text +M11 ✅ COMPLETE +``` + +--- + +# 12. M12 — Observability / performance logging + +To je bil velik milestone. + +Dodali smo nginx JSON performance log: + +```text +/var/log/nginx/skinbase-performance.log +``` + +Dodali smo: + +* Laravel slow HTTP middleware, +* threshold okoli 750 ms, +* `Server-Timing`, +* 14-dnevno slow log spremljanje, +* analyzer: + +```text +scripts/analyze-http-performance.php +``` + +Analyzer zna: + +* p50 +* p95 +* p99 +* max +* total time +* count +* 4xx +* 5xx +* 499 +* rotated logs +* gzip logs +* `--since` +* `--until` +* coverage checking + +```text +M12 ✅ COMPLETE +``` + +--- + +# 13. M12.5 — Vector / Similar-AI incident + +Odkrili smo konkretno težavo: + +Skinbase je poslal Qdrantu: + +```text +limit = 121 +``` + +Qdrant pa dovoljuje največ: + +```text +100 +``` + +Gateway je nato napačno preslikoval Qdrant 422 v 502, kar je odpiralo circuit breaker. + +## M12.5A + +Na Skinbase strani smo clampali similarity limit: + +```text +max 99 + exclusion = max 100 +``` + +## M12.5B + +Gateway smo popravili: + +* Qdrant 4xx ostane 4xx, +* Qdrant 5xx/transport → 502, +* sanitized error details. + +Incident je bil nato zaprt. + +```text +M12.5 ✅ COMPLETE +Vector incident ✅ CLOSED +``` + +--- + +# 14. M12.6 — Analyzer izboljšave + +Analyzer smo nadgradili za: + +* rotated loge, +* `.gz`, +* exact status, +* status classes, +* `--since`, +* `--until`, +* coverage. + +```text +M12.6 ✅ COMPLETE +``` + +--- + +# 15. M13 — X-Accel artwork download + +Velik performance win. + +Prej je Laravel/PHP dejansko serviral datoteko. + +Sedaj: + +```text +/download/artwork/{id} +``` + +Laravel naredi authorization/path resolution, nato pa nginx prevzame file transfer prek: + +```text +X-Accel-Redirect +``` + +Dodali smo tudi: + +* internal nginx alias, +* range support, +* SHA preverjanje, +* rate limiting, +* Cloudflare real IP handling. + +```text +M13 ✅ COMPLETE +``` + +--- + +# 16. M14 — Scanner fast reject + +Uvedli smo: + +```text +15-skinbase-scanner-fast-reject.conf +``` + +Nginx zdaj prestreže očitne probe še preden pridejo do PHP: + +* WordPress +* phpMyAdmin +* PHPUnit +* secrets/config +* Docker +* Terraform +* private keys +* Swagger +* GraphQL scanner paths +* Spring actuator +* Grafana +* Elasticsearch +* itd. + +Namerno nismo naredili blanket pravil, ki bi lahko blokirala legitimen Skinbase traffic. + +```text +M14 ✅ COMPLETE +``` + +--- + +# 17. M15 — Legacy download cleanup + +## M15A + +Stari: + +```text +/download/{id} +``` + +je bil še vedno bombardiran z requesti, čeprav route ne obstaja. + +Zdaj nginx vrne instant 404 brez PHP. + +## M15B + +Artwork download route: + +* public session samo kjer jo potrebujemo, +* `TrackOnlineVisitor` se izloči, +* invalid ID ne povzroča nepotrebnega Laravel overhead-a. + +```text +M15 ✅ COMPLETE +``` + +--- + +# 18. M16 — Legacy/scanner endpoints + +Nginx fast reject smo dodali za: + +```text +/zoom.php +/dist/manifest.json +``` + +Pomembno: + +```text +/build/manifest.json +``` + +ostaja normalen Vite manifest. + +```text +M16 ✅ COMPLETE +``` + +--- + +# 19. M17 — Legacy cleanup + +M17 je bil precej velik. + +## M17A — auth/admin scanner paths + +Dodani exact/narrow rejects za: + +```text +/secure +/users/login +/signin +/sign-in +/signup +/console +/backoffice +/panel +/portal +/api/env +... +/wp/v2 +``` + +Nismo blokirali legitimnih: + +```text +/reset-password +/auth +/account +/profile/* +/photo/* +/following/* +/livewire/update +... +``` + +## M17B — legacy photo URLs + +Implementiran: + +```text +LegacyArtworkPhotoController +``` + +* base62 decode, +* public/published artwork only, +* legacy photo path compatibility, +* direct 404 za neobstoječe datoteke, +* brez nepotrebnega visitor/session overhead-a. + +## M17B2 — legacy followers URL + +Legacy: + +```text +/following/{id}/{slug?} +``` + +sedaj naredi 301 na canonical: + +```text +/@{username}/followers +``` + +Dodani: + +* followers tab, +* pagination, +* canonical usernames, +* guest support. + +## M17C — ThumbnailService fix + +Odstranili smo napačen: + +```php +orWhere('legacy_id', $id) +``` + +in ostali pri PK lookupu: + +```php +find($id) +``` + +Ker novi importer že ohranja legacy ID kot primary key. + +```text +M17 ✅ COMPLETE +``` + +--- + +# 20. M18 — Artwork detail optimization + +To je trenutni glavni performance program. + +M18A audit je pokazal, da `/art/{id}/{slug}` povzroča velik browser fan-out: + +```text +/art page +POST /api/art/{id}/view +GET navigation +GET similar-ai +GET rank +GET reactions +GET /page prefetch +... +``` + +Od tu naprej smo sistematično odstranjevali requeste. + +--- + +# 21. M18B1 — Comment reaction N+1 + +Prej je frontend za vsak comment/reply posebej klical: + +```text +/api/comments/{id}/reactions +``` + +To je bil HTTP N+1. + +Zdaj comments API že vsebuje reaction aggregate. + +Rezultat: + +```text +automatic reaction GETs → praktično 0 +``` + +Stari endpoint še vedno ostaja kompatibilen. + +```text +M18B1 ✅ COMPLETE +``` + +--- + +# 22. M18B2 — Duplicate artwork lookup + +Odstranili smo drugi artwork lookup v public detail flowu. + +Prej: + +```text +2 artwork lookupa +``` + +Po: + +```text +1 artwork lookup +``` + +Tudi `Schema::hasTable()` smo request-local memoizirali. + +```text +M18B2 ✅ COMPLETE +``` + +--- + +# 23. M18B3 — Lightweight neighbor prefetch + +Prej je neighbor preload uporabljal skoraj full artwork page payload. + +Dodali smo: + +```text +/api/artworks/{id}/page?prefetch=1 +``` + +ki vrne samo: + +```text +id +title +slug +thumbs.md +thumbs.lg +``` + +Payload je padel približno: + +```text +~5776 B +→ +~1034 B +``` + +SQL: + +```text +guest ~1 +auth ~2 +``` + +```text +M18B3 ✅ COMPLETE +``` + +--- + +# 24. OPS-M18.1 — SSR process ownership + +Odkrili smo race condition pri Inertia SSR. + +Prej deploy: + +```text +stop SSR +nohup start SSR +``` + +kar je povzročalo: + +```text +EADDRINUSE +``` + +Zdaj je SSR pod: + +```text +Supervisor +``` + +Config: + +```text +/etc/supervisor/conf.d/skinbase-ssr.conf +``` + +Deploy samo naredi kontroliran supervisor restart. + +```text +OPS-M18.1 ✅ COMPLETE +``` + +--- + +# 25. M18B4 — Similar-AI defer + +Similar-AI se prej avtomatsko sprožil pri vsakem artwork mountu. + +Dodali smo: + +```text +IntersectionObserver +``` + +z: + +```text +rootMargin: 800px 0px +``` + +in JS-session registry: + +* shared in-flight, +* reuse completed result, +* no duplicate request, +* retry after error, +* correct subscriber handling. + +```text +M18B4 ✅ COMPLETE +``` + +Kasnejši M18C pa je pokazal, da se request count ni zmanjšal toliko, kot smo pričakovali. + +--- + +# 26. M18B5 — full `/page` audit + +Pregledali smo možnost zmanjšanja full `/page` payload/queryjev. + +Zaključek: + +```text +NO-OP +``` + +Ker full `/page` contract dejansko potrebuje: + +* viewer +* stats +* categories +* maturity +* evolution +* file +* itd. + +Ni bilo varne optimizacije brez spremembe contracta. + +```text +M18B5 ✅ COMPLETE — NO-OP +``` + +--- + +# 27. M18B6 — navigation SQL audit + +Poskus optimizacije navigation queryja. + +Ugotovili smo, da bi candidate izboljšava pomagala samo v zelo posebnem primeru z enim published artworkom. + +Za normalen production workload ni imela smisla. + +Original wrap-around semantics smo ohranili. + +```text +M18B6 ✅ COMPLETE — NO-OP +``` + +--- + +# 28. M18C — pravi production baseline + +Zbrali smo približno: + +```text +26 h 47 min +79,402 requestov +``` + +Največji stroški: + +```text +/art/{id}/{slug} +count 24377 +avg 0.615 s +total ~14999 s +``` + +Similar-AI: + +```text +count 2362 +avg 1.707 s +total ~4032 s +``` + +Artwork page: + +```text +count 4508 +avg 0.298 s +total ~1343 s +``` + +Navigation: + +```text +count 2713 +avg 0.297 s +total ~805 s +``` + +M18C nam je dal zelo dobro osnovo za naslednje korake. + +```text +M18C ✅ COMPLETE +``` + +--- + +# 29. OPS-M18.2 — legacy CometChat + +Našli smo: + +```text +/cometchat4/cometchat_receive.php +``` + +To je ostanek starega Skinbase/CometChat sistema. + +SkinbaseNova ga ne uporablja. + +M18C: + +```text +1831 requestov +1831 x 404 +~0.365 s avg +~669 s PHP časa +``` + +Dodali smo exact nginx fast reject. + +Sedaj: + +```text +404 +request_time 0.000 +no upstream +``` + +```text +OPS-M18.2 ✅ COMPLETE +``` + +--- + +# 30. M18D — odstranitev neighbor prefetch requestov + +Prej je: + +```text +/api/artworks/navigation/{id} +``` + +vrnil samo IDs/URLs. + +Frontend je nato naredil še: + +```text +/page?prefetch=1 +/page?prefetch=1 +``` + +za oba soseda. + +M18D je navigation response razširil z: + +```text +prev_preview +next_preview +``` + +ki vsebujeta: + +```text +id +title +slug +thumbs.md +thumbs.lg +``` + +Zato frontend ne potrebuje več dodatnih prefetch HTTP requestov. + +Tipičen initial artwork: + +```text +PREJ +1 navigation +2 prefetch + +ZDAJ +1 navigation +0 prefetch +``` + +Production/browser acceptance je prestal. + +```text +M18D ✅ COMPLETE +``` + +Production measurement boundary: + +```text +2026-08-28T18:44:04+02:00 +``` + +Ta baseline trenutno še teče. + +--- + +# 31. M18E — trenutno v delu + +Codex trenutno dela na tem. + +Cilj je odstraniti še standalone: + +```text +/api/artworks/navigation/{id} +``` + +iz normalnega browser flowa. + +Ideja: + +## Initial page + +Namesto: + +```text +GET /art/A +GET /api/artworks/navigation/A +``` + +želimo: + +```text +GET /art/A +``` + +SSR že vrne navigation payload. + +## Client navigation + +Namesto: + +```text +GET /api/artworks/B/page +GET /api/artworks/navigation/B +``` + +želimo: + +```text +GET /api/artworks/B/page +``` + +Full `/page` response bo vseboval tudi navigation payload. + +Standalone endpoint ostane fallback. + +Če uspe: + +```text +initial artwork: +-1 HTTP request + +vsak A → B: +-1 HTTP request +``` + +```text +M18E 🟡 IN PROGRESS +``` + +--- + +# Kaj še imamo v planu + +Od tu naprej bi šel po podatkih, ne po vnaprej fiksnem seznamu. + +## M18F — M18D/M18E production validation + +Najprej bomo izmerili učinek. + +Za M18D imamo baseline: + +```text +START +2026-08-28T18:44:04+02:00 +``` + +Primerjali bomo: + +```text +/page count +/navigation count +page/navigation ratio +total upstream time +p50/p95/p99 +``` + +Po M18E bomo naredili nov boundary in enako meritev. + +Glavni cilj: + +```text +navigation request count → skoraj 0 v normalnem UI flowu +``` + +--- + +# M18G — Similar-AI druga faza + +Similar-AI je še vedno drugi največji konkretni hotspot. + +M18C: + +```text +2362 requestov +~4032 s total +avg 1.707 s +``` + +B4 ga je deferred, vendar ratio: + +```text +similar-ai / artwork detail +≈ 9.7 % +``` + +je bil skoraj isti kot prej (~10 %). + +Zato moramo raziskati: + +* zakaj `IntersectionObserver` praktično vedno triggera, +* ali je sekcija preblizu viewporta, +* ali `rootMargin=800px` povzroča skoraj instant load, +* ali je smiselno AI recommendations loadati šele po dejanskem scrollu, +* ali uporabnik sploh vidi sekcijo, +* ali lahko rezultat reuseamo dlje v client sessionu. + +Pomembno: ne bomo slepo spreminjali TTL-jev, breakerja ali Qdrant nastavitev. + +--- + +# M18H — artwork `/art` SSR cost + +Po odstranitvi browser fan-outa ostane največji posamezni strošek: + +```text +/art/{id}/{slug} +``` + +M18C: + +```text +count 24377 +avg ~615 ms +p95 ~1.11 s +p99 ~2.38 s +``` + +Takrat bomo naredili bolj natančen query/profile audit samega SSR requesta: + +* relations, +* duplicate model serialization, +* avatar/profile lookup, +* categories, +* stats, +* metadata, +* SEO, +* props, +* Inertia serialization. + +Cilj bo zmanjšati: + +```text +~0.615 s +``` + +brez spreminjanja funkcionalnosti. + +--- + +# M18I — view tracking + +Trenutno imamo: + +```text +POST /api/art/{id}/view +``` + +M18C: + +```text +2715 requestov +avg ~390 ms +total ~1060 s +``` + +To je precejšen strošek za nekaj, kar je sekundarna funkcija. + +Možne smeri: + +* batch/deferred write, +* cheap queue dispatch, +* Redis counter + async persist, +* bolj minimalen middleware stack. + +Vendar bomo pred tem preverili semantics, da ne pokvarimo unique view counting. + +--- + +# M18J — rank endpoint + +M18C: + +```text +/api/rank/category/{id} +count 2444 +avg ~379 ms +total ~927 s +``` + +To je naslednji očiten artwork fan-out kandidat. + +Preverili bomo: + +* ali se rank lahko doda v že obstoječ artwork response, +* ali se lahko request defera, +* ali je rank nujen na initial paint, +* cache hit/miss rate, +* ali je mogoče odstraniti standalone HTTP request. + +--- + +# M18K — comments endpoint + +M18B1 je odstranil reaction N+1, vendar še vedno imamo: + +```text +/api/artworks/{id}/comments +``` + +M18C: + +```text +988 requestov +avg ~312 ms +``` + +Če bo po artwork optimizacijah še relevanten, pogledamo: + +* lazy loading, +* payload size, +* pagination, +* avatar/profile serialization, +* relation queryje. + +--- + +# M19 — profile pages + +M18C je pokazal tudi precejšen strošek: + +```text +/@{user} +count ~2061 +avg ~648 ms +total ~1335 s +``` + +in: + +```text +/@{user}/gallery +~969 requestov +avg ~623 ms +``` + +Ko zaključimo artwork flow, je profil verjetno naslednji večji frontend/SSR kandidat. + +--- + +# M20 — homepage / discover + +Potencialni naslednji kandidati: + +```text +/ +discover/fresh +discover/trending +discover/rising +``` + +Trenutno niso kritični kot artwork, ampak imajo dovolj prometa, da se bodo prihranki poznali. + +--- + +# M21 — remaining legacy traffic cleanup + +Performance log nam še vedno kaže stare/scanner URL-je. + +Primeri: + +```text +/cometchat4/* +old downloads +wp probes +legacy PHP paths +old profiles +scanner APIs +``` + +Pristop ostaja isti: + +```text +dokazano ne obstaja ++ +ni legitimne funkcije ++ +veliko prometa += +exact nginx fast reject +``` + +Nikoli blanket reject brez audita. + +--- + +# M22 — final infrastructure re-audit + +Ko app-level optimizacije zaključimo: + +* PHP-FPM saturation, +* CPU, +* MySQL latency, +* Redis latency, +* queue latency, +* nginx timings, +* slow requests, +* 5xx, +* 499, +* SSR health. + +Šele če podatki pokažejo potrebo, bi razmišljali o spremembah workerjev ali infrastrukture. + +--- + +# Trenutna slika + +Če to skrčimo na status: + +```text +M1–M17 ✅ COMPLETE + +M18A ✅ audit +M18B1 ✅ reactions HTTP N+1 +M18B2 ✅ duplicate artwork lookup +M18B3 ✅ lightweight prefetch +OPS-M18.1 ✅ Supervisor SSR +M18B4 ✅ deferred Similar-AI +M18B5 ✅ audit / no-op +M18B6 ✅ audit / no-op +M18C ✅ production baseline +OPS-M18.2 ✅ legacy CometChat nginx reject +M18D ✅ remove neighbor prefetch HTTP +M18E ✅ collapse navigation HTTP request +M18F ✅ production measurement +M18G ✅ Similar-AI phase 2 + +NEXT: +M18H artwork SSR optimization +M18I view tracking +M18J rank request +M18K comments +M19 profiles +M20 discover/home +M21 remaining legacy traffic +M22 final infra re-audit +``` + +Največja sprememba v celotnem projektu je, da smo začeli z običajnim “optimiziraj PHP/MySQL” pristopom, zdaj pa smo prišli do precej bolj učinkovitega modela: **iz production telemetry identificiramo konkretne stroške, odstranimo nepotrebne HTTP requeste in Laravel bootstrape, nato rezultat ponovno izmerimo**. Tako imamo pri skoraj vsakem koraku dokazljiv before/after.