Wire admin studio SSR and search infrastructure
This commit is contained in:
@@ -36,10 +36,10 @@ We provide an example Supervisor config at `deploy/supervisor/skinbase-queue.con
|
||||
The example worker listens on the application queues currently used by Skinbase features, including AI and discovery jobs:
|
||||
|
||||
```text
|
||||
forum-security,forum-moderation,vision,recommendations,discovery,mail,default
|
||||
search,forum-security,forum-moderation,vision,recommendations,discovery,mail,default
|
||||
```
|
||||
|
||||
If you split workloads across dedicated workers, make sure any queues configured through `VISION_QUEUE`, `RECOMMENDATIONS_QUEUE`, or `DISCOVERY_QUEUE` are explicitly covered by at least one worker process.
|
||||
If you split workloads across dedicated workers, make sure the Scout/Meilisearch `search` queue and any queues configured through `VISION_QUEUE`, `RECOMMENDATIONS_QUEUE`, or `DISCOVERY_QUEUE` are explicitly covered by at least one worker process.
|
||||
|
||||
To use it on a Debian/Ubuntu server:
|
||||
|
||||
@@ -79,7 +79,7 @@ Adjust `WorkingDirectory` and `User` in the unit to match your deployment.
|
||||
- Start a one-off worker (foreground):
|
||||
|
||||
```bash
|
||||
php artisan queue:work --sleep=3 --tries=3
|
||||
php artisan queue:work --sleep=3 --tries=3 --queue=search,default
|
||||
```
|
||||
|
||||
- Restart all workers gracefully (useful after deployments):
|
||||
@@ -108,6 +108,7 @@ php artisan queue:flush
|
||||
- Ensure `QUEUE_CONNECTION` in `.env` matches the driver you've configured.
|
||||
- If using `database` driver, the `jobs` and `failed_jobs` tables must exist.
|
||||
- The mailable used for contact submissions is queued; if the queue worker is not running mails will accumulate in the queue table (or Redis).
|
||||
- Scout search syncing uses the `search` queue from `config/scout.php`. If workers only consume `default`, `artworks:search-rebuild` will dispatch jobs but new artwork documents will never reach Meilisearch.
|
||||
- AI auto-tagging, embedding generation, vector index repair, and recommendation cache regeneration are all queued. If you move them onto dedicated queues, the shipped example worker commands must be updated to consume those queue names.
|
||||
|
||||
8) AI and discovery queue checklist
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Skinbase26 CLI Reference
|
||||
|
||||
Last updated: 2026-04-18
|
||||
Last updated: 2026-04-25
|
||||
|
||||
This document lists the repository-specific command-line entry points in Skinbase26.
|
||||
|
||||
@@ -27,7 +27,8 @@ Examples below are representative. For the full option list of any Artisan comma
|
||||
|
||||
| Script | Why it is used | Example |
|
||||
| --- | --- | --- |
|
||||
| `scripts/deploy-production.sh` | Full production deployment workflow including build, rsync, migrations, and optional sync steps | `bash scripts/deploy-production.sh --mode=normal` |
|
||||
| `scripts/deploy-production.sh` | Full production deployment workflow using retained release directories and a server-side current-release switch | `bash scripts/deploy-production.sh --mode=normal` |
|
||||
| `scripts/rollback-production.sh` | Switch the active production release to a previously retained server-side deployment | `bash scripts/rollback-production.sh --previous` |
|
||||
| `scripts/push-db-to-prod.sh` | Push the local database to production with remote backup controls | `bash scripts/push-db-to-prod.sh --force` |
|
||||
| `scripts/render-nova-card.cjs` | Render a Nova card screenshot through Playwright | `node scripts/render-nova-card.cjs --url=https://example.test/card --out=tmp/card.png` |
|
||||
| `scripts/vision-smoke.ps1` | Smoke-test the configured Vision API from PowerShell | `powershell -ExecutionPolicy Bypass -File scripts/vision-smoke.ps1` |
|
||||
@@ -299,4 +300,10 @@ These are useful, but they read more like maintenance probes and one-off utiliti
|
||||
|
||||
- The deploy mirror under `.deploy/artwork-evolution-release` contains equivalent command classes for deployment packaging, but this reference is sourced from the main repository tree.
|
||||
- For commands that mutate production data, prefer running a dry-run mode first when one exists.
|
||||
- For one-off scripts, read the file before using it. Several are intentionally ad hoc and may contain hardcoded IDs, paths, or assumptions.
|
||||
- For one-off scripts, read the file before using it. Several are intentionally ad hoc and may contain hardcoded IDs, paths, or assumptions.
|
||||
|
||||
## Scheduler
|
||||
|
||||
- **Active scheduler file:** The application's active runtime schedule is consolidated in [routes/console.php](routes/console.php). Do not duplicate recurring entries in [app/Console/Kernel.php](app/Console/Kernel.php) — the app boots command routes from `bootstrap/app.php` and registering the same cron bodies in multiple places can lead to duplicate runs.
|
||||
- **Validation:** After editing schedules run `php artisan schedule:list` to confirm Laravel parses the schedule and to view the resolved cron timings. This command also surfaces invalid frequency methods (for example `everyHour()` or `everyFiveMinute()`) which will break schedule introspection.
|
||||
- **Best practice:** Stagger heavy jobs away from the same minute boundaries (avoid piling multiple CPU-heavy commands on `:00`, `:10`, `:15`, `:30`). Prefer cron offsets or `->cron()` expressions in `routes/console.php` and use `->withoutOverlapping()` and `->runInBackground()` where appropriate.
|
||||
@@ -15,17 +15,21 @@ If you launch `bash sync.sh` from WSL against this Windows checkout, the script
|
||||
This will:
|
||||
|
||||
- build frontend assets locally with `npm run build`
|
||||
- rsync the application files to production
|
||||
- run `composer install --no-dev` on the server before entering maintenance mode
|
||||
- bring the app down only for the short critical section that runs `php artisan migrate --force`, `php artisan optimize:clear`, and `php artisan optimize`
|
||||
- stage the new code into a versioned release directory on the production server
|
||||
- run `composer install --no-dev` inside that staged release before switching traffic
|
||||
- keep the fixed production app path pointed at the active release through a server-side `current` symlink
|
||||
- bring the app down only for the short critical section that switches the active release and runs `php artisan migrate --force`, `php artisan optimize:clear`, and `php artisan optimize`
|
||||
- bring the app back up immediately after that critical section finishes
|
||||
- warm the guest homepage cache with `php artisan homepage:warm-guest-cache`
|
||||
- warm the post trending cache with `php artisan posts:warm-trending`
|
||||
- restart queue workers with `php artisan queue:restart`
|
||||
- gracefully restart Horizon with `php artisan horizon:terminate`
|
||||
|
||||
This is now the low-downtime default path for normal code and feature deploys.
|
||||
|
||||
Each deploy generates a release ID automatically from UTC time and the local Git revision. Releases are retained under `REMOTE_RELEASE_ROOT/releases/<release-id>`, and production switches between them on the server by updating `REMOTE_RELEASE_ROOT/current`. The public/runtime path stays fixed at `REMOTE_FOLDER`, which is now treated as a stable symlink to the active release.
|
||||
|
||||
On the first deploy with this layout, the existing live folder is adopted into the release archive automatically and `REMOTE_FOLDER` is converted into that stable symlink path. After that, switching back to an older release does not require any local re-upload.
|
||||
|
||||
## Full upgrade
|
||||
|
||||
Use a full upgrade when the release also needs broad Meilisearch work or non-code service operations.
|
||||
@@ -63,12 +67,15 @@ bash sync.sh --skip-build
|
||||
bash sync.sh --skip-migrate
|
||||
bash sync.sh --no-maintenance
|
||||
bash sync.sh --mode=full-upgrade
|
||||
bash sync.sh --keep-releases=8
|
||||
bash sync.sh --release-id=release-2026-04-25
|
||||
```
|
||||
|
||||
Environment overrides:
|
||||
|
||||
```bash
|
||||
REMOTE_SERVER=user@example.com REMOTE_FOLDER=/var/www/app bash sync.sh
|
||||
REMOTE_RELEASE_ROOT=/var/www/app.releases RELEASE_RETENTION=8 bash sync.sh
|
||||
```
|
||||
|
||||
You can also override the local build command explicitly:
|
||||
@@ -85,6 +92,40 @@ FULL_UPGRADE_PRE_HOOK='sudo systemctl stop reverb' bash sync.sh --full-upgrade
|
||||
FULL_UPGRADE_POST_HOOK='sudo systemctl restart reverb meilisearch' bash sync.sh --full-upgrade
|
||||
```
|
||||
|
||||
## Rollback and release history
|
||||
|
||||
List retained releases on production:
|
||||
|
||||
```bash
|
||||
bash scripts/rollback-production.sh --list
|
||||
```
|
||||
|
||||
Switch production to the previous retained release:
|
||||
|
||||
```bash
|
||||
bash scripts/rollback-production.sh --previous
|
||||
```
|
||||
|
||||
Switch production to a specific retained release:
|
||||
|
||||
```bash
|
||||
bash scripts/rollback-production.sh --release-id=20260425-132455-a1b2c3d
|
||||
```
|
||||
|
||||
Preview a release switch without changing the server:
|
||||
|
||||
```bash
|
||||
bash scripts/rollback-production.sh --previous --dry-run
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
|
||||
- Rollback now means switching the active server release, not re-syncing files from local.
|
||||
- Each retained release already contains its own code and vendor tree, so rollback is primarily a symlink switch plus cache refresh and `queue:restart`.
|
||||
- Rollback does not reverse database migrations. If a release includes incompatible schema changes, handle the database separately.
|
||||
- Release retention defaults to 5 releases and can be changed with `--keep-releases` or `RELEASE_RETENTION`.
|
||||
- Release data lives outside the active app path by default at `REMOTE_FOLDER.releases`, so switching releases happens entirely on the production server.
|
||||
|
||||
## Replace production database from local
|
||||
|
||||
This is intentionally separate from a normal deploy because it overwrites production data.
|
||||
@@ -139,4 +180,24 @@ LOCAL_MYSQLDUMP_COMMAND='mysqldump --host=10.0.0.5 --port=3306 --user=app dbname
|
||||
- Normal deployments should use `bash sync.sh` without `--with-db`.
|
||||
- Use `bash sync.sh --full-upgrade` only when the release also includes Meilisearch-wide refreshes or remote service changes.
|
||||
- Use database replacement only for first-time bootstrap, staging, or an intentional full production reset.
|
||||
- Route caching now runs through `php artisan optimize` in deploy automation; if that starts failing again, fix the route definitions instead of dropping route caching from deploy.
|
||||
- Use `bash scripts/rollback-production.sh --previous` for a fast server-side release switch when the last deploy needs to be reverted.
|
||||
- Route caching now runs through `php artisan optimize` in deploy automation; if that starts failing again, fix the route definitions instead of dropping route caching from deploy.
|
||||
|
||||
## Nginx upstream error pages
|
||||
|
||||
Laravel now owns the Nova-style HTML error pages for normal application responses, but true upstream failures such as `502 Bad Gateway` and `504 Gateway Timeout` still have to be handled by nginx because PHP/FPM is unavailable in that state.
|
||||
|
||||
The repo includes a ready-to-include snippet at `deploy/nginx/upstream-error-pages.conf` and the matching static page at `public/errors/upstream-gateway.html`.
|
||||
|
||||
To enable it on production:
|
||||
|
||||
1. Include the snippet inside the Skinbase `server {}` block.
|
||||
2. Add `fastcgi_intercept_errors on;` to every FastCGI location that should use the static fallback.
|
||||
3. Keep the static file available in the live public path so nginx can serve it without Laravel.
|
||||
|
||||
On the current `skinbase.top` vhost, the required FastCGI locations are:
|
||||
|
||||
- `location ^~ /api/uploads/`
|
||||
- `location = /index.php`
|
||||
|
||||
This intentionally intercepts only `502` and `504`, so Laravel remains responsible for normal `404`, `419`, `429`, `500`, and `503` rendering when the application is actually running.
|
||||
473
docs/slow-query-optimization-plan.md
Normal file
473
docs/slow-query-optimization-plan.md
Normal file
@@ -0,0 +1,473 @@
|
||||
# MySQL Slow Query Optimization Plan
|
||||
|
||||
**Source:** `/var/log/mysql/slow.log` — 68,950 total queries, 151,576s total exec time
|
||||
**Period:** 2026-04-04 → 2026-04-26 (22 days)
|
||||
**Threshold:** 500ms
|
||||
**Server:** server3 / database: `skinbase`
|
||||
|
||||
---
|
||||
|
||||
## Summary Stats
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Total queries logged | 68,950 |
|
||||
| Unique query fingerprints | 139 |
|
||||
| Total execution time | 151,576s (~42h) |
|
||||
| Average exec time | 2s |
|
||||
| 95th percentile | 3s |
|
||||
| Total rows examined | 15.39B |
|
||||
| Total bytes sent | 40.79GB |
|
||||
|
||||
---
|
||||
|
||||
## Priority 1 — Critical (fix immediately)
|
||||
|
||||
### P1-A: Correlated subquery counting artworks per tag (Query 8)
|
||||
**Total time:** 1,004s · **Calls:** 1,138 · **Rows examined/call:** ~240k
|
||||
|
||||
**Current query:**
|
||||
```sql
|
||||
SELECT tags.*,
|
||||
(SELECT count(*) FROM artworks
|
||||
INNER JOIN artwork_tag ON artworks.id = artwork_tag.artwork_id
|
||||
WHERE tags.id = artwork_tag.tag_id AND artworks.deleted_at IS NULL)
|
||||
AS artworks_count
|
||||
FROM tags
|
||||
ORDER BY artworks_count DESC
|
||||
LIMIT 10
|
||||
```
|
||||
|
||||
**Problem:** N+1 correlated subquery — one full `artworks JOIN artwork_tag` count per tag row.
|
||||
|
||||
**Fix options (pick one):**
|
||||
1. **Best — cached counter column:** Add `artworks_count INT DEFAULT 0` to `tags`, maintained by an Eloquent observer on `artwork_tag` attach/detach. Query becomes `SELECT * FROM tags ORDER BY artworks_count DESC LIMIT 10` — instant.
|
||||
2. **Quick — JOIN + GROUP BY:**
|
||||
```sql
|
||||
SELECT t.*, COALESCE(cnt.c, 0) AS artworks_count
|
||||
FROM tags t
|
||||
LEFT JOIN (
|
||||
SELECT at.tag_id, COUNT(*) AS c
|
||||
FROM artwork_tag at
|
||||
JOIN artworks a ON a.id = at.artwork_id AND a.deleted_at IS NULL
|
||||
GROUP BY at.tag_id
|
||||
) cnt ON cnt.tag_id = t.id
|
||||
ORDER BY artworks_count DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
**Migration needed:** `php artisan make:migration add_artworks_count_to_tags`
|
||||
|
||||
---
|
||||
|
||||
### P1-B: Correlated subquery counting artworks per user (Query 33)
|
||||
**Total time:** 85s · **Calls:** 131 · **Rows examined/call:** ~61k
|
||||
|
||||
**Current query:**
|
||||
```sql
|
||||
SELECT users.*,
|
||||
(SELECT count(*) FROM artworks
|
||||
WHERE users.id = artworks.user_id AND is_approved=1 AND is_public=1 ...)
|
||||
AS artworks_count
|
||||
FROM users
|
||||
HAVING artworks_count > 0
|
||||
ORDER BY artworks_count DESC
|
||||
LIMIT 6
|
||||
```
|
||||
|
||||
**Problem:** Same correlated N+1 pattern per user. `HAVING` on a subquery forces a full users scan.
|
||||
|
||||
**Fix:** Use the existing `user_statistics` table. Add `public_artworks_count INT DEFAULT 0` if not present, maintained by artwork publish/unpublish observer. Then:
|
||||
```sql
|
||||
SELECT users.*, us.public_artworks_count AS artworks_count
|
||||
FROM users
|
||||
JOIN user_statistics us ON us.user_id = users.id
|
||||
WHERE us.public_artworks_count > 0 AND users.deleted_at IS NULL
|
||||
ORDER BY us.public_artworks_count DESC
|
||||
LIMIT 6;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P1-C: Jobs table LIKE scan on JSON payload (Query 38)
|
||||
**Total time:** 64s · **Calls:** 58 · **Rows examined/call:** ~55k
|
||||
|
||||
**Current query:**
|
||||
```sql
|
||||
SELECT count(*) FROM jobs
|
||||
WHERE payload LIKE '%AutoTagArtworkJob%' AND payload LIKE '%69756%'
|
||||
```
|
||||
|
||||
**Problem:** Full scan of the `jobs` table JSON payload column — no index possible on LIKE '%...%'.
|
||||
|
||||
**Fix:** Replace this deduplication check with a Redis key or a dedicated `job_dedup` table with an indexed `(job_class, subject_id)` column. Example:
|
||||
```php
|
||||
// Instead of scanning jobs table:
|
||||
if (Cache::has("auto-tag-queued:{$artworkId}")) return;
|
||||
Cache::put("auto-tag-queued:{$artworkId}", true, now()->addHours(1));
|
||||
AutoTagArtworkJob::dispatch($artwork);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P1-D: `SELECT DISTINCT artwork_id` from snapshots with no index (Query 12)
|
||||
**Total time:** 923s · **Calls:** 280 · **Rows examined/call:** ~3.97M (max 8.3M!)
|
||||
|
||||
**Current query:**
|
||||
```sql
|
||||
SELECT DISTINCT artwork_id
|
||||
FROM artwork_metric_snapshots_hourly
|
||||
WHERE bucket_hour BETWEEN '...' AND '...'
|
||||
```
|
||||
|
||||
**Problem:** No covering index on `(bucket_hour, artwork_id)`. Full or large range scan every time.
|
||||
|
||||
**Fix:** Add a compound index:
|
||||
```sql
|
||||
ALTER TABLE artwork_metric_snapshots_hourly
|
||||
ADD INDEX idx_bucket_artwork (bucket_hour, artwork_id);
|
||||
```
|
||||
|
||||
Migration: `php artisan make:migration add_index_bucket_artwork_to_metric_snapshots`
|
||||
|
||||
---
|
||||
|
||||
### P1-E: Full-text LIKE searches on artworks title + description (Queries 50, 53)
|
||||
**Total time:** ~62s combined · **Calls:** ~48 · Pattern: `WHERE title LIKE '%keyword%' OR description LIKE '%keyword%' OR ...` (20+ OR conditions)
|
||||
|
||||
**Problem:** Leading wildcard LIKE cannot use B-tree indexes. Full table scan every time.
|
||||
|
||||
**Fix:** Use **Meilisearch** (already in use for artwork search). Route AI-tag search queries through `ArtworkSearchService` instead of raw LIKE. For any fallback that must stay in MySQL:
|
||||
```sql
|
||||
-- Add FULLTEXT index:
|
||||
ALTER TABLE artworks ADD FULLTEXT INDEX ft_artwork_text (title, description);
|
||||
-- Then use MATCH..AGAINST instead of LIKE:
|
||||
WHERE MATCH(title, description) AGAINST ('+moon +lunar' IN BOOLEAN MODE)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Priority 2 — High Impact
|
||||
|
||||
### P2-A: Artwork aggregate stats queries — top 2 time consumers (Queries 1 & 2)
|
||||
**Total time:** 83,766s + 34,068s = **117,834s** (78% of all slow query time)
|
||||
**Calls:** 37,124 + 13,789 = ~51k · **Avg:** 2.2s · **Rows examined/call:** ~215–234k
|
||||
|
||||
These are the same heavy SELECT pattern loading per-artwork stats from multiple tables:
|
||||
```sql
|
||||
SELECT a.id, a.user_id, a.published_at, a.is_public, a.is_approved,
|
||||
(a.thumb_ext IS NOT NULL AND a.thumb_ext != '') AS has_thumbnail,
|
||||
COALESCE(ast.views, 0) AS views_all,
|
||||
COALESCE(ast.downloads, 0) AS downloads_all,
|
||||
COALESCE(ast.favorites, 0) AS favourites_all,
|
||||
COALESCE(cc.cnt, 0) AS comments_count,
|
||||
COALESCE(sc.cnt, 0) AS shares_count,
|
||||
COALESCE(ast.views_7d, 0) AS views_7d,
|
||||
...
|
||||
FROM artworks a
|
||||
LEFT JOIN artwork_stats ast ON ast.artwork_id = a.id
|
||||
LEFT JOIN (SELECT artwork_id, COUNT(*) cnt FROM artwork_favourites WHERE created_at >= ...) fav7 ...
|
||||
LEFT JOIN (SELECT artwork_id, COUNT(*) cnt FROM artwork_comments ...) cc ...
|
||||
LEFT JOIN (SELECT artwork_id, COUNT(*) cnt FROM artwork_shares ...) sc ...
|
||||
WHERE a.is_public = 1 AND a.is_approved = 1 AND a.deleted_at IS NULL
|
||||
AND a.published_at <= NOW()
|
||||
ORDER BY a.id ASC
|
||||
LIMIT 500
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Inline derived subqueries for `fav7`, `cc`, `sc` run per page — not cached.
|
||||
- `ORDER BY a.id` with filters requires index on `(deleted_at, is_public, is_approved, published_at, id)`.
|
||||
- 51k calls/22 days = ~2,300 calls/day = every ~37 seconds, all day long. This is a **scheduled job or background process** churning through artworks.
|
||||
|
||||
**Fix — composite index (immediate):**
|
||||
```sql
|
||||
ALTER TABLE artworks
|
||||
ADD INDEX idx_public_approved_published (deleted_at, is_public, is_approved, published_at, id);
|
||||
```
|
||||
|
||||
**Fix — pre-aggregate counts (medium term):**
|
||||
Ensure `artwork_stats` already stores `favorites_7d`, `comments_count`, `shares_count`. If so, move all counts to `artwork_stats` maintenance jobs and remove the inline derived joins. The SELECT becomes a single fast `LEFT JOIN artwork_stats`.
|
||||
|
||||
**Fix — reduce call frequency:**
|
||||
If this is a scheduler-driven scan, batch it into chunks with exponential backoff and persist cursor position so it doesn't re-scan from scratch every run.
|
||||
|
||||
---
|
||||
|
||||
### P2-B: User stats calculation — 28 second average (Query 3)
|
||||
**Total time:** 8,847s · **Calls:** 320 · **Avg:** 28s · **Max:** 180s!
|
||||
|
||||
Complex query joining `users`, `user_xp_logs`, `user_followers`, `artwork_likes`, `artworks`, `artwork_metric_snapshots_hourly`.
|
||||
|
||||
**Fix:** This should **never run on demand**. Route it to:
|
||||
1. A scheduled background job that pre-aggregates into `user_statistics` (runs every N minutes).
|
||||
2. The controller/service reads from `user_statistics` only — single-row lookup by `user_id`.
|
||||
|
||||
Also ensure `user_statistics` has `INDEX idx_user_id (user_id)`.
|
||||
|
||||
---
|
||||
|
||||
### P2-C: `artwork_metric_snapshots_hourly` heavy join query (Query 4)
|
||||
**Total time:** 6,328s · **Calls:** 237 · **Avg:** 27s · **Max:** 52s
|
||||
|
||||
```sql
|
||||
SELECT artworks.*, ...
|
||||
FROM artworks
|
||||
JOIN artwork_metric_snapshots_hourly amsh ON amsh.artwork_id = artworks.id
|
||||
JOIN artwork_likes al ON ...
|
||||
JOIN artwork_downloads ad ON ...
|
||||
JOIN artwork_comments ac ON ...
|
||||
WHERE amsh.bucket_hour BETWEEN ... AND ...
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
1. Add index from P1-D: `idx_bucket_artwork (bucket_hour, artwork_id)`.
|
||||
2. Pre-aggregate hourly snapshots into daily/weekly summary tables and query those instead.
|
||||
3. Reduce time range of `BETWEEN` clause if querying recent data only.
|
||||
|
||||
---
|
||||
|
||||
### P2-D: rank_artwork_scores queries — ~10 variants (Queries 26, 27, 29, 37, 39, 41, 42, 43, 46)
|
||||
**Total time:** ~750s combined · ~870 combined calls · **Avg:** ~750ms · **Rows examined/call:** ~140–150k
|
||||
|
||||
Pattern:
|
||||
```sql
|
||||
SELECT ras.artwork_id, a.user_id, ras.score_trending
|
||||
FROM rank_artwork_scores ras
|
||||
INNER JOIN artworks a ON a.id = ras.artwork_id AND a.is_public=1 AND a.is_approved=1 AND a.deleted_at IS NULL
|
||||
WHERE ras.model_version = 'rank_v2'
|
||||
ORDER BY ras.score_trending DESC
|
||||
LIMIT 200
|
||||
```
|
||||
|
||||
**Problem:** `WHERE model_version = 'rank_v2' ORDER BY score_X DESC` — no composite index covers both.
|
||||
|
||||
**Fix — add partial/composite indexes:**
|
||||
```sql
|
||||
ALTER TABLE rank_artwork_scores
|
||||
ADD INDEX idx_mv_trending (model_version, score_trending DESC),
|
||||
ADD INDEX idx_mv_new_hot (model_version, score_new_hot DESC),
|
||||
ADD INDEX idx_mv_best (model_version, score_best DESC),
|
||||
ADD INDEX idx_mv_score (model_version, score_new_hot, score_trending, score_best);
|
||||
```
|
||||
|
||||
**Fix — cache ranking results:**
|
||||
These are pre-computed ranking scores. Cache the TOP 200 list per `(model_version, score_column)` for 5–15 minutes in Redis. The ranking job already runs on a schedule — warm the cache at the end of each ranking job run.
|
||||
|
||||
---
|
||||
|
||||
### P2-E: `artworks` public count via full scan (Query 30)
|
||||
**Total time:** 100s · **Calls:** 127 · **Avg:** 790ms · **Rows examined/call:** 97k
|
||||
|
||||
```sql
|
||||
SELECT count(*) FROM artworks
|
||||
WHERE deleted_at IS NULL AND is_approved=1 AND is_public=1
|
||||
AND published_at IS NOT NULL AND published_at <= NOW()
|
||||
```
|
||||
|
||||
**Fix — maintain a counter cache:**
|
||||
```php
|
||||
// In config or cache:
|
||||
Cache::remember('artworks.public_count', 300, fn() => Artwork::public()->published()->count());
|
||||
```
|
||||
|
||||
Or store the count in a `site_statistics` / `system_settings` table, updated by the publish observer.
|
||||
|
||||
---
|
||||
|
||||
### P2-F: artworks ORDER BY id scanner for publish pipeline (Queries 18, 22, 23, 40, 47)
|
||||
**Total time:** ~650s combined · Many calls · Pattern: `SELECT * FROM artworks WHERE ... ORDER BY id ASC` or `ORDER BY trending_score_7d DESC`
|
||||
|
||||
**Problem:** `SELECT *` loads all columns including large `description` blobs. The publish/ranking pipeline only needs IDs.
|
||||
|
||||
**Fix:**
|
||||
- Use `SELECT id` or `SELECT id, user_id` instead of `SELECT *`.
|
||||
- Ensure `trending_score_7d` is indexed if using `ORDER BY trending_score_7d`.
|
||||
- Add index from P2-A: `idx_public_approved_published`.
|
||||
|
||||
---
|
||||
|
||||
## Priority 3 — Medium Impact
|
||||
|
||||
### P3-A: Popular tags query with tag_interaction_daily_metrics (Query 6)
|
||||
**Total time:** 2,234s · **Calls:** 2,349 · **Avg:** 951ms
|
||||
|
||||
Joining `artworks → artwork_tag → tags → tag_interaction_daily_metrics`. 246k rows examined/call.
|
||||
|
||||
**Fix:**
|
||||
- Cache the result: popular tags change slowly — cache 5–15 minutes.
|
||||
- Add index on `tag_interaction_daily_metrics (tag_id, metric_date)`.
|
||||
- Precompute `tag_interaction_daily_metrics` aggregates into a `tag_trending_scores` table.
|
||||
|
||||
---
|
||||
|
||||
### P3-B: Browse/gallery category + tag joins (Query 7)
|
||||
**Total time:** 1,758s · **Calls:** 2,344 · **Avg:** 750ms
|
||||
|
||||
`artworks + categories + artwork_category + artwork_tag` — 80M total rows examined.
|
||||
|
||||
**Fix:**
|
||||
- Verify indexes: `artwork_category(category_id, artwork_id)`, `artwork_tag(tag_id, artwork_id)`.
|
||||
- Pagination: ensure cursor/keyset pagination is used, not `OFFSET`.
|
||||
- Cache browse results per category (already partially done in HomepageService).
|
||||
|
||||
---
|
||||
|
||||
### P3-C: artwork_metric_snapshots_hourly backup full scan (Query 15)
|
||||
**Total time:** 646s · **Calls:** 93 · **Max:** 127s · **User:** `backuper`
|
||||
|
||||
`SELECT /*!40001 SQL_NO_CACHE */ * FROM artwork_metric_snapshots_hourly` — mysqldump reading full table.
|
||||
|
||||
**Fix (ops):**
|
||||
- Partition `artwork_metric_snapshots_hourly` by month on `bucket_hour`. Backup only reads the active partition.
|
||||
- Or: exclude this table from hot backup and back it up separately during low-traffic window (02:00–04:00 UTC).
|
||||
- Archive data older than 90 days to a cold table.
|
||||
|
||||
---
|
||||
|
||||
### P3-D: `artworks LEFT JOIN artwork_stats` with OR condition (Query 32)
|
||||
**Total time:** 86s · **Calls:** 110
|
||||
|
||||
```sql
|
||||
WHERE (artworks.created_at >= '...' OR (s.ranking_score IS NOT NULL AND s.ranking_score > 0))
|
||||
```
|
||||
|
||||
**Problem:** OR prevents index usage on `created_at`.
|
||||
|
||||
**Fix:** Rewrite as UNION:
|
||||
```sql
|
||||
SELECT id FROM artworks WHERE created_at >= '...' AND deleted_at IS NULL AND is_approved=1
|
||||
UNION
|
||||
SELECT a.id FROM artworks a JOIN artwork_stats s ON s.artwork_id = a.id
|
||||
WHERE s.ranking_score > 0 AND a.deleted_at IS NULL AND a.is_approved=1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### P3-E: GROUP BY user_id from artworks (Query 31)
|
||||
**Total time:** 92s · **Calls:** 110
|
||||
|
||||
```sql
|
||||
SELECT a.user_id, COALESCE(us.followers_count, 0), COALESCE(us.favorites_received_count, 0)
|
||||
FROM artworks a
|
||||
LEFT JOIN user_statistics us ON us.user_id = a.user_id
|
||||
WHERE a.is_public=1 AND a.is_approved=1 AND a.deleted_at IS NULL
|
||||
GROUP BY a.user_id, us.followers_count, us.favorites_received_count
|
||||
```
|
||||
|
||||
**Problem:** Full artworks scan + GROUP BY without a covering index.
|
||||
|
||||
**Fix:** Add index `(deleted_at, is_public, is_approved, user_id)` on artworks. Or pre-aggregate into `user_statistics` and query that directly without touching `artworks`.
|
||||
|
||||
---
|
||||
|
||||
### P3-F: Forum posts moderation scan (Query 62)
|
||||
**Total time:** 12s · **Calls:** 15
|
||||
|
||||
```sql
|
||||
SELECT * FROM forum_posts
|
||||
WHERE (moderation_checked = 0 OR last_ai_scan_at IS NULL OR updated_at > last_ai_scan_at)
|
||||
AND id IS NOT NULL AND deleted_at IS NULL
|
||||
ORDER BY id ASC LIMIT 50
|
||||
```
|
||||
|
||||
**Fix:** Add a partial index for unmoderated posts:
|
||||
```sql
|
||||
ALTER TABLE forum_posts ADD INDEX idx_needs_moderation (moderation_checked, last_ai_scan_at, id)
|
||||
WHERE moderation_checked = 0;
|
||||
```
|
||||
Or maintain a `moderation_queue` table with only pending IDs.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Index Migrations
|
||||
|
||||
Apply these in order (fastest wins first):
|
||||
|
||||
```bash
|
||||
php artisan make:migration add_performance_indexes_batch1
|
||||
```
|
||||
|
||||
```php
|
||||
public function up(): void
|
||||
{
|
||||
// P1-D: snapshot bucket+artwork lookup
|
||||
Schema::table('artwork_metric_snapshots_hourly', function (Blueprint $table) {
|
||||
$table->index(['bucket_hour', 'artwork_id'], 'idx_bucket_artwork');
|
||||
});
|
||||
|
||||
// P2-A + P2-F: public artwork scans & ORDER BY id
|
||||
Schema::table('artworks', function (Blueprint $table) {
|
||||
$table->index(
|
||||
['deleted_at', 'is_public', 'is_approved', 'published_at', 'id'],
|
||||
'idx_public_approved_published_id'
|
||||
);
|
||||
$table->index(
|
||||
['deleted_at', 'is_public', 'is_approved', 'user_id'],
|
||||
'idx_public_approved_user'
|
||||
);
|
||||
});
|
||||
|
||||
// P2-D: rank_artwork_scores per model_version
|
||||
Schema::table('rank_artwork_scores', function (Blueprint $table) {
|
||||
$table->index(['model_version', 'score_trending'], 'idx_mv_trending');
|
||||
$table->index(['model_version', 'score_new_hot'], 'idx_mv_new_hot');
|
||||
$table->index(['model_version', 'score_best'], 'idx_mv_best');
|
||||
$table->index(['model_version', 'score_new_hot'], 'idx_mv_score');
|
||||
});
|
||||
|
||||
// P3-A: tag daily metrics
|
||||
Schema::table('tag_interaction_daily_metrics', function (Blueprint $table) {
|
||||
$table->index(['tag_id', 'metric_date'], 'idx_tag_date');
|
||||
});
|
||||
|
||||
// P1-A: tag artworks_count cached column
|
||||
Schema::table('tags', function (Blueprint $table) {
|
||||
$table->unsignedInteger('artworks_count')->default(0)->after('slug');
|
||||
$table->index('artworks_count', 'idx_artworks_count');
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Caching Quick Wins (no schema change needed)
|
||||
|
||||
| Surface | Cache Key | TTL | Notes |
|
||||
|---|---|---|---|
|
||||
| Popular tags | `tags.popular.10` | 15 min | Currently: correlated subquery per request |
|
||||
| Ranking top-200 lists | `rank.{model}.{score}.200` | 10 min | Warm at end of ranking job |
|
||||
| Public artwork count | `artworks.public_count` | 5 min | Used in sitemaps, stats |
|
||||
| User artworks_count | `user.{id}.artworks_count` | 5 min | Warm on publish/unpublish |
|
||||
| Group leaderboard | `leaderboard.group.monthly.5` | 30 min | Already in leaderboards service |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Roadmap
|
||||
|
||||
| Phase | Items | Effort | Expected gain |
|
||||
|---|---|---|---|
|
||||
| **Week 1** | P1-A correlated tag query, P1-C jobs LIKE, P1-D snapshot index, all P2-D rank indexes | Low–Med | ~15–20% reduction in slow queries |
|
||||
| **Week 2** | P2-A artworks composite index, P1-B user artworks_count, caching quick wins table | Med | ~40–50% reduction; kills #1 and #2 slow query families |
|
||||
| **Week 3** | P2-B user stats background job, P2-C snapshot pre-aggregation, P1-E fulltext index | High | ~65–75% reduction; kills 28s queries |
|
||||
| **Month 2** | P2-F SELECT * → SELECT id, P3-C partition snapshots table, P3-D OR→UNION rewrite | High | Remaining tail |
|
||||
|
||||
---
|
||||
|
||||
## Monitoring After Changes
|
||||
|
||||
```bash
|
||||
# Reset slow query log on server
|
||||
mysql -e "FLUSH SLOW LOGS;"
|
||||
|
||||
# Re-run pt-query-digest after 1 week:
|
||||
pt-query-digest /var/log/mysql/slow.log > /tmp/slow-report-after.txt
|
||||
|
||||
# Check query plan for top queries:
|
||||
EXPLAIN SELECT ... \G
|
||||
```
|
||||
|
||||
Key metrics to watch:
|
||||
- Total slow query count per day (target: -50% in week 2)
|
||||
- `Rows_examined` for artwork queries (target: <10k instead of 234k)
|
||||
- MySQL CPU usage during ranking job windows
|
||||
Reference in New Issue
Block a user