SkinbaseNova/docs/optimization-m5.1-metric-snapshot-correctness.md

# M5.1 — Metric Snapshot Correctness & 30-Day Readiness

```text
STATUS: COMPLETE (local; not deployed)
PRODUCTION WRITES: none
OPTIMIZE TABLE: not run
PARTITIONING: not added
NEW TABLES: none
```

---

## Verdict

```text
Safe to change production retention 7 → 30 after this code ships: YES
Recommended env: ARTWORK_METRIC_HOURLY_RETENTION_DAYS=30
M5 prune batches: verified
Hourly writer vs prune overlap: staggered (prune 04:25, snapshot hourlyAt 2)
Monthly GROUP BY at ~36M rows: acceptable as a scheduled job; not request-path
```

Production prune is already at 30 days (`eligible=0`); **this KPI/delta code is still local and must ship before treating Studio/monthly numbers as correct 30-day metrics.**

---

## 1. Cumulative-counter bugs found

| Location | Bug | Status |
| -------- | --- | ------ |
| `StudioMetricsService::getDashboardKpis` | `SUM(views_count)` over cumulative hourly rows | **FIXED** |
| Same method | If SUM was 0, **fallback to lifetime** `artwork_stats.views` as `views_30d` | **REMOVED** |
| Same method | `favourites_30d` / `shares_30d` were **lifetime** totals | **FIXED** |
| `CreatorStudioOverviewService` (live Studio) | 30d KPI keys were **lifetime** module totals | **FIXED** (artwork hourly window + coverage) |
| `LeaderboardService` artwork daily/weekly/monthly | `MAX−MIN` **inside** the window only — undercounts **new** artworks (first snapshot already has views) | **FIXED** (latest − pre-window baseline, or 0 if born in period) |
| Heat `RecalculateHeatCommand` | Single snapshot without earlier hour → 0 heat | **intentional** (momentum, not 30d KPI) |
| Rising homepage/discover/RSS | `MAX−MIN` over 24h | **acceptable** for 24h momentum; first hour of a brand-new work is undercounted |
| Nova cards / collections `SUM(views_count)` | Other tables | **not a snapshot bug** |

No remaining `SUM` of `artwork_metric_snapshots_hourly` cumulative columns.

---

## 2. Studio `views_30d` fix

For each artwork:

```text
latest = MAX(counter) in [start, now]
baseline =
  0  if published_at/created_at >= start
  else snapshot at/immediately before start (last 36h before start)
  else MIN(counter) in window   -- warm-up / missing hours; NOT lifetime
delta = GREATEST(latest - baseline, 0)
```

Then SUM those **deltas** across the creator’s artworks.

- Created during the period: first snapshot of 40 + later 90 → **90**
- Older work with pre-window 200 and latest 250 → **50**
- Older work, one in-window snapshot of 400 (no baseline) → **0** (do not report lifetime)
- Zero growth stays **0** (no lifetime fallback)

Coverage is exposed so a 7.5-day table is not labeled as a full 30 days.

---

## 3. Warm-up behavior

Production history today starts **2026-08-16** (~7.5 days). After env=30, history stays incomplete until ~2026-09-15.

UI/API now:

- Still compute the delta from **available** hours (do not invent 30d).
- Expose `kpis.snapshot_window`:
  - `requested_days` (30)
  - `table_coverage_days`
  - `user_coverage_days`
  - `window_complete` (true only when table span ≥ 30 days minus one hour)
  - oldest/newest buckets
- Studio dashboard hint: **“Based on 7.5 of 30 days”** until complete; then **“Last 30 days”**.

Monthly leaderboards already score from `bucket_hour >= now-1 month` (whatever rows exist). They are internally consistent but **short** until warm-up ends. No separate leaderboard badge in this milestone (not a Studio KPI). Do not treat those boards as a full calendar month until coverage is complete.

---

## 4. Creator journey

`biggestDownloadSpike` loaded **all** retained hours for every public artwork (unbounded vs retention). After 30d that is ~720 hours × N artworks.

**Decision:** explicit lookback = `metrics.hourly_snapshot_retention_days` (default 30).

- Career “all-time spike” cannot exist beyond retention anyway.
- Query size stays bounded when retention is raised later.
- Milestone copy now says the retained hourly window.
- Fixture snapshots moved to recent hours so the test still sees a spike.

---

## 5. Monthly leaderboard at 30-day scale

M5 EXPLAIN (production, 8.9M rows, 30d predicate covering the whole table):

```text
type=index  key=uq_artwork_bucket  rows~8.87M  Using where
GROUP BY artwork_id  MAX-MIN
```

At 36M rows the plan stays a covering unique-index scan, ~4× current row volume. `leaderboards:refresh` is hourly, `withoutOverlapping`, not on the HTTP path.

**Acceptable** for a scheduled job. If wall time exceeds a few minutes after warm-up, rewrite to range on `idx_bucket_artwork` or pre-aggregate — not required now. No load test was run on production.

24h rising/heat already use `idx_bucket_artwork` range (~1.2M rows) and stay fine.

---

## 6. Pruning (verified in code)

| Requirement | Implementation |
| ----------- | ---------------- |
| Default 30 days | `config/metrics.php` / `ARTWORK_METRIC_HOURLY_RETENTION_DAYS` |
| Bounded deletes | `--chunk` default **5000**, PK `id IN (...)` |
| Sleep | `--sleep-ms` default **50** |
| No giant transaction | one DELETE per batch |
| No self-overlap | `withoutOverlapping(120)` |
| Logs | start, each batch, completed |
| Resume next day | cutoff is `now - keep-days`; leftover rows delete on the next run |
| Scheduler time | **04:25** (was 04:00) |

---

## 7. Snapshot writer vs prune

| Job | Time |
| --- | ---- |
| `nova:metrics-snapshot-hourly` | `hourlyAt(2)` → **04:02** upserts **current** hour (~50k rows) |
| `nova:prune-metric-snapshots` | **04:25** deletes `bucket_hour < now-30d` |

They no longer share 04:00–04:02. Locks: prune deletes old PK/`bucket_hour` range; writer upserts `(artwork_id, current hour)`. Different unique-key tuples. Residual risk is IO only, not a correctness lock.

`withoutOverlapping` is per schedule **name**, so the two jobs can still run together if prune overruns into 05:02. Stagger + 50ms sleeps keep that unlikely.

---

## 8. Deployment sequence (later; not M5.1)

1. Deploy this code (window helper, Studio KPIs, journey lookback, prune 04:25).
2. Set `ARTWORK_METRIC_HOURLY_RETENTION_DAYS=30`.
3. `config:cache`.
4. Optional: `php artisan nova:prune-metric-snapshots --dry-run` (read-only).
5. Do **not** OPTIMIZE TABLE.
6. Expect table growth toward ~7.5 GB over ~3 weeks; Studio shows “N of 30 days” until then.

**Rollback:** env=7; Studio KPIs still MAX−MIN (correct on whatever history exists).

---

## 9. Tests

- `tests/Feature/Metrics/ArtworkHourlySnapshotWindowTest.php` (SUM vs delta, new artwork, missing baseline, partial history, zero growth)
- `tests/Feature/Metrics/LeaderboardSnapshotDeltaTest.php` (daily/weekly/monthly)
- `tests/Feature/StudioTest.php` (snapshot_window on dashboard)
- creator journey retention window
- existing prune tests

---

## 10. Files changed

```text
app/Services/Metrics/ArtworkHourlySnapshotWindow.php
app/Services/LeaderboardService.php
app/Services/Studio/StudioMetricsService.php
app/Services/Studio/CreatorStudioOverviewService.php
app/Services/Profile/CreatorJourneyService.php
resources/js/Pages/Studio/StudioDashboard.jsx
routes/console.php
config/metrics.php
tests/Feature/Metrics/ArtworkHourlySnapshotWindowTest.php
tests/Feature/Metrics/LeaderboardSnapshotDeltaTest.php
tests/Feature/StudioTest.php
tests/Feature/Profile/CreatorJourneyTest.php
docs/optimization-m5.1-metric-snapshot-correctness.md
```

---

## 11. Risks

- Studio 30d KPIs now **artwork-only** (hourly snapshots). Cards/collections/stories lifetime totals no longer inflate those tiles.
- Cached `studio.kpi.{id}` 5 minutes.
- Table-level `window_complete` is global, not per-user (correct for production warm-up).
- Monthly boards remain unlabeled for incomplete months.

M5+M5.1 together are **safe to deploy** with env=30 after this code is on the server. Do not deploy env=30 onto the old SUM path.