SkinbaseNova/docs/optimization-m5.1-metric-snapshot-correctness.md

M5.1 — Metric Snapshot Correctness & 30-Day Readiness

STATUS: COMPLETE (local; not deployed)
PRODUCTION WRITES: none
OPTIMIZE TABLE: not run
PARTITIONING: not added
NEW TABLES: none

---

Verdict

Safe to change production retention 7 → 30 after this code ships: YES
Recommended env: ARTWORK_METRIC_HOURLY_RETENTION_DAYS=30
M5 prune batches: verified
Hourly writer vs prune overlap: staggered (prune 04:25, snapshot hourlyAt 2)
Monthly GROUP BY at ~36M rows: acceptable as a scheduled job; not request-path

Production prune is already at 30 days (eligible=0); this KPI/delta code is still local and must ship before treating Studio/monthly numbers as correct 30-day metrics.

---

1. Cumulative-counter bugs found

Location	Bug	Status
StudioMetricsService::getDashboardKpis	SUM(views_count) over cumulative hourly rows	FIXED
Same method	If SUM was 0, fallback to lifetime artwork_stats.views as views_30d	REMOVED
Same method	favourites_30d / shares_30d were lifetime totals	FIXED
CreatorStudioOverviewService (live Studio)	30d KPI keys were lifetime module totals	FIXED (artwork hourly window + coverage)
LeaderboardService artwork daily/weekly/monthly	MAX−MIN inside the window only — undercounts new artworks (first snapshot already has views)	FIXED (latest − pre-window baseline, or 0 if born in period)
Heat RecalculateHeatCommand	Single snapshot without earlier hour → 0 heat	intentional (momentum, not 30d KPI)
Rising homepage/discover/RSS	MAX−MIN over 24h	acceptable for 24h momentum; first hour of a brand-new work is undercounted
Nova cards / collections SUM(views_count)	Other tables	not a snapshot bug

No remaining SUM of artwork_metric_snapshots_hourly cumulative columns.

---

2. Studio views_30d fix

For each artwork:

latest = MAX(counter) in [start, now]
baseline =
  0  if published_at/created_at >= start
  else snapshot at/immediately before start (last 36h before start)
  else MIN(counter) in window   -- warm-up / missing hours; NOT lifetime
delta = GREATEST(latest - baseline, 0)

Then SUM those deltas across the creator’s artworks.

• Created during the period: first snapshot of 40 + later 90 → 90
• Older work with pre-window 200 and latest 250 → 50
• Older work, one in-window snapshot of 400 (no baseline) → 0 (do not report lifetime)
• Zero growth stays 0 (no lifetime fallback)

Coverage is exposed so a 7.5-day table is not labeled as a full 30 days.

---

3. Warm-up behavior

Production history today starts 2026-08-16 (~7.5 days). After env=30, history stays incomplete until ~2026-09-15.

UI/API now:

• Still compute the delta from available hours (do not invent 30d).
• Expose kpis.snapshot_window:
  • requested_days (30)
  • table_coverage_days
  • user_coverage_days
  • window_complete (true only when table span ≥ 30 days minus one hour)
  • oldest/newest buckets
• Studio dashboard hint: “Based on 7.5 of 30 days” until complete; then “Last 30 days”.

Monthly leaderboards already score from bucket_hour >= now-1 month (whatever rows exist). They are internally consistent but short until warm-up ends. No separate leaderboard badge in this milestone (not a Studio KPI). Do not treat those boards as a full calendar month until coverage is complete.

---

4. Creator journey

biggestDownloadSpike loaded all retained hours for every public artwork (unbounded vs retention). After 30d that is ~720 hours × N artworks.

Decision: explicit lookback = metrics.hourly_snapshot_retention_days (default 30).

• Career “all-time spike” cannot exist beyond retention anyway.
• Query size stays bounded when retention is raised later.
• Milestone copy now says the retained hourly window.
• Fixture snapshots moved to recent hours so the test still sees a spike.

---

5. Monthly leaderboard at 30-day scale

M5 EXPLAIN (production, 8.9M rows, 30d predicate covering the whole table):

type=index  key=uq_artwork_bucket  rows~8.87M  Using where
GROUP BY artwork_id  MAX-MIN

At 36M rows the plan stays a covering unique-index scan, ~4× current row volume. leaderboards:refresh is hourly, withoutOverlapping, not on the HTTP path.

Acceptable for a scheduled job. If wall time exceeds a few minutes after warm-up, rewrite to range on idx_bucket_artwork or pre-aggregate — not required now. No load test was run on production.

24h rising/heat already use idx_bucket_artwork range (~1.2M rows) and stay fine.

---

6. Pruning (verified in code)

Requirement	Implementation
Default 30 days	config/metrics.php / ARTWORK_METRIC_HOURLY_RETENTION_DAYS
Bounded deletes	--chunk default 5000, PK id IN (...)
Sleep	--sleep-ms default 50
No giant transaction	one DELETE per batch
No self-overlap	withoutOverlapping(120)
Logs	start, each batch, completed
Resume next day	cutoff is now - keep-days; leftover rows delete on the next run
Scheduler time	04:25 (was 04:00)

---

7. Snapshot writer vs prune

Job	Time
nova:metrics-snapshot-hourly	hourlyAt(2) → 04:02 upserts current hour (~50k rows)
nova:prune-metric-snapshots	04:25 deletes bucket_hour < now-30d

They no longer share 04:00–04:02. Locks: prune deletes old PK/bucket_hour range; writer upserts (artwork_id, current hour). Different unique-key tuples. Residual risk is IO only, not a correctness lock.

withoutOverlapping is per schedule name, so the two jobs can still run together if prune overruns into 05:02. Stagger + 50ms sleeps keep that unlikely.

---

8. Deployment sequence (later; not M5.1)

1. Deploy this code (window helper, Studio KPIs, journey lookback, prune 04:25).
2. Set ARTWORK_METRIC_HOURLY_RETENTION_DAYS=30.
3. config:cache.
4. Optional: php artisan nova:prune-metric-snapshots --dry-run (read-only).
5. Do not OPTIMIZE TABLE.
6. Expect table growth toward ~7.5 GB over ~3 weeks; Studio shows “N of 30 days” until then.

Rollback: env=7; Studio KPIs still MAX−MIN (correct on whatever history exists).

---

9. Tests

• tests/Feature/Metrics/ArtworkHourlySnapshotWindowTest.php (SUM vs delta, new artwork, missing baseline, partial history, zero growth)
• tests/Feature/Metrics/LeaderboardSnapshotDeltaTest.php (daily/weekly/monthly)
• tests/Feature/StudioTest.php (snapshot_window on dashboard)
• creator journey retention window
• existing prune tests

---

10. Files changed

app/Services/Metrics/ArtworkHourlySnapshotWindow.php
app/Services/LeaderboardService.php
app/Services/Studio/StudioMetricsService.php
app/Services/Studio/CreatorStudioOverviewService.php
app/Services/Profile/CreatorJourneyService.php
resources/js/Pages/Studio/StudioDashboard.jsx
routes/console.php
config/metrics.php
tests/Feature/Metrics/ArtworkHourlySnapshotWindowTest.php
tests/Feature/Metrics/LeaderboardSnapshotDeltaTest.php
tests/Feature/StudioTest.php
tests/Feature/Profile/CreatorJourneyTest.php
docs/optimization-m5.1-metric-snapshot-correctness.md

---

11. Risks

• Studio 30d KPIs now artwork-only (hourly snapshots). Cards/collections/stories lifetime totals no longer inflate those tiles.
• Cached studio.kpi.{id} 5 minutes.
• Table-level window_complete is global, not per-user (correct for production warm-up).
• Monthly boards remain unlabeled for incomplete months.

M5+M5.1 together are safe to deploy with env=30 after this code is on the server. Do not deploy env=30 onto the old SUM path.