# Deployment

This repository uses a Bash-based production deploy flow with a Windows Command Prompt wrapper.

## Normal deploy

Preferred entrypoints:

```bat
deploy.cmd
```

```bash
bash deploy.sh
```

`deploy.cmd` is the Windows Command Prompt entrypoint. It switches into WSL and runs `deploy.sh`.
`deploy.sh` is the canonical bash entrypoint and delegates to the safe production deploy wrapper, which stages each deploy into a versioned release directory and switches traffic by updating the server-side `current` symlink.

`bash sync.sh` remains as a legacy alias for the same flow.

Production deploys require a reproducible Git source tree by default
(`REQUIRE_CLEAN_GIT=1`). The preflight inspects staged, tracked, and
deployable untracked files; untracked paths excluded by rsync are ignored.
The local Vite build may refresh the tracked generated SSR bundle under
`bootstrap/ssr/`, but source/config/deploy changes made during preparation
abort before the release is switched. The local Git `HEAD` is captured before
build and must remain unchanged through rsync.

Run the local-only guard when validating a release without creating a remote
release or running rsync:

```bash
bash sync.sh --preflight-only
```

`--skip-build` is rejected for a clean production deploy unless
`deploy.cmd` has just completed the Windows build and exported
`WINDOWS_FRONTEND_BUILT=1`. An explicit `REQUIRE_CLEAN_GIT=0` is reserved for
non-production/custom workflows and is not the production default.

`deploy.cmd` runs `npm.cmd run build` on Windows first, then enters WSL for rsync/ssh. That is required when the Ubuntu distro cannot execute Windows `.exe` files (`Exec format error` on `powershell.exe`). If WSL interop does work, `bash deploy.sh` can still launch `npm.cmd` through PowerShell. Local Linux `php`/`composer` are not required for a normal deploy; Artisan and Composer run on the production server. `--with-tests` uses WSL `php` when present, otherwise Windows `php.exe`.

This will:

- build frontend assets locally with `npm run build`
- stage the new code into a versioned release directory on the production server
- run `composer install --no-dev` inside that staged release before switching traffic
- keep the fixed production app path pointed at the active release through a server-side `current` symlink
- bring the app down only for the short critical section that switches the active release and runs `php artisan migrate --force`, `php artisan optimize:clear`, and `php artisan optimize`
- bring the app back up immediately after that critical section finishes
- warm the guest homepage cache with `php artisan homepage:warm-guest-cache`
- warm the post trending cache with `php artisan posts:warm-trending`
- restart queue workers with `php artisan queue:restart`

This is now the low-downtime default path for normal code and feature deploys.

Each deploy generates:

- a monotonic local **build number** (stored in `var/deploy/build-number`)
- a **release ID** from UTC time + build number + Git revision, for example `20260829-141522-b42-a1b2c3d`
- local deploy history under `var/deploy/` (`latest.json`, `history.jsonl`, and per-run logs in `var/deploy/logs/`)
- remote metadata under `REMOTE_RELEASE_ROOT/deployments/<release-id>.json` and `current-release.json`

Console output includes step progress with elapsed time, rsync transfer progress, and a final duration summary. Override the build number with `--build-number N` or `BUILD_NUMBER=N` when needed. Dry-runs preview the next build number without consuming it.

Releases are retained under `REMOTE_RELEASE_ROOT/releases/<release-id>`, and production switches between them on the server by updating `REMOTE_RELEASE_ROOT/current`. The public/runtime path stays fixed at `REMOTE_FOLDER`, which is now treated as a stable symlink to the active release.

On the first deploy with this layout, the existing live folder is adopted into the release archive automatically and `REMOTE_FOLDER` is converted into that stable symlink path. After that, switching back to an older release does not require any local re-upload.

## Full upgrade

Use a full upgrade when the release also needs broad Meilisearch work or non-code service operations.

```bat
deploy.cmd --full-upgrade
```

```bash
bash deploy.sh --full-upgrade
```

Full-upgrade mode:

- keeps the normal deploy steps
- forces a full Meilisearch import for all searchable models unless you explicitly pass `--skip-meilisearch`
- allows optional remote upgrade hooks for service-level work

Example with service hooks:

```bash
bash deploy.sh --full-upgrade \
	--upgrade-pre-hook='sudo systemctl stop reverb' \
	--upgrade-post-hook='sudo systemctl restart reverb meilisearch'
```

You can also provide those hooks through environment variables instead of CLI flags:

```bash
FULL_UPGRADE_PRE_HOOK='sudo systemctl stop reverb' \
FULL_UPGRADE_POST_HOOK='sudo systemctl restart reverb meilisearch' \
bash deploy.sh --full-upgrade
```

## Deploy options

```bat
deploy.cmd --skip-build
deploy.cmd --skip-migrate
deploy.cmd --no-maintenance
deploy.cmd --mode=full-upgrade
deploy.cmd --keep-releases=8
deploy.cmd --release-id=release-2026-04-25
deploy.cmd --build-number=100
deploy.cmd --no-rsync-progress
```

```bash
bash deploy.sh --skip-build
bash deploy.sh --skip-migrate
bash deploy.sh --no-maintenance
bash deploy.sh --mode=full-upgrade
bash deploy.sh --keep-releases=8
bash deploy.sh --release-id=release-2026-04-25
bash deploy.sh --build-number=100
bash deploy.sh --no-rsync-progress
```

Environment overrides:

```bash
REMOTE_SERVER=user@example.com REMOTE_FOLDER=/var/www/app bash deploy.sh
REMOTE_RELEASE_ROOT=/var/www/app.releases RELEASE_RETENTION=8 bash deploy.sh
```

You can also override the local build command explicitly:

```bash
LOCAL_BUILD_COMMAND='npm run build' bash deploy.sh
LOCAL_BUILD_COMMAND='pnpm build' bash deploy.sh
```

Upgrade hooks can also be supplied via environment variables:

```bash
FULL_UPGRADE_PRE_HOOK='sudo systemctl stop reverb' bash deploy.sh --full-upgrade
FULL_UPGRADE_POST_HOOK='sudo systemctl restart reverb meilisearch' bash deploy.sh --full-upgrade
```

## Rollback and release history

List retained releases on production:

```bash
bash scripts/rollback-production.sh --list
```

Switch production to the previous retained release:

```bash
bash scripts/rollback-production.sh --previous
```

Switch production to a specific retained release:

```bash
bash scripts/rollback-production.sh --release-id=20260425-132455-a1b2c3d
```

Preview a release switch without changing the server:

```bash
bash scripts/rollback-production.sh --previous --dry-run
```

Operational notes:

- Rollback now means switching the active server release, not re-syncing files from local.
- Each retained release already contains its own code and vendor tree, so rollback is primarily a symlink switch plus cache refresh and `queue:restart`.
- Rollback does not reverse database migrations. If a release includes incompatible schema changes, handle the database separately.
- Release retention defaults to 5 releases and can be changed with `--keep-releases` or `RELEASE_RETENTION`.
- Reldeploy.sh --with-db-from=local
```

When run interactively, the deploy script will ask you to confirm the exact remote server and type a confirmation phrase before replacing production data.

For non-interactive use, pass both confirmations explicitly:

```bash
bash deploy.sh --with-db-from=local \
	--confirm-db-sync-target=klevze@server3.klevze.si \
	--confirm-db-sync-phrase='replace production db from local'
```

Legacy compatibility still exists for:

```bash
bash deployinteractively, the deploy script will ask you to confirm the exact remote server and type a confirmation phrase before replacing production data.

For non-interactive use, pass both confirmations explicitly:

```bash
bash sync.sh --with-db-from=local \
	--confirm-db-sync-target=klevze@server3.klevze.si \
	--confirm-db-sync-phrase='replace production db from local'
```

Legacy compatibility still exists for:

```bash
bash sync.sh --with-db --force-db-sync
```

But the safer `--with-db-from=local` flow should be preferred.

The database sync script will:

- read local DB credentials from the local `.env`
- create a local `mysqldump` export
- upload the dump to the production server
- create a backup of the current production database under `storage/app/deploy-backups`
- import the local dump into the production database
- run `php artisan migrate --force` unless `--skip-migrate` is passed

If you run the deploy from WSL while your local MySQL server is running on Windows with `DB_HOST=127.0.0.1` or `localhost`, the DB sync script will automatically use Windows `mysqldump.exe` so it can still reach the local database.

You can override the dump command explicitly if needed:

```bash
LOCAL_MYSQLDUMP_COMMAND='mysqldump --host=10.0.0.5 --port=3306 --user=app dbname' bash scripts/push-db-to-prod.sh --force
```

## Safety notes

- Normal deployments should use `deploy.cmd` or `bash deploy.sh` without `--with-db`.
- Use `--full-upgrade` only when the release also includes Meilisearch-wide refreshes or remote service changes.
- Use database replacement only for first-time bootstrap, staging, or an intentional full production reset.
- Use `bash scripts/rollback-production.sh --previous` for a fast server-side release switch when the last deploy needs to be reverted.
- Route caching now runs through `php artisan optimize` in deploy automation; if that starts failing again, fix the route definitions instead of dropping route caching from deploy.
- On Windows, prefer `deploy.cmd` so the process always enters WSL before rsync/ssh/php. It defaults to the local `Ubuntu` WSL distro. The production server itself runs Debian 13; that is separate from the local WSL choice. Override with `DEPLOY_WSL_DISTRO` only if needed. If WSL has no usable `~/.ssh` keys (common when the distro runs as root and keys live in `C:\Users\<you>\.ssh`), the script copies those Windows identities into a 0600 temp dir and uses them for BatchMode SSH. Override the source with `WINDOWS_SSH_DIR`.
- SSH still authenticates as `REMOTE_SERVER` (for example `klevze@host`), but remote rsync/composer/artisan/release work runs as `REMOTE_APP_USER` (default `skinbase`) via `sudo -n -u skinbase`. This keeps release files and runtime dirs owned by the app user and avoids prune blockers like `owner=skinbase mode=2700 .config`. Set `REMOTE_APP_USER=-` only to disable that and run as the SSH login user.
- Remote app-user shells start in `/tmp` (not the SSH user's home). The stable `REMOTE_FOLDER` symlink is left alone once it already points at `.../current`; only `current` is rewritten each deploy. That avoids `Permission denied` on root-owned parents like `/opt/www/virtual`.
- The deploy script refuses missing Vite/SSR build artifacts by default (`REQUIRE_BUILD_MANIFEST=1`), verifies SSH non-interactively first, and takes a local deploy lock so two overlapping deploys do not race.
- Inertia SSR restart is owned exclusively by Supervisor program `skinbase-ssr` (`deploy/supervisor/skinbase-ssr.conf`). The remote application phase runs as `skinbase`, while the restart is issued separately over the SSH deployment session as the privileged login account with `sudo -n /usr/bin/supervisorctl`. The deploy fails if privileged Supervisor access or the configured program is unavailable, and verifies that the program reaches `RUNNING`; it never starts or stops SSR through Artisan.

## Nginx upstream error pages

Laravel now owns the Nova-style HTML error pages for normal application responses, but true upstream failures such as `502 Bad Gateway` and `504 Gateway Timeout` still have to be handled by nginx because PHP/FPM is unavailable in that state.

The repo includes a ready-to-include snippet at `deploy/nginx/upstream-error-pages.conf` and the matching static page at `public/errors/upstream-gateway.html`.

To enable it on production:

1. Include the snippet inside the Skinbase `server {}` block.
2. Add `fastcgi_intercept_errors on;` to every FastCGI location that should use the static fallback.
3. Keep the static file available in the live public path so nginx can serve it without Laravel.

On the current `skinbase.org` vhost, the required FastCGI locations are:

- `location ^~ /api/uploads/`
- `location = /index.php`

This intentionally intercepts only `502` and `504`, so Laravel remains responsible for normal `404`, `419`, `429`, `500`, and `503` rendering when the application is actually running.