580 lines
14 KiB
Markdown
580 lines
14 KiB
Markdown
Skinbase Enhance Setup Guide
|
|
===========================
|
|
|
|
This guide explains how to set up, enable, verify, and operate the Skinbase Enhance module end to end.
|
|
|
|
Use this document when you need to:
|
|
|
|
- enable Enhance in a local environment
|
|
- switch from stub mode to the external worker
|
|
- run the worker in Pillow mode or Real-ESRGAN mode
|
|
- configure queues, cleanup, and health checks
|
|
- understand what the module stores and how it behaves in production
|
|
|
|
What the module does
|
|
--------------------
|
|
|
|
Enhance accepts an uploaded or selected source image, creates an Enhance job, processes that job through the configured engine, stores the generated output on the Enhance storage disk, and keeps the original source file untouched.
|
|
|
|
Current supported engines on the Laravel side:
|
|
|
|
- `ENHANCE_ENGINE=stub`
|
|
- `ENHANCE_ENGINE=external_worker`
|
|
|
|
Current supported worker engines:
|
|
|
|
- `WORKER_ENGINE=pillow`
|
|
- `WORKER_ENGINE=realesrgan-ncnn`
|
|
- `WORKER_ENGINE=realesrgan`
|
|
|
|
`WORKER_ENGINE=realesrgan` currently aliases to `realesrgan-ncnn`.
|
|
|
|
Architecture
|
|
------------
|
|
|
|
The Enhance module is split into two layers.
|
|
|
|
Laravel application:
|
|
|
|
- accepts the Enhance request
|
|
- validates allowed file types, dimensions, scales, and modes
|
|
- stores job records
|
|
- dispatches the job to the queue
|
|
- owns permanent storage for Enhance sources and outputs
|
|
- exposes moderation, cleanup, and health commands
|
|
|
|
Optional external worker:
|
|
|
|
- downloads the copied Enhance source image
|
|
- upscales it with Pillow or Real-ESRGAN
|
|
- exposes a temporary internal result URL
|
|
- deletes its temporary result after Laravel confirms download
|
|
|
|
Laravel remains the source of truth. The worker is temporary processing only.
|
|
|
|
Default behavior and limits
|
|
---------------------------
|
|
|
|
Default config values from [config/enhance.php](config/enhance.php):
|
|
|
|
- disk: `ENHANCE_DISK` or the app filesystem default
|
|
- source prefix: `enhance/sources`
|
|
- output prefix: `enhance/outputs`
|
|
- preview prefix: `enhance/previews`
|
|
- default engine: `stub`
|
|
- allowed MIME types: `image/jpeg`, `image/png`, `image/webp`
|
|
- allowed modes: `standard`, `artwork`, `photo`, `illustration`
|
|
- allowed scales: `2`, `4`
|
|
- daily limit: `10`
|
|
- default queue: `default`
|
|
|
|
Lifecycle defaults:
|
|
|
|
- completed jobs expire after `30` days
|
|
- failed jobs expire after `7` days
|
|
- deleted job files get a `1` day grace period
|
|
|
|
Health defaults:
|
|
|
|
- processing jobs are considered stuck after `30` minutes
|
|
- queued jobs are considered stale after `60` minutes
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
Laravel requirements:
|
|
|
|
- the application boots normally
|
|
- database migrations are current
|
|
- the configured filesystem disk is writable
|
|
- a queue worker or Horizon is running for the queue you assign to Enhance
|
|
- Laravel scheduler is enabled if you want automatic cleanup
|
|
|
|
Worker requirements for `external_worker` mode:
|
|
|
|
- Docker or a Python runtime for `services/enhance-worker`
|
|
- network access from Laravel to the worker URL
|
|
- a shared bearer token between Laravel and the worker
|
|
- for Real-ESRGAN mode, the `realesrgan-ncnn-vulkan` binary and model files
|
|
|
|
Setup paths
|
|
-----------
|
|
|
|
There are three practical ways to run Enhance.
|
|
|
|
1. Stub mode
|
|
|
|
- safest local starting point
|
|
- exercises the Laravel flow without a real AI runtime
|
|
- no worker needed
|
|
|
|
2. External worker with Pillow
|
|
|
|
- good for local integration testing and CI-like validation
|
|
- real HTTP worker contract
|
|
- deterministic fallback upscale path
|
|
- no Real-ESRGAN runtime files required
|
|
|
|
3. External worker with Real-ESRGAN
|
|
|
|
- production-oriented path
|
|
- uses the `realesrgan-ncnn-vulkan` CLI runtime
|
|
- requires runtime files and a verified worker host
|
|
|
|
Laravel setup
|
|
-------------
|
|
|
|
Minimum env for stub mode:
|
|
|
|
```env
|
|
ENHANCE_ENGINE=stub
|
|
ENHANCE_QUEUE=default
|
|
```
|
|
|
|
Recommended env for external worker mode:
|
|
|
|
```env
|
|
ENHANCE_ENGINE=external_worker
|
|
ENHANCE_QUEUE=enhance
|
|
ENHANCE_WORKER_URL=http://127.0.0.1:8095
|
|
ENHANCE_WORKER_TIMEOUT=900
|
|
ENHANCE_WORKER_TOKEN=change-this-token
|
|
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
|
|
```
|
|
|
|
Optional Laravel env keys you may tune:
|
|
|
|
```env
|
|
ENHANCE_DISK=public
|
|
ENHANCE_SOURCE_PREFIX=enhance/sources
|
|
ENHANCE_OUTPUT_PREFIX=enhance/outputs
|
|
ENHANCE_PREVIEW_PREFIX=enhance/previews
|
|
|
|
ENHANCE_MAX_UPLOAD_MB=20
|
|
ENHANCE_MAX_INPUT_WIDTH=4096
|
|
ENHANCE_MAX_INPUT_HEIGHT=4096
|
|
ENHANCE_MAX_OUTPUT_WIDTH=8192
|
|
ENHANCE_MAX_OUTPUT_HEIGHT=8192
|
|
|
|
ENHANCE_DAILY_LIMIT=10
|
|
|
|
ENHANCE_COMPLETED_EXPIRES_AFTER_DAYS=30
|
|
ENHANCE_FAILED_EXPIRES_AFTER_DAYS=7
|
|
ENHANCE_DELETED_FILE_GRACE_DAYS=1
|
|
ENHANCE_CLEANUP_CHUNK_SIZE=100
|
|
|
|
ENHANCE_STUCK_PROCESSING_AFTER_MINUTES=30
|
|
ENHANCE_STUCK_QUEUED_AFTER_MINUTES=60
|
|
|
|
ENHANCE_STUB_SHOW_WARNING=true
|
|
```
|
|
|
|
After changing env:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
```
|
|
|
|
Queue setup
|
|
-----------
|
|
|
|
Enhance jobs run on `ENHANCE_QUEUE`.
|
|
|
|
If you keep the default queue:
|
|
|
|
```bash
|
|
php artisan queue:work --queue=default
|
|
```
|
|
|
|
If you use a dedicated Enhance queue:
|
|
|
|
```bash
|
|
php artisan queue:work --queue=enhance,default
|
|
```
|
|
|
|
If Horizon or workers do not consume the configured queue, Enhance jobs will stay queued.
|
|
|
|
Scheduler and cleanup
|
|
---------------------
|
|
|
|
Enhance cleanup is scheduled from [routes/console.php](routes/console.php) and runs:
|
|
|
|
```bash
|
|
php artisan enhance:cleanup --force
|
|
```
|
|
|
|
Useful cleanup and health commands:
|
|
|
|
```bash
|
|
php artisan enhance:health
|
|
php artisan enhance:health --json
|
|
php artisan enhance:cleanup --dry-run
|
|
php artisan enhance:cleanup --force
|
|
```
|
|
|
|
Cleanup only removes files under the configured Enhance prefixes. It does not delete artwork originals or unrelated storage paths.
|
|
|
|
Enable stub mode
|
|
----------------
|
|
|
|
Use stub mode first if you want to validate the Laravel module without introducing worker runtime variables.
|
|
|
|
1. Set:
|
|
|
|
```env
|
|
ENHANCE_ENGINE=stub
|
|
ENHANCE_QUEUE=default
|
|
```
|
|
|
|
2. Clear config:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
```
|
|
|
|
3. Start a queue worker:
|
|
|
|
```bash
|
|
php artisan queue:work --queue=default
|
|
```
|
|
|
|
4. Check health:
|
|
|
|
```bash
|
|
php artisan enhance:health
|
|
```
|
|
|
|
5. Open `/enhance/create`, submit a small image, and verify the job completes.
|
|
|
|
Enable external worker mode with Pillow
|
|
---------------------------------------
|
|
|
|
This is the safest real integration path because it exercises the HTTP worker contract without requiring Real-ESRGAN files.
|
|
|
|
1. Start the worker:
|
|
|
|
```bash
|
|
cd services/enhance-worker
|
|
docker compose -f docker-compose.example.yml up --build
|
|
```
|
|
|
|
2. Confirm worker health:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8095/health
|
|
```
|
|
|
|
Expected result:
|
|
|
|
- `status: ok`
|
|
- `engine: pillow`
|
|
|
|
3. Set Laravel env:
|
|
|
|
```env
|
|
ENHANCE_ENGINE=external_worker
|
|
ENHANCE_QUEUE=enhance
|
|
ENHANCE_WORKER_URL=http://127.0.0.1:8095
|
|
ENHANCE_WORKER_TIMEOUT=600
|
|
ENHANCE_WORKER_TOKEN=change-this-token
|
|
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
|
|
```
|
|
|
|
4. Clear config and start queue workers:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
php artisan queue:work --queue=enhance,default
|
|
```
|
|
|
|
5. Verify the Laravel side:
|
|
|
|
```bash
|
|
php artisan enhance:health
|
|
php artisan test --filter=EnhanceExternalWorker
|
|
```
|
|
|
|
Enable external worker mode with Real-ESRGAN
|
|
--------------------------------------------
|
|
|
|
This is the production-oriented setup.
|
|
|
|
1. Install or mount the runtime files inside [services/enhance-worker](services/enhance-worker):
|
|
|
|
```bash
|
|
cd services/enhance-worker
|
|
bash scripts/download-realesrgan-ncnn.sh
|
|
bash scripts/verify-realesrgan.sh
|
|
```
|
|
|
|
Required file locations:
|
|
|
|
- binary: `bin/realesrgan-ncnn-vulkan`
|
|
- models: `models/*.param` and `models/*.bin`
|
|
|
|
2. Start the Real-ESRGAN worker:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.realesrgan.example.yml up --build
|
|
```
|
|
|
|
3. Confirm worker health:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8095/health
|
|
```
|
|
|
|
Expected result when ready:
|
|
|
|
- `status: ok`
|
|
- `engine: realesrgan-ncnn`
|
|
|
|
Expected result when runtime files are missing or invalid:
|
|
|
|
- `status: degraded`
|
|
|
|
4. Set Laravel env:
|
|
|
|
```env
|
|
ENHANCE_ENGINE=external_worker
|
|
ENHANCE_QUEUE=enhance
|
|
ENHANCE_WORKER_URL=http://127.0.0.1:8095
|
|
ENHANCE_WORKER_TIMEOUT=900
|
|
ENHANCE_WORKER_TOKEN=change-this-token
|
|
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
|
|
```
|
|
|
|
5. Clear config and start the queue worker:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
php artisan queue:work --queue=enhance,default
|
|
```
|
|
|
|
6. Verify through the application by submitting a small image from `/enhance/create`.
|
|
|
|
Worker configuration
|
|
--------------------
|
|
|
|
Main worker env values:
|
|
|
|
```env
|
|
WORKER_HOST=0.0.0.0
|
|
WORKER_PORT=8095
|
|
WORKER_TOKEN=change-this-token
|
|
|
|
WORKER_ENGINE=pillow
|
|
WORKER_DEVICE=cpu
|
|
|
|
WORKER_MAX_UPLOAD_MB=20
|
|
WORKER_MAX_INPUT_WIDTH=4096
|
|
WORKER_MAX_INPUT_HEIGHT=4096
|
|
WORKER_MAX_OUTPUT_WIDTH=8192
|
|
WORKER_MAX_OUTPUT_HEIGHT=8192
|
|
|
|
WORKER_TMP_DIR=/app/storage/tmp
|
|
WORKER_OUTPUT_DIR=/app/storage/output
|
|
WORKER_RESULT_TTL_MINUTES=60
|
|
```
|
|
|
|
Real-ESRGAN-specific worker env values:
|
|
|
|
```env
|
|
WORKER_REALESRGAN_BIN=/app/bin/realesrgan-ncnn-vulkan
|
|
WORKER_REALESRGAN_MODEL_DIR=/app/models
|
|
WORKER_REALESRGAN_DEFAULT_MODEL=realesrgan-x4plus
|
|
WORKER_REALESRGAN_ANIME_MODEL=realesrgan-x4plus-anime
|
|
WORKER_REALESRGAN_TILE=0
|
|
WORKER_REALESRGAN_TTA=false
|
|
WORKER_REALESRGAN_VERBOSE=false
|
|
WORKER_REALESRGAN_TIMEOUT_SECONDS=900
|
|
WORKER_REALESRGAN_PREPROCESS_MAX_PIXELS=16777216
|
|
WORKER_REALESRGAN_OUTPUT_EXT=webp
|
|
WORKER_REALESRGAN_ALLOW_MODEL_FALLBACK=true
|
|
```
|
|
|
|
Compatibility values kept by the worker:
|
|
|
|
```env
|
|
WORKER_MODEL_DIR=/app/app/models
|
|
WORKER_DEFAULT_MODEL=realesrgan-x4plus
|
|
```
|
|
|
|
Worker behavior
|
|
---------------
|
|
|
|
Worker request contract:
|
|
|
|
- endpoint: `POST /v1/upscale`
|
|
- bearer auth required
|
|
- accepted output formats: `webp`, `png`, `jpg`
|
|
- allowed scales: `2`, `4`
|
|
- allowed modes: `standard`, `artwork`, `photo`, `illustration`
|
|
|
|
Health and temp file endpoints:
|
|
|
|
- `GET /health`
|
|
- `GET /v1/results/{filename}`
|
|
- `DELETE /v1/results/{filename}`
|
|
|
|
Mode and scale behavior
|
|
-----------------------
|
|
|
|
Real-ESRGAN mode mapping:
|
|
|
|
- `standard` -> default model
|
|
- `artwork` -> default model
|
|
- `photo` -> default model
|
|
- `illustration` -> anime model when available
|
|
|
|
Fallback behavior:
|
|
|
|
- if the requested model exists, it is used
|
|
- if it is missing and fallback is enabled, the default model is used
|
|
- if it is missing and fallback is disabled, processing fails safely
|
|
|
|
Scale behavior:
|
|
|
|
- `4x` returns native 4x output
|
|
- `2x` currently runs the 4x model and then downsamples to 2x
|
|
|
|
Storage and data flow
|
|
---------------------
|
|
|
|
Laravel stores Enhance files under the configured prefixes:
|
|
|
|
- sources under `enhance/sources`
|
|
- outputs under `enhance/outputs`
|
|
- previews under `enhance/previews`
|
|
|
|
The worker does not permanently store Enhance results.
|
|
|
|
When `ENHANCE_ENGINE=external_worker`:
|
|
|
|
1. Laravel prepares a copied Enhance source file.
|
|
2. Laravel sends the worker a temporary URL or a temporary signed internal route.
|
|
3. The worker downloads the source file.
|
|
4. The worker processes the image.
|
|
5. The worker exposes a temporary result URL.
|
|
6. Laravel downloads, validates, and stores the final output.
|
|
7. Laravel instructs the worker to delete the temporary result.
|
|
|
|
Verification checklist
|
|
----------------------
|
|
|
|
Laravel verification:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
php artisan enhance:health
|
|
php artisan enhance:health --json
|
|
```
|
|
|
|
Worker verification:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:8095/health
|
|
```
|
|
|
|
Queue verification:
|
|
|
|
```bash
|
|
php artisan queue:work --queue=enhance,default
|
|
```
|
|
|
|
Application verification:
|
|
|
|
1. Open `/enhance/create`.
|
|
2. Upload or choose a small source image.
|
|
3. Select `2x` or `4x` and a valid mode.
|
|
4. Submit the job.
|
|
5. Confirm the job transitions from queued to completed.
|
|
6. Confirm output exists on the Enhance disk.
|
|
7. Confirm the source file and original artwork remain untouched.
|
|
|
|
Health states
|
|
-------------
|
|
|
|
Laravel health command helps identify:
|
|
|
|
- configured engine
|
|
- queue usage
|
|
- stuck jobs
|
|
- lifecycle status
|
|
|
|
Worker `/health` helps identify:
|
|
|
|
- current worker engine
|
|
- maximum input and output limits
|
|
- Real-ESRGAN binary availability
|
|
- Real-ESRGAN model directory readiness
|
|
- available Real-ESRGAN models
|
|
|
|
If the worker is in Real-ESRGAN mode and health returns `degraded`, do not treat the runtime as production-ready yet.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
`Worker URL is missing.`
|
|
|
|
- set `ENHANCE_WORKER_URL`
|
|
- clear config
|
|
|
|
`Worker token is missing.`
|
|
|
|
- set `ENHANCE_WORKER_TOKEN`
|
|
- make sure the worker uses the same `WORKER_TOKEN`
|
|
|
|
`Worker is unavailable.`
|
|
|
|
- confirm the worker is reachable
|
|
- confirm the worker container is running
|
|
- confirm the URL points to the worker base URL
|
|
|
|
`Upscale engine is not available. Check model files and worker installation.`
|
|
|
|
- confirm `WORKER_ENGINE=realesrgan-ncnn`
|
|
- confirm `bin/realesrgan-ncnn-vulkan` exists and is executable
|
|
- confirm the required `.param` and `.bin` model files exist
|
|
- run `bash scripts/verify-realesrgan.sh`
|
|
|
|
Jobs stay queued
|
|
|
|
- confirm queue workers consume `ENHANCE_QUEUE`
|
|
- if using `enhance`, run workers with `--queue=enhance,default`
|
|
|
|
`status: degraded` from worker health
|
|
|
|
- verify binary and model directory paths
|
|
- verify runtime files are mounted inside the container
|
|
- fall back to `WORKER_ENGINE=pillow` until the runtime is fixed
|
|
|
|
Operations and safety notes
|
|
---------------------------
|
|
|
|
- Keep the worker bound to `127.0.0.1` or a private container network.
|
|
- Do not expose the worker publicly.
|
|
- Use a strong shared token.
|
|
- Keep Enhance on a dedicated queue when load increases.
|
|
- Keep cleanup enabled so stale outputs and failed files do not accumulate.
|
|
- Do not commit Real-ESRGAN binary or model weight files unless explicitly approved.
|
|
- The worker only serves generated files from its own output directory.
|
|
- The worker rejects unsupported source URLs and unsafe output paths.
|
|
- Original artwork files are never replaced by the Enhance flow.
|
|
|
|
Production rollout recommendation
|
|
---------------------------------
|
|
|
|
Recommended rollout sequence:
|
|
|
|
1. Enable `ENHANCE_ENGINE=stub` and verify the Laravel workflow.
|
|
2. Move to `ENHANCE_ENGINE=external_worker` with `WORKER_ENGINE=pillow`.
|
|
3. Verify queue, storage, cleanup, and health behavior.
|
|
4. Install Real-ESRGAN runtime files and switch the worker to `WORKER_ENGINE=realesrgan-ncnn`.
|
|
5. Confirm worker health is `ok` before calling the runtime production-ready.
|
|
|
|
Related docs
|
|
------------
|
|
|
|
- operational notes: [docs/enhance.md](docs/enhance.md)
|
|
- worker runtime docs: [services/enhance-worker/README.md](services/enhance-worker/README.md) |