209 lines
6.9 KiB
Markdown
209 lines
6.9 KiB
Markdown
# Skinbase Enhance Worker
|
|
|
|
This service is the private HTTP worker for the Skinbase Enhance `external_worker` engine. Laravel remains responsible for job state, permissions, queueing, moderation, storage, and cleanup. The worker only downloads the copied Enhance source image, upscales it, exposes a temporary result, and deletes that temporary result when Laravel is done.
|
|
|
|
## Local build and run
|
|
|
|
```bash
|
|
cd services/enhance-worker
|
|
docker compose -f docker-compose.example.yml up --build
|
|
curl http://127.0.0.1:8095/health
|
|
```
|
|
|
|
The example compose file uses `WORKER_ENGINE=pillow` so local development works without Real-ESRGAN weights.
|
|
|
|
## Real-ESRGAN ncnn runtime
|
|
|
|
The worker supports these engine values:
|
|
|
|
- `WORKER_ENGINE=pillow`
|
|
- `WORKER_ENGINE=realesrgan-ncnn`
|
|
- `WORKER_ENGINE=realesrgan`
|
|
|
|
`realesrgan` currently aliases to the ncnn-vulkan CLI path. Pillow mode remains the safe local and CI fallback.
|
|
|
|
The worker expects the Real-ESRGAN ncnn runtime at `/app/bin/realesrgan-ncnn-vulkan` and model files under `/app/models`. These files are not committed. Keep `bin/` and `models/` private to your deployment or local environment.
|
|
|
|
## Endpoints
|
|
|
|
- `GET /health`
|
|
- `POST /v1/upscale` with `Authorization: Bearer <WORKER_TOKEN>`
|
|
- `GET /v1/results/{filename}` for temporary internal result download
|
|
- `DELETE /v1/results/{filename}` with bearer token for Laravel cleanup
|
|
|
|
## Worker env
|
|
|
|
```env
|
|
WORKER_HOST=0.0.0.0
|
|
WORKER_PORT=8095
|
|
WORKER_TOKEN=change-this-token
|
|
|
|
WORKER_ENGINE=pillow
|
|
WORKER_DEVICE=cpu
|
|
|
|
WORKER_MAX_UPLOAD_MB=20
|
|
WORKER_MAX_INPUT_WIDTH=4096
|
|
WORKER_MAX_INPUT_HEIGHT=4096
|
|
WORKER_MAX_OUTPUT_WIDTH=8192
|
|
WORKER_MAX_OUTPUT_HEIGHT=8192
|
|
|
|
WORKER_TMP_DIR=/app/storage/tmp
|
|
WORKER_OUTPUT_DIR=/app/storage/output
|
|
WORKER_RESULT_TTL_MINUTES=60
|
|
|
|
WORKER_MODEL_DIR=/app/app/models
|
|
WORKER_DEFAULT_MODEL=realesrgan-x4plus
|
|
|
|
WORKER_REALESRGAN_BIN=/app/bin/realesrgan-ncnn-vulkan
|
|
WORKER_REALESRGAN_MODEL_DIR=/app/models
|
|
WORKER_REALESRGAN_DEFAULT_MODEL=realesrgan-x4plus
|
|
WORKER_REALESRGAN_ANIME_MODEL=realesrgan-x4plus-anime
|
|
WORKER_REALESRGAN_TILE=0
|
|
WORKER_REALESRGAN_TTA=false
|
|
WORKER_REALESRGAN_VERBOSE=false
|
|
WORKER_REALESRGAN_TIMEOUT_SECONDS=900
|
|
WORKER_REALESRGAN_PREPROCESS_MAX_PIXELS=16777216
|
|
WORKER_REALESRGAN_OUTPUT_EXT=webp
|
|
WORKER_REALESRGAN_ALLOW_MODEL_FALLBACK=true
|
|
```
|
|
|
|
## Laravel env
|
|
|
|
```env
|
|
ENHANCE_ENGINE=external_worker
|
|
ENHANCE_WORKER_URL=http://127.0.0.1:8095
|
|
ENHANCE_WORKER_TIMEOUT=600
|
|
ENHANCE_WORKER_TOKEN=change-this-token
|
|
ENHANCE_WORKER_MAX_DOWNLOAD_MB=60
|
|
ENHANCE_QUEUE=enhance
|
|
```
|
|
|
|
After updating Laravel env:
|
|
|
|
```bash
|
|
php artisan config:clear
|
|
php artisan enhance:health
|
|
php artisan queue:work --queue=enhance,default
|
|
php artisan test --filter=EnhanceExternalWorker
|
|
```
|
|
|
|
## Example request
|
|
|
|
```bash
|
|
curl -X POST http://127.0.0.1:8095/v1/upscale \
|
|
-H "Authorization: Bearer change-this-token" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"job_id": 1,
|
|
"source_url": "https://example.com/test.webp",
|
|
"scale": 2,
|
|
"mode": "artwork",
|
|
"output_format": "webp"
|
|
}'
|
|
```
|
|
|
|
## Installing runtime files
|
|
|
|
```bash
|
|
cd services/enhance-worker
|
|
bash scripts/download-realesrgan-ncnn.sh
|
|
bash scripts/verify-realesrgan.sh
|
|
```
|
|
|
|
You can override the upstream release tag with:
|
|
|
|
```bash
|
|
bash scripts/download-realesrgan-ncnn.sh --version v0.2.0
|
|
```
|
|
|
|
If you already manage the binary and models yourself, place them in `bin/` and `models/` instead of using the helper script.
|
|
|
|
## Model files
|
|
|
|
Expected examples:
|
|
|
|
- `realesrgan-x4plus.param`
|
|
- `realesrgan-x4plus.bin`
|
|
- `realesrgan-x4plus-anime.param`
|
|
- `realesrgan-x4plus-anime.bin`
|
|
|
|
Illustration mode prefers the anime model. If that model is missing and `WORKER_REALESRGAN_ALLOW_MODEL_FALLBACK=true`, the worker falls back to the default model and records the fallback in response metadata.
|
|
|
|
## Running in Pillow mode
|
|
|
|
- `docker compose -f docker-compose.example.yml up --build`
|
|
- health should report `engine: pillow`
|
|
- metadata reports `real_ai_upscale: false`
|
|
|
|
## Running in Real-ESRGAN mode
|
|
|
|
```bash
|
|
cd services/enhance-worker
|
|
docker compose -f docker-compose.realesrgan.example.yml up --build
|
|
curl http://127.0.0.1:8095/health
|
|
```
|
|
|
|
If the binary or models are missing, health will report `status: degraded` and the worker will return a safe unavailable error for upscale requests.
|
|
|
|
## Health checks
|
|
|
|
`GET /health` returns general worker status for every engine.
|
|
|
|
When using `realesrgan-ncnn`, health also reports:
|
|
|
|
- whether the binary is configured
|
|
- whether the binary exists
|
|
- whether it is executable
|
|
- whether the model directory exists
|
|
- the available model list
|
|
- the configured default model
|
|
|
|
## Engines
|
|
|
|
- `WORKER_ENGINE=pillow`: local development and tests. Metadata reports `real_ai_upscale: false`.
|
|
- `WORKER_ENGINE=realesrgan-ncnn`: runs the ncnn-vulkan CLI through a safe `subprocess.run(..., shell=False)` path.
|
|
- `WORKER_ENGINE=realesrgan`: alias to `realesrgan-ncnn`.
|
|
|
|
2x handling uses the 4x model output and a deterministic high-quality downsample. 4x requests return the native model scale.
|
|
|
|
## CPU and GPU
|
|
|
|
CPU mode is supported by default.
|
|
|
|
Optional GPU deployment later:
|
|
|
|
- install NVIDIA Container Toolkit
|
|
- use a CUDA-compatible image and runtime
|
|
- set `WORKER_DEVICE=cuda`
|
|
- install Real-ESRGAN and its GPU-capable dependencies
|
|
|
|
Do not remove CPU fallback.
|
|
|
|
For `realesrgan-ncnn`, `WORKER_DEVICE=vulkan` is the intended production setting when the host has compatible drivers and GPU access.
|
|
|
|
## Performance notes
|
|
|
|
- 2x requests still pay much of the 4x cost because the worker currently runs the 4x model then downsamples.
|
|
- Very large images may need tiling. `WORKER_REALESRGAN_TILE=0` keeps the runtime default behavior.
|
|
- `WORKER_REALESRGAN_TTA=true` may improve quality but is slower.
|
|
- CPU-only Real-ESRGAN is suitable for verification and low-volume use, but usually too slow for production.
|
|
- Start production with conservative input limits and a dedicated `enhance` queue.
|
|
|
|
## Security notes
|
|
|
|
- Keep the worker on `127.0.0.1` or a private container network.
|
|
- Do not expose the worker publicly.
|
|
- Use a strong `WORKER_TOKEN`.
|
|
- The worker only accepts `http` and `https` source URLs.
|
|
- The result endpoint only serves sanitized filenames from the worker output directory.
|
|
- Temporary inputs are deleted after processing, and stale temporary outputs are pruned by TTL.
|
|
- Real-ESRGAN model names are selected from an internal allowlist only. Users cannot inject arbitrary CLI flags.
|
|
- The worker uses `subprocess.run` without `shell=True`.
|
|
|
|
## Troubleshooting
|
|
|
|
- `401 Unauthorized`: check `WORKER_TOKEN` and Laravel `ENHANCE_WORKER_TOKEN`.
|
|
- `Upscale engine is not available...`: the worker is in `realesrgan-ncnn` mode but the binary or models are missing, not executable, or the CLI run failed.
|
|
- `Worker rejected the image.`: invalid mode, scale, output format, MIME type, dimensions, or download size.
|
|
- `status: degraded` on `/health`: use `bash scripts/verify-realesrgan.sh` and either fix the runtime files or switch back to `WORKER_ENGINE=pillow`.
|
|
- Jobs stuck in `queued`: if `ENHANCE_QUEUE=enhance`, make sure workers or Horizon consume the `enhance` queue. |