Files
docker-images/python-tools/README.md
T
ajp_anton 1c713ff635
Build custom container images / build (map[base_image:php:8-fpm-alpine build_args:PHP_VERSION=8 context:php8-pgsql fingerprint_command:{ apk info -v | LC_ALL=C sort; find /usr/local/lib/php/extensions /usr/local/etc/php/conf.d -type f -exec sha256sum {} + | LC_ALL=C sort; } name:ph… (push) Successful in 50s
Build custom container images / build (map[base_image:postgres:18 build_args:PG_VERSION=18 POSTGIS_VERSION=3 VCHORD_VERSION=0.5.3 context:postgres fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; find /usr/lib/postgresql -type f -exec sha256su… (push) Successful in 50s
Build custom container images / build (map[base_image:python:3 build_args:PYTHON_VERSION=3 context:python-tools fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:python-tools oci_labels:org.opencontainers.ima… (push) Successful in 1m37s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim context:linkki-tiedotus fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:linkki-tiedotus oci_labels:… (push) Successful in 36s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim context:timepoll fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:timepoll oci_labels:org.opencontai… (push) Successful in 42s
Simplify task input handling and tighten runner coordination
2026-11-01 16:44:11 +00:00

281 lines
14 KiB
Markdown

# Python Tools
This image is a Python 3 task runner with ExifTool and a small web control panel
for running known, mounted Python maintenance tasks. It starts the control panel
with `python -m app`.
## Control Panel
The application discovers direct `*.py` files in each non-hidden directory
under `/opt/tasks`. Each directory is a task group. A group is exclusive while
one of its tasks runs; unrelated groups may run concurrently. Tasks can declare
additional shared resources to prevent conflicts across groups. It records run history in
SQLite and combined output in log files under `/var/lib/server-maintenance`,
and supports cancellation of active task process groups.
The image contains application code only. Mount these paths at runtime:
- `/opt/tasks` read-only: reviewed task scripts and their input manifests.
- `/opt/credentials` read-only: named credential files used by those scripts.
- `/var/lib/server-maintenance`: persistent SQLite data, logs, and task state.
Each group's durable task data lives under
`/var/lib/server-maintenance/tasks/<group-id>`. When a task group disappears
from the mounted task directory, its state is retained for 14 days and then
removed automatically. Tasks that need more focused retention should clean up
their own state within that group.
Task processes receive only runner-provided paths for input, artifacts,
credentials, and state. Keep task-specific non-secret configuration in an
optional `task-config.json` beside that group's scripts, and credentials in
`/opt/credentials`. The runner does not interpret `task-config.json`; its
schema belongs to the task group that reads it.
Tasks are never imported during discovery. Their optional description is the
first line of their module docstring. A task must not require stdin or command
line arguments, should write useful output, and must return a non-zero exit
code on failure.
## Task definitions
Each non-hidden directory directly below `/opt/tasks` is a task group. Its
direct `*.py` files are the exposed tasks. Use an optional
`task-inputs.json` beside those files to define the group's title, task titles
and descriptions, task order, and web form fields. It is parsed as data only
and currently uses version `1`:
```json
{
"version": 1,
"title": "Example Tasks",
"tasks": {
"inspect.py": {
"title": "Inspect item",
"description": "Checks one item without changing it.",
"inputs": [
{"name": "identifier", "label": "Identifier", "type": "text", "required": true},
{"name": "attempts", "label": "Attempts", "type": "integer", "minimum": 1, "maximum": 5, "default": 2},
{"name": "mode", "label": "Mode", "type": "choice", "options": ["standard", "extended"]},
{"name": "alerts", "label": "Alerts", "type": "multi_choice", "options": ["email", "webhook"]},
{"name": "review_csv", "label": "Review CSV", "type": "file", "accept": [".csv"], "maximum_bytes": 10485760}
],
"wait_for_result": true,
"execution": {
"field": "mode",
"dry_run_value": "review",
"dry_run_label": "Review",
"execute_value": "apply",
"execute_label": "Execute"
}
}
}
}
```
`title` is optional on a group and task. A group otherwise uses its directory
name; a task otherwise uses its filename without `.py`, with underscores made
readable. `description` is optional on a task and otherwise comes from the
first non-empty line of its module docstring. The order of keys in `tasks`
controls the listed tasks' display order. Include a task as `{}` when only
ordering is needed. Unlisted direct Python files follow alphabetically.
Use `\n` for a line break or `\n\n` for a paragraph gap in descriptions.
Each task declaration may contain `title`, `description`, `inputs`,
`wait_for_result`, `download_artifacts`, and `execution`. Set
`wait_for_result` for tasks whose result should appear beside the form instead
of immediately opening the run-detail page. Running and conflicting controls
are disabled, but navigation is allowed. Opening or reloading the page restores
the current status and optional live progress from the server. `download_artifacts` additionally
starts the first non-JSON artifact download after a successful browser run;
reserve it for files such as archives that the browser should download
automatically.
Every input has `name`, `label`, and `type`; `required` defaults to `false`.
`default` is allowed for every type except `file`.
- `text`: optional `pattern`, a Python-compatible regular expression.
- `integer`: optional `minimum`, `maximum`, and positive `step`.
- `date` and `datetime`: ISO-formatted browser date controls.
- `choice`: `options` is required, as strings or `{ "value", "label" }` objects.
- `multi_choice`: the same required `options`; its `default` is an array.
- `file`: optional lowercase extension list in `accept` and positive byte cap
in `maximum_bytes`.
Set `sensitive` to `true` on text or file inputs that must not appear in run
history. Text is masked, the database records `[redacted]`, and the original
value is available only through the run's private work-directory input file. Interrupted
recoverable tasks retain that input and their uploads until recovery finishes.
For review-first tasks, use `execution`. It produces one split button: click
the main part to use its current mode, or use the arrow to select the other
mode. A successful review run can be repeated with its saved inputs in the
execution mode from the run-detail page. The task must still revalidate the
live filesystem before changing anything.
Execution creates a separate run; the original review and its output remain
accessible through Run history on either run's detail page.
For tasks with file inputs, successful reviews keep the uploaded files privately
for up to 24 hours. Execute copies those files into the new run, then removes
the review's copies. Missing or expired uploads require a new review. Tasks
with sensitive inputs cannot use this replay feature.
```json
{
"field": "mode",
"dry_run_value": "review",
"dry_run_label": "Review",
"execute_value": "apply",
"execute_label": "Execute"
}
```
The server validates every submitted value before a run is queued. It records
the validated object in the run database and writes it to a per-run file.
Sensitive fields are redacted in the database and use a private `0600`
execution file instead. Tasks read the applicable file path from
`SERVER_MAINTENANCE_INPUT`; no user value is appended to the command line. A
task without an entry in the manifest gets an empty JSON object. Uploaded files
are stored in a private per-run directory; their absolute paths are provided in
the input JSON. Files are removed when the task finishes, fails, or is cancelled,
except for successful review uploads retained temporarily for execution as above.
Interrupted tasks with recovery support retain their files privately.
## Shutdown And Recovery
The Compose example grants 120 seconds for container shutdown. The runner stops
accepting work, pauses the background scheduler, and asks all task subprocesses
to save and stop at safe boundaries. It waits up to 110 seconds by default, then
terminates remaining process groups. The HTTP server gets a final short cleanup
window. These are maximum waits; idle or cooperative jobs stop sooner. The
Compose example also uses Docker's init process to reap orphaned subprocesses.
`SERVER_MAINTENANCE_SHUTDOWN_GRACE_SECONDS` changes the application deadline; keep
the container's `stop_grace_period` longer than that deadline.
Tasks opt in by defining a top-level, synchronous `recover()` function with no
arguments in their own Python file. Discovery reads its syntax without importing
the task. On startup, a new linked recovery run enters the ordinary task queue
and calls that function in an isolated worker process. Original logs and status
remain unchanged. Cancelled tasks never recover automatically. Tasks without a
hook remain interrupted. Historical runs created before this recovery protocol
are not automatically replayed.
Recovery waits and retries transient connection failures while services start,
with a maximum 30-second interval. Authentication failures and task validation
errors are reported rather than automatically repeated.
Recovery code must inspect current files or remote state before repeating an
operation. A lost response does not prove that the operation failed. Save an
intent before an irreversible operation and reconcile its outcome after restart.
For repeatable scans, a periodic checkpoint is sufficient. The runner cannot
serialize Python stacks or reconstruct task-specific state automatically.
Use `from app import task_runtime as runtime` in task code:
- `runtime.checkpoint_data()` reads the saved dictionary on recovery.
- `runtime.checkpoint(data)` updates the in-memory snapshot and persists ordinary
repeatable work at most once a minute. Use `force=True` for important intents,
completed results, and other state that must precede an external mutation.
- `runtime.check_stop()` observes a pause at a safe boundary, saves the latest
snapshot, and unwinds the task. Do not catch its `TaskPaused` exception.
Exit code 75 is reserved for this cooperative pause.
- `runtime.critical_section()` defers a pause through a small consistency-sensitive
action. Do not wrap an entire job or an unbounded transfer in it.
- `runtime.sleep(seconds)` observes pauses during waits.
- `runtime.was_clean_shutdown()` distinguishes a completed shutdown checkpoint
from a periodic checkpoint. External state must still be checked in both cases.
- `runtime.work_directory()` is a private directory shared by the interrupted run
and its recovery chain. Use it for durable intermediate files, not artifacts.
Keep the usual `if __name__ == "__main__":` guard. The recovery worker loads the
module under a different name, then calls `recover()`; module import must not
start work. A recovery hook may call the task's regular implementation with a
recovery flag, provided that implementation performs the necessary reconciliation.
Work directories live at `runs/<original-run-id>.work`. Exact inputs and uploads
remain protected while a run is active or awaits recovery. Raw work inputs are
removed on completion or cancellation; finished checkpoint directories are
retained for 14 days. Do not put passwords or other credentials in checkpoint or
progress dictionaries. Read credentials from their normal mounted files.
### Shared Resources
Declare optional resource names as a literal tuple at module level:
```python
RESOURCES = ("shared-files",)
```
Two tasks sharing a resource cannot run together, including during automatic
recovery. The backend rejects conflicting submissions with HTTP 409 and identifies
the blocking run. Every task also holds its group's resource. Resource names are
parsed as data; no task code executes during discovery.
### Live Progress
`runtime.progress(text)` publishes optional plain text, including line breaks,
to the task and run pages. Updates are throttled to once a second; use
`force=True` at stage boundaries. Progress is independent of log output, so it
does not replace selected log text or require the browser to stay open.
For simple item-count percentages and per-step estimates:
```python
tracker = runtime.Progress({"Scanning files": 100, "Updating records": 20})
tracker.update("Scanning files", 0)
# After processing five files:
tracker.update("Scanning files", 5)
```
Each step shows completed/total items, a percentage and an approximate ETA based
on items processed since starting or resuming. Stage starts and completion are
published immediately. Pending steps have no ETA. Tasks may instead publish their
own text when totals are unknown; no global estimate is required.
A group may also include a private `task-config.json` for stable local details
such as mounted filesystem roots. This is not part of discovery or the web
form schema: task code reads and validates it itself. Keeping it beside the
group makes a task's runtime requirements clear and avoids a global
task-environment allowlist.
## Artifacts
Tasks can write files to the directory named by
`SERVER_MAINTENANCE_ARTIFACTS`. Run-detail pages show them when the task
finishes. JSON artifacts offer an **Open** link that renders the JSON in a new
tab and leaves the artifact available; other artifacts offer a download link.
The server deletes an artifact after its download stream completes; interrupted
downloads remain available for retry. Active runs' artifacts never expire.
After a run finishes, artifacts expire after 24 hours. Failed or cancelled
runs keep valid JSON diagnostic reports, but discard incomplete downloads.
Ordinary input files are removed when execution ends; saved inputs remain in
SQLite for history and review replay. Run pages fetch only appended log bytes.
A browser cannot confirm that a downloaded file was retained on the client device.
## Background tasks
An optional `background-tasks.json` beside a group's task files declares
internal recurring scripts. These scripts must live below the group's
`internal/` directory, are not exposed in the UI, and are launched at most once
per configured interval. Their latest output is retained below the persistent
state directory rather than being added to normal run history.
```json
{
"version": 1,
"tasks": [
{"id": "poll", "script": "internal/poll.py", "interval_seconds": 60}
]
}
```
`compose.example.yaml` is suitable for adding the service to an existing Compose
stack. Set `PYTHON_TOOLS_VOLUME_ROOT` to the host directory that will hold
`state`, `tasks`, and `credentials`; set
`SERVER_MAINTENANCE_ALLOWED_PROXY_IPS` to the reverse proxy's direct peer
address. Keep the application behind an authenticated reverse proxy.
Tasks receive a group-specific writable directory in
`SERVER_MAINTENANCE_STATE`. Tasks which intentionally maintain application-wide
state, such as run-history maintenance, can use
`SERVER_MAINTENANCE_STATE_ROOT`; normal task state should remain
group-specific.