Add cooperative task recovery and persistent live status
Build custom container images / build (map[base_image:php:8-fpm-alpine build_args:PHP_VERSION=8 context:php8-pgsql fingerprint_command:{ apk info -v | LC_ALL=C sort; find /usr/local/lib/php/extensions /usr/local/etc/php/conf.d -type f -exec sha256sum {} + | LC_ALL=C sort; } name:ph… (push) Successful in 40s
Build custom container images / build (map[base_image:postgres:18 build_args:PG_VERSION=18 POSTGIS_VERSION=3 VCHORD_VERSION=0.5.3 context:postgres fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; find /usr/lib/postgresql -type f -exec sha256su… (push) Successful in 49s
Build custom container images / build (map[base_image:python:3 build_args:PYTHON_VERSION=3 context:python-tools fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:python-tools oci_labels:org.opencontainers.ima… (push) Successful in 1m37s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim context:linkki-tiedotus fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:linkki-tiedotus oci_labels:… (push) Successful in 36s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim context:timepoll fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; } name:timepoll oci_labels:org.opencontai… (push) Successful in 42s

This commit is contained in:
ajp_anton committed 2026-10-29 22:12:04 +00:00
1 parent 034fecbd51
commit 3bfd199b9c
28 files changed
+2356 -172

No files matched your search

+99 -7
View File
@@ -7,8 +7,9 @@ with `python -m app`.
## Control Panel
The application discovers direct `*.py` files in each non-hidden directory
under `/opt/tasks`. Each directory is a task group. Tasks in one group run
sequentially; different groups may run concurrently. It records run history in
under `/opt/tasks`. Each directory is a task group. A group is exclusive while
one of its tasks runs; unrelated groups may run concurrently. Tasks can declare
additional shared resources to prevent conflicts across groups. It records run history in
SQLite and combined output in log files under `/var/lib/server-maintenance`,
and supports cancellation of active task process groups.
@@ -81,9 +82,10 @@ Use `\n` for a line break or `\n\n` for a paragraph gap in descriptions.
Each task declaration may contain `title`, `description`, `inputs`,
`wait_for_result`, `download_artifacts`, and `execution`. Set
`wait_for_result` for short tasks where the form should remain blocked until it
reports success or failure. The control panel disables its controls and warns
before page unload while such a task runs. `download_artifacts` additionally
`wait_for_result` for tasks whose result should appear beside the form instead
of immediately opening the run-detail page. Running and conflicting controls
are disabled, but navigation is allowed. Opening or reloading the page restores
the current status and optional live progress from the server. `download_artifacts` additionally
starts the first non-JSON artifact download after a successful browser run;
reserve it for files such as archives that the browser should download
automatically.
@@ -101,8 +103,8 @@ Every input has `name`, `label`, and `type`; `required` defaults to `false`.
Set `sensitive` to `true` on text or file inputs that must not appear in run
history. Text is masked, the database records `[redacted]`, and the original
value is available only through a private execution-input file. Uploaded files
are likewise retained privately for the run and then removed.
value is available only through a private execution-input file. Interrupted
recoverable tasks retain that input and their uploads until recovery finishes.
For review-first tasks, use `execution`. It produces one split button: click
the main part to use its current mode, or use the arrow to select the other
@@ -135,6 +137,96 @@ task without an entry in the manifest gets an empty JSON object. Uploaded files
are stored in a private per-run directory; their absolute paths are provided in
the input JSON. Files are removed when the task finishes, fails, or is cancelled,
except for successful review uploads retained temporarily for execution as above.
Interrupted tasks with recovery support retain their files privately.
## Shutdown And Recovery
The Compose example grants 120 seconds for container shutdown. The runner stops
accepting work, pauses the background scheduler, and asks all task subprocesses
to save and stop at safe boundaries. It waits up to 110 seconds by default, then
terminates remaining process groups. The HTTP server gets a final short cleanup
window. These are maximum waits; idle or cooperative jobs stop sooner. The
Compose example also uses Docker's init process to reap orphaned subprocesses.
`SERVER_MAINTENANCE_SHUTDOWN_GRACE_SECONDS` changes the application deadline; keep
the container's `stop_grace_period` longer than that deadline.
Tasks opt in by defining a top-level, synchronous `recover()` function with no
arguments in their own Python file. Discovery reads its syntax without importing
the task. On startup, a new linked recovery run enters the ordinary task queue
and calls that function in an isolated worker process. Original logs and status
remain unchanged. Cancelled tasks never recover automatically. Tasks without a
hook remain interrupted. Historical runs created before this recovery protocol
are not automatically replayed.
Recovery waits and retries transient connection failures while services start,
with a maximum 30-second interval. Authentication failures and task validation
errors are reported rather than automatically repeated.
Recovery code must inspect current files or remote state before repeating an
operation. A lost response does not prove that the operation failed. Save an
intent before an irreversible operation and reconcile its outcome after restart.
For repeatable scans, a periodic checkpoint is sufficient. The runner cannot
serialize Python stacks or reconstruct task-specific state automatically.
Use `from app import task_runtime as runtime` in task code:
- `runtime.checkpoint_data()` reads the saved dictionary on recovery.
- `runtime.checkpoint(data)` updates the in-memory snapshot and persists ordinary
repeatable work at most once a minute. Use `force=True` for important intents,
completed results, and other state that must precede an external mutation.
- `runtime.check_stop()` observes a pause at a safe boundary, saves the latest
snapshot, and unwinds the task. Do not catch its `TaskPaused` exception.
Exit code 75 is reserved for this cooperative pause.
- `runtime.critical_section()` defers a pause through a small consistency-sensitive
action. Do not wrap an entire job or an unbounded transfer in it.
- `runtime.sleep(seconds)` observes pauses during waits.
- `runtime.was_clean_shutdown()` distinguishes a completed shutdown checkpoint
from a periodic checkpoint. External state must still be checked in both cases.
- `runtime.work_directory()` is a private directory shared by the interrupted run
and its recovery chain. Use it for durable intermediate files, not artifacts.
Keep the usual `if __name__ == "__main__":` guard. The recovery worker loads the
module under a different name, then calls `recover()`; module import must not
start work. A recovery hook may call the task's regular implementation with a
recovery flag, provided that implementation performs the necessary reconciliation.
Work directories live at `runs/<original-run-id>.work`. Exact inputs and uploads
remain protected while a run is active or awaits recovery. Raw work inputs are
removed on completion or cancellation; finished checkpoint directories are
retained for 14 days. Do not put passwords or other credentials in checkpoint or
progress dictionaries. Read credentials from their normal mounted files.
### Shared Resources
Declare optional resource names as a literal tuple at module level:
```python
RESOURCES = ("shared-files",)
```
Two tasks sharing a resource cannot run together, including during automatic
recovery. The backend rejects conflicting submissions with HTTP 409 and identifies
the blocking run. Every task also holds its group's resource. Resource names are
parsed as data; no task code executes during discovery.
### Live Progress
`runtime.progress(text)` publishes optional plain text, including line breaks,
to the task and run pages. Updates are throttled to once a second; use
`force=True` at stage boundaries. Progress is independent of log output, so it
does not replace selected log text or require the browser to stay open.
For simple item-count percentages and per-step estimates:
```python
tracker = runtime.Progress({"Scanning files": 100, "Updating records": 20})
tracker.update("Scanning files", 0, force=True)
# After processing five files:
tracker.update("Scanning files", 5)
```
Each step has its own percentage and an approximate ETA based only on processed
items and elapsed time. Pending steps have no ETA. Tasks may instead publish their
own text when totals are unknown; no global estimate is required.
A group may also include a private `task-config.json` for stable local details
such as mounted filesystem roots. This is not part of discovery or the web