Add cooperative task recovery and persistent live status
Build custom container images / build (map[base_image:php:8-fpm-alpine build_args:PHP_VERSION=8
context:php8-pgsql fingerprint_command:{ apk info -v | LC_ALL=C sort; find /usr/local/lib/php/extensions /usr/local/etc/php/conf.d -type f -exec sha256sum {} + | LC_ALL=C sort; }
name:ph… (push) Successful in 40s
Build custom container images / build (map[base_image:postgres:18 build_args:PG_VERSION=18
POSTGIS_VERSION=3
VCHORD_VERSION=0.5.3
context:postgres fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; find /usr/lib/postgresql -type f -exec sha256su… (push) Successful in 49s
Build custom container images / build (map[base_image:python:3 build_args:PYTHON_VERSION=3
context:python-tools fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:python-tools oci_labels:org.opencontainers.ima… (push) Successful in 1m37s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim
context:linkki-tiedotus fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:linkki-tiedotus oci_labels:… (push) Successful in 36s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim
context:timepoll fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:timepoll oci_labels:org.opencontai… (push) Successful in 42s
Build custom container images / build (map[base_image:php:8-fpm-alpine build_args:PHP_VERSION=8
context:php8-pgsql fingerprint_command:{ apk info -v | LC_ALL=C sort; find /usr/local/lib/php/extensions /usr/local/etc/php/conf.d -type f -exec sha256sum {} + | LC_ALL=C sort; }
name:ph… (push) Successful in 40s
Build custom container images / build (map[base_image:postgres:18 build_args:PG_VERSION=18
POSTGIS_VERSION=3
VCHORD_VERSION=0.5.3
context:postgres fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; find /usr/lib/postgresql -type f -exec sha256su… (push) Successful in 49s
Build custom container images / build (map[base_image:python:3 build_args:PYTHON_VERSION=3
context:python-tools fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:python-tools oci_labels:org.opencontainers.ima… (push) Successful in 1m37s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim
context:linkki-tiedotus fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:linkki-tiedotus oci_labels:… (push) Successful in 36s
Build custom container images / build (map[base_image:python:3.12-slim build_args:PYTHON_VERSION=3.12-slim
context:timepoll fingerprint_command:{ dpkg-query -W -f='${binary:Package}=${Version}\n' | LC_ALL=C sort; pip freeze | LC_ALL=C sort; }
name:timepoll oci_labels:org.opencontai… (push) Successful in 42s
This commit is contained in:
1 parent
034fecbd51
commit
3bfd199b9c
28 files changed
+2356
-172
No files matched your search
+99
-7
@@ -7,8 +7,9 @@ with `python -m app`.
|
||||
## Control Panel
|
||||
|
||||
The application discovers direct `*.py` files in each non-hidden directory
|
||||
under `/opt/tasks`. Each directory is a task group. Tasks in one group run
|
||||
sequentially; different groups may run concurrently. It records run history in
|
||||
under `/opt/tasks`. Each directory is a task group. A group is exclusive while
|
||||
one of its tasks runs; unrelated groups may run concurrently. Tasks can declare
|
||||
additional shared resources to prevent conflicts across groups. It records run history in
|
||||
SQLite and combined output in log files under `/var/lib/server-maintenance`,
|
||||
and supports cancellation of active task process groups.
|
||||
|
||||
@@ -81,9 +82,10 @@ Use `\n` for a line break or `\n\n` for a paragraph gap in descriptions.
|
||||
|
||||
Each task declaration may contain `title`, `description`, `inputs`,
|
||||
`wait_for_result`, `download_artifacts`, and `execution`. Set
|
||||
`wait_for_result` for short tasks where the form should remain blocked until it
|
||||
reports success or failure. The control panel disables its controls and warns
|
||||
before page unload while such a task runs. `download_artifacts` additionally
|
||||
`wait_for_result` for tasks whose result should appear beside the form instead
|
||||
of immediately opening the run-detail page. Running and conflicting controls
|
||||
are disabled, but navigation is allowed. Opening or reloading the page restores
|
||||
the current status and optional live progress from the server. `download_artifacts` additionally
|
||||
starts the first non-JSON artifact download after a successful browser run;
|
||||
reserve it for files such as archives that the browser should download
|
||||
automatically.
|
||||
@@ -101,8 +103,8 @@ Every input has `name`, `label`, and `type`; `required` defaults to `false`.
|
||||
|
||||
Set `sensitive` to `true` on text or file inputs that must not appear in run
|
||||
history. Text is masked, the database records `[redacted]`, and the original
|
||||
value is available only through a private execution-input file. Uploaded files
|
||||
are likewise retained privately for the run and then removed.
|
||||
value is available only through a private execution-input file. Interrupted
|
||||
recoverable tasks retain that input and their uploads until recovery finishes.
|
||||
|
||||
For review-first tasks, use `execution`. It produces one split button: click
|
||||
the main part to use its current mode, or use the arrow to select the other
|
||||
@@ -135,6 +137,96 @@ task without an entry in the manifest gets an empty JSON object. Uploaded files
|
||||
are stored in a private per-run directory; their absolute paths are provided in
|
||||
the input JSON. Files are removed when the task finishes, fails, or is cancelled,
|
||||
except for successful review uploads retained temporarily for execution as above.
|
||||
Interrupted tasks with recovery support retain their files privately.
|
||||
|
||||
## Shutdown And Recovery
|
||||
|
||||
The Compose example grants 120 seconds for container shutdown. The runner stops
|
||||
accepting work, pauses the background scheduler, and asks all task subprocesses
|
||||
to save and stop at safe boundaries. It waits up to 110 seconds by default, then
|
||||
terminates remaining process groups. The HTTP server gets a final short cleanup
|
||||
window. These are maximum waits; idle or cooperative jobs stop sooner. The
|
||||
Compose example also uses Docker's init process to reap orphaned subprocesses.
|
||||
`SERVER_MAINTENANCE_SHUTDOWN_GRACE_SECONDS` changes the application deadline; keep
|
||||
the container's `stop_grace_period` longer than that deadline.
|
||||
|
||||
Tasks opt in by defining a top-level, synchronous `recover()` function with no
|
||||
arguments in their own Python file. Discovery reads its syntax without importing
|
||||
the task. On startup, a new linked recovery run enters the ordinary task queue
|
||||
and calls that function in an isolated worker process. Original logs and status
|
||||
remain unchanged. Cancelled tasks never recover automatically. Tasks without a
|
||||
hook remain interrupted. Historical runs created before this recovery protocol
|
||||
are not automatically replayed.
|
||||
Recovery waits and retries transient connection failures while services start,
|
||||
with a maximum 30-second interval. Authentication failures and task validation
|
||||
errors are reported rather than automatically repeated.
|
||||
|
||||
Recovery code must inspect current files or remote state before repeating an
|
||||
operation. A lost response does not prove that the operation failed. Save an
|
||||
intent before an irreversible operation and reconcile its outcome after restart.
|
||||
For repeatable scans, a periodic checkpoint is sufficient. The runner cannot
|
||||
serialize Python stacks or reconstruct task-specific state automatically.
|
||||
|
||||
Use `from app import task_runtime as runtime` in task code:
|
||||
|
||||
- `runtime.checkpoint_data()` reads the saved dictionary on recovery.
|
||||
- `runtime.checkpoint(data)` updates the in-memory snapshot and persists ordinary
|
||||
repeatable work at most once a minute. Use `force=True` for important intents,
|
||||
completed results, and other state that must precede an external mutation.
|
||||
- `runtime.check_stop()` observes a pause at a safe boundary, saves the latest
|
||||
snapshot, and unwinds the task. Do not catch its `TaskPaused` exception.
|
||||
Exit code 75 is reserved for this cooperative pause.
|
||||
- `runtime.critical_section()` defers a pause through a small consistency-sensitive
|
||||
action. Do not wrap an entire job or an unbounded transfer in it.
|
||||
- `runtime.sleep(seconds)` observes pauses during waits.
|
||||
- `runtime.was_clean_shutdown()` distinguishes a completed shutdown checkpoint
|
||||
from a periodic checkpoint. External state must still be checked in both cases.
|
||||
- `runtime.work_directory()` is a private directory shared by the interrupted run
|
||||
and its recovery chain. Use it for durable intermediate files, not artifacts.
|
||||
|
||||
Keep the usual `if __name__ == "__main__":` guard. The recovery worker loads the
|
||||
module under a different name, then calls `recover()`; module import must not
|
||||
start work. A recovery hook may call the task's regular implementation with a
|
||||
recovery flag, provided that implementation performs the necessary reconciliation.
|
||||
|
||||
Work directories live at `runs/<original-run-id>.work`. Exact inputs and uploads
|
||||
remain protected while a run is active or awaits recovery. Raw work inputs are
|
||||
removed on completion or cancellation; finished checkpoint directories are
|
||||
retained for 14 days. Do not put passwords or other credentials in checkpoint or
|
||||
progress dictionaries. Read credentials from their normal mounted files.
|
||||
|
||||
### Shared Resources
|
||||
|
||||
Declare optional resource names as a literal tuple at module level:
|
||||
|
||||
```python
|
||||
RESOURCES = ("shared-files",)
|
||||
```
|
||||
|
||||
Two tasks sharing a resource cannot run together, including during automatic
|
||||
recovery. The backend rejects conflicting submissions with HTTP 409 and identifies
|
||||
the blocking run. Every task also holds its group's resource. Resource names are
|
||||
parsed as data; no task code executes during discovery.
|
||||
|
||||
### Live Progress
|
||||
|
||||
`runtime.progress(text)` publishes optional plain text, including line breaks,
|
||||
to the task and run pages. Updates are throttled to once a second; use
|
||||
`force=True` at stage boundaries. Progress is independent of log output, so it
|
||||
does not replace selected log text or require the browser to stay open.
|
||||
|
||||
For simple item-count percentages and per-step estimates:
|
||||
|
||||
```python
|
||||
tracker = runtime.Progress({"Scanning files": 100, "Updating records": 20})
|
||||
tracker.update("Scanning files", 0, force=True)
|
||||
# After processing five files:
|
||||
tracker.update("Scanning files", 5)
|
||||
```
|
||||
|
||||
Each step has its own percentage and an approximate ETA based only on processed
|
||||
items and elapsed time. Pending steps have no ETA. Tasks may instead publish their
|
||||
own text when totals are unknown; no global estimate is required.
|
||||
|
||||
A group may also include a private `task-config.json` for stable local details
|
||||
such as mounted filesystem roots. This is not part of discovery or the web
|
||||
|
||||
Reference in new issue
Block a user