Local disk is not progress.

The durability rule

For continuity mode checkpointed, a workload can resume only from a verified checkpoint manifest. Local disk, process memory, and partially uploaded files are never progress. Final artifacts for artifact-producing work use the same protocol: completed requires a committed final manifest.

restartable and ephemeral workloads do not use this path as primary progress (cursors, or none). They may still emit final manifests when the work produces durable outputs.

The runner must never claim a checkpoint is complete just because local files exist.

Stage contract

• Owns: checkpoint coordinator
• Before: PrepareCheckpoint validates generation ownership → short-lived scoped upload credentials
• After: immutable verified manifest; CAS advances latest-manifest pointer; checkpoint.committed
• Transport: gRPC + direct signed object-storage upload
• Durable ack: CommitCheckpoint verifies every object SHA-256, writes manifest, then compare-and-set pointer

Checkpointing is concurrent with execution, not a post-run stage. Storage is tenant-isolated, encrypted in transit and at rest, and independently durable from the active supplier.

Prepare, commit, CAS

# Concurrent with execution
1. Runner reaches a safe application boundary
2. PrepareCheckpoint
     validate gen ownership + runner token
     return fencing_token + staging prefix + presigned PUT
3. Upload parts → object store (S3-compatible)
4. CommitCheckpoint
     VerifySHA256 each file
     write immutable checkpoint_manifests row
     CAS advance latest_manifest (gen↑ or same gen seq↑)
     emit checkpoint.committed

A fenced generation cannot prepare or commit. Token and fence epoch must match. A stale (generation, sequence) loses the compare-and-set and surfaces as conflict. Integrity failure fences the generation and blocks restore until an operator or automated validator establishes a valid prior manifest.

The latest-manifest pointer is a small transactional record on (workload_id, stage_id), never inferred from object-list ordering. Redis is not a correctness boundary for this pointer.

What a manifest carries

Manifests are immutable, content-addressed, checksum verified, and retained under an explicit lifecycle.

{
  "workload_id": "wl_...",
  "stage_id": "train",
  "generation": 4,
  "sequence": 18,
  "files": [{"uri": "...", "sha256": "...", "bytes": 123}],
  "runtime_contract_version": "runner-v1",
  "restore": {"command": ["python", "train.py", "--resume", "..."]},
  "outputs": { "handoff": { "uri": "...", "sha256": "...", "bytes": 123 } },
  "final": false
}

• final: true completes the stage; all stages done → workload completed
• outputs enable multi-stage handoffs without shared process state
• Partial checkpoints are garbage-collected only after unreachable from every committed manifest and outside forensic retention

Policy is an RPO / RTO trade

Default checkpoint cadence is chosen from expected interruption risk, checkpoint duration and size, restore time, remaining budget, and deadline. Customers may tighten the interval; an interval that consumes too much execution time is surfaced as a route tradeoff, not silently accepted.

Internally we publish, per workload class, checkpoint RPO (maximum acceptable lost progress) and restore RTO (target time from route availability to resumed runner). The chosen cadence must be explainable as the cost/risk trade required to meet those objectives: the same objective function that feeds cost_to_complete in routing.

How this closes the loop

Market finds capacity. Route picks a path and a ceiling. Bidding holds a lease. Execution runs gen N. Checkpoint makes progress portable. When reclaim arrives, fencing kills gen N’s write authority, recovery restores from the latest verified manifest, and the same market → route → bid → execute path starts gen N+1.

Independence from supplier reclaim is the whole point. Progress survives the machine that produced it.

What ships next

Pilot checkpoints live in Postgres BYTEA / Neon for small blobs; object storage (R2/S3) is the production path once sizes grow. Restore drills under reclaim, with and without warning, are the gate before we claim automatic recovery. Field note 007 will cover fencing and the reclaim saga end to end: signed webhook persisted before Temporal signal, compensating actions at every step, and the postmortem record every recovery must leave behind.

Previous: 005 · Execution. Next in series (coming): fencing and the reclaim saga.