Decide what continuation means
There are two different goals: recovering approximately where training stopped and reproducing the uninterrupted trajectory. The second requires stricter control of randomness, data order and software versions. Write down which one the experiment needs.
PyTorch describes saving a general checkpoint with model and optimizer state plus progress such as the epoch. Add the application-specific state your loop needs, including scheduler progress or data position when relevant. A checkpoint format should have a version, just like a data format.
References: PyTorch model saving and loading
Choose the interval from recovery cost
Suppose an illustrative job writes a checkpoint every 20 minutes and takes one minute to write it. You spend roughly a minute of each interval on saving, while a failure can discard almost a full interval of unsaved work. Shortening the interval reduces loss but adds storage traffic.
Measure the save operation on the storage system you intend to use. Saving locally can look fast while leaving everything on the machine you are about to lose. Track the time until the durable copy is complete.
Run a failure drill before buying a long window
Start a short training run, create a durable checkpoint, then terminate the worker deliberately. Launch a fresh worker from the saved container version and restore the checkpoint. Check progress, optimizer settings and the next batch to confirm the resume path.
Use unique run identifiers and atomic checkpoint publication so a reader does not pick up a half-written file. Keep at least one previous complete checkpoint. Test what happens when the latest file is corrupt or unavailable.
Treat interruption notices as extra time
AWS documents Spot interruption notices with a two-minute warning for most interruptions, with a different behavior for hibernation. That is useful for shutdown handling, but a large checkpoint may take longer to copy. Save periodically during normal operation.
A retry should not duplicate an external side effect, such as publishing the same result or charging a downstream task twice. Separate durable progress from disposable worker state, and make result publication idempotent.
References: EC2 Spot interruption notices
Questions readers ask
Is saving model weights enough to resume training?
It may be enough for inference or a fresh training start. Continuing the same training run commonly needs optimizer and progress state as well.
Where should checkpoints live?
On durable storage that survives loss of the worker, with a restore path tested from another machine.
Sources and editorial notes
Primary documentation reviewed October 9, 2026. Provider descriptions reflect published documentation. Recommendations are our analysis; worked figures are illustrative unless explicitly stated otherwise. No independent provider benchmark or uptime ranking is claimed.
Read the editorial policy. NeuraMesh is a prototype, not live verified GPU inventory.