Batch Jobs Overview
Batch Jobs let you run Jupyter notebooks as unattended Kubernetes Job resources, so execution continues in the background even after you close your browser.
How Batch Jobs Work
Section titled “How Batch Jobs Work”Batch Jobs use Papermill to execute notebooks programmatically:
Notebook Path → Papermill → Kubernetes Job → Output Notebook → Shared Workspace- You pick a notebook from the project’s shared workspace
- Optionally provide parameters to inject at runtime
- The platform creates a Kubernetes
Jobthat runs Papermill - The output notebook is written back to the shared workspace under
batch_outputs/<jobID>/ - You can access the output notebook from JupyterLab or download it via the UI
Shared Workspace
Section titled “Shared Workspace”Each project has a shared workspace — a ReadWriteMany PersistentVolumeClaim (10Gi) mounted at /home/jovyan/work/shared/ in all notebook instances. Batch job outputs are written here, making them immediately visible in JupyterLab without any manual transfer.
Use Cases
Section titled “Use Cases”- Scheduled reports — run a data aggregation notebook on demand or via the API
- Parameterized experiments — run the same notebook with different inputs (hyperparameters, date ranges, datasets)
- Data pipelines — chain batch jobs together to process data end-to-end
- Model evaluation — run evaluation notebooks against new model checkpoints
Job Lifecycle
Section titled “Job Lifecycle”| Status | Meaning |
|---|---|
pending | Job has been submitted and is waiting to be scheduled |
running | Notebook is currently executing |
succeeded | Notebook completed successfully |
failed | Notebook execution encountered an error |
Jobs are automatically cleaned up from Kubernetes after 24 hours (ttlSecondsAfterFinished: 86400), but the output notebook and job record in the UI persist.