Container Deployment

The weather pipeline runs inside a container on HPC clusters (Apptainer) and VMs/workstations (Docker). The image contains only the conda environment and system tools — source code is bind-mounted at runtime.

This means:

Code changes require only an scp and re-run (seconds).
Dependency changes (weather_env.yml) require an image rebuild (~10 min).

Image Contents

Layer	What
Base	`continuumio/miniconda3:latest`
System tools	`lbzip2`, `curl`
Conda env	`weather_env` from `weather_env.yml` (xarray, cfgrib, dask, netCDF4, eccodes, pvlib, etc.)
Source code	Not included — bind-mounted at runtime

Apptainer (HPC / Snellius)

Build (only when weather_env.yml changes):

cd ~/buem
bash src/buem/weather/shell_scripts/build_container.sh def

This submits a SLURM job to the pbuild partition. The SIF image appears at ~/buem/buem_weather.sif when the job finishes.

Run:

sbatch src/buem/weather/shell_scripts/run_pipeline_container.sh --months 1

The run script bind-mounts:

Host path	Container path	Purpose
`~/buem/src`	`/app/src`	Python source code
`~/buem_weather`	`/data`	Download, decompress, and output directories

Update code without rebuilding:

# From local machine:
scp -r src/buem/weather ssahoo@snellius.surf.nl:~/buem/src/buem/

# On Snellius:
sbatch src/buem/weather/shell_scripts/run_pipeline_container.sh --months 1

Interactive debugging:

apptainer shell --bind ~/buem/src:/app/src ~/buem/buem_weather.sif
# Inside the container:
python -m buem.weather info
python -m buem.weather validate

Environment variables (override defaults):

Variable	Default	Description
`COSMO_SIF_PATH`	`~/buem/buem_weather.sif`	Path to the SIF image
`COSMO_WORK_DIR`	`~/buem_weather`	Host data directory
`COSMO_SRC_DIR`	`~/buem/src`	Host source directory

Docker (VMs / Workstations)

Build:

docker build -f src/buem/weather/Dockerfile.weather -t buem-weather .

Run (mount source + data):

docker run --rm \
    -v $(pwd)/src:/app/src \
    -v ~/buem_weather:/data \
    -e COSMO_WORK_DIR=/data \
    buem-weather \
    python -m buem.weather run --months 1

Push to registry (for apptainer pull on HPC):

docker tag buem-weather docker.io/<user>/buem-weather:latest
docker push docker.io/<user>/buem-weather:latest

Build Methods

build_container.sh supports four methods:

Method	Command	Where
`def`	Build SIF from `weather.def`	Snellius (pbuild) or local with root
`docker`	Build Docker image	Local (Docker required)
`convert`	Docker image → SIF	Local (Apptainer required)
`pull`	Registry → SIF	Snellius login node (no root)

Key Files

File	Purpose
`weather.def`	Apptainer definition (deps-only image)
`Dockerfile.weather`	Docker multi-stage build (deps-only image)
`shell_scripts/build_container.sh`	Build script (4 methods)
`shell_scripts/run_pipeline_container.sh`	SLURM job script for containerised runs
`weather_env.yml`	Conda environment specification