Command line#

pimm is installed with the package. Run it from the checkout as uv run pimm <command>; every command takes -h.

Command

Does

pimm ls

list training configs

pimm launch

run training locally or inside an allocation

pimm submit

submit training to Slurm through submitit

pimm watchdog

manage supervisors for chained interactive runs

pimm trace

inspect or export structured trace logs

pimm export

export model weights (optionally push to the HF Hub)

pimm ls#

List training configs below an optional directory. The names are paths under configs/ without .py, the form --train.config takes.

usage: pimm ls [DIRECTORY ...]

pimm launch#

Run pimm training locally or inside an existing allocation.

usage: pimm launch [FLAGS] [-- KEY=VALUE ...]

Flags set the launch config; KEY=VALUE tokens after a bare -- override the experiment config (see Configs and overrides). Defaults below come from launch/defaults.yaml and the local site profile; --site and --recipe load others.

General#

Flag

Type

Meaning

Default

--site

str

—

local

--executor

{local,batch,interactive}

Execution mode; the launch/submit command sets this.

local

--interactive / --no-interactive

bool

Use a blocking Slurm allocation for pimm submit.

False

--setup

str ...

Shell lines that bootstrap the environment (e.g. mamba activate pimm, module load ...) before pimm runs. Applied in two places: on the compute node before scripts/train.sh (inside the container, if one is configured), and on the login host before a remote pimm submit (when submit.host is set). Runs regardless of container.runtime.

—

--recipe

str

—

None

--dry-run / --no-dry-run

bool

—

False

--output

str

—

None

--paths.*#

Flag

Type

Meaning

Default

--paths.repo-root

str

—

.

--paths.exp-root

str

—

'{repo_root}/exp'

--resources.*#

Flag

Type

Meaning

Default

--resources.scheduler

{local,slurm}

—

local

--resources.nnodes

int

—

1

--resources.nproc-per-node

int | {auto}

GPUs/processes per node; ‘auto’ is local-only.

auto

--resources.cpus-per-proc

int

—

8

--resources.account

str

—

None

--resources.partition

str

—

None

--resources.qos

str

—

None

--resources.constraint

str

—

None

--resources.dependency

str

—

None

--resources.time

str | int

—

24:00:00

--resources.mem

str

—

None

--resources.output

str

—

slurm_logs/slurm-%j.out

--resources.error

str

—

None

--resources.gpu-directive

{gres,gpus-per-node}

—

gres

--resources.job-name

str

—

None

--resources.signal-delay-s

int

—

120

--run.*#

Flag

Type

Meaning

Default

--run.name

str

—

None

--run.timestamp / --run.no-timestamp

bool

—

True

--run.wandb-name

str

—

None

--run.wandb-project

str

—

None

--run.wandb-api-key

str

—

None

--train.*#

Flag

Type

Meaning

Default

--train.config

str

—

None

--train.weight

str

—

None

--train.resume / --train.no-resume

bool

—

False

--train.code-copy / --train.no-code-copy

bool

—

True

--train.python

str

—

None

--container.*#

Flag

Type

Meaning

Default

--container.runtime

{none,singularity,apptainer,shifter,docker}

—

none

--container.image

str

—

None

--container.module

str

—

None

--container.binds

str ...

—

—

--container.repo-mount

str

—

/opt/pimm/src

--container.unset-env

str ...

—

—

--container.interpreter

str

Absolute interpreter path inside the image (e.g. /opt/conda/bin/python). When set, used as train.sh -p so the image’s python wins regardless of a host conda leak. None -> fall back to the runtime default / sys.executable.

None

--submit.*#

Flag

Type

Meaning

Default

--submit.host

str

—

None

--submit.folder

str

—

None

--chain.*#

Flag

Type

Meaning

Default

--chain.jobs

int

—

1

--chain.resume-first / --chain.no-resume-first

bool

—

False

--chain.wandb-group

str

—

None

--chain.wandb-job-type

str

—

train-job

--watchdog.*#

Flag

Type

Meaning

Default

--watchdog.qos

str

—

cron

--watchdog.interval-min

int

—

5

--watchdog.time

str

—

1-00:00:00

--rdzv.*#

Flag

Type

Meaning

Default

--rdzv.endpoint

str

—

None

--rdzv.id

str

—

None

--rdzv.backend

str

—

None

pimm submit#

Submit pimm training to Slurm through submitit.

usage: pimm submit [FLAGS] [-- KEY=VALUE ...]

pimm submit takes every pimm launch flag, with --site defaulting to s3df, plus:

Flag

Meaning

Default

--no-remote / --no-no-remote

—

False

pimm watchdog#

Manage watchdogs that supervise chained interactive runs.

usage: pimm watchdog [-h] {ls,rm} ...

pimm watchdog ls#

List installed watchdogs and their state

usage: pimm watchdog ls [-h]

pimm watchdog rm#

Remove a run’s watchdog (stop supervising)

usage: pimm watchdog rm [-h] [--scancel] run

Argument

Meaning

Default

RUN

run name (as shown by pimm watchdog ls)

—

--scancel

also scancel the running driver and its interactive slot

—

pimm trace#

Inspect and export pimm structured trace logs.

usage: pimm trace [-h] {summarize,export} ...

pimm trace summarize#

Summarize rank progress, open spans, and phase durations.

usage: pimm trace summarize [-h] run_dir

Argument

Meaning

Default

RUN_DIR

Experiment directory or its structured_logs directory.

—

pimm trace export#

Merge per-rank JSONL logs into a Perfetto-compatible trace.

usage: pimm trace export [-h] [-o OUTPUT] run_dir

Argument

Meaning

Default

RUN_DIR

Experiment directory or its structured_logs directory.

—

-o / --output

Output JSON path (default: RUN_DIR/analysis/structured_trace.json).

—

pimm export#

Export pimm checkpoints to portable pretrained model directories.

usage: pimm export [-h] [--run-dir RUN_DIR] [--config CONFIG]
                   [--model-card MODEL_CARD] [--push-to-hub PUSH_TO_HUB]
                   [--public] [--token TOKEN] [--device DEVICE]
                   [--no-safe-serialization] [--dry-run]
                   [checkpoint] [output_dir]

Argument

Meaning

Default

CHECKPOINT

Checkpoint path, DCP directory, or checkpoint name under <run-dir>/model. Defaults to last when –run-dir is used.

last

OUTPUT_DIR

Directory to write exported artifacts.

—

--run-dir

Experiment directory containing config.py and model/.

—

--config

Config file to use when it cannot be inferred from the run dir.

—

--model-card

README.md text file to include as the model card.

—

--push-to-hub

Optional Hugging Face Hub repo id to upload after export.

—

--public

Create a public Hub repo (default: private).

—

--token

Hugging Face token for upload.

—

--device

Device used while consolidating checkpoint tensors.

cpu

--no-safe-serialization

Write model.bin (torch pickle) instead of model.safetensors.

—

--dry-run

Print resolved paths and exit without writing.

—