Command-line reference

pimm exposes six commands. Training-config values are intentionally not typed launcher flags: put them after a bare -- as key=value tokens.

Command	Use it for	Does not do
pimm ls	list Python training configs below an optional config directory	inspect launch recipes or site profiles
pimm launch	run on the current node or inside an existing allocation	submit a scheduler job
pimm submit	submit through Slurm with Submitit, or request an interactive allocation	evaluate a completed experiment
pimm watchdog	inspect or remove supervisors for chained interactive runs	install a supervisor directly; pimm submit does that
pimm trace	summarize or export per-rank structured trace logs	replace experiment metrics or an operator-level profiler
pimm export	consolidate model weights into a portable directory and optionally upload it	save resumable trainer state

Run commands through the checkout’s locked environment:

uv run pimm --help
uv run pimm ls panda/pretrain
uv run pimm launch --help
uv run pimm submit --help
uv run pimm watchdog --help
uv run pimm trace --help
uv run pimm export --help

The override boundary

uv run pimm launch \
  --train.config tests/tiny_semseg \
  --resources.nproc-per-node 1 \
  --run.name smoke \
  -- \
  epoch=1 \
  batch_size=4 \
  data.train.max_len=32

Everything before -- configures execution. Everything after it patches the Python experiment config. Post-separator tokens must contain = and must not start with --; values are parsed with YAML scalar/list/map syntax. Dotted keys patch nested mappings.

pimm launch

pimm launch [launcher options] -- [EXPERIMENT.KEY=VALUE ...]

launch always selects the local executor, even if a different --executor value is passed. “Local” means run in the current shell context: this can be a workstation, one compute node, or an allocation obtained separately.

# Resolve without starting training
uv run pimm launch \
  --train.config panda/pretrain/pretrain-sonata-v1m1-pilarnet-smallmask \
  --resources.nproc-per-node 4 \
  --dry-run

The local site defaults resources.nproc_per_node to auto, which asks scripts/train.sh to use all visible GPUs. Set an integer for a predictable world size.

pimm submit

pimm submit [launcher and Slurm options] -- [EXPERIMENT.KEY=VALUE ...]

Always pass --site explicitly in scripts. The parser currently defaults submit to the repository’s s3df profile, which is convenient for that site but not portable.

uv run pimm submit \
  --site nersc \
  --resources.nnodes 2 \
  --resources.nproc-per-node 4 \
  --resources.time 02:00:00 \
  --train.config panda/pretrain/pretrain-sonata-v1m1-pilarnet-smallmask \
  --dry-run

--interactive uses salloc ... srun ... instead of queueing a Submitit batch job. A single slot runs in the terminal. With --chain.jobs N greater than one, pimm installs a scron watchdog that supervises sequential, resuming interactive slots; the site must provide scrontab and a suitable login-node QOS.

If --submit.host HOST is set, pimm SSHes to that host and invokes pimm submit there. --no-remote prevents that second hop; pimm adds it internally to avoid recursion.

Launcher option groups

Hyphens in flags map to underscores in the launch dataclasses. Boolean flags have positive and negative forms, for example --run.timestamp and --run.no-timestamp.

Group	Flags users commonly set
selection/output	--site, --recipe, --dry-run, --output
paths	--paths.repo-root, --paths.exp-root
resources	--resources.scheduler, --resources.nnodes, --resources.nproc-per-node, --resources.cpus-per-proc, --resources.time, --resources.mem, --resources.account, --resources.partition, --resources.qos, --resources.constraint, --resources.dependency, --resources.output, --resources.error, --resources.gpu-directive, --resources.job-name, --resources.signal-delay-s
run identity	--run.name, --run.[no-]timestamp, --run.wandb-name, --run.wandb-project, --run.wandb-api-key
training handoff	--train.config, --train.weight, --train.[no-]resume, --train.[no-]code-copy, --train.python
container	--container.runtime, --container.image, --container.module, --container.binds, --container.repo-mount, --container.unset-env, --container.interpreter
environment bootstrap	--setup
requeue	--chain.jobs, --chain.[no-]resume-first, --chain.wandb-group, --chain.wandb-job-type
interactive watchdog	--watchdog.qos, --watchdog.interval-min, --watchdog.time
rendezvous	--rdzv.endpoint, --rdzv.id, --rdzv.backend
remote submit	--submit.host, --submit.folder, --no-remote

--recipe is a YAML path relative to the repository root, for example --recipe launch/runs/e050_tail.yaml. --output PATH writes the redacted local script, Submitit manifest, or interactive salloc command. It does not redirect training logs.

Three advanced mappings are deliberately absent from Tyro’s flag list: env, train.options, and resources.scheduler_options. Put them in launch YAML. Prefer the post--- tail over train.options for one-off experiment overrides.

resources.scheduler accepts only local and slurm. pimm launch always runs locally and ignores scheduler-only resource fields; pimm submit requires resources.scheduler: slurm.

Caution

--executor is visible for schema compatibility, but the command verb wins: launch forces local and submit forces batch or interactive Slurm. Choose the verb rather than setting this field.

pimm watchdog

Chained interactive submission installs the supervisor; this command manages installed supervisors without loading a launch config:

uv run pimm watchdog ls
uv run pimm watchdog rm <run-name>
uv run pimm watchdog rm <run-name> --scancel

rm removes the run’s scrontab entry. --scancel also cancels the active watchdog driver and interactive allocation by Slurm job name.

pimm trace

pimm trace summarize RUN_DIR
pimm trace export RUN_DIR [-o OUTPUT]

RUN_DIR can be an experiment directory or its structured_logs/ directory. summarize reports the last step and event for every traced process, unmatched spans, and p50/p95/maximum phase durations. export merges base and rotated JSONL segments into a Perfetto-compatible Chrome Trace document. Its default destination is <run>/analysis/structured_trace.json.

Structured logging is disabled by default. Enable it through the experiment config:

uv run pimm launch \
  --train.config <config> \
  -- structured_logging.enabled=true

See Structured logging for the event schema, storage controls, and failure-diagnosis workflow.

pimm export

pimm export [checkpoint] [output_dir] [options]

With --run-dir, the checkpoint positional argument defaults to last and is resolved under <run-dir>/model/. The output directory is required unless --dry-run is set.

uv run pimm export \
  --run-dir exp/panda/semseg/my-run \
  last \
  artifacts/panda-semantic

Option	Meaning
--run-dir DIR	infer checkpoint names and the run config from an experiment directory
--config FILE	provide a config when it cannot be inferred
--model-card FILE	copy Markdown into the export as README.md
--device DEVICE	device used while consolidating tensors; default cpu
--no-safe-serialization	write model.bin instead of model.safetensors
--push-to-hub ORG/REPO	upload the completed export
--public	create a public Hub repository; default is private
--token TOKEN	explicit Hub token; prefer hf auth login or HF_TOKEN
--dry-run	print resolved paths and exit without writing

See Export and publication for artifact semantics and a strict round-trip check.

Evaluation is currently a script

There is no pimm evaluate subcommand. The supported standalone path is:

uv run sh scripts/test.sh \
  -c panda/semseg/semseg-pt-v3m2-pilarnet-ft-5cls-fft \
  -n <experiment-name> \
  -w model_best

The script accepts -p, -c, -n, -w, -g, and -m, but its current execution path is single-process. Its -c value selects a config from the current checkout, not the saved run’s config.py. Read Evaluate an experiment for the exact saved-config command before reporting metrics.

Exit-before-spend checks

Use these before a job with data or GPUs:

uv run pimm launch --train.config <config> --dry-run
uv run pimm submit --site <site> --train.config <config> --dry-run
uv run pimm export --run-dir <run> --dry-run

Dry runs validate cheap path/config conditions and render the command. They do not construct the dataset or model, test credentials, inspect scheduler state, or prove that a container/data mount is reachable on compute nodes.