ResourceUtilizationLogger

class ResourceUtilizationLogger(log_frequency=10, prefix='resources', log_per_gpu=False, log_cpu=True, log_system_memory=True, per_process=True)

Bases: HookBase

Log GPU and CPU/RAM utilization to the writer over the course of training.

In before_train it probes optional dependencies (psutil for CPU/RAM, pynvml for system-wide GPU stats), warning/falling back to torch.cuda when unavailable. Runs in after_step every log_frequency steps on rank 0: collects GPU memory (and, system-wide, utilization %) plus CPU and system/process memory metrics and writes them to the writer under {prefix}/.... after_train shuts down NVML if it was initialized. Registered as ResourceUtilizationLogger.

Parameters:

• log_frequency (int) – Log metrics every this many steps. Defaults to 10.

• prefix (str) – Namespace prefix for the logged keys. Defaults to "resources".

• log_per_gpu (bool) – In system-wide mode, log metrics for every visible GPU instead of just the local one. Defaults to False.

• log_cpu (bool) – Log CPU utilization metrics. Defaults to True.

• log_system_memory (bool) – Log RAM usage metrics. Defaults to True.

• per_process (bool) – If True, report only this process’s CPU/RAM and torch.cuda GPU memory (recommended on shared nodes); if False, report system-wide metrics (using pynvml for GPU when available). Defaults to True.

Note

Logs on rank 0 only and no-ops when trainer.writer is absent/None. per_process=True ignores log_per_gpu/pynvml and uses torch.cuda memory counters for this process.

Example

Add to cfg.hooks; every log_frequency steps (rank 0) it writes GPU/CPU/RAM utilization to the writer:

hooks = [dict(type="ResourceUtilizationLogger", log_frequency=50)]
# → every 50 steps writes "resources/gpu_memory_allocated_gb",
#   "resources/gpu_memory_reserved_gb", "resources/process_cpu_percent",
#   "resources/process_rss_gb", … to the writer (per-process by default)

after_step()

after_train()

before_train()