WeightDecayExclusion

class WeightDecayExclusion(exclude_bias_from_wd=True, exclude_norm_from_wd=True, exclude_gamma_from_wd=True, exclude_token_from_wd=True, exclude_ndim_1_from_wd=True)

Bases: HookBase

Rewrite optimizer param groups to exclude selected params from weight decay.

Runs once in before_train: walks the optimizer’s existing parameter groups (preserving any layer-wise learning rates) and splits each into a decay group and a no-decay group based on parameter name/shape. Excluded parameters get weight_decay=0.0 and are flagged apply_wd=False; kept parameters are flagged apply_wd=True. These flags are honoured by WeightDecayScheduler. Logs the resulting group counts. Registered as WeightDecayExclusion.

Parameters:

• exclude_bias_from_wd (bool) – Exclude parameters whose name ends in .bias. Defaults to True.

• exclude_norm_from_wd (bool) – Exclude parameters with "norm" in their name. Defaults to True.

• exclude_gamma_from_wd (bool) – Exclude parameters with "gamma" in their name. Defaults to True.

• exclude_token_from_wd (bool) – Exclude parameters with "token" in their name. Defaults to True.

• exclude_ndim_1_from_wd (bool) – Exclude any 1-D parameter (biases, norm/scale vectors, etc.). Defaults to True.

Note

Mutates trainer.optimizer.param_groups in place before training. Pair with WeightDecayScheduler to schedule weight decay only on the kept (apply_wd=True) groups; place this hook before the scheduler.

Example

Add to cfg.hooks before the weight-decay scheduler; once in before_train it rewrites the optimizer’s parameter groups:

hooks = [
    dict(type="WeightDecayExclusion"),
    dict(type="WeightDecayScheduler", base_value=0.04,
         final_value=0.2),
]
# → splits each optimizer param group into an apply_wd=True group and an
#   apply_wd=False group (weight_decay=0.0) holding biases/norm/gamma/
#   token/1-D params; logs the resulting with/without-wd group counts

before_train()