lightrft.trainer.kl_controller

class lightrft.trainer.kl_controller.AdaptiveKLController(init_kl_coef: float, target: float, horizon: int)

Bases: object

Adaptive KL controller for PPO training.

Implements the adaptive KL penalty coefficient adjustment described in: “Fine-Tuning Language Models from Human Preferences” (https://arxiv.org/pdf/1909.08593.pdf)

This controller dynamically adjusts the KL penalty coefficient based on how the current KL divergence compares to a target value, helping maintain stable training while preventing the policy from deviating too far from the reference.

update(current: float, n_steps: int) → None

Update KL coefficient using adaptive algorithm.

Parameters:

• current (float) – Current KL divergence value.

• n_steps (int) – Number of training steps taken.

class lightrft.trainer.kl_controller.FixedKLController(kl_coef: float)

Bases: object

Fixed KL controller that maintains a constant KL penalty coefficient.

Unlike the adaptive controller, this keeps the KL coefficient fixed throughout training, providing consistent regularization strength.

update(current: float, n_steps: int) → None

Update KL controller state (no-op for fixed KL).

Parameters:

• current (float) – Current KL divergence value (unused).

• n_steps (int) – Number of training steps (unused).