eval

CarlaBenchmarkEvaluator

class core.eval.CarlaBenchmarkEvaluator(cfg: Dict, env: BaseEnvManager, policy: Any, tb_logger: SummaryWriter | None = None, exp_name: str | None = 'default_experiment', instance_name: str | None = 'benchmark_evaluator')

Evaluator used to evaluate a policy with Carla benchmark evaluation suites. It uses several environments in EnvManager to evaluate policy. For every suites provided by user, evaluator will first find and store all available reset params from Benchmark files and store them in a queue such that each reset param is evaluated once and only once. The evaluation results are stored in a ‘.csv’ file with reward, success and failure status and reset param of the episode.

Note:

Env manager must run WITHOUT auto reset.

Arguments:

• cfg (Dict): Config dict.

• env (BaseEnvManager): Env manager used to evaluate.

• policy (Any): Policy to evaluate. Must have forward method.

• tb_logger (SummaryWriter, optional): Tensorboard writter to store values in tensorboard. Defaults to None.

• exp_name (str, optional): Name of the experiments. Used to build logger. Defaults to ‘default_experiment’.

• instance_name (str, optional): Name of the evaluator. Used to build logger. Defaults to ‘benchmark_evaluator’.

Interfaces:

reset, eval, close, should_eval

Properties:

• env (BaseEnvManager): Env manager with several environments used to evaluate.

• policy (Any): Policy instance to interact with envs.

close() → None

Close the collector and the env manager if not closed.

eval(save_ckpt_fn: Callable | None = None, train_iter: int = -1, envstep: int = -1, policy_kwargs: Dict | None = None, n_episode: int | None = None) → float

Run evaluation with provided policy arguments. It will evaluate all available episodes of the benchmark suite unless episode_per_suite is set in config.

Arguments:

• save_ckpt_fn (Callable, optional): Function to save ckpt. Will be called if at best performance.

  Defaults to None.

• train_iter (int, optional): Current training iterations. Defaults to -1.

• envstep (int, optional): Current env steps. Defaults to -1.

• policy_kwargs (Dict, optional): Additional arguments in policy forward. Defaults to None.

• n_episode: (int, optional): Episodes to eval. By default it is set in config.

Returns:

Tuple[bool, float]: Whether reach stop value and success rate.

reset() → None

Reset evaluator and policies.

should_eval(train_iter: int) → bool

Judge if the training iteration is at frequency value to run evaluation.

Arguments:

• train_iter (int): Current training iteration

Returns:

bool: Whether should run iteration

SingleCarlaEvaluator

class core.eval.SingleCarlaEvaluator(cfg: Dict, env: Any, policy: Any, tb_logger: SummaryWriter | None = None, exp_name: str | None = 'default_experiment', instance_name: str | None = 'single_evaluator')

Carla evaluator used to evaluate a single environment. It is mainly used to visualize the evaluation results. It uses a environment in DI-engine form and can be rendered in the runtime.

Arguments:

• cfg (Dict): Config dict

• env (Any): Carla env, should be in DI-engine form

• policy (Any): the policy to pe evaluated

• exp_name (str, optional): Name of the experiments. Used to build logger. Defaults to ‘default_experiment’.

• instance_name (str, optional): Name of the evaluator. Used to build logger. Defaults to ‘single_evaluator’.

Interfaces:

reset, eval, close

Properties:

• env (BaseDriveEnv): Environment used to evaluate.

• policy (Any): Policy instance to interact with envs.

close() → None

Close evaluator. It will close the EnvManager

eval(reset_param: Dict | None = None) → float

Running one episode evaluation with provided reset params.

Arguments:

• reset_param (Dict, optional): Reset parameter for environment. Defaults to None.

Returns:

bool: Whether evaluation succeed.

SerialEvaluator

class core.eval.SerialEvaluator(cfg: Dict, env: BaseEnvManager, policy: Any, tb_logger: SummaryWriter | None = None, exp_name: str | None = 'default_experiment', instance_name: str | None = 'serial_evaluator')

Evaluator used to serially evaluate a policy for defined times. It is mainly used when training a policy to get the evaluator performance frequently and store the best iterations. Different from serial evaluator in DI-engine, this evaluator compares the performance of iterations by the success rate rather than rewards. You can provide a tensorboard logger to save scalars when training.

Note:

Env manager must run WITH auto reset.

Arguments:

• cfg (Dict): Config dict.

• env (BaseEnvManager): Env manager used to evaluate.

• policy (Any): Policy to evaluate. Must have forward method.

• tb_logger (SummaryWriter, optional): Tensorboard writter to store values in tensorboard. Defaults to None.

• exp_name (str, optional): Name of the experiments. Used to build logger. Defaults to ‘default_experiment’.

• instance_name (str, optional): [description]. Defaults to ‘serial_evaluator’.

Interfaces:

reset, eval, close, should_eval

Properties:

• env (BaseEnvManager): Env manager with several environments used to evaluate.

• policy (Any): Policy instance to interact with envs.

close() → None

Close the collector and the env manager if not closed.

eval(save_ckpt_fn: Callable | None = None, train_iter: int = -1, envstep: int = -1, policy_kwargs: Dict | None = None, n_episode: int | None = None) → Tuple[bool, float]

Run evaluation with provided policy arguments. It will evaluate all available episodes of the benchmark suite unless episode_per_suite is set in config.

Arguments:

• save_ckpt_fn (Callable, optional): Function to save ckpt. Will be called if at best performance.

  Defaults to None.

• train_iter (int, optional): Current training iterations. Defaults to -1.

• envstep (int, optional): Current env steps. Defaults to -1.

• policy_kwargs (Dict, optional): Additional arguments in policy forward. Defaults to None.

• n_episode: (int, optional): Episodes to eval. By default it is set in config.

Returns:

Tuple[bool, float]: Whether reach stop value and success rate.

reset() → None

Reset evaluator and policies.

should_eval(train_iter: int) → bool

Judge if the training iteration is at frequency value to run evaluation.

Arguments:

• train_iter (int): Current training iteration

Returns:

bool: Whether should run iteration