推测性解码

面向用户的API，用于将模型转换为modelopt.torch.speculative.MedusaModel。

函数

convert

将基础模型转换为推测解码模型的主要转换函数。

convert(model, mode)

将基础模型转换为推测解码模型的主要转换函数。

Parameters:

• model (Module) – 要使用的基础模型。

• mode (_ModeDescriptor | str | List[_ModeDescriptor | str] | List[Tuple[str, Dict[str, Any]]]) –

  A (list of) string(s) or Mode(s) or a list of tuples containing the mode and its config indicating the desired mode(s) (and configurations) for the convert process. Modes set up the model for different algorithms for model optimization. The following modes are available:

  • "medusa": The model will be converted into a medusa model with added medusa head. The mode’s config is described in MedusaConfig.

  • "eagle": The model will be converted into a eagle model with added eagle weights. The mode’s config is described in EagleConfig.

  If the mode argument is specified as a dictionary, the keys should indicate the mode and the values specify the per-mode configuration.

Returns:

MedusaModel 或 EagleModel 的一个实例或其子类。

Return type:

模块