跳到主要内容

vLLM

vLLM 是一个本地运行的代理和推理服务器，提供与OpenAI兼容的API。由于它同时执行代理和推理，因此您不需要安装额外的推理服务器。

注意：vLLM 不支持 OpenAI 的 Function Calling （可与 AutoGen 一起使用）。不过，它正在开发中，可能在你阅读此文时已经可用。

运行此堆栈需要安装：

  1. AutoGen (安装说明)
  2. vLLM

注意：我们建议为你的技术栈使用虚拟环境，请参阅这篇文章获取指导。

安装vLLM​

在您的终端中：

pip install vllm

选择模型​

vLLM 在运行服务器时会下载新的模型。

模型来自Hugging Face，过滤后的文本生成模型列表在这里，vLLM有一个常用模型的列表。请使用完整的模型名称，例如mistralai/Mistral-7B-Instruct-v0.2。

聊天模板​

vLLM 使用预定义的聊天模板，除非模型在 Hugging Face 的配置文件中定义了聊天模板。如果聊天模板不允许像 AutoGen 中使用的 'role' : 'system' 消息，这可能会导致问题。

因此，我们将为正在使用的Mistral.AI Mistral 7B模型创建一个聊天模板，允许'user'、'assistant'和'system'角色。

创建一个文件名为autogenmistraltemplate.jinja，内容如下：

{{ bos_token }}
{% for message in messages %}
    {% if ((message['role'] == 'user' or message['role'] == 'system') != (loop.index0 % 2 == 0)) %}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif %}

    {% if (message['role'] == 'user' or message['role'] == 'system') %}
        {{ '[INST] ' + message['content'] + ' [/INST]' }}
    {% elif message['role'] == 'assistant' %}
        {{ message['content'] + eos_token}}
    {% else %}
        {{ raise_exception('Only system, user and assistant roles are supported!') }}
    {% endif %}
{% endfor %}
warning

聊天模板是针对特定模型/模型系列的。这里展示的示例适用于基于Mistral的模型，如Mistral 7B和Mixtral 8x7B。

vLLM 提供了许多示例模板，可以作为你聊天模板的起点。只需记住，模板可能需要调整以支持“系统”角色消息。

运行vLLM代理服务器​

要在终端中运行具有所选模型和我们的聊天模板的vLLM：

python -m vllm.entrypoints.openai.api_server --model mistralai/Mistral-7B-Instruct-v0.2 --chat-template autogenmistraltemplate.jinja

默认情况下，vLLM将在'http://0.0.0.0:8000'上运行。

使用vLLM与AutoGen​

现在我们有了vLLM代理服务器的URL，你可以在AutoGen中以与OpenAI或基于云的代理服务器相同的方式使用它。

由于您在本地运行此代理服务器，因此不需要API密钥。由于api_key是AutoGen中配置的必填字段，我们按照以下示例为其设置了一个虚拟值。

尽管我们在运行 vLLM 命令时指定了模型，但我们仍然需要将其放入 vLLM 的 model 值中。

from autogen import UserProxyAgent, ConversableAgent

local_llm_config={
    "config_list": [
        {
            "model": "mistralai/Mistral-7B-Instruct-v0.2", # Same as in vLLM command
            "api_key": "NotRequired", # Not needed
            "base_url": "http://0.0.0.0:8000/v1"  # Your vLLM URL, with '/v1' added
        }
    ],
    "cache_seed": None # Turns off caching, useful for testing different models
}

# Create the agent that uses the LLM.
assistant = ConversableAgent("agent", llm_config=local_llm_config,system_message="")

# Create the agent that represents the user in the conversation.
user_proxy = UserProxyAgent("user", code_execution_config=False,system_message="")

# Let the assistant start the conversation.  It will end when the user types exit.
assistant.initiate_chat(user_proxy, message="How can I help you today?")

输出：

agent (to user):

How can I help you today?

--------------------------------------------------------------------------------
Provide feedback to agent. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: Why is the sky blue?
user (to agent):

Why is the sky blue?

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
agent (to user):


The sky appears blue due to a phenomenon called Rayleigh scattering. As sunlight reaches Earth's atmosphere, it interacts with molecules and particles in the air, causing the scattering of light. Blue light has a shorter wavelength and gets scattered more easily than other colors, which is why the sky appears blue during a clear day.

However, during sunrise and sunset, the sky can appear red, orange, or purple due to a different type of scattering called scattering by dust, pollutants, and water droplets, which scatter longer wavelengths of light more effectively.

--------------------------------------------------------------------------------
Provide feedback to agent. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: and why does it turn red?
user (to agent):

and why does it turn red?

--------------------------------------------------------------------------------

>>>>>>>> USING AUTO REPLY...
agent (to user):


During sunrise and sunset, the angle of the sun's rays in the sky is lower, and they have to pass through more of the Earth's atmosphere before reaching an observer. This additional distance results in more scattering of sunlight, which preferentially scatters the longer wavelengths (red, orange, and yellow) more than the shorter wavelengths (blue and green).

The scattering of sunlight by the Earth's atmosphere causes the red, orange, and yellow colors to be more prevalent in the sky during sunrise and sunset, resulting in the beautiful display of colors often referred to as a sunrise or sunset.

As the sun continues to set, the sky can transition to various shades of purple, pink, and eventually dark blue or black, as the available sunlight continues to decrease and the longer wavelengths are progressively scattered less effectively.

--------------------------------------------------------------------------------
Provide feedback to agent. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: exit