vLLM

使用vLLM有两种模式：本地模式和远程模式。让我们从前者开始，它要求本地具备可用的CUDA环境。

安装 vLLM

pip install vllm
或者如果您想要编译，可以从源码编译

Orca-7b 补全示例

%pip install llama-index-llms-vllm

import os

os.environ["HF_HOME"] = "model/"

from llama_index.llms.vllm import Vllm, VllmServer

llm = Vllm(
    model="microsoft/Orca-2-7b",
    tensor_parallel_size=4,
    max_new_tokens=100,
    vllm_kwargs={"swap_space": 1, "gpu_memory_utilization": 0.5},
)

llm.complete("[INST]You are a helpful assistant[/INST] What is a black hole ?")

LLama-2-7b 补全示例

llm = Vllm(
    model="codellama/CodeLlama-7b-hf",
    dtype="float16",
    tensor_parallel_size=4,
    temperature=0,
    max_new_tokens=100,
    vllm_kwargs={
        "swap_space": 1,
        "gpu_memory_utilization": 0.5,
        "max_model_len": 4096,
    },
)

llm.complete("import socket\n\ndef ping_exponential_backoff(host: str):")

Mistral 聊天 7b 完成示例

llm = Vllm(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    dtype="float16",
    tensor_parallel_size=4,
    temperature=0,
    max_new_tokens=100,
    vllm_kwargs={
        "swap_space": 1,
        "gpu_memory_utilization": 0.5,
        "max_model_len": 4096,
    },
)

Vllm mock initialized

llm.complete(" What is a black hole ?")

通过HTTP调用vLLM

在此模式下，无需安装 vllm 模型，也无需在本地配置 CUDA。要设置 vLLM API，您可以按照此处的指南进行操作。注意：llama-index-llms-vllm 模块是 vllm.entrypoints.api_server 的客户端，它仅作为演示使用。
如果 vLLM 服务器以 vllm.entrypoints.openai.api_server 作为 OpenAI 兼容服务器启动，或通过 Docker 启动，您需要从 llama-index-llms-openai-like 模块中获取 OpenAILike 类。

完成响应

from llama_index.core.llms import ChatMessage

llm = VllmServer(
    api_url="http://localhost:8000/generate", max_new_tokens=100, temperature=0
)

llm.complete("what is a black hole ?")

message = [ChatMessage(content="hello", role="user")]
llm.chat(message)

流式响应

list(llm.stream_complete("what is a black hole"))[-1]

message = [ChatMessage(content="what is a black hole", role="user")]
[x for x in llm.stream_chat(message)][-1]

异步响应

import asyncio

await llm.acomplete("What is a black hole")

await llm.achat(message)

[x async for x in await llm.astream_complete("what is a black hole")][-1]

[x async for x in await llm.astream_chat(message)][-1]