Skip to main content

使用视觉模型

快速开始​

示例：向模型传递图像

import os 
from litellm import completion

os.environ["OPENAI_API_KEY"] = "your-api-key"

# openai 调用
response = completion(
    model = "gpt-4-vision-preview", 
    messages=[
        {
            "role": "user",
            "content": [
                            {
                                "type": "text",
                                "text": "这张图片里有什么？"
                            },
                            {
                                "type": "image_url",
                                "image_url": {
                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                                }
                            }
                        ]
        }
    ],
)

  1. 在 config.yaml 中定义视觉模型
model_list:
  - model_name: gpt-4-vision-preview # OpenAI gpt-4-vision-preview
    litellm_params:
      model: openai/gpt-4-vision-preview
      api_key: os.environ/OPENAI_API_KEY
  - model_name: llava-hf          # 自定义 OpenAI 兼容模型
    litellm_params:
      model: openai/llava-hf/llava-v1.6-vicuna-7b-hf
      api_base: http://localhost:8000
      api_key: fake-key
    model_info:
      supports_vision: True        # 将 supports_vision 设置为 True，以便 /model/info 返回此属性为 True

  1. 运行代理服务器
litellm --config config.yaml
  1. 使用 OpenAI Python SDK 进行测试
import os 
from openai import OpenAI

client = OpenAI(
    api_key="sk-1234", # 你的 litellm 代理 api 密钥
)

response = client.chat.completions.create(
    model = "gpt-4-vision-preview",  # 使用 model="llava-hf" 测试自定义 OpenAI 端点
    messages=[
        {
            "role": "user",
            "content": [
                            {
                                "type": "text",
                                "text": "这张图片里有什么？"
                            },
                            {
                                "type": "image_url",
                                "image_url": {
                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                                }
                            }
                        ]
        }
    ],
)

检查模型是否支持 vision​

使用 litellm.supports_vision(model="") -> 如果模型支持 vision 则返回 True，否则返回 False

assert litellm.supports_vision(model="gpt-4-vision-preview") == True
assert litellm.supports_vision(model="gemini-1.0-pro-vision") == True
assert litellm.supports_vision(model="gpt-3.5-turbo") == False
  1. 在 config.yaml 中定义视觉模型
model_list:
  - model_name: gpt-4-vision-preview # OpenAI gpt-4-vision-preview
    litellm_params:
      model: openai/gpt-4-vision-preview
      api_key: os.environ/OPENAI_API_KEY
  - model_name: llava-hf          # 自定义 OpenAI 兼容模型
    litellm_params:
      model: openai/llava-hf/llava-v1.6-vicuna-7b-hf
      api_base: http://localhost:8000
      api_key: fake-key
    model_info:
      supports_vision: True        # 将 supports_vision 设置为 True，以便 /model/info 返回此属性为 True
  1. 运行代理服务器
litellm --config config.yaml
  1. 调用 /model_group/info 检查你的模型是否支持 vision
curl -X 'GET' \
  'http://localhost:4000/model_group/info' \
  -H 'accept: application/json' \
  -H 'x-api-key: sk-1234'

预期响应

{
  "data": [
    {
      "model_group": "gpt-4-vision-preview",
      "providers": ["openai"],
      "max_input_tokens": 128000,
      "max_output_tokens": 4096,
      "mode": "chat",
      "supports_vision": true, # 👈 supports_vision 为 true
      "supports_function_calling": false
    },
    {
      "model_group": "llava-hf",
      "providers": ["openai"],
      "max_input_tokens": null,
      "max_output_tokens": null,
      "mode": null,
      "supports_vision": true, # 👈 supports_vision 为 true
      "supports_function_calling": false
    }
  ]
}