Skip to main content

Nvidia NIM

https://docs.api.nvidia.com/nim/reference/

tip

我们支持所有Nvidia NIM模型，只需在发送litellm请求时将model=nvidia_nim/<任意nvidia_nim上的模型>设置为前缀

API密钥​

# 环境变量
os.environ['NVIDIA_NIM_API_KEY']

示例用法​

from litellm import completion
import os

os.environ['NVIDIA_NIM_API_KEY'] = ""
response = completion(
    model="nvidia_nim/meta/llama3-70b-instruct",
    messages=[
        {
            "role": "user",
            "content": "今天波士顿的天气如何，以华氏度为单位？",
        }
    ],
    temperature=0.2,        # 可选
    top_p=0.9,              # 可选
    frequency_penalty=0.1,  # 可选
    presence_penalty=0.1,   # 可选
    max_tokens=10,          # 可选
    stop=["\n\n"],          # 可选
)
print(response)

示例用法 - 流式传输​

from litellm import completion
import os

os.environ['NVIDIA_NIM_API_KEY'] = ""
response = completion(
    model="nvidia_nim/meta/llama3-70b-instruct",
    messages=[
        {
            "role": "user",
            "content": "今天波士顿的天气如何，以华氏度为单位？",
        }
    ],
    stream=True,
    temperature=0.2,        # 可选
    top_p=0.9,              # 可选
    frequency_penalty=0.1,  # 可选
    presence_penalty=0.1,   # 可选
    max_tokens=10,          # 可选
    stop=["\n\n"],          # 可选
)

for chunk in response:
    print(chunk)

用法 - 嵌入​

import litellm
import os

response = litellm.embedding(
    model="nvidia_nim/nvidia/nv-embedqa-e5-v5",               # 添加`nvidia_nim/`前缀以便litellm知道路由到Nvidia NIM
    input=["good morning from litellm"],
    encoding_format = "float", 
    user_id = "user-1234",

    # Nvidia NIM特定参数
    input_type = "passage", # 可选
    truncate = "NONE" # 可选
)
print(response)

用法 - LiteLLM代理服务器​

以下是如何使用LiteLLM代理服务器调用Nvidia NIM端点

  1. 修改config.yaml 

    model_list:
      - model_name: my-model
        litellm_params:
          model: nvidia_nim/<your-model-name>  # 添加nvidia_nim/前缀以路由为Nvidia NIM提供者
          api_key: api-key                 # 发送到模型的API密钥

  1. 启动代理 

    $ litellm --config /path/to/config.yaml

  2. 向LiteLLM代理服务器发送请求

    import openai
    client = openai.OpenAI(
        api_key="sk-1234",             # 如果使用虚拟密钥，请传递litellm代理密钥
        base_url="http://0.0.0.0:4000" # litellm-proxy-base url
    )

    response = client.chat.completions.create(
        model="my-model",
        messages = [
            {
                "role": "user",
                "content": "你是哪种llm"
            }
        ],
    )

    print(response)

支持的模型 - 💥 所有Nvidia NIM模型都支持！​

我们支持所有nvidia_nim模型，只需在发送完成请求时将nvidia_nim/设置为前缀

模型名称函数调用
nvidia/nemotron-4-340b-rewardcompletion(model="nvidia_nim/nvidia/nemotron-4-340b-reward", messages)
01-ai/yi-largecompletion(model="nvidia_nim/01-ai/yi-large", messages)
aisingapore/sea-lion-7b-instructcompletion(model="nvidia_nim/aisingapore/sea-lion-7b-instruct", messages)
databricks/dbrx-instructcompletion(model="nvidia_nim/databricks/dbrx-instruct", messages)
google/gemma-7bcompletion(model="nvidia_nim/google/gemma-7b", messages)
google/gemma-2bcompletion(model="nvidia_nim/google/gemma-2b", messages)
google/codegemma-1.1-7bcompletion(model="nvidia_nim/google/codegemma-1.1-7b", messages)
google/codegemma-7bcompletion(model="nvidia_nim/google/codegemma-7b", messages)
google/recurrentgemma-2bcompletion(model="nvidia_nim/google/recurrentgemma-2b", messages)
ibm/granite-34b-code-instructcompletion(model="nvidia_nim/ibm/granite-34b-code-instruct", messages)
ibm/granite-8b-code-instructcompletion(model="nvidia_nim/ibm/granite-8b-code-instruct", messages)
mediatek/breeze-7b-instructcompletion(model="nvidia_nim/mediatek/breeze-7b-instruct", messages)
meta/codellama-70bcompletion(model="nvidia_nim/meta/codellama-70b", messages)
| meta/llama2-70b | `completion(model="nvidia_nim/meta/llama2-70b", messages)` |
| meta/llama3-8b | `completion(model="nvidia_nim/meta/llama3-8b", messages)` |
| meta/llama3-70b | `completion(model="nvidia_nim/meta/llama3-70b", messages)` |
| microsoft/phi-3-medium-4k-instruct | `completion(model="nvidia_nim/microsoft/phi-3-medium-4k-instruct", messages)` |
| microsoft/phi-3-mini-128k-instruct | `completion(model="nvidia_nim/microsoft/phi-3-mini-128k-instruct", messages)` |
| microsoft/phi-3-mini-4k-instruct | `completion(model="nvidia_nim/microsoft/phi-3-mini-4k-instruct", messages)` |
| microsoft/phi-3-small-128k-instruct | `completion(model="nvidia_nim/microsoft/phi-3-small-128k-instruct", messages)` |
| microsoft/phi-3-small-8k-instruct | `completion(model="nvidia_nim/microsoft/phi-3-small-8k-instruct", messages)` |
| mistralai/codestral-22b-instruct-v0.1 | `completion(model="nvidia_nim/mistralai/codestral-22b-instruct-v0.1", messages)` |
| mistralai/mistral-7b-instruct | `completion(model="nvidia_nim/mistralai/mistral-7b-instruct", messages)` |
| mistralai/mistral-7b-instruct-v0.3 | `completion(model="nvidia_nim/mistralai/mistral-7b-instruct-v0.3", messages)` |
| mistralai/mixtral-8x7b-instruct | `completion(model="nvidia_nim/mistralai/mixtral-8x7b-instruct", messages)` |
| mistralai/mixtral-8x22b-instruct | `completion(model="nvidia_nim/mistralai/mixtral-8x22b-instruct", messages)` |
| mistralai/mistral-large | `completion(model="nvidia_nim/mistralai/mistral-large", messages)` |
| nvidia/nemotron-4-340b-instruct | `completion(model="nvidia_nim/nvidia/nemotron-4-340b-instruct", messages)` |
| seallms/seallm-7b-v2.5 | `completion(model="nvidia_nim/seallms/seallm-7b-v2.5", messages)` |
| snowflake/arctic | `completion(model="nvidia_nim/snowflake/arctic", messages)` |
| upstage/solar-10.7b-instruct | `completion(model="nvidia_nim/upstage/solar-10.7b-instruct", messages)` |
优云智算