🚀 DeepEval - 带追踪的开源评估工具

本代码教程展示了如何轻松追踪和评估您的LlamaIndex智能体。您可以在此处了解更多关于DeepEval框架的信息：https://docs.confident-ai.com/docs/getting-started

LlamaIndex 与 DeepEval 的集成使您能够追踪您的 LlamaIndex 智能体并使用 DeepEval 的默认指标对它们进行评估。在此处阅读有关集成的更多信息：https://deepeval.com/integrations/frameworks/langchain

欢迎随时查看我们在GitHub上的代码库：https://github.com/confident-ai/deepeval

快速入门

安装以下软件包：

!pip install -q -q llama-index
!pip install -U -q deepeval

此步骤是可选的，仅当您需要服务器托管的仪表板时才需要！（悄悄说一句，我觉得您应该需要！）

!deepeval login

端到端评估

deepevaldeepeval 使您能够在一分钟内端到端地评估LlamaIndex应用程序。

创建一个包含您希望使用的指标列表的 FunctionAgent，并将其传递给您的 LlamaIndex 应用程序的 run 方法。

import asyncio

from llama_index.llms.openai import OpenAI
import llama_index.core.instrumentation as instrument

from deepeval.integrations.llama_index import (
    instrument_llama_index,
    FunctionAgent,
)
from deepeval.metrics import AnswerRelevancyMetric

instrument_llama_index(instrument.get_dispatcher())


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b


answer_relevancy_metric = AnswerRelevancyMetric()

agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
    metrics=[answer_relevancy_metric],
)


async def llm_app(input: str):
    return await agent.run(input)


asyncio.run(llm_app("What is 2 * 3?"))

评估支持 LlamaIndex FunctionAgent、ReActAgent 和 CodeActAgent。仅具有LLM参数输入和输出的指标符合评估条件。

同步

创建一个包含您希望使用的指标列表的FunctionAgent，并将其传递给您的LlamaIndex应用程序的run方法。

from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(
    goldens=[Golden(input="What is 3 * 12?"), Golden(input="What is 4 * 13?")]
)

for golden in dataset.evals_iterator():
    task = asyncio.create_task(llm_app(golden.input))
    dataset.evaluate(task)

异步

from deepeval.dataset import EvaluationDataset, Golden
import asyncio

dataset = EvaluationDataset(
    goldens=[Golden(input="What's 7 * 8?"), Golden(input="What's 7 * 6?")]
)

for golden in dataset.evals_iterator():
    task = asyncio.create_task(llm_app(golden.input))
    dataset.evaluate(task)

⚠️ 警告：DeepEval使用事件循环来管理异步操作。

Jupyter notebooks 已经维护了自己的事件循环，直接在 notebook 单元格中运行 DeepEval 示例可能会导致意外行为、卡顿或运行时错误。

建议：为避免此类问题，请在独立的Python脚本（.py文件）中运行您的DeepEval示例，而非在Jupyter Notebook内运行。

示例

以下是一些示例脚本。

# Synchronous (End-to-End Evals)
import os
import deepeval
import asyncio

from llama_index.llms.openai import OpenAI
import llama_index.core.instrumentation as instrument

from deepeval.integrations.llama_index import instrument_llama_index
from deepeval.integrations.llama_index import FunctionAgent
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset, Golden

from dotenv import load_dotenv

load_dotenv()

deepeval.login(os.getenv("CONFIDENT_API_KEY"))
instrument_llama_index(instrument.get_dispatcher())


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b


answer_relevancy_metric = AnswerRelevancyMetric()
agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
    metrics=[answer_relevancy_metric],
)


async def llm_app(input: str):
    return await agent.run(input)


dataset = EvaluationDataset(
    goldens=[Golden(input="What is 3 * 12?"), Golden(input="What is 4 * 13?")]
)
for golden in dataset.evals_iterator():
    task = asyncio.create_task(llm_app(golden.input))
    dataset.evaluate(task)

# Asynchronous (End-to-End Evals)
import os
from deepeval.integrations.llama_index import instrument_llama_index
import llama_index.core.instrumentation as instrument
from deepeval.integrations.llama_index import FunctionAgent
from llama_index.llms.openai import OpenAI
import asyncio
import time

import deepeval
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.dataset import EvaluationDataset, Golden
from dotenv import load_dotenv

load_dotenv()


# Don't forget to setup tracing
deepeval.login(os.getenv("CONFIDENT_API_KEY"))
instrument_llama_index(instrument.get_dispatcher())


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b


answer_relevancy_metric = AnswerRelevancyMetric()
agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
    metrics=[answer_relevancy_metric],
)

goldens = [Golden(input="What's 7 * 8?"), Golden(input="What's 7 * 6?")]


async def llm_app(golden: Golden):
    await agent.run(golden.input)


def main():
    dataset = EvaluationDataset(goldens=goldens)
    for golden in dataset.evals_iterator():
        task = asyncio.create_task(llm_app(golden))
        dataset.evaluate(task)


if __name__ == "__main__":
    main()

import os
from deepeval.integrations.llama_index import instrument_llama_index
import llama_index.core.instrumentation as instrument
from deepeval.integrations.llama_index import FunctionAgent
from llama_index.llms.openai import OpenAI
import asyncio

import deepeval
from dotenv import load_dotenv

load_dotenv()

# Don't forget to setup tracing
deepeval.login(os.getenv("CONFIDENT_API_KEY"))
instrument_llama_index(instrument.get_dispatcher())


def multiply(a: float, b: float) -> float:
    """Useful for multiplying two numbers."""
    return a * b


agent = FunctionAgent(
    tools=[multiply],
    llm=OpenAI(model="gpt-4o-mini"),
    system_prompt="You are a helpful assistant that can perform calculations.",
    metric_collection="test_collection_1",
)


async def llm_app(golden: Golden):
    await agent.run(golden.input)


asyncio.run(llm_app(Golden(input="What is 3 * 12?")))