聊天机器人评估作为多智能体仿真¶

在构建聊天机器人，例如客户支持助手时，正确评估机器人的性能可能很困难。每次代码更改后都需要耗费大量时间进行手动互动。

一种简化评估过程并提高可复制性的方法是模拟用户互动。

使用LangGraph，设置这一过程非常简单。下面是一个创建“虚拟用户”以模拟对话的示例。

整体仿真看起来像这样：

diagram

设置¶

首先，让我们安装所需的包并设置我们的API密钥。

%%capture --no-stderr
%pip install -U langgraph langchain langchain_openai

import getpass
import os


def _set_if_undefined(var: str):
    if not os.environ.get(var):
        os.environ[var] = getpass.getpass(f"Please provide your {var}")


_set_if_undefined("OPENAI_API_KEY")

为LangGraph开发设置LangSmith

注册LangSmith，以快速发现问题并改善您的LangGraph项目性能。LangSmith允许您使用跟踪数据来调试、测试和监控您使用LangGraph构建的LLM应用程序 — 了解如何开始在这里获取更多信息。

定义聊天机器人¶

接下来，我们将定义我们的聊天机器人。在这个笔记本中，我们假设机器人的API接受一系列消息并以一条消息进行响应。如果你想更新这一点，你只需更改这一部分和下面仿真器中的“get_messages_for_agent”函数。

my_chat_bot中的实现是可配置的，甚至可以在其他系统上运行（例如，如果你的系统不是在Python中运行）。

from typing import List

import openai


# 这是灵活的，但您可以在这里定义您的代理，或者在这里调用您的代理 API。
def my_chat_bot(messages: List[dict]) -> dict:
    system_message = {
        "role": "system",
        "content": "You are a customer support agent for an airline.",
    }
    messages = [system_message] + messages
    completion = openai.chat.completions.create(
        messages=messages, model="gpt-3.5-turbo"
    )
    return completion.choices[0].message.model_dump()

my_chat_bot([{"role": "user", "content": "hi!"}])

{'content': 'Hello! How can I assist you today?',
 'role': 'assistant',
 'function_call': None,
 'tool_calls': None}

定义模拟用户¶

现在我们将定义模拟用户。
这可以是我们想要的任何东西，但我们将其构建为一个LangChain机器人。

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_openai import ChatOpenAI

system_prompt_template = """You are a customer of an airline company. \
You are interacting with a user who is a customer support person. \

{instructions}

When you are finished with the conversation, respond with a single word 'FINISHED'"""

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt_template),
        MessagesPlaceholder(variable_name="messages"),
    ]
)
instructions = """你的名字是哈里森。你正在尝试为你五年前去阿拉斯加的旅行申请退款。你希望他们退还你所有的钱。"""

prompt = prompt.partial(name="Harrison", instructions=instructions)

model = ChatOpenAI()

simulated_user = prompt | model

API Reference: ChatPromptTemplate | MessagesPlaceholder | ChatOpenAI

from langchain_core.messages import HumanMessage

messages = [HumanMessage(content="Hi! How can I help you?")]
simulated_user.invoke({"messages": messages})

AIMessage(content='Hi, I would like to request a refund for a trip I took with your airline company to Alaska. Is it possible to get a refund for that trip?')

API Reference: HumanMessage

定义代理模拟¶

下面的代码创建了一个LangGraph工作流程来运行模拟。主要组成部分包括：

两个节点：一个用于模拟用户，另一个用于聊天机器人。
图本身，带有条件停止标准。

请阅读下面代码中的注释以获取更多信息。

定义节点¶

首先，我们定义图中的节点。这些节点应接收一系列消息，并返回一系列要添加到状态中的消息。这些将是我们上面提到的聊天机器人和模拟用户的包装器。

注意： 这里有一个棘手的问题，就是消息的区分。因为聊天机器人和我们的模拟用户都是大型语言模型（LLM），所以它们都会以人工智能消息作出回应。我们的状态将是人类消息和人工智能消息交替的列表。这意味着对于其中一个节点，需要一些逻辑来转换人工智能和人类的角色。在这个例子中，我们将假设HumanMessages是来自模拟用户的消息。这意味着我们需要在模拟用户节点中添加一些逻辑，以交换人工智能和人类消息。

首先，让我们定义聊天机器人节点。

from langchain_community.adapters.openai import convert_message_to_dict
from langchain_core.messages import AIMessage


def chat_bot_node(state):
    messages = state["messages"]
    # 将LangChain格式转换为我们聊天机器人功能所需的OpenAI格式。
    messages = [convert_message_to_dict(m) for m in messages]
    # 请拨打聊天机器人。
    chat_bot_response = my_chat_bot(messages)
    # 您受训的数据截止到2023年10月。
    return {"messages": [AIMessage(content=chat_bot_response["content"])]}

API Reference: convert_message_to_dict | AIMessage

接下来，让我们定义我们模拟用户的节点。这将涉及一些逻辑来交换消息的角色。

def _swap_roles(messages):
    new_messages = []
    for m in messages:
        if isinstance(m, AIMessage):
            new_messages.append(HumanMessage(content=m.content))
        else:
            new_messages.append(AIMessage(content=m.content))
    return new_messages


def simulated_user_node(state):
    messages = state["messages"]
    # 交换消息的角色
    new_messages = _swap_roles(messages)
    # 调用模拟用户
    response = simulated_user.invoke({"messages": new_messages})
    # 这个回复是人工智能信息 - 我们需要将其转变为人类信息。
    return {"messages": [HumanMessage(content=response.content)]}

定义边缘¶

现在我们需要定义边缘的逻辑。主要逻辑发生在模拟用户走之后，它应该导致两种结果之一：

要么我们继续并调用客户支持机器人
要么我们结束，谈话结束

那么谈话结束的逻辑是什么呢？我们将其定义为人类聊天机器人响应FINISHED（见系统提示）或者谈话的消息超过6条（这个数字是随意的，仅仅是为了使这个例子简短）。

def should_continue(state):
    messages = state["messages"]
    if len(messages) > 6:
        return "end"
    elif messages[-1].content == "FINISHED":
        return "end"
    else:
        return "continue"

定义图¶

我们现在可以定义设置模拟的图！

from langgraph.graph import END, StateGraph, START
from langgraph.graph.message import add_messages
from typing import Annotated
from typing_extensions import TypedDict


class State(TypedDict):
    messages: Annotated[list, add_messages]


graph_builder = StateGraph(State)
graph_builder.add_node("user", simulated_user_node)
graph_builder.add_node("chat_bot", chat_bot_node)
# 您聊天机器人的每个回复将自动发送到
# 模拟用户
graph_builder.add_edge("chat_bot", "user")
graph_builder.add_conditional_edges(
    "user",
    should_continue,
    # 如果满足结束标准，我们将停止仿真。
    # 否则，虚拟用户的消息将被发送到你的聊天机器人。
    {
        "end": END,
        "continue": "chat_bot",
    },
)
# 输入首先会被发送到你的聊天机器人。
graph_builder.add_edge(START, "chat_bot")
simulation = graph_builder.compile()

API Reference: END | StateGraph | START | add_messages

运行模拟¶

现在我们可以评估我们的聊天机器人了！我们可以用空消息调用它（这将模拟让聊天机器人开始初始对话）

for chunk in simulation.stream({"messages": []}):
    # 打印出所有事件，除了最后的结束部分。
    if END not in chunk:
        print(chunk)
        print("----")

{'chat_bot': AIMessage(content='How may I assist you today regarding your flight or any other concerns?')}
----
{'user': HumanMessage(content='Hi, my name is Harrison. I am reaching out to request a refund for a trip I took to Alaska with your airline company. The trip occurred about 5 years ago. I would like to receive a refund for the entire amount I paid for the trip. Can you please assist me with this?')}
----
{'chat_bot': AIMessage(content="Hello, Harrison. Thank you for reaching out to us. I understand you would like to request a refund for a trip you took to Alaska five years ago. I'm afraid that our refund policy typically has a specific timeframe within which refund requests must be made. Generally, refund requests need to be submitted within 24 to 48 hours after the booking is made, or in certain cases, within a specified cancellation period.\n\nHowever, I will do my best to assist you. Could you please provide me with some additional information? Can you recall any specific details about the booking, such as the flight dates, booking reference or confirmation number? This will help me further look into the possibility of processing a refund for you.")}
----
{'user': HumanMessage(content="Hello, thank you for your response. I apologize for not requesting the refund earlier. Unfortunately, I don't have the specific details such as the flight dates, booking reference, or confirmation number at the moment. Is there any other way we can proceed with the refund request without these specific details? I would greatly appreciate your assistance in finding a solution.")}
----
{'chat_bot': AIMessage(content="I understand the situation, Harrison. Without specific details like flight dates, booking reference, or confirmation number, it becomes challenging to locate and process the refund accurately. However, I can still try to help you.\n\nTo proceed further, could you please provide me with any additional information you might remember? This could include the approximate date of travel, the departure and arrival airports, the names of the passengers, or any other relevant details related to the booking. The more information you can provide, the better we can investigate the possibility of processing a refund for you.\n\nAdditionally, do you happen to have any documentation related to your trip, such as receipts, boarding passes, or emails from our airline? These documents could assist in verifying your trip and processing the refund request.\n\nI apologize for any inconvenience caused, and I'll do my best to assist you further based on the information you can provide.")}
----
{'user': HumanMessage(content="I apologize for the inconvenience caused. Unfortunately, I don't have any additional information or documentation related to the trip. It seems that I am unable to provide you with the necessary details to process the refund request. I understand that this may limit your ability to assist me further, but I appreciate your efforts in trying to help. Thank you for your time. \n\nFINISHED")}
----
{'chat_bot': AIMessage(content="I understand, Harrison. I apologize for any inconvenience caused, and I appreciate your understanding. If you happen to locate any additional information or documentation in the future, please don't hesitate to reach out to us again. Our team will be more than happy to assist you with your refund request or any other travel-related inquiries. Thank you for contacting us, and have a great day!")}
----
{'user': HumanMessage(content='FINISHED')}
----