使用AgentOps进行代理跟踪
AgentOps 提供会话重放、指标和AI代理的监控。
在高层次上，AgentOps 使您能够监控LLM调用、成本、延迟、代理故障、多代理交互、工具使用情况、会话范围的统计信息等。欲了解更多信息，请查看AgentOps 仓库。
概览仪表板
会话回放
将AgentOps添加到现有的Autogen服务中。
要开始使用，你需要安装AgentOps包并设置一个API密钥。
AgentOps 在初始化时会自动进行配置，这意味着您的代理运行数据会立即被跟踪并记录到您的 AgentOps 账户中。
Requirements
设置一个API密钥
默认情况下，AgentOps 的 init() 函数会查找名为 AGENTOPS_API_KEY 的环境变量。或者，你也可以将其作为一个可选参数传入。
创建一个账户并在 AgentOps.ai获取一个API密钥
import agentops
from autogen import ConversableAgent, UserProxyAgent, config_list_from_json
agentops.init(api_key="...")
🖇 AgentOps: Session Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4
UUID('8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4')
Autogen 现在将自动开始跟踪 - LLM 提示和完成 - 令牌使用和成本 - 代理名称和操作 - 代理之间的通信 - 工具使用 - 错误
简单聊天示例
import agentops
# When initializing AgentOps, you can pass in optional tags to help filter sessions
agentops.init(tags=["simple-autogen-example"])
# Create the agent that uses the LLM.
config_list = config_list_from_json(env_or_file="OAI_CONFIG_LIST")
assistant = ConversableAgent("agent", llm_config={"config_list": config_list})
# Create the agent that represents the user in the conversation.
user_proxy = UserProxyAgent("user", code_execution_config=False)
# Let the assistant start the conversation. It will end when the user types "exit".
assistant.initiate_chat(user_proxy, message="How can I help you today?")
# Close your AgentOps session to indicate that it completed.
agentops.end_session("Success")
agent (to user):
How can I help you today?
--------------------------------------------------------------------------------
user (to agent):
2+2
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
agent (to user):
2 + 2 equals 4.
--------------------------------------------------------------------------------
🖇 AgentOps: This run's cost $0.000960
🖇 AgentOps: Session Replay: https://app.agentops.ai/drilldown?session_id=8bfaeed1-fd51-4c68-b3ec-276b1a3ce8a4
您可以在此运行中查看数据，访问 app.agentops.ai。
仪表板将显示每个代理发送的每条消息的LLM事件，包括由人类用户发送的消息。
工具示例
AgentOps 还会跟踪 Autogen 代理何时使用工具。您可以在tool-use.ipynb中找到有关此示例的更多信息。
from typing import Annotated, Literal
from autogen import ConversableAgent, config_list_from_json, register_function
agentops.start_session(tags=["autogen-tool-example"])
Operator = Literal["+", "-", "*", "/"]
def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int:
if operator == "+":
return a + b
elif operator == "-":
return a - b
elif operator == "*":
return a * b
elif operator == "/":
return int(a / b)
else:
raise ValueError("Invalid operator")
config_list = config_list_from_json(env_or_file="OAI_CONFIG_LIST")
# Create the agent that uses the LLM.
assistant = ConversableAgent(
name="Assistant",
system_message="You are a helpful AI assistant. "
"You can help with simple calculations. "
"Return 'TERMINATE' when the task is done.",
llm_config={"config_list": config_list},
)
# The user proxy agent is used for interacting with the assistant agent
# and executes tool calls.
user_proxy = ConversableAgent(
name="User",
llm_config=False,
is_termination_msg=lambda msg: msg.get("content") is not None and "TERMINATE" in msg["content"],
human_input_mode="NEVER",
)
assistant.register_for_llm(name="calculator", description="A simple calculator")(calculator)
user_proxy.register_for_execution(name="calculator")(calculator)
# Register the calculator function to the two agents.
register_function(
calculator,
caller=assistant, # The assistant agent can suggest calls to the calculator.
executor=user_proxy, # The user proxy agent can execute the calculator calls.
name="calculator", # By default, the function name is used as the tool name.
description="A simple calculator", # A description of the tool.
)
# Let the assistant start the conversation. It will end when the user types "exit".
user_proxy.initiate_chat(assistant, message="What is (1423 - 123) / 3 + (32 + 23) * 5?")
agentops.end_session("Success")
🖇 AgentOps: Session Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d
/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2489: UserWarning: Function 'calculator' is being overridden.
warnings.warn(f"Function '{tool_sig['function']['name']}' is being overridden.", UserWarning)
/Users/braelynboynton/Developer/agentops/autogen/autogen/agentchat/conversable_agent.py:2408: UserWarning: Function 'calculator' is being overridden.
warnings.warn(f"Function '{name}' is being overridden.", UserWarning)
🖇 AgentOps: This run's cost $0.001800
🖇 AgentOps: Session Replay: https://app.agentops.ai/drilldown?session_id=880c206b-751e-4c23-9313-8684537fc04d
User (to Assistant):
What is (1423 - 123) / 3 + (32 + 23) * 5?
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
***** Suggested tool call (call_aINcGyo0Xkrh9g7buRuhyCz0): calculator *****
Arguments:
{
"a": 1423,
"b": 123,
"operator": "-"
}
***************************************************************************
--------------------------------------------------------------------------------
>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):
User (to Assistant):
***** Response from calling tool (call_aINcGyo0Xkrh9g7buRuhyCz0) *****
1300
**********************************************************************
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
***** Suggested tool call (call_prJGf8V0QVT7cbD91e0Fcxpb): calculator *****
Arguments:
{
"a": 1300,
"b": 3,
"operator": "/"
}
***************************************************************************
--------------------------------------------------------------------------------
>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):
User (to Assistant):
***** Response from calling tool (call_prJGf8V0QVT7cbD91e0Fcxpb) *****
433
**********************************************************************
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
***** Suggested tool call (call_CUIgHRsySLjayDKuUphI1TGm): calculator *****
Arguments:
{
"a": 32,
"b": 23,
"operator": "+"
}
***************************************************************************
--------------------------------------------------------------------------------
>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):
User (to Assistant):
***** Response from calling tool (call_CUIgHRsySLjayDKuUphI1TGm) *****
55
**********************************************************************
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
***** Suggested tool call (call_L7pGtBLUf9V0MPL90BASyesr): calculator *****
Arguments:
{
"a": 55,
"b": 5,
"operator": "*"
}
***************************************************************************
--------------------------------------------------------------------------------
>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):
User (to Assistant):
***** Response from calling tool (call_L7pGtBLUf9V0MPL90BASyesr) *****
275
**********************************************************************
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
***** Suggested tool call (call_Ygo6p4XfcxRjkYBflhG3UVv6): calculator *****
Arguments:
{
"a": 433,
"b": 275,
"operator": "+"
}
***************************************************************************
--------------------------------------------------------------------------------
>>>>>>>> EXECUTING FUNCTION calculator...
User (to Assistant):
User (to Assistant):
***** Response from calling tool (call_Ygo6p4XfcxRjkYBflhG3UVv6) *****
708
**********************************************************************
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
The result of the calculation is 708.
--------------------------------------------------------------------------------
User (to Assistant):
--------------------------------------------------------------------------------
>>>>>>>> USING AUTO REPLY...
Assistant (to User):
TERMINATE
--------------------------------------------------------------------------------
你可以在app.agentops.ai看到你的运行情况。在这个示例中，AgentOps仪表板将显示：- 代理之间的对话 - 每次使用calculator工具的情况 - 每次调用OpenAI以使用LLM的情况
