https://colab.research.google.com/assets/colab-badge.svg” alt=”在Colab中打开”/>
SylphAI-Inc/AdalFlow” target=”_blank” style=”display: flex; align-items: center;”>
追踪#
特别是，我们提供了两种跟踪方法来帮助您开发和改进生成器：
1. 在开发过程中追踪提示的历史变化（状态）。开发者通常会经历一个漫长的提示优化过程，当当前的更改实际上使性能变得更差时，失去对提示更改的跟踪是非常令人沮丧的。
我们创建了一个GeneratorStateLogger来处理日志记录并保存到json文件中。为了进一步简化开发者的流程，我们提供了一个类装饰器trace_generator_states，只需在您的任务组件中添加一行代码即可。它将自动跟踪任何类型为Generator的属性。
from adalflow.tracing import trace_generator_states
from adalflow.core import Component, Generator
@trace_generator_states()
class SimpleQA(Component):
def __init__(self):
super().__init__()
self.generator = Generator(...)
self.generator_2 = Generator(...)
def call(...):
默认情况下，将在当前工作目录中创建一个目录来存储日志文件。 默认的项目名称为SimpleQA，日志文件将被命名为generator_state_trace.json， 其中generator和generator_2都将被记录。 日志目录的结构如下：
.
├── traces
│ ├── SimpleQA
│ │ ├── generator_state_trace.json
这是一个示例日志文件：
{
"generator": [
{
"prompt_states": {
"_components": {},
"_parameters": {},
"training": false,
"_template_string": "{# task desc #}\n{% if task_desc_str %}\n{{task_desc_str}}\n{% else %}\nAnswer user query.\n{% endif %}\n{# output format #}\n{% if output_format_str %}\n<OUTPUT_FORMAT>\n{{output_format_str}}\n</OUTPUT_FORMAT>\n{% endif %}\n{# tools #}\n{% if tools_str %}\n<TOOLS>\n{{tools_str}}\n</TOOLS>\n{% endif %}\n{# example #}\n{% if examples_str %}\n<EXAMPLES>\n{{examples_str}}\n</EXAMPLES>\n{% endif %}\n{# chat history #}\n{% if chat_history_str %}\n<CHAT_HISTORY>\n{{chat_history_str}}\n</CHAT_HISTORY>\n{% endif %}\n{#contex#}\n{% if context_str %}\n<CONTEXT>\n{{context_str}}\n</CONTEXT>\n{% endif %}\n{# steps #}\n{% if steps_str %}\n<STEPS>\n{{steps_str}}\n</STEPS>\n{% endif %}\n{% if input_str %}\n<Inputs>\n{{input_str}}\n</Inputs>\n{% endif %}\n{% if output_str %}\n<Outputs>\n{{output_str}}\n</Outputs>\n{% endif %}\n",
"prompt_variables": [
"chat_history_str",
"context_str",
"examples_str",
"input_str",
"output_format_str",
"output_str",
"steps_str",
"task_desc_str",
"tools_str"
],
"preset_prompt_kwargs": {
"task_desc_str": "You are a helpful assistant and with a great sense of humor."
}
},
"time_stamp": "2024-06-02T15:55:21.765794"
},
{
"prompt_states": {
"_components": {},
"_parameters": {},
"training": false,
"_template_string": "{# task desc #}\n{% if task_desc_str %}\n{{task_desc_str}}\n{% else %}\nAnswer user query.\n{% endif %}\n{# output format #}\n{% if output_format_str %}\n<OUTPUT_FORMAT>\n{{output_format_str}}\n</OUTPUT_FORMAT>\n{% endif %}\n{# tools #}\n{% if tools_str %}\n<TOOLS>\n{{tools_str}}\n</TOOLS>\n{% endif %}\n{# example #}\n{% if examples_str %}\n<EXAMPLES>\n{{examples_str}}\n</EXAMPLES>\n{% endif %}\n{# chat history #}\n{% if chat_history_str %}\n<CHAT_HISTORY>\n{{chat_history_str}}\n</CHAT_HISTORY>\n{% endif %}\n{#contex#}\n{% if context_str %}\n<CONTEXT>\n{{context_str}}\n</CONTEXT>\n{% endif %}\n{# steps #}\n{% if steps_str %}\n<STEPS>\n{{steps_str}}\n</STEPS>\n{% endif %}\n{% if input_str %}\n<Inputs>\n{{input_str}}\n</Inputs>\n{% endif %}\n{% if output_str %}\n<Outputs>\n{{output_str}}\n</Outputs>\n{% endif %}\n",
"prompt_variables": [
"chat_history_str",
"context_str",
"examples_str",
"input_str",
"output_format_str",
"output_str",
"steps_str",
"task_desc_str",
"tools_str"
],
"preset_prompt_kwargs": {
"task_desc_str": "You are a helpful assistant and with a great sense of humor. Second edition."
}
},
"time_stamp": "2024-06-02T15:56:37.756148"
}
],
"generator2": [
{
"prompt_states": {
"_components": {},
"_parameters": {},
"training": false,
"_template_string": "{# task desc #}\n{% if task_desc_str %}\n{{task_desc_str}}\n{% else %}\nAnswer user query.\n{% endif %}\n{# output format #}\n{% if output_format_str %}\n<OUTPUT_FORMAT>\n{{output_format_str}}\n</OUTPUT_FORMAT>\n{% endif %}\n{# tools #}\n{% if tools_str %}\n<TOOLS>\n{{tools_str}}\n</TOOLS>\n{% endif %}\n{# example #}\n{% if examples_str %}\n<EXAMPLES>\n{{examples_str}}\n</EXAMPLES>\n{% endif %}\n{# chat history #}\n{% if chat_history_str %}\n<CHAT_HISTORY>\n{{chat_history_str}}\n</CHAT_HISTORY>\n{% endif %}\n{#contex#}\n{% if context_str %}\n<CONTEXT>\n{{context_str}}\n</CONTEXT>\n{% endif %}\n{# steps #}\n{% if steps_str %}\n<STEPS>\n{{steps_str}}\n</STEPS>\n{% endif %}\n{% if input_str %}\n<Inputs>\n{{input_str}}\n</Inputs>\n{% endif %}\n{% if output_str %}\n<Outputs>\n{{output_str}}\n</Outputs>\n{% endif %}\n",
"prompt_variables": [
"chat_history_str",
"context_str",
"examples_str",
"input_str",
"output_format_str",
"output_str",
"steps_str",
"task_desc_str",
"tools_str"
],
"preset_prompt_kwargs": {
"task_desc_str": "You are the second generator."
}
},
"time_stamp": "2024-06-03T16:44:45.223220"
}
]
}
追踪所有失败的LLM预测以进一步改进。
同样地，tracing.generator_call_logger.GeneratorCallLogger 被创建用于记录生成器调用的输入参数和输出结果。
提供了trace_generator_call装饰器，用于提供一行设置来跟踪调用，默认情况下只会记录失败的预测。
在上面的例子中添加第二个装饰器：
from tracing import trace_generator_errors
@trace_generator_call()
@trace_generator_states()
class SimpleQA(Component):
def __init__(self):
super().__init__()
self.generator = Generator(...)
self.generator_2 = Generator(...)
def call(...):
现在，日志目录中将创建三个更多的文件：
.
├── traces
│ ├── SimpleQA
│ │ ├── logger_metadata.json
│ │ ├── generator_call.jsonl
│ │ ├── generator_2_call.jsonl
logger_metadata.json 文件包含记录器的元数据，它看起来像这样：
{
"generator": "./traces/SimpleQA/generator_call.jsonl",
"generator2": "./traces/SimpleQA/generator2_call.jsonl"
}
generator_call.jsonl 文件包含了对生成器的所有调用的日志，它看起来像这样：
{"prompt_kwargs": {"input_str": "What is the capital of France?"}, "model_kwargs": {}, "output": {"data": "Bonjour!\n\nThe capital of France is Paris, of course! But did you know that the Eiffel Tower in Paris is actually the most-visited paid monument in the world? Mind-blowing, right?\n\nNow, would you like to know some more fun facts or perhaps ask another question? I'm all ears (or should I say, all eyes?)", "error_message": null, "raw_response": "Bonjour!\n\nThe capital of France is Paris, of course! But did you know that the Eiffel Tower in Paris is actually the most-visited paid monument in the world? Mind-blowing, right?\n\nNow, would you like to know some more fun facts or perhaps ask another question? I'm all ears (or should I say, all eyes?)"}, "time_stamp": "2024-06-03T16:44:45.582859"}
注意
通常，让评估在评估上运行以收集尽可能多的失败预测，这对于手动提示或自动提示工程（APE）都非常有帮助。