与PDF聊天 - 测试、评估和实验

作者：   

我们将引导您了解如何使用prompt flow Python SDK来测试、评估和实验“与PDF聊天”流程。

0. 安装依赖

%pip install -r requirements.txt

1. 创建连接

在提示流中的连接用于管理应用程序行为的设置，包括如何与不同的服务（例如Azure OpenAI）进行通信。

import promptflow

pf = promptflow.PFClient()

# List all the available connections
for c in pf.connections.list():
    print(c.name + " (" + c.type + ")")

您需要有一个名为“open_ai_connection”的连接才能运行chat_with_pdf流程。

# create needed connection
from promptflow.entities import AzureOpenAIConnection, OpenAIConnection

try:
    conn_name = "open_ai_connection"
    conn = pf.connections.get(name=conn_name)
    print("using existing connection")
except:
    # Follow https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal to create an Azure OpenAI resource.
    connection = AzureOpenAIConnection(
        name=conn_name,
        api_key="<user-input>",
        api_base="<test_base>",
        api_type="azure",
        api_version="<test_version>",
    )

    # use this if you have an existing OpenAI account
    # connection = OpenAIConnection(
    #     name=conn_name,
    #     api_key="<user-input>",
    # )
    conn = pf.connections.create_or_update(connection)
    print("successfully created connection")

print(conn)

2. 测试流程

注意: 此示例使用 预下载的 PDFs 和 预构建的 FAISS 索引 来加快执行时间。 您可以删除文件夹以开始新的运行。

# ./chat_with_pdf/.pdfs/ stores predownloaded PDFs
# ./chat_with_pdf/.index/ stores prebuilt index files

output = pf.flows.test(
    ".",
    inputs={
        "chat_history": [],
        "pdf_url": "https://arxiv.org/pdf/1810.04805.pdf",
        "question": "what is BERT?",
    },
)
print(output)

3. 使用数据文件运行流程

flow_path = "."
data_path = "./data/bert-paper-qna-3-line.jsonl"

config_2k_context = {
    "EMBEDDING_MODEL_DEPLOYMENT_NAME": "text-embedding-ada-002",
    "CHAT_MODEL_DEPLOYMENT_NAME": "gpt-4",  # change this to the name of your deployment if you're using Azure OpenAI
    "PROMPT_TOKEN_LIMIT": 2000,
    "MAX_COMPLETION_TOKENS": 256,
    "VERBOSE": True,
    "CHUNK_SIZE": 1024,
    "CHUNK_OVERLAP": 64,
}

column_mapping = {
    "question": "${data.question}",
    "pdf_url": "${data.pdf_url}",
    "chat_history": "${data.chat_history}",
    "config": config_2k_context,
}
run_2k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)
pf.stream(run_2k_context)

print(run_2k_context)

pf.get_details(run_2k_context)

4. 评估“基础性”

eval-groundedness flow 使用 ChatGPT/GPT4 模型来评估由 chat-with-pdf flow 生成的答案。

eval_groundedness_flow_path = "../../evaluation/eval-groundedness/"
eval_groundedness_2k_context = pf.run(
    flow=eval_groundedness_flow_path,
    run=run_2k_context,
    column_mapping={
        "question": "${run.inputs.question}",
        "answer": "${run.outputs.answer}",
        "context": "${run.outputs.context}",
    },
    display_name="eval_groundedness_2k_context",
)
pf.stream(eval_groundedness_2k_context)

print(eval_groundedness_2k_context)

pf.get_details(eval_groundedness_2k_context)

pf.get_metrics(eval_groundedness_2k_context)

pf.visualize(eval_groundedness_2k_context)

您将看到一个这样的网页。它为您提供了每行如何评分的详细信息，甚至还包括评估运行执行的详细信息：

5. 尝试不同的配置并再次评估 - 实验

注意：由于我们在这个示例中只使用了3行测试数据，并且由于LLM的非确定性特性，如果您在运行此过程时看到完全相同的指标，请不要感到惊讶。

config_3k_context = {
    "EMBEDDING_MODEL_DEPLOYMENT_NAME": "text-embedding-ada-002",
    "CHAT_MODEL_DEPLOYMENT_NAME": "gpt-4",  # change this to the name of your deployment if you're using Azure OpenAI
    "PROMPT_TOKEN_LIMIT": 3000,
    "MAX_COMPLETION_TOKENS": 256,
    "VERBOSE": True,
    "CHUNK_SIZE": 1024,
    "CHUNK_OVERLAP": 64,
}

run_3k_context = pf.run(flow=flow_path, data=data_path, column_mapping=column_mapping)
pf.stream(run_3k_context)

print(run_3k_context)

eval_groundedness_3k_context = pf.run(
    flow=eval_groundedness_flow_path,
    run=run_3k_context,
    column_mapping={
        "question": "${run.inputs.question}",
        "answer": "${run.outputs.answer}",
        "context": "${run.outputs.context}",
    },
    display_name="eval_groundedness_3k_context",
)
pf.stream(eval_groundedness_3k_context)

print(eval_groundedness_3k_context)

pf.get_details(eval_groundedness_3k_context)

pf.visualize([eval_groundedness_2k_context, eval_groundedness_3k_context])