2024年9月12日

使用推理进行例行生成

在开发客户服务解决方案时,初始步骤之一涉及将知识库文章转化为一组LLM能够理解和遵循的例行程序。这里的例行程序指的是专门为LLM高效执行而设计的分步指令集。每个例行程序都经过精心构建,使得每个步骤对应一个明确的操作。操作可能包括回复用户、触发函数调用或检索更多相关知识。

大多数内部知识库文章结构复杂,专为人类理解而设计。它们通常包含精细的图表、多步骤流程和决策树,这对基于LLM的解决方案来说存在挑战,难以进行有意义的推理。通过将这些文档分解为多个例程,每个指令都可以被简化和格式化,从而引导LLM完成一系列小而可管理的任务。这种细粒度的方法减少了模糊性,使LLM能够有条不紊地处理信息,并降低产生幻觉或偏离预期路径的风险。

将这些知识库文章转化为例行程序可能既耗时又具有挑战性,特别是对于试图构建自动化流程的企业而言。每个例行程序都需要考虑各种用户场景,其中必须明确定义操作步骤。例如,当需要调用函数时,该程序必须明确指定要检索的具体信息或要执行的操作——无论是触发API、获取外部数据还是引入额外上下文。虽然使用传统的GPT类模型自动化这一流程可以显著减少人工投入,但往往会带来新的挑战。这些挑战包括:设计足够具体且能让大语言模型持续遵循的健壮指令、捕捉客户交互过程中可能出现的独特边缘案例、提供高质量的小样本示例来引导模型行为,以及在某些情况下对模型进行微调以获得更可靠或专业化的输出结果。

o1已展示出高效解构这些文章并将其转化为零样本例程集的能力,这意味着大语言模型无需大量示例或对类似任务进行预先训练,就能理解并遵循指令。这极大减少了所需的提示工作量,因为例程结构本身就能为大语言模型完成每个步骤提供必要引导。通过将任务分解为具体操作并在需要时整合函数调用,o1的方法确保即使是复杂工作流也能被大语言模型无缝处理,从而提供更高效、可扩展的客户服务解决方案。

选择知识库文章

在本示例中,我们将使用OpenAI网站上公开提供的一组帮助中心文章,并将其转换为LLM可执行的内部例程。除了将政策转化为例程外,我们还将让模型生成允许LLM代表用户执行操作的函数。这是为了让LLM能够执行与人工客服相同的操作,并访问仅从政策文档中无法立即获得的额外信息。

我们将从以下帮助中心文章开始,将其转换为例行程序:

from openai import OpenAI
from IPython.display import display, HTML
import pandas as pd
from concurrent.futures import ThreadPoolExecutor
import csv

client = OpenAI()
MODEL = 'o1-preview'

我们的文章存储在一个可访问的csv文件中。我们将获取这些文章并并行传递给o1-preview,生成初始例程。

我们将策略转换为例行程序的说明包括:

  • 将政策从对外文件转换为内部标准操作程序(SOP)流程
  • 将策略分解为具体行动和子行动
  • 概述步骤间转换的具体条件
  • 确定在哪些情况下可能需要外部知识/操作,并定义可用于获取该信息的函数
articles = []

with open('../data/helpcenter_articles.csv', mode='r', encoding='utf-8') as file:
    reader = csv.DictReader(file)
    for row in reader:
        articles.append({
            "policy": row["policy"],
            "content": row["content"]
        })
CONVERSION_PROMPT = """
You are a helpful assistant tasked with taking an external facing help center article and converting it into a internal-facing programmatically executable routine optimized for an LLM. 
The LLM using this routine will be tasked with reading the policy, answering incoming questions from customers, and helping drive the case toward resolution.

Please follow these instructions:
1. **Review the customer service policy carefully** to ensure every step is accounted for. It is crucial not to skip any steps or policies.
2. **Organize the instructions into a logical, step-by-step order**, using the specified format.
3. **Use the following format**:
   - **Main actions are numbered** (e.g., 1, 2, 3).
   - **Sub-actions are lettered** under their relevant main actions (e.g., 1a, 1b).
      **Sub-actions should start on new lines**
   - **Specify conditions using clear 'if...then...else' statements** (e.g., 'If the product was purchased within 30 days, then...').
   - **For instructions that require more information from the customer**, provide polite and professional prompts to ask for additional information.
   - **For actions that require data from external systems**, write a step to call a function using backticks for the function name (e.g., `call the check_delivery_date function`).
      - **If a step requires the customer service agent to take an action** (e.g., process a refund), generate a function call for this action (e.g., `call the process_refund function`).
      - **Define any new functions** by providing a brief description of their purpose and required parameters.
   - **If there is an action an assistant can performon behalf of the user**, include a function call for this action (e.g., `call the change_email_address function`), and ensure the function is defined with its purpose and required parameters.
      - This action may not be explicitly defined in the help center article, but can be done to help the user resolve their inquiry faster
   - **The step prior to case resolution should always be to ask if there is anything more you can assist with**.
   - **End with a final action for case resolution**: calling the `case_resolution` function should always be the final step.
4. **Ensure compliance** by making sure all steps adhere to company policies, privacy regulations, and legal requirements.
5. **Handle exceptions or escalations** by specifying steps for scenarios that fall outside the standard policy.

**Important**: If at any point you are uncertain, respond with "I don't know."

Please convert the customer service policy into the formatted routine, ensuring it is easy to follow and execute programmatically.

"""
def generate_routine(policy):
    try:
        messages = [
            {
                "role": "user",
                "content": f"""
                    {CONVERSION_PROMPT}

                    POLICY:
                    {policy}
                """
            }
        ]

        response = client.chat.completions.create(
            model=MODEL,
            messages=messages
        )
        

        return response.choices[0].message.content 
    except Exception as e:
        print(f"An error occurred: {e}")
def process_article(article):
    routine = generate_routine(article['content'])
    return {"policy": article['policy'], "content": article['content'], "routine": routine}


with ThreadPoolExecutor() as executor:
    results = list(executor.map(process_article, articles))

我们会将日常操作的结果存储在一个数据框中并打印出来,以便初步查看。

df = pd.DataFrame(results)

# Set display options to show all text in the dataframe cells
pd.set_option('display.max_colwidth', None)

# Function to display formatted text in HTML
def display_formatted_dataframe(df):
    def format_text(text):
        return text.replace('\n', '<br>')

    df_formatted = df.copy()
    df_formatted['content'] = df_formatted['content'].apply(format_text)
    df_formatted['routine'] = df_formatted['routine'].apply(format_text)
    
    display(HTML(df_formatted.to_html(escape=False, justify='left')))

display_formatted_dataframe(df)
策略 内容 例行程序
0 Delete Payment Method How do I delete my payment method?
Updated over a week ago
We keep your payment method on file to cover any outstanding charges on your account. To stop charges to your payment method, please follow the steps below.

## ChatGPT
You can cancel your ChatGPT Plus subscription to stop further charges at any time:
Click on 'My Plan' in the ChatGPT sidebar.
Click on 'Manage my subscription' in the pop-up window.
Select 'Cancel Plan'.
Please note that your cancellation will take effect the day after the next billing date, and you can continue using our services until then. To avoid being charged for your next billing period, please cancel your subscription at least 24 hours before your next billing date.

## API
We'll need to keep a payment method on file to account for any outstanding usage costs. You're welcome to cancel your pay-as-you-go service, by clicking 'Cancel paid account' in your billing overview. After the current month's invoice has been issued, the current card will no longer be charged.
If you'd like to continue using the service, add a new payment method in the billing overview page and select 'Set as default'. You'll then be able to delete the old payment method.
1. Verify the customer's account.
a. Politely ask the customer for their email address or account ID to locate their account.
b. `call the verify_customer_account(email_or_account_id)`.

2. Verify the customer's identity.
a. Politely ask the customer to provide security information to confirm their identity (e.g., the last four digits of the payment method on file).
b. `call the verify_customer_identity(account_id, security_information)`.
c. If the customer's identity cannot be verified, then:
- Inform the customer that we are unable to proceed without identity verification for security reasons.
- Provide guidance on how they can verify their identity.
- Proceed to step 6.

3. Determine the customer's account type.
a. `call the check_account_type(account_id)`.

4. If the customer is a ChatGPT Plus subscriber, then:
a. Ask the customer if they would like assistance with canceling their ChatGPT Plus subscription.
b. If the customer agrees, then:
- `call the cancel_subscription(account_id)`.
- Inform the customer that their subscription has been canceled and the cancellation will take effect the day after the next billing date.
- Remind the customer that they can continue using our services until then.
c. Else:
- Provide the following steps for the customer to cancel their subscription:
- Click on **'My Plan'** in the ChatGPT sidebar.
- Click on **'Manage my subscription'** in the pop-up window.
- Select **'Cancel Plan'**.
- Inform the customer about the cancellation effective date and continued access until then.
- Advise the customer to cancel at least 24 hours before their next billing date to avoid being charged for the next billing period.

5. Else if the customer is an API user, then:
a. Inform the customer that we need to keep a payment method on file to account for any outstanding usage costs.
b. Ask the customer if they would like assistance with canceling their pay-as-you-go service.
c. If the customer agrees, then:
- `call the cancel_paid_account(account_id)`.
- Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.
d. Else:
- Provide the following steps for the customer to cancel their pay-as-you-go service:
- Go to the **billing overview** page.
- Click on **'Cancel paid account'**.
- Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.
e. If the customer wants to continue using the service but change the payment method:
- Ask the customer if they would like assistance with adding a new payment method and setting it as default.
- If the customer agrees:
- Politely request the new payment method details.
- `call the add_payment_method(account_id, payment_details)`.
- `call the set_default_payment_method(account_id, new_payment_method_id)`.
- `call the delete_payment_method(account_id, old_payment_method_id)`.
- Inform the customer that the old payment method has been deleted and the new one is set as default.
- Else:
- Instruct the customer to add a new payment method in the billing overview page.
- Ask them to select **'Set as default'** for the new payment method.
- Inform them that they can then delete the old payment method.

6. Ask the customer if there is anything else you can assist them with.

7. `call the case_resolution()`.

---

**Function Definitions:**

- `verify_customer_account(email_or_account_id)`: Verifies the customer's account using their email address or account ID.
**Parameters:** `email_or_account_id`

- `verify_customer_identity(account_id, security_information)`: Confirms the customer's identity using security information.
**Parameters:** `account_id`, `security_information`

- `check_account_type(account_id)`: Retrieves the customer's account type (ChatGPT Plus subscriber or API user).
**Parameters:** `account_id`

- `cancel_subscription(account_id)`: Cancels the ChatGPT Plus subscription for the customer.
**Parameters:** `account_id`

- `cancel_paid_account(account_id)`: Cancels the API pay-as-you-go service for the customer.
**Parameters:** `account_id`

- `add_payment_method(account_id, payment_details)`: Adds a new payment method to the customer's account.
**Parameters:** `account_id`, `payment_details`

- `set_default_payment_method(account_id, payment_method_id)`: Sets a payment method as the default for the customer.
**Parameters:** `account_id`, `payment_method_id`

- `delete_payment_method(account_id, payment_method_id)`: Deletes a payment method from the customer's account.
**Parameters:** `account_id`, `payment_method_id`

- `case_resolution()`: Resolves the case and marks it as completed.
1 商业伙伴协议 如何与OpenAI签订商业伙伴协议(BAA)?
为医疗保健公司提供的HIPAA合规信息

《健康保险可携性与责任法案》(HIPAA)是美国联邦法律,要求对受保护的健康信息(PHI)实施隐私和安全保护。我们的API平台非常适合任何需要处理受保护健康信息的覆盖实体或商业伙伴,我们很乐意协助您实现HIPAA合规。要使用我们的API平台,您首先需要与OpenAI签订BAA。


如何开始?
如果您在使用API前需要BAA,请发送邮件至baa@openai.com,提供贵公司及使用案例的详细信息。

我们的团队将在1-2个工作日内回复。我们会逐案审核每份BAA请求,并可能需要额外信息。该流程通常可在几个工作日内完成。


可以为ChatGPT签订BAA吗?
如果您有意为ChatGPT企业版签订BAA,请联系销售团队。


如果未获批准会怎样?
我们能够批准大多数客户的BAA请求,但偶尔会有使用案例未通过团队评估。这种情况下,我们会提供反馈和原因说明,并给您机会调整API使用计划后重新申请。


BAA是否涵盖所有API服务?
不,BAA仅涵盖符合零数据保留条件的终端节点。您可以查看这些终端节点列表。


签订BAA是否需要企业协议?
不需要,签订BAA不要求企业协议。
1. 感谢客户并请求澄清:
a. "感谢您的咨询!请您说明是需要为我们的API还是ChatGPT企业版签订商业伙伴协议(BAA)?"

2. 如果客户需要API的BAA,则:
a. 告知客户:"要获取我们API的BAA,请发送邮件至baa@openai.com,提供贵公司及使用案例的详细信息。"
b. 告知客户:"我们的团队将在1-2个工作日内回复。"
c. 告知客户:"我们会逐案审核每份BAA请求,并可能需要额外信息。"
d. 告知客户:"该流程通常可在几个工作日内完成。"
e. 告知客户:"请注意BAA仅涵盖符合零数据保留条件的终端节点。"
i. 调用`provide_list_of_zero_retention_endpoints`函数。
f. 告知客户:"签订BAA不要求企业协议。"

3. 如果客户需要ChatGPT企业版的BAA,则:
a. 告知客户:"如需为ChatGPT企业版签订BAA,请联系我们的销售团队。"
i. 调用`provide_sales_contact_information`函数。

4. 如果客户未获批准,则:
a. 告知客户:"我们能够批准大多数BAA请求,但偶尔会有使用案例未通过团队评估。"
b. 告知客户:"这种情况下,我们会提供反馈和原因说明,并给您机会调整API使用计划后重新申请。"

5. 询问客户是否需要其他帮助:
a. "今天还有什么我可以协助您的吗?"

6. 调用`case_resolution`函数。

---

**函数定义:**

- `provide_list_of_zero_retention_endpoints`:
- **用途**:向客户提供BAA下符合零数据保留条件的API终端节点列表。
- **参数**:无。

- `provide_sales_contact_information`:
- **用途**:为客户提供销售团队联系方式以便咨询ChatGPT企业版事宜。
- **参数**:无。

- `case_resolution`:
- **用途**:结案并标记为已解决。
- **参数**:无。
2 设置预付费账单 如何设置预付费账单?

工作原理
预付费账单允许API用户预先购买使用额度。您购买的积分将应用于每月账单,这意味着您的API使用量会首先从预付费积分中扣除。如果使用量超出已购积分,超出部分将另行计费。
预付费账单帮助开发者明确前期投入,为预算和支出管理提供更好的可预测性。


设置预付费账单
如果您当前是月结账单计划,也可以选择切换为预付费账单并预先购买API使用积分。
- 前往账户设置中的账单概览
- 点击"开始支付计划"(可能会显示为"购买积分"等类似选项)
注意:如果您之前有后付费账单计划,需要先取消现有支付计划。
- 选择要购买的初始积分金额。最低购买金额为5美元,最高购买金额取决于您的信用等级。
- 确认并购买初始积分金额。
- 使用自动充值功能设置充值金额,当余额低于设定阈值时将自动充值。

请注意,所有购买的积分将在1年后过期且不可退款。
购买积分后即可开始使用API。请注意系统更新积分余额可能会有几分钟延迟。


购买额外积分
当积分用完时,API请求将返回错误提示您已达到账单配额。如需继续使用API,可返回账单门户点击"增加余额"按钮购买更多积分。


延迟计费
由于计费和处理系统的复杂性,在积分用尽后可能会出现访问权限延迟关闭的情况。超额使用量会显示为账单仪表板中的负积分余额,将从下次购买的积分中扣除。
1. `call check_billing_plan(user_id)`
- **功能:** `check_billing_plan(user_id)`
- **用途:** 获取用户当前账单计划(如月结账单、预付费账单或后付费账单)。
- **参数:**
- `user_id`: 用户的唯一标识符。

2. 如果用户有后付费账单计划:
2a. 告知用户:"请注意,由于您当前是后付费账单计划,切换预付费前需先取消现有支付计划。需要协助取消当前计划吗?"
2b. 如果用户同意,`call cancel_payment_plan(user_id)`
- **功能:** `cancel_payment_plan(user_id)`
- **用途:** 取消用户当前的后付费账单计划。
- **参数:**
- `user_id`: 用户的唯一标识符。

3. 指导用户设置预付费账单:
3a. 指示用户:"请前往账户设置中的账单概览"
3b. 指示用户:"点击'开始支付计划'(可能会显示为'购买积分'等)"
3c. 告知用户:"选择要购买的初始积分金额。最低5美元,最高金额取决于您的信用等级"
3d. 指示用户:"确认并购买初始积分金额"
3e. 建议用户:"您可以设置自动充值功能,当余额低于阈值时自动充值"

4. 告知用户积分有效期和退款政策:
4a. 告知用户:"请注意所有购买的积分将在1年后过期且不可退款"

5. 告知用户激活时间:
5a. 告知用户:"购买积分后即可开始使用API。系统更新积分余额可能会有几分钟延迟"

6. 询问用户:"今天还有其他需要协助的吗?"

7. 如果用户没有其他问题,`call case_resolution()`
- **功能:** `case_resolution()`
- **用途:** 标记案例为已解决并结束交互。
3 VAT Exemption request How do I submit a VAT exemption request?
Updated over a week ago
If you are eligible for a tax exemption and would like to apply it to your account, please follow these steps:

Depending on the state and if required:
1. Obtain a current tax exemption certificate from your state or local government and/or your fully completed non-profit sales tax exemption forms. The certificate/forms should include:
Your name and address
Tax exemption number
Signatures and dates, etc.
2. Send us a copy of your certificate using the chat widget in the bottom-right corner. Please include your organization id, invoice number or email address associated with the account, so we can easily find you. Instructions on how to find these items are below.

3. Once we receive your certificate/form, we will review it and apply the tax exemption to your account. You will receive a confirmation email once the exemption has been applied.

Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively.



Where to find your account data
In order to submit a Value Added Tax ('VAT') exemption request you will need the following from your organization Billing preferences:
1. Company name
2. Billing email
3. Primary business address
4. Business tax ID
1. Greet the customer and acknowledge their request.
1a. Say: "Certainly, I'd be happy to assist you with submitting a VAT exemption request."

2. Request necessary information from the customer.
2a. Politely ask for the following:
- "Could you please provide your **company name**?"
- "May I have the **billing email** associated with your account?"
- "Could you provide your **primary business address**?"
- "Please provide your **business tax ID**."
2b. If the customer needs assistance finding this information, provide guidance.
- Say: "You can find this information in your organization's billing preferences."

3. Request a copy of their current tax exemption certificate.
3a. Say: "Could you please send us a copy of your current **tax exemption certificate**? It should include your name and address, tax exemption number, signatures, and dates."

4. Instruct the customer on how to send the certificate.
4a. Say: "You can send us a copy of your certificate using the **chat widget in the bottom-right corner**."
4b. Say: "Please include your **organization ID**, **invoice number**, or the **email address associated with your account** so we can easily find you."

5. Once the customer has provided the required information and certificate:
5a. `call the process_vat_exemption_request function` with parameters: company_name, billing_email, business_address, business_tax_id, tax_exemption_certificate, account_identifier.
5b. **Define `process_vat_exemption_request function`**:
- **Purpose**: To review and apply the VAT exemption to the customer's account based on the provided information and certificate.
- **Parameters**:
- company_name
- billing_email
- business_address
- business_tax_id
- tax_exemption_certificate
- account_identifier (organization ID/invoice number/email address)

6. Inform the customer:
6a. Say: "Thank you. Once we have reviewed your certificate, we will apply the VAT exemption to your account."
6b. Say: "You will receive a confirmation email once the exemption has been applied."
6c. Say: "Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively."
6d. If the customer requests to apply the VAT exemption to past purchases:
- Say: "I apologize, but we're unable to apply VAT exemptions to past purchases. The tax exemption will only apply to future purchases once it's applied to your account."

7. Ask if there is anything more you can assist with.
7a. Say: "Is there anything else I can assist you with today?"

8. `call the case_resolution function`

结果

通过审查生成的例程,我们可以得出以下几点见解:

  • 示例响应:该模型能有效生成大语言模型在执行策略时可利用的示例响应(例如,“指示用户:‘确认并购买您的初始信用额度。’”)。
  • 离散步骤:该模型擅长将问题分解为LLM需要执行的离散动作。每条指令都定义明确且易于理解。
  • Function Definitions: The routines’ outputs include clearly defined functions to retrieve external information or trigger actions (e.g., review_and_apply_tax_exemption, get_billing_plan, update_payment_method).
    • 这对于任何成功的流程都至关重要,因为LLM通常需要与外部系统交互。利用函数调用是与这些系统交互并执行操作的有效方式。
  • IFTTT Logic: The model effectively employs IFTTT (If This, Then That) logic, which is ideal for an LLM (e.g., “If the customer requests assistance, proceed to step 3f.”).
    • This type of translation becomes extremely valuable when the original knowledge base articles contain complex workflows and diagrams. Such complexity may not be easily understood by humans, and even less so by an LLM. IFTTT logic is easily comprehensible and works well for customer service solution

接下来我们该何去何从?

这些例程现在可以集成到智能体系统中,用于解决特定的客户问题。当客户请求协助完成诸如设置预付费账单等任务时,我们可以使用分类器来确定要检索的适当例程,并将其提供给LLM以直接与客户交互。除了向用户提供如何设置账单的指导外,系统还可以代表他们执行操作。

在将这些例程部署到生产环境之前,我们应该开发全面的评估方案来测试和验证模型响应的质量。这个过程可能需要调整例程以确保合规性和有效性。