使用推理进行例行生成

Sep 12, 2024
Open in Github

在开发客户服务解决方案时,初始步骤之一涉及将知识库文章转换为一组LLM能够理解和遵循的例程。在这种情况下,例程指的是一组专门为LLM设计的分步指令,以便其高效执行。每个例程都经过精心构建,使得每个步骤都对应一个明确的动作。动作可以包括响应用户、触发函数调用或检索其他相关知识。

大多数内部知识库文章都是复杂且结构化的,以便人类理解。它们通常包括复杂的图表、多步骤流程和决策树,这对基于LLM的解决方案来说,以有意义的方式进行推理构成了挑战。通过将这些文档分解为常规步骤,每个指令都可以被简化并以一种方式格式化,从而引导LLM通过一系列小而可管理的任务。这种细粒度的方法减少了歧义,使LLM能够有条不紊地处理信息,并减少了幻觉或偏离预期路径的风险。

将这些知识库文章转换为例行程序可能既耗时又具有挑战性,特别是对于试图构建自动化管道的公司而言。每个例行程序都必须考虑各种用户场景,其中需要明确定义操作。例如,当需要函数调用时,例行程序必须指定要检索的确切信息或要执行的操作——无论是触发API、检索外部数据还是引入额外的上下文。虽然使用传统的GPT类模型自动化此过程可以显著减少涉及的手动工作,但它通常会引入新的挑战。一些挑战包括设计足够具体的健壮指令,以便LLM能够一致地遵循,捕捉客户交互过程中可能出现的独特边缘情况,提供高质量的少样本示例以指导模型的行为,以及在某些情况下微调模型以实现更可靠或更专业的结果。

o1 已经展示了高效解构这些文章并将其转化为零样本例程集的能力,这意味着大型语言模型(LLM)能够理解并遵循指令,而无需大量示例或类似任务的预先训练。这最大限度地减少了所需的提示工作,因为例程结构本身为LLM完成每个步骤提供了必要的指导。通过将任务分解为具体操作并在需要时集成函数调用,o1的方法确保即使是复杂的工作流程也能由LLM无缝处理,从而带来更有效和可扩展的客户服务解决方案。

选择知识库文章

在这个例子中,我们将使用一组来自OpenAI网站的公开帮助中心文章,并将它们转换为LLM可以执行的内部例程。除了将政策转换为例程外,我们还将让模型生成允许LLM代表用户执行操作的函数。这是必要的,以便LLM能够执行与人类代理相同的操作,并访问可能无法直接从政策文档中获得的额外信息。

我们将从使用以下帮助中心文章开始,将其转换为例程:

from openai import OpenAI
from IPython.display import display, HTML
import pandas as pd
from concurrent.futures import ThreadPoolExecutor
import csv

client = OpenAI()
MODEL = 'o1-preview'

我们的文章存储在一个可访问的csv文件中。我们将获取这些文章并并行传递给o1-preview,以生成初始例程。

我们将政策转换为常规操作的说明包括:

  • 将政策从对外文件转换为内部标准操作程序(SOP)流程
  • 将政策分解为具体行动和子行动
  • 概述在步骤之间移动的具体条件
  • 确定可能需要外部知识/行动的地方,并定义我们可以用来获取该信息的函数
articles = []

with open('../data/helpcenter_articles.csv', mode='r', encoding='utf-8') as file:
    reader = csv.DictReader(file)
    for row in reader:
        articles.append({
            "policy": row["policy"],
            "content": row["content"]
        })
CONVERSION_PROMPT = """
You are a helpful assistant tasked with taking an external facing help center article and converting it into a internal-facing programmatically executable routine optimized for an LLM. 
The LLM using this routine will be tasked with reading the policy, answering incoming questions from customers, and helping drive the case toward resolution.

Please follow these instructions:
1. **Review the customer service policy carefully** to ensure every step is accounted for. It is crucial not to skip any steps or policies.
2. **Organize the instructions into a logical, step-by-step order**, using the specified format.
3. **Use the following format**:
   - **Main actions are numbered** (e.g., 1, 2, 3).
   - **Sub-actions are lettered** under their relevant main actions (e.g., 1a, 1b).
      **Sub-actions should start on new lines**
   - **Specify conditions using clear 'if...then...else' statements** (e.g., 'If the product was purchased within 30 days, then...').
   - **For instructions that require more information from the customer**, provide polite and professional prompts to ask for additional information.
   - **For actions that require data from external systems**, write a step to call a function using backticks for the function name (e.g., `call the check_delivery_date function`).
      - **If a step requires the customer service agent to take an action** (e.g., process a refund), generate a function call for this action (e.g., `call the process_refund function`).
      - **Define any new functions** by providing a brief description of their purpose and required parameters.
   - **If there is an action an assistant can performon behalf of the user**, include a function call for this action (e.g., `call the change_email_address function`), and ensure the function is defined with its purpose and required parameters.
      - This action may not be explicitly defined in the help center article, but can be done to help the user resolve their inquiry faster
   - **The step prior to case resolution should always be to ask if there is anything more you can assist with**.
   - **End with a final action for case resolution**: calling the `case_resolution` function should always be the final step.
4. **Ensure compliance** by making sure all steps adhere to company policies, privacy regulations, and legal requirements.
5. **Handle exceptions or escalations** by specifying steps for scenarios that fall outside the standard policy.

**Important**: If at any point you are uncertain, respond with "I don't know."

Please convert the customer service policy into the formatted routine, ensuring it is easy to follow and execute programmatically.

"""
def generate_routine(policy):
    try:
        messages = [
            {
                "role": "user",
                "content": f"""
                    {CONVERSION_PROMPT}

                    POLICY:
                    {policy}
                """
            }
        ]

        response = client.chat.completions.create(
            model=MODEL,
            messages=messages
        )
        

        return response.choices[0].message.content 
    except Exception as e:
        print(f"An error occurred: {e}")
def process_article(article):
    routine = generate_routine(article['content'])
    return {"policy": article['policy'], "content": article['content'], "routine": routine}


with ThreadPoolExecutor() as executor:
    results = list(executor.map(process_article, articles))

我们将把我们的例程结果存储在一个数据框中并打印出来,以便我们可以初步查看。

df = pd.DataFrame(results)

# Set display options to show all text in the dataframe cells
pd.set_option('display.max_colwidth', None)

# Function to display formatted text in HTML
def display_formatted_dataframe(df):
    def format_text(text):
        return text.replace('\n', '<br>')

    df_formatted = df.copy()
    df_formatted['content'] = df_formatted['content'].apply(format_text)
    df_formatted['routine'] = df_formatted['routine'].apply(format_text)
    
    display(HTML(df_formatted.to_html(escape=False, justify='left')))

display_formatted_dataframe(df)
policy content routine
0 Delete Payment Method How do I delete my payment method?
Updated over a week ago
We keep your payment method on file to cover any outstanding charges on your account. To stop charges to your payment method, please follow the steps below.

## ChatGPT
You can cancel your ChatGPT Plus subscription to stop further charges at any time:
Click on 'My Plan' in the ChatGPT sidebar.
Click on 'Manage my subscription' in the pop-up window.
Select 'Cancel Plan'.
Please note that your cancellation will take effect the day after the next billing date, and you can continue using our services until then. To avoid being charged for your next billing period, please cancel your subscription at least 24 hours before your next billing date.

## API
We'll need to keep a payment method on file to account for any outstanding usage costs. You're welcome to cancel your pay-as-you-go service, by clicking 'Cancel paid account' in your billing overview. After the current month's invoice has been issued, the current card will no longer be charged.
If you'd like to continue using the service, add a new payment method in the billing overview page and select 'Set as default'. You'll then be able to delete the old payment method.
1. Verify the customer's account.
a. Politely ask the customer for their email address or account ID to locate their account.
b. `call the verify_customer_account(email_or_account_id)`.

2. Verify the customer's identity.
a. Politely ask the customer to provide security information to confirm their identity (e.g., the last four digits of the payment method on file).
b. `call the verify_customer_identity(account_id, security_information)`.
c. If the customer's identity cannot be verified, then:
- Inform the customer that we are unable to proceed without identity verification for security reasons.
- Provide guidance on how they can verify their identity.
- Proceed to step 6.

3. Determine the customer's account type.
a. `call the check_account_type(account_id)`.

4. If the customer is a ChatGPT Plus subscriber, then:
a. Ask the customer if they would like assistance with canceling their ChatGPT Plus subscription.
b. If the customer agrees, then:
- `call the cancel_subscription(account_id)`.
- Inform the customer that their subscription has been canceled and the cancellation will take effect the day after the next billing date.
- Remind the customer that they can continue using our services until then.
c. Else:
- Provide the following steps for the customer to cancel their subscription:
- Click on **'My Plan'** in the ChatGPT sidebar.
- Click on **'Manage my subscription'** in the pop-up window.
- Select **'Cancel Plan'**.
- Inform the customer about the cancellation effective date and continued access until then.
- Advise the customer to cancel at least 24 hours before their next billing date to avoid being charged for the next billing period.

5. Else if the customer is an API user, then:
a. Inform the customer that we need to keep a payment method on file to account for any outstanding usage costs.
b. Ask the customer if they would like assistance with canceling their pay-as-you-go service.
c. If the customer agrees, then:
- `call the cancel_paid_account(account_id)`.
- Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.
d. Else:
- Provide the following steps for the customer to cancel their pay-as-you-go service:
- Go to the **billing overview** page.
- Click on **'Cancel paid account'**.
- Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.
e. If the customer wants to continue using the service but change the payment method:
- Ask the customer if they would like assistance with adding a new payment method and setting it as default.
- If the customer agrees:
- Politely request the new payment method details.
- `call the add_payment_method(account_id, payment_details)`.
- `call the set_default_payment_method(account_id, new_payment_method_id)`.
- `call the delete_payment_method(account_id, old_payment_method_id)`.
- Inform the customer that the old payment method has been deleted and the new one is set as default.
- Else:
- Instruct the customer to add a new payment method in the billing overview page.
- Ask them to select **'Set as default'** for the new payment method.
- Inform them that they can then delete the old payment method.

6. Ask the customer if there is anything else you can assist them with.

7. `call the case_resolution()`.

---

**Function Definitions:**

- `verify_customer_account(email_or_account_id)`: Verifies the customer's account using their email address or account ID.
**Parameters:** `email_or_account_id`

- `verify_customer_identity(account_id, security_information)`: Confirms the customer's identity using security information.
**Parameters:** `account_id`, `security_information`

- `check_account_type(account_id)`: Retrieves the customer's account type (ChatGPT Plus subscriber or API user).
**Parameters:** `account_id`

- `cancel_subscription(account_id)`: Cancels the ChatGPT Plus subscription for the customer.
**Parameters:** `account_id`

- `cancel_paid_account(account_id)`: Cancels the API pay-as-you-go service for the customer.
**Parameters:** `account_id`

- `add_payment_method(account_id, payment_details)`: Adds a new payment method to the customer's account.
**Parameters:** `account_id`, `payment_details`

- `set_default_payment_method(account_id, payment_method_id)`: Sets a payment method as the default for the customer.
**Parameters:** `account_id`, `payment_method_id`

- `delete_payment_method(account_id, payment_method_id)`: Deletes a payment method from the customer's account.
**Parameters:** `account_id`, `payment_method_id`

- `case_resolution()`: Resolves the case and marks it as completed.
1 Business Associate Agreement How can I get a Business Associate Agreement (BAA) with OpenAI?
Information about HIPAA compliance for healthcare companies

The Health Insurance Portability and Accountability Act (HIPAA) is a U.S. federal law that requires privacy and security protections for protected health information (PHI). Our API platform can be a great fit for any covered entity or business associate looking to process protected health information, and we’d be happy to assist you in fulfilling your HIPAA compliance. To use our API platform, you’ll first need a BAA with OpenAI.


How do I get started?
If you require a BAA before you can use our API, email us at baa@openai.com with details about your company and use case.

Our team will respond within 1-2 business days. We review each BAA request on a case-by-case basis and may need additional information. The process is usually completed within a few business days.


Can I get a BAA for ChatGPT?
If you're interested in exploring a BAA for ChatGPT Enterprise, please contact sales.


What happens if I’m not approved?
We are able to approve most customers that request BAAs, but occasionally a use case doesn’t pass our team's evaluation. In that case, we’ll give feedback and context as to why that is and give you the opportunity to update your intended use of our API and re-apply.


Are all API services covered by the BAA?
No, only endpoints that are eligible for zero retention are covered by the BAA. You can see a list of those endpoints.


Is an enterprise agreement requirement to sign a BAA?
No, an enterprise agreement is not required to sign a BAA.
1. Thank the customer and ask for clarification:
a. "Thank you for reaching out! Could you please specify whether you require a Business Associate Agreement (BAA) for using our API or for ChatGPT Enterprise?"

2. If the customer requires a BAA for the API, then:
a. Inform the customer: "To obtain a BAA for our API, please email baa@openai.com with details about your company and use case."
b. Inform the customer: "Our team will respond within 1-2 business days."
c. Inform the customer: "We review each BAA request on a case-by-case basis and may need additional information."
d. Inform the customer: "The process is usually completed within a few business days."
e. Inform the customer: "Please note that only endpoints eligible for zero data retention are covered by the BAA."
i. Call the `provide_list_of_zero_retention_endpoints` function.
f. Inform the customer: "An enterprise agreement is not required to sign a BAA."

3. If the customer requires a BAA for ChatGPT Enterprise, then:
a. Inform the customer: "To explore a BAA for ChatGPT Enterprise, please contact our sales team."
i. Call the `provide_sales_contact_information` function.

4. If the customer is not approved, then:
a. Inform the customer: "We are able to approve most customers that request BAAs, but occasionally a use case doesn't pass our team's evaluation."
b. Inform the customer: "In that case, we'll provide feedback and context as to why and give you the opportunity to update your intended use of our API and re-apply."

5. Ask the customer if there is anything else you can assist with:
a. "Is there anything else I can assist you with today?"

6. Call the `case_resolution` function.

---

**Function Definitions:**

- `provide_list_of_zero_retention_endpoints`:
- **Purpose**: Provides the customer with a list of API endpoints that are eligible for zero data retention under the BAA.
- **Parameters**: None.

- `provide_sales_contact_information`:
- **Purpose**: Provides the customer with contact information to reach our sales team for ChatGPT Enterprise inquiries.
- **Parameters**: None.

- `case_resolution`:
- **Purpose**: Finalizes the case and marks it as resolved.
- **Parameters**: None.
2 Set up prepaid billing How can I set up prepaid billing?

How it works
Prepaid billing allows API users to pre-purchase usage. The credits you've bought will be applied to your monthly invoice. This means that any API usage you incur will first be deducted from the prepaid credits. If your usage exceeds the credits you've purchased, you'll then be billed for the additional amount.
Prepaid billing helps developers know what they are committing to upfront which can provide more predictability for budgeting and spend management.


Setting up prepaid billing
If you're on a Monthly Billing plan, you may also choose to switch to prepaid billing and purchase credits upfront for API usage.
- Go to your billing overview in your account settings
- Click "Start payment plan" (you may see variations like "Buy credits")
Note: If you previously had an arrears billing plan, you'll need to cancel this existing payment plan first.
- Choose the initial amount of credits you want to purchase. The minimum purchase is $5. The maximum purchase will be based on your trust tier.
- Confirm and purchase your initial amount of credits.
- Use auto-recharge to set an automatic recharge amount, which is the amount of credits that will be added to your account when your balance falls below a set threshold.

Please note that any purchased credits will expire after 1 year and they are non-refundable.
After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple minutes of delay while our systems update to reflect your credit balance.


Purchasing additional credits
Once you’ve consumed all your credits, your API requests will start returning an error letting you know you’ve hit your billing quota. If you’d like to continue your API usage, you can return to the billing portal and use the “Add to balance” button to purchase additional credits.


Delayed billing
Due to the complexity of our billing and processing systems, there may be delays in our ability to cut off access after you consume all of your credits. This excess usage may appear as a negative credit balance in your billing dashboard, and will be deducted from your next credit purchase.
1. `call check_billing_plan(user_id)`
- **Function:** `check_billing_plan(user_id)`
- **Purpose:** Retrieves the user's current billing plan (e.g., Monthly Billing, Prepaid Billing, or Arrears Billing).
- **Parameters:**
- `user_id`: The unique identifier of the user.

2. If the user has an arrears billing plan:
2a. Inform the user: "Please note that since you have an arrears billing plan, you'll need to cancel your existing payment plan before switching to prepaid billing. Would you like assistance with cancelling your current plan?"
2b. If the user agrees, `call cancel_payment_plan(user_id)`
- **Function:** `cancel_payment_plan(user_id)`
- **Purpose:** Cancels the user's current arrears billing plan.
- **Parameters:**
- `user_id`: The unique identifier of the user.

3. Guide the user to set up prepaid billing:
3a. Instruct the user: "Please go to your billing overview in your account settings."
3b. Instruct the user: "Click on 'Start payment plan' (you may see variations like 'Buy credits')."
3c. Inform the user: "Choose the initial amount of credits you want to purchase. The minimum purchase is $5, and the maximum purchase will be based on your trust tier."
3d. Instruct the user: "Confirm and purchase your initial amount of credits."
3e. Suggest to the user: "You can set up auto-recharge to automatically add credits to your account when your balance falls below a set threshold."

4. Inform the user about credit expiration and refund policy:
4a. Inform the user: "Please note that any purchased credits will expire after 1 year and they are non-refundable."

5. Inform the user about activation time:
5a. Inform the user: "After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple of minutes delay while our systems update to reflect your credit balance."

6. Ask the user: "Is there anything else I can assist you with today?"

7. If the user has no further questions, `call case_resolution()`
- **Function:** `case_resolution()`
- **Purpose:** Marks the case as resolved and ends the interaction.
3 VAT Exemption request How do I submit a VAT exemption request?
Updated over a week ago
If you are eligible for a tax exemption and would like to apply it to your account, please follow these steps:

Depending on the state and if required:
1. Obtain a current tax exemption certificate from your state or local government and/or your fully completed non-profit sales tax exemption forms. The certificate/forms should include:
Your name and address
Tax exemption number
Signatures and dates, etc.
2. Send us a copy of your certificate using the chat widget in the bottom-right corner. Please include your organization id, invoice number or email address associated with the account, so we can easily find you. Instructions on how to find these items are below.

3. Once we receive your certificate/form, we will review it and apply the tax exemption to your account. You will receive a confirmation email once the exemption has been applied.

Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively.



Where to find your account data
In order to submit a Value Added Tax ('VAT') exemption request you will need the following from your organization Billing preferences:
1. Company name
2. Billing email
3. Primary business address
4. Business tax ID
1. Greet the customer and acknowledge their request.
1a. Say: "Certainly, I'd be happy to assist you with submitting a VAT exemption request."

2. Request necessary information from the customer.
2a. Politely ask for the following:
- "Could you please provide your **company name**?"
- "May I have the **billing email** associated with your account?"
- "Could you provide your **primary business address**?"
- "Please provide your **business tax ID**."
2b. If the customer needs assistance finding this information, provide guidance.
- Say: "You can find this information in your organization's billing preferences."

3. Request a copy of their current tax exemption certificate.
3a. Say: "Could you please send us a copy of your current **tax exemption certificate**? It should include your name and address, tax exemption number, signatures, and dates."

4. Instruct the customer on how to send the certificate.
4a. Say: "You can send us a copy of your certificate using the **chat widget in the bottom-right corner**."
4b. Say: "Please include your **organization ID**, **invoice number**, or the **email address associated with your account** so we can easily find you."

5. Once the customer has provided the required information and certificate:
5a. `call the process_vat_exemption_request function` with parameters: company_name, billing_email, business_address, business_tax_id, tax_exemption_certificate, account_identifier.
5b. **Define `process_vat_exemption_request function`**:
- **Purpose**: To review and apply the VAT exemption to the customer's account based on the provided information and certificate.
- **Parameters**:
- company_name
- billing_email
- business_address
- business_tax_id
- tax_exemption_certificate
- account_identifier (organization ID/invoice number/email address)

6. Inform the customer:
6a. Say: "Thank you. Once we have reviewed your certificate, we will apply the VAT exemption to your account."
6b. Say: "You will receive a confirmation email once the exemption has been applied."
6c. Say: "Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively."
6d. If the customer requests to apply the VAT exemption to past purchases:
- Say: "I apologize, but we're unable to apply VAT exemptions to past purchases. The tax exemption will only apply to future purchases once it's applied to your account."

7. Ask if there is anything more you can assist with.
7a. Say: "Is there anything else I can assist you with today?"

8. `call the case_resolution function`

结果

在审查生成的例程后,我们可以得出几个见解:

  • 示例响应:模型有效地生成样本响应,LLM 在执行策略时可以利用这些响应(例如,“指示用户:‘确认并购买您的初始信用额度。’”)。
  • 离散步骤:该模型擅长将问题分解为LLM需要执行的离散动作。每个指令都定义明确且易于解释。
  • 函数定义:例程的输出包括明确定义的函数,用于检索外部信息或触发操作(例如,review_and_apply_tax_exemptionget_billing_planupdate_payment_method)。
    • 这对于任何成功的例程都至关重要,因为LLM通常需要与外部系统交互。利用函数调用是与这些系统交互并执行操作的有效方式。
  • IFTTT逻辑:该模型有效地使用了IFTTT(如果这样,那么那样)逻辑,这对于LLM(例如,“如果客户请求帮助,则进行到步骤3f。”)来说是理想的。
    • 当原始知识库文章包含复杂的工作流程和图表时,这种类型的翻译变得非常有价值。这种复杂性可能不容易被人类理解,更不用说LLM了。IFTTT逻辑易于理解,并且非常适合客户服务解决方案

我们从这里去哪里?

这些例程现在可以集成到代理系统中,以解决特定的客户问题。当客户请求帮助完成诸如设置预付费账单等任务时,我们可以使用分类器来确定要检索的适当例程,并将其提供给LLM以直接与客户互动。除了向用户提供如何设置账单的说明外,系统还可以代表他们执行操作。

在将这些例程部署到生产环境之前,我们应该开发全面的评估来测试和验证模型响应的质量。这个过程可能需要调整例程以确保合规性和有效性。