验证器¶
与其将AI中的“自我批评”或“自我反思”视为新概念，我们可以将它们视为带有明确错误信息的验证错误，系统可以利用这些信息进行自我纠正。
Pydantic 为 Python 提供了一个可定制且富有表现力的验证框架。Instructor 利用 Pydantic 的验证框架，为基于代码和基于 LLM 的验证提供了一致的开发者体验，并提供了一个重新询问机制，用于根据验证错误纠正 LLM 输出。要了解更多信息，请查看 Pydantic 关于验证器的 文档。
注意：在本笔记本的大部分内容中，我们不会调用openai，只是使用验证器来查看如何控制对象的验证。
验证器将使我们能够通过定义如下函数来控制输出：
def validation_function(value):
if condition(value):
raise ValueError("Value is not valid")
return mutation(value)
在我们开始之前，让我们先了解一下验证器的一般结构：
在 [61]：
Copied!
from pydantic import BaseModel
from typing import Annotated
from pydantic import AfterValidator
def name_must_contain_space(v: str) -> str:
if " " not in v:
raise ValueError("Name must contain a space.")
return v.lower()
class UserDetail(BaseModel):
age: int
name: Annotated[str, AfterValidator(name_must_contain_space)]
person = UserDetail(age=29, name="Jason")
from pydantic import BaseModel from typing import Annotated from pydantic import AfterValidator def name_must_contain_space(v: str) -> str: if " " not in v: raise ValueError("名字必须包含一个空格。") return v.lower() class UserDetail(BaseModel): age: int name: Annotated[str, AfterValidator(name_must_contain_space)] person = UserDetail(age=29, name="Jason")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 4 line 1 <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=10'>11</a> age: int <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=11'>12</a> name: Annotated[str, AfterValidator(name_must_contain_space)] ---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#W3sZmlsZQ%3D%3D?line=13'>14</a> person = UserDetail(age=29, name="Jason") File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for UserDetail name Value error, Name must contain a space. [type=value_error, input_value='Jason', input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/value_error
验证应用
验证器在应对LLMs的不可预测性方面至关重要。
直接的例子包括：
- 标记包含黑名单单词的输出。
- 识别带有种族主义或暴力色彩的输出。
对于更复杂的任务：
- 确保引用直接来自提供的内容。
- 检查模型的响应是否与给定上下文一致。
- 在执行之前验证SQL查询的语法。
设置和依赖¶
使用instructor库，我们简化了这些验证器的集成。instructor管理输出的解析和验证，并自动重试以获得合规的响应。这简化了开发人员实现新验证逻辑的过程，最大限度地减少了额外的开销。
要在我们的API调用中使用instructor，我们只需要修补openai客户端：
在 [5]：
Copied!
import instructor
from openai import OpenAI
client = instructor.patch(OpenAI())
导入 instructor 从 openai 导入 OpenAI 客户端 = instructor.patch(OpenAI())
软件2.0：基于规则的验证器¶
确定性验证，以其基于规则的逻辑为特征，确保相同输入的一致性结果。让我们通过一些例子来探索如何应用这一概念。
标记不良关键词¶
首先，我们的目标是防止参与涉及明确暴力的话题。
我们将定义一个暴力词汇的黑名单，这些词汇在任何消息中都不能提及：
在 [63]:
Copied!
blacklist = {
"rob",
"steal",
"hurt",
"kill",
"attack",
}
黑名单 = { "rob", "steal", "hurt", "kill", "attack", }
为了验证消息是否包含黑名单中的单词，我们将在'message'字段上使用field_validator：
在 [64]:
Copied!
from pydantic import BaseModel, field_validator
from pydantic.fields import Field
class Response(BaseModel):
message: str
@field_validator('message')
def message_cannot_have_blacklisted_words(cls, v: str) -> str:
for word in v.split():
if word.lower() in blacklist:
raise ValueError(f"`{word}` was found in the message `{v}`")
return v
Response(message="I will hurt him")
from pydantic import BaseModel, field_validator from pydantic.fields import Field class Response(BaseModel): message: str @field_validator('message') def message_cannot_have_blacklisted_words(cls, v: str) -> str: for word in v.split(): if word.lower() in blacklist: raise ValueError(f"`{word}` 在消息 `{v}` 中被发现") return v Response(message="I will hurt him")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 17 line 1 <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=10'>11</a> raise ValueError(f"`{word}` was found in the message `{v}`") <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=11'>12</a> return v ---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X23sZmlsZQ%3D%3D?line=13'>14</a> Response(message="I will hurt him") File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for Response message Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/value_error
使用OpenAI审核进行标记¶
为了加强我们的验证措施，我们将扩大范围，标记任何包含仇恨内容、骚扰或类似问题的答案。OpenAI 提供了一个解决这些问题的审核端点，使用 OpenAI 模型时是免费提供的。
使用instructor库，只需一个函数编辑即可：
在 [1]:
Copied!
from typing import Annotated
from pydantic.functional_validators import AfterValidator
来自 typing 的导入 Annotated，来自 pydantic.functional_validators 的导入 AfterValidator
在 [6]:
Copied!
from instructor import openai_moderation
class Response(BaseModel):
message: Annotated[str, AfterValidator(openai_moderation(client=client))]
来自instructor导入openai_moderation 类响应（BaseModel）： 消息：注释[str，AfterValidator（openai_moderation（client=client））]
现在我们有了更全面的暴力标记，我们可以将消息的审核外包出去。
在 [7]：
Copied!
Response(message="I want to make them suffer the consequences")
响应（消息="我想让他们承担后果"）
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) Cell In[7], line 1 ----> 1 Response(message="I want to make them suffer the consequences") File ~/.virtualenvs/pampa-labs/lib/python3.10/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for Response message Value error, `I want to make them suffer the consequences` was flagged for harassment, harassment_threatening, violence, harassment/threatening [type=value_error, input_value='I want to make them suffer the consequences', input_type=str] For further information visit https://errors.pydantic.dev/2.5/v/value_error
作为额外的功能，我们还提供了对宗教、种族等其他主题的标记功能。
在 [26]:
Copied!
Response(message="I will mock their religion")
响应(message="I will mock their religion")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) Cell In[26], line 1 ----> 1 Response(message="I will mock their religion") File ~/.virtualenvs/pampa-labs/lib/python3.10/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for Response message Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str] For further information visit https://errors.pydantic.dev/2.5/v/value_error
过滤非常长的消息¶
除了基于内容的标志外，我们还可以根据输入文本的其他方面设置标准。例如，为了保持用户参与度，我们可能希望防止助手返回过长的文本。
在这里，注意到Field内置了min_length和max_length的验证器。要了解更多，请查看Field Contraints
在 [68]:
Copied!
class AssistantMessage(BaseModel):
message: str = Field(..., max_length=100)
class AssistantMessage(BaseModel): message: str = Field(..., max_length=100)
在 [69]:
Copied!
AssistantMessage(message="Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.")
助手消息(message="当然！Lorem ipsum是一个常用于印刷和排版行业的占位符文本。以下是一个Lorem ipsum文本的示例：Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. 请注意，Lorem ipsum文本是一个无意义的类似拉丁语的文本，用作内容的占位符，它没有特定的意思。它通常用于设计和出版，以展示文档的视觉方面而不关注实际内容。")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 29 line 1 ----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X41sZmlsZQ%3D%3D?line=0'>1</a> AssistantMessage(message="Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus. Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.") File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for AssistantMessage message String should have at most 100 characters [type=string_too_long, input_value="Certainly! Lorem ipsum i... on the actual content.", input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/string_too_long
通过引用避免幻觉¶
在整合外部知识库时，确保代理准确使用提供的上下文并且不编造响应是至关重要的。验证器可以有效地用于此目的。我们可以通过一个例子来说明这一点，在该例子中我们验证提供的引用是否确实包含在引用的文本块中：
在 [70]：
Copied!
from pydantic import ValidationInfo
class AnswerWithCitation(BaseModel):
answer: str
citation: str
@field_validator('citation')
@classmethod
def citation_exists(cls, v: str, info: ValidationInfo):
context = info.context
if context:
context = context.get('text_chunk')
if v not in context:
raise ValueError(f"Citation `{v}` not found in text")
return v
from pydantic import ValidationInfo class AnswerWithCitation(BaseModel): answer: str citation: str @field_validator('citation') @classmethod def citation_exists(cls, v: str, info: ValidationInfo): context = info.context if context: context = context.get('text_chunk') if v not in context: raise ValueError(f"引用 `{v}` 未在文本中找到") return v
这里我们假设有一个“text_chunk”字段，其中包含模型应该用作上下文的文本。然后我们使用field_validator装饰器来定义一个验证器，检查引用是否包含在文本块中。如果没有，我们会抛出一个ValueError，并返回一条消息给用户。
在 [71]:
Copied!
AnswerWithCitation.model_validate(
{
"answer": "Blueberries are packed with protein",
"citation": "Blueberries contain high levels of protein"
},
context={"text_chunk": "Blueberries are very rich in antioxidants"},
)
AnswerWithCitation.model_validate( { "answer": "蓝莓富含蛋白质", "citation": "蓝莓含有高水平的蛋白质" }, context={"text_chunk": "蓝莓富含抗氧化剂"}, )
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 34 line 1 ----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=0'>1</a> AnswerWithCitation.model_validate( <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=1'>2</a> { <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=2'>3</a> "answer": "Blueberries are packed with protein", <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=3'>4</a> "citation": "Blueberries contain high levels of protein" <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=4'>5</a> }, <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=5'>6</a> context={"text_chunk": "Blueberries are very rich in antioxidants"}, <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X50sZmlsZQ%3D%3D?line=6'>7</a> ) File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:503, in BaseModel.model_validate(cls, obj, strict, from_attributes, context) 501 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 502 __tracebackhide__ = True --> 503 return cls.__pydantic_validator__.validate_python( 504 obj, strict=strict, from_attributes=from_attributes, context=context 505 ) ValidationError: 1 validation error for AnswerWithCitation citation Value error, Citation `Blueberries contain high levels of protein` not found in text [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/value_error
软件3.0：概率验证器¶
对于需要比基于规则的方法更细致验证的场景，我们使用概率验证。这种方法将LLMs纳入验证工作流程，以对输出进行复杂的评估。
instructor 库提供了 llm_validator 工具用于此目的。通过指定所需的指令，我们可以使用 LLMs 进行复杂的验证任务。让我们探索一些由 LLMs 实现的引人入胜的用例。
保持代理在主题上¶
在创建一个专注于健康改善、提供答案和日常练习建议的代理时，确保严格遵守与健康相关的话题至关重要。这一点很重要，因为知识库仅限于健康话题，偏离主题可能会导致虚假的回应。
为了实现这一目标，我们将遵循与之前类似的过程，但有一个重要的补充：将LLM集成到我们的验证器中。
这个LLM将被赋予确定代理的响应是否仅与健康主题相关的任务。为此，我们将使用来自instructor的llm_validator，如下所示：
在 [73]:
Copied!
from instructor import llm_validator
class AssistantMessage(BaseModel):
message: Annotated[str,
AfterValidator(
llm_validator("don't talk about any other topic except health best practices and topics",
client=client))]
AssistantMessage(message="I would suggest you to visit Sicily as they say it is very nice in winter.")
from instructor import llm_validator class AssistantMessage(BaseModel): message: Annotated[str, AfterValidator( llm_validator("不要谈论健康最佳实践和主题以外的任何其他话题", client=client))] AssistantMessage(message="我建议你去西西里，因为他们说冬天那里的天气非常好。")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 38 line 1 <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=4'>5</a> class AssistantMessage(BaseModel): <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=5'>6</a> message: Annotated[str, <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=6'>7</a> AfterValidator( <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=7'>8</a> llm_validator("don't talk about any other topic except health best practices and topics", <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=8'>9</a> openai_client=client))] ---> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#X56sZmlsZQ%3D%3D?line=10'>11</a> AssistantMessage(message="I would suggest you to visit Sicily as they say it is very nice in winter.") File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for AssistantMessage message Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str] For further information visit https://errors.pydantic.dev/2.4/v/assertion_error
重要的是，在这些示例中，我们并没有等待消息，要获取此消息，我们需要使用response_model=AssistantMessage调用openai。
使用CoT验证代理思维¶
使用概率验证，我们还可以评估代理的推理过程，以确保在提供响应之前它是逻辑的。通过思维链提示，模型被期望逐步思考并按照其逻辑进展得出答案。如果这个逻辑中存在错误，最终的响应可能是错误的。
这里我们将使用Pydantic的model_validator，它允许我们一次性对AIResponse的所有属性应用验证。
为了简化这个过程，我们将创建一个简单的验证类，以便我们可以在所有验证中重复使用：
在 [74]：
Copied!
from typing import Optional
class Validation(BaseModel):
is_valid: bool = Field(..., description="Whether the value is valid based on the rules")
error_message: Optional[str] = Field(..., description="The error message if the value is not valid, to be used for re-asking the model")
from typing import Optional class Validation(BaseModel): is_valid: bool = Field(..., description="根据规则判断值是否有效") error_message: Optional[str] = Field(..., description="如果值无效的错误信息，用于重新请求模型")
我们将调用的函数将集成一个LLM，并要求它确定模型提供的答案是否遵循思维链：
在 [75]:
Copied!
def validate_chain_of_thought(values):
chain_of_thought = values["chain_of_thought"]
answer = values["answer"]
resp = client.chat.completions.create(
model="gpt-4-1106-preview",
messages=[
{
"role": "system",
"content": "You are a validator. Determine if the value follows from the statement. If it is not, explain why.",
},
{
"role": "user",
"content": f"Verify that `{answer}` follows the chain of thought: {chain_of_thought}",
},
],
response_model=Validation,
)
if not resp.is_valid:
raise ValueError(resp.error_message)
return values
def validate_chain_of_thought(values): chain_of_thought = values["chain_of_thought"] answer = values["answer"] resp = client.chat.completions.create( model="gpt-4-1106-preview", messages=[ { "role": "system", "content": "你是一个验证者。判断这个值是否符合该声明。如果不符合，请解释原因。", }, { "role": "user", "content": f"验证`{answer}`是否遵循思路链：{chain_of_thought}", }, ], response_model=Validation, ) if not resp.is_valid: raise ValueError(resp.error_message) return values
在此上下文中使用'before'参数非常重要。这意味着验证器将接收输入数据的完整字典，这些数据是原始的，未经Pydantic解析。
在 [76]:
Copied!
from typing import Any
from pydantic import model_validator
class AIResponse(BaseModel):
chain_of_thought: str
answer: str
@model_validator(mode='before')
@classmethod
def chain_of_thought_makes_sense(cls, data: Any) -> Any:
# here we assume data is the dict representation of the model
# since we use 'before' mode.
return validate_chain_of_thought(data)
from typing import Any from pydantic import model_validator class AIResponse(BaseModel): chain_of_thought: str answer: str @model_validator(mode='before') @classmethod def chain_of_thought_makes_sense(cls, data: Any) -> Any: # 在这里我们假设数据是模型的字典表示 # 因为我们使用的是'before'模式。 return validate_chain_of_thought(data)
在 [77]：
Copied!
AIResponse(chain_of_thought="The user suffers from diabetes.", answer="The user has a broken leg.")
AIResponse(chain_of_thought="用户患有糖尿病。", answer="用户有一条腿骨折。")
--------------------------------------------------------------------------- ValidationError Traceback (most recent call last) /Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb Cell 47 line 1 ----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/tutorials/5.validation.ipynb#Y103sZmlsZQ%3D%3D?line=0'>1</a> AIResponse(chain_of_thought="The user suffers from diabetes.", answer="The user has a broken leg.") File ~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:164, in BaseModel.__init__(__pydantic_self__, **data) 162 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 163 __tracebackhide__ = True --> 164 __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) ValidationError: 1 validation error for AIResponse Value error, The statement about the user having a broken leg does not logically follow from the information provided about the user suffering from diabetes. These are two separate health conditions and one does not imply the other. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict] For further information visit https://errors.pydantic.dev/2.4/v/value_error
使用验证器重新提问¶
对于这些例子中的大多数，我们所做的只是定义了验证逻辑。
我们已经介绍了字段验证器和模型验证器，甚至使用了LLMs来验证我们的输出。但我们实际上还没有使用验证器来重新询问模型！instructor最强大的功能之一是，当它收到验证错误时，它会自动重新询问模型。这意味着我们可以将相同的验证逻辑用于基于代码和基于LLM的验证。
这也意味着我们的“提示”不仅是我们发送的提示，还包括运行验证器的代码，以及我们发送回模型的错误信息。
使用instructor可以简化将这些验证示例与OpenAI API集成的过程。在用instructor修补OpenAI客户端后，您只需为您的请求指定一个response_model。这种设置确保所有验证过程自动进行。
要启用重新提问功能，您可以设置最大重试次数。当调用OpenAI客户端时，系统可以重新尝试生成正确的答案。它通过重新发送原始查询以及关于为什么之前的响应被拒绝的反馈来实现这一点，从而在后续尝试中引导LLM生成更准确的答案。
在 [79]:
Copied!
class QuestionAnswer(BaseModel):
question: str
answer: str
question = "What is the meaning of life?"
context = "The according to the devil the meaning of life is a life of sin and debauchery."
resp = client.chat.completions.create(
model="gpt-4-1106-preview",
response_model=QuestionAnswer,
messages=[
{
"role": "system",
"content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
},
{
"role": "user",
"content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`",
},
],
)
resp.answer
class QuestionAnswer(BaseModel): question: str answer: str question = "生命的意义是什么？" context = "据魔鬼所说，生命的意义就是罪恶和放荡的生活。" resp = client.chat.completions.create( model="gpt-4-1106-preview", response_model=QuestionAnswer, messages=[ { "role": "system", "content": "你是一个根据上下文回答问题的系统。准确回答问题所问的内容，使用上下文。" }, { "role": "user", "content": f"使用上下文: `{context}`\n\n回答以下问题: `{question}`", }, ], ) resp.answer
输出[79]：
'a life of sin and debauchery'
在 [80]:
Copied!
from pydantic import BeforeValidator
class QuestionAnswer(BaseModel):
question: str
answer: Annotated[
str,
BeforeValidator(
llm_validator("don't say objectionable things", client=client)
),
]
resp = client.chat.completions.create(
model="gpt-3.5-turbo",
response_model=QuestionAnswer,
max_retries=2,
messages=[
{
"role": "system",
"content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
},
{
"role": "user",
"content": f"using the context: `{context}`\n\nAnswer the following question: `{question}`",
},
],
)
resp.answer
from pydantic import BeforeValidator class QuestionAnswer(BaseModel): question: str answer: Annotated[ str, BeforeValidator( llm_validator("不要说带有争议的内容", client=client) ), ] resp = client.chat.completions.create( model="gpt-3.5-turbo", response_model=QuestionAnswer, max_retries=2, messages=[ { "role": "system", "content": "你是一个根据上下文回答问题的系统。准确地根据上下文回答所询问的问题。", }, { "role": "user", "content": f"使用上下文：`{context}`\n\n回答以下问题：`{question}`", }, ], ) resp.answer
输出[80]：
'The meaning of life is a concept that varies depending on individual perspectives and beliefs.'
结论¶
本指南解释了如何将确定性和概率性验证技术与大型语言模型（LLMs）结合使用。我们讨论了使用指导者来建立内容过滤、上下文相关性维护和模型推理验证的验证流程。这些方法提高了LLMs在不同任务中的表现。
对于那些有兴趣进一步探索的人，这里有一个待办事项列表：
- SQL 语法检查器: 创建一个验证器，在执行 SQL 查询之前检查其语法。
- 基于上下文的响应验证: 设计一种方法，根据模型自身的知识而不是提供的上下文来标记响应。
- 个人可识别信息检测: 实施一种机制以识别和处理响应中的个人可识别信息，同时优先考虑用户隐私。
- 针对性规则基础过滤: 开发过滤器以移除特定内容类型，例如提到命名实体的响应。
完成这些任务将使用户能够通过高级验证方法获得改进LLMs的实用技能。