跳转到内容
LlamaIndex Python Documentation
TypeScript

Activeloop 深度记忆

我们如何为文档问答实现+15%的检索增强生成命中率提升？

检索增强生成器（RAGs）近来获得了广泛关注。随着高级RAG技术和智能体的出现，它们拓展了RAGs所能实现的潜力。然而，若干挑战可能限制RAGs在生产环境中的集成。在生产环境中实施RAGs时需要考虑的主要因素是准确率（召回率）、成本和延迟。对于基础使用场景，OpenAI的Ada模型搭配简单相似性搜索即可产生令人满意的结果。但对于需要更高搜索准确率或召回率的场景，可能需要采用高级检索技术。这些方法可能涉及不同的数据块大小、多次查询重写等，这可能会增加延迟和成本。Activeloop的深度记忆作为Activeloop Deep Lake用户可用的功能，通过引入一个经过训练的微型神经网络层来解决这些问题，该层能够将用户查询与语料库中的相关数据进行匹配。虽然这种附加功能在搜索过程中仅产生最小延迟，但能将检索准确率提升高达27%，同时保持成本效益和易用性，无需任何额外的高级rag技术。

%pip install llama-index-vector-stores-deeplake
%pip install llama-index-llms-openai
import nest_asyncio
import os
import getpass


nest_asyncio.apply()
!pip install deeplake beautifulsoup4 html2text tiktoken openai llama-index python-dotenv

在本教程中，我们将解析deeplake文档，并创建一个能够回答文档中问题的RAG系统。

本教程可分为以下几个部分：

  1. 数据集创建与上传
  2. 生成合成查询并训练深度记忆
  3. 评估深度记忆性能
  4. 深度记忆推理

1. 数据集创建与导入

Section titled “1. Dataset Creation and ingestion”

让我使用 BeautifulSoup 解析所有链接并将其转换为 LlamaIndex 文档：

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin




def get_all_links(url):
    response = requests.get(url)
    if response.status_code != 200:
        print(f"Failed to retrieve the page: {url}")
        return []


    soup = BeautifulSoup(response.content, "html.parser")


    # Finding all 'a' tags which typically contain href attribute for links
    links = [
        urljoin(url, a["href"])
        for a in soup.find_all("a", href=True)
        if a["href"]
    ]


    return links
from langchain.document_loaders import AsyncHtmlLoader
from langchain.document_transformers import Html2TextTransformer
from llama_index.core import Document




def load_documents(url):
    all_links = get_all_links(url)
    loader = AsyncHtmlLoader(all_links)
    docs = loader.load()


    html2text = Html2TextTransformer()
    docs_transformed = html2text.transform_documents(docs)
    docs = [Document.from_langchain_format(doc) for doc in docs_transformed]
    return docs




docs = load_documents("https://docs.deeplake.ai/en/latest/")
Fetching pages: 100%|##########| 120/120 [00:13<00:00,  8.70it/s]
len(docs)
120
from llama_index.core.evaluation import generate_question_context_pairs
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
)
from llama_index.vector_stores.deeplake import DeepLakeVectorStore
from llama_index.core.node_parser import SimpleNodeParser
from llama_index.llms.openai import OpenAI




os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API token: ")
# # activeloop token is needed if you are not signed in using CLI: `activeloop login -u <USERNAME> -p <PASSWORD>`
os.environ["ACTIVELOOP_TOKEN"] = getpass.getpass(
    "Enter your ActiveLoop API token: "
)  # Get your API token from https://app.activeloop.ai, click on your profile picture in the top right corner, and select "API Tokens"


token = os.getenv("ACTIVELOOP_TOKEN")


vector_store = DeepLakeVectorStore(
    dataset_path="hub://activeloop-test/deeplake_docs_deepmemory2",
    overwrite=False,  # set to True to overwrite the existing dataset
    runtime={"tensor_db": True},
    token=token,
)
Deep Lake Dataset in hub://activeloop-test/deeplake_docs_deepmemory2 already exists, loading from the storage
def create_modules(vector_store, docs=[], populate_vector_store=True):
    if populate_vector_store:
        node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
        nodes = node_parser.get_nodes_from_documents(docs)
    else:
        nodes = []


    # by default, the node ids are set to random uuids. To ensure same id's per run, we manually set them.
    for idx, node in enumerate(nodes):
        node.id_ = f"node_{idx}"


    llm = OpenAI(model="gpt-4")
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    return storage_context, nodes, llm
(
    storage_context,
    nodes,
    llm,
) = create_modules(
    docs=docs,
    vector_store=vector_store,
    # populate_vector_store=False, # uncomment this line to skip populating the vector store
)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
deep_memory_retriever = vector_index.as_retriever(
    similarity_top_k=4, deep_memory=True
)

2. 训练深度记忆

Section titled “2. Training Deep Memory”

Image description

在上方，我们展示了 deep_memory 的整体工作原理架构。如您所见，为了训练它，您需要相关性数据、查询以及语料库数据（我们想要查询的数据）。语料库数据已在上一节中填充；这里，我们将生成问题和相关性数据。

  1. questionsquestions - 是一个字符串文本，其中每个字符串代表一个查询。
  2. relevancerelevance - 包含每个问题对应标准答案的链接。可能存在多个文档包含给定问题的答案。因此，相关性字段为List[List[tuple[str, float]]]，其中外层列表表示查询，内层列表表示相关文档。元组包含字符串与浮点数的配对，其中字符串表示源文档的标识（对应数据集中的id张量），而浮点数表示当前文档与问题的相关程度。
from llama_index.core.evaluation import (
    generate_question_context_pairs,
    EmbeddingQAFinetuneDataset,
)
import random




def create_train_test_datasets(
    number_of_samples=600, llm=None, nodes=None, save=False
):
    random_indices = random.sample(range(len(nodes)), number_of_samples)


    ratio = int(len(random_indices) * 0.8)


    train_indices = random_indices[:ratio]
    test_indices = random_indices[ratio:]


    train_nodes = [nodes[i] for i in train_indices]
    test_nodes = [nodes[i] for i in test_indices]


    train_qa_dataset = generate_question_context_pairs(
        train_nodes, llm=llm, num_questions_per_chunk=1
    )


    test_qa_dataset = generate_question_context_pairs(
        test_nodes, llm=llm, num_questions_per_chunk=1
    )


    # [optional] save
    if save:
        train_qa_dataset.save_json(
            f"deeplake_docs_{number_of_samples}_train.json"
        )
        test_qa_dataset.save_json(
            f"deeplake_docs_{number_of_samples}_test.json"
        )
    return train_qa_dataset, test_qa_dataset
train_qa_dataset, test_qa_dataset = create_train_test_datasets(
    number_of_samples=600, llm=llm, nodes=nodes, save=True
)
  4%|▍         | 19/480 [02:25<1:04:00,  8.33s/it]
train_qa_dataset = EmbeddingQAFinetuneDataset.from_json(
    "deeplake_docs_600_train.json"
)
test_qa_dataset = EmbeddingQAFinetuneDataset.from_json(
    "deeplake_docs_600_test.json"
)
def create_query_relevance(qa_dataset):
    """Function for converting llama-index dataset to correct format for deep memory training"""
    queries = [text for _, text in qa_dataset.queries.items()]
    relevant_docs = qa_dataset.relevant_docs
    relevance = []
    for doc in relevant_docs:
        relevance.append([(relevant_docs[doc][0], 1)])
    return queries, relevance
train_queries, train_relevance = create_query_relevance(train_qa_dataset)
test_queries, test_relevance = create_query_relevance(test_qa_dataset)
train_queries[:3]
['In the context of creating a bounding box tensor in a dataset, explain the significance of the "coords" argument and its keys "type" and "mode". What does the "type" key specify about the bounding box coordinates?',
 'Explain the process of creating an intrinsics tensor and appending intrinsics matrices in the context of computer vision. What are the dimensions of the intrinsics parameters and what do they represent? Also, describe the concept of a Segmentation Mask Htype and its role in image processing.',
 'In the context of querying for images in the MNIST Train Dataset using `ds.query`, what does the command "select * where labels == 0" signify and what is the expected output?']
train_relevance[:3]
[[('node_788', 1)], [('node_861', 1)], [('node_82', 1)]]
test_queries[:3]
['What are the steps to update the information of keypoints and connections in a tensor, and what types of data can be appended to keypoints?',
 'What is the command to create a mesh tensor in DeepLake and what are the supported compressions? Also, explain how to append a ply file containing mesh data to this tensor.',
 'What is a Sequence htype in the context of tensors and how does it function as a wrapper for other htypes? Provide examples.']
test_relevance[:3]
[[('node_933', 1)], [('node_671', 1)], [('node_471', 1)]]
from langchain.embeddings.openai import OpenAIEmbeddings


embeddings = OpenAIEmbeddings()




job_id = vector_store.vectorstore.deep_memory.train(
    queries=train_queries,
    relevance=train_relevance,
    embedding_function=embeddings.embed_documents,
)
Starting DeepMemory training job
Your Deep Lake dataset has been successfully created!








Preparing training data for deepmemory:




Creating 483 embeddings in 1 batches of size 483:: 100%|██████████| 1/1 [00:03<00:00,  3.67s/it]




DeepMemory training job started. Job ID: 65421a5003888c9ca36c72e8
vector_store.vectorstore.deep_memory.status(job_id)
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/adilkhan/deeplake_docs_deepmemory2
--------------------------------------------------------------
|                  65421a5003888c9ca36c72e8                  |
--------------------------------------------------------------
| status                     | completed                     |
--------------------------------------------------------------
| progress                   | eta: 12.2 seconds             |
|                            | recall@10: 67.01% (+18.56%)   |
--------------------------------------------------------------
| results                    | recall@10: 67.01% (+18.56%)   |
--------------------------------------------------------------

3. 深度记忆评估

Section titled “3. DeepMemory Evaluation”

太棒了！训练带来了显著的改进！现在，让我们在测试集上评估它的性能。

recalls = vector_store.vectorstore.deep_memory.evaluate(
    queries=test_queries,
    relevance=test_relevance,
    embedding_function=embeddings.embed_documents,
)
info Wed Nov  1 09:32:44 2023 GMT         Added distance metric `deepmemory_distance`.
Embedding queries took 0.95 seconds
---- Evaluating without Deep Memory ----
Recall@1:    12.5%
Recall@3:    23.3%
Recall@5:    30.8%
Recall@10:    50.8%
Recall@50:    94.2%
Recall@100:    95.8%
---- Evaluating with Deep Memory ----
Recall@1:    11.7%
Recall@3:    27.5%
Recall@5:    40.8%
Recall@10:    65.0%
Recall@50:    96.7%
Recall@100:    98.3%

令人印象深刻！我们在测试集上观察到召回率提升了15%。接下来，让我们使用RetrieverEvaluator来检查MRR（平均倒数排名）和命中率。

import pandas as pd




def display_results(eval_results):
    """Display results from evaluate."""
    hit_rates = []
    mrrs = []
    names = []
    for name, eval_result in eval_results.items():
        metric_dicts = []
        for er in eval_result:
            metric_dict = er.metric_vals_dict
            metric_dicts.append(metric_dict)


        full_df = pd.DataFrame(metric_dicts)


        hit_rate = full_df["hit_rate"].mean()
        mrr = full_df["mrr"].mean()


        hit_rates.append(hit_rate)
        mrrs.append(mrr)
        names.append(name)


    metric_df = pd.DataFrame(
        [
            {"retrievers": names[i], "hit_rate": hit_rates[i], "mrr": mrrs[i]}
            for i in range(2)
        ],
    )


    return metric_df

评估使用深度记忆的检索性能：

from llama_index.core.evaluation import RetrieverEvaluator


deep_memory_retriever = vector_index.as_retriever(
    similarity_top_k=10, vector_store_kwargs={"deep_memory": True}
)
dm_retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=deep_memory_retriever
)


dm_eval_results = await dm_retriever_evaluator.aevaluate_dataset(
    test_qa_dataset, retriever=dm_retriever_evaluator
)
from llama_index.core.evaluation import RetrieverEvaluator


naive_retriever = vector_index.as_retriever(similarity_top_k=10)
naive_retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=naive_retriever
)


naive_eval_results = await naive_retriever_evaluator.aevaluate_dataset(
    test_qa_dataset, retriever=naive_retriever
)
eval_results = {
    f"{mode} with Deep Memory top-10 eval": eval_result
    for mode, eval_result in zip(
        ["with", "without"], [dm_eval_results, naive_eval_results]
    )
}


display_results(eval_results)
.dataframe tbody tr th {
 vertical-align: top;
}


.dataframe thead th {
 text-align: right;
}
检索器 hit_rate 平均倒数排名
0 使用 Deep Memory 前10评估 0.650000 0.244775
1 无Deep Memory前10评估 0.508333 0.215129

不仅命中率有所提升，平均倒数排名也有所提高

4. 深度记忆推理

Section titled “4. Deep Memory Inference”
query_engine = vector_index.as_query_engine(
    vector_store_kwargs={"deep_memory": True}, llm=llm
)
response = query_engine.query(
    "How can you connect your own storage to the deeplake?"
)
print(response)
info Wed Nov  1 11:37:33 2023 GMT         Can't find any metric in the dataset.
You can connect your own storage to deeplake by using the `connect()` function in the deeplake API.
query_engine = vector_index.as_query_engine(
    vector_store_kwargs={"deep_memory": False}, llm=llm
)
response = query_engine.query(
    "How can you connect your own storage to the deeplake?"
)
print(response)
The context does not provide information on how to connect your own storage to Deep Lake.

根据我们的观察，在没有“深度记忆”的情况下，我们的模型倾向于产生不准确的结果，因为它检索了错误的上下文。