跳至内容

自定义存储#

默认情况下，LlamaIndex隐藏了复杂性，让您用不到5行代码就能查询数据：

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the documents.")

在底层实现中，LlamaIndex还支持可替换的存储层，允许您自定义存储位置，包括已处理的文档（即Node对象）、嵌入向量和索引元数据。

底层API#

为此，不使用高级API，

index = VectorStoreIndex.from_documents(documents)

我们使用一个更低级别的API，提供更细粒度的控制：

from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.storage.index_store import SimpleIndexStore
from llama_index.core.vector_stores import SimpleVectorStore
from llama_index.core.node_parser import SentenceSplitter

# create parser and parse document into nodes
parser = SentenceSplitter()
nodes = parser.get_nodes_from_documents(documents)

# create storage context using default stores
storage_context = StorageContext.from_defaults(
    docstore=SimpleDocumentStore(),
    vector_store=SimpleVectorStore(),
    index_store=SimpleIndexStore(),
)

# create (or load) docstore and add nodes
storage_context.docstore.add_documents(nodes)

# build index
index = VectorStoreIndex(nodes, storage_context=storage_context)

# save index
index.storage_context.persist(persist_dir="<persist_dir>")

# can also set index_id to save multiple indexes to the same folder
index.set_index_id("<index_id>")
index.storage_context.persist(persist_dir="<persist_dir>")

# to load index later, make sure you setup the storage context
# this will loaded the persisted stores from persist_dir
storage_context = StorageContext.from_defaults(persist_dir="<persist_dir>")

# then load the index object
from llama_index.core import load_index_from_storage

loaded_index = load_index_from_storage(storage_context)

# if loading an index from a persist_dir containing multiple indexes
loaded_index = load_index_from_storage(storage_context, index_id="<index_id>")

# if loading multiple indexes from a persist dir
loaded_indicies = load_index_from_storage(
    storage_context, index_ids=["<index_id>", ...]
)

只需一行代码修改即可自定义底层存储，实例化不同的文档存储、索引存储和向量存储。 详情请参阅Document Stores、Vector Stores和Index Stores指南。

向量存储集成与存储#

我们的大多数向量存储集成将整个索引（向量+文本）存储在向量存储本身中。这带来了一个主要优势，即无需像上面所示那样显式持久化索引，因为向量存储已经托管并持久化了我们索引中的数据。

支持此实践的向量存储包括：

  • AzureAISearchVectorStore
  • ChatGPTRetrievalPluginClient
  • CassandraVectorStore
  • ChromaVectorStore
  • EpsillaVectorStore
  • DocArrayHnswVectorStore
  • DocArrayInMemoryVectorStore
  • JaguarVectorStore
  • LanceDBVectorStore
  • MetalVectorStore
  • MilvusVectorStore
  • MyScale向量存储
  • OpensearchVectorStore
  • PineconeVectorStore
  • QdrantVectorStore
  • TablestoreVectorStore
  • RedisVectorStore
  • UpstashVectorStore
  • WeaviateVectorStore

下面是一个使用Pinecone的小例子:

import pinecone
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.vector_stores.pinecone import PineconeVectorStore

# Creating a Pinecone index
api_key = "api_key"
pinecone.init(api_key=api_key, environment="us-west1-gcp")
pinecone.create_index(
    "quickstart", dimension=1536, metric="euclidean", pod_type="p1"
)
index = pinecone.Index("quickstart")

# construct vector store
vector_store = PineconeVectorStore(pinecone_index=index)

# create storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# load documents
documents = SimpleDirectoryReader("./data").load_data()

# create index, which will insert documents/vectors to pinecone
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

如果您已经有一个包含数据的现有向量存储，可以按以下方式连接它并直接创建VectorStoreIndex：

index = pinecone.Index("quickstart")
vector_store = PineconeVectorStore(pinecone_index=index)
loaded_index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
优云智算