使用向量存储
LlamaIndex 提供了与向量存储/向量数据库的多个集成点:
- LlamaIndex 可以将向量存储本身用作索引。与其他任何索引一样,该索引可以存储文档并用于回答查询。
- LlamaIndex 可以从向量存储中加载数据,类似于任何其他数据连接器。这些数据随后可在 LlamaIndex 数据结构中使用。
LlamaIndex 还支持不同的向量存储作为 VectorStoreIndex 的存储后端。
- 阿里云开放搜索 (
AlibabaCloudOpenSearchStore). 快速开始. - Amazon Neptune - Neptune Analytics (
NeptuneAnalyticsVectorStore)。在Neptune Analytics中使用向量相似性。 - 通过CQL使用Apache Cassandra®和Astra DB(
CassandraVectorStore)。安装指南 快速开始 - Astra DB (
AstraDBVectorStore)。快速入门。 - AWS Document DB (
AWSDocDbVectorStore)。快速入门。 - Azure AI 搜索 (
AzureAISearchVectorStore)。快速入门 - Azure Cosmos DB Mongo vCore(
AzureCosmosDBMongoDBVectorSearch). 快速入门 - Azure Cosmos DB NoSql (
AzureCosmosDBNoSqlVectorSearch). 快速入门 - Chroma (
ChromaVectorStore) 安装 - ClickHouse (
ClickHouseVectorStore) 安装 - Couchbase (
CouchbaseSearchVectorStore) 安装 - DashVector (
DashVectorStore)。安装指南。 - DeepLake (
DeepLakeVectorStore) 安装 - DocArray (
DocArrayHnswVectorStore,DocArrayInMemoryVectorStore)。安装/Python客户端。 - Elasticsearch (
ElasticsearchStore) 安装 - Epsilla (
EpsillaVectorStore) Installation/Quickstart - Faiss (
FaissVectorStore). 安装. - Google AlloyDB for PostgreSQL (
AlloyDBVectorStore)。快速入门。 - Google Cloud SQL for PostgreSQL (
PostgresVectorStore). 快速入门 - Hnswlib(
HnswlibVectorStore)。安装指南。 - txtai (
TxtaiVectorStore). 安装. - 捷豹 (
JaguarVectorStore). 安装. - Lantern (
LanternVectorStore). 快速入门. - MariaDB (
MariaDBVectorStore). MariaDB 向量概述 - Milvus (
MilvusVectorStore). 安装 - MongoDB Atlas (
MongoDBAtlasVectorSearch). 安装/快速开始. - MyScale (
MyScaleVectorStore)。快速开始。安装/Python客户端。 - Neo4j (
Neo4jVectorIndex). 安装. - OceanBase (
OceanBaseVectorStore)。OceanBase 概述。快速开始。Python 客户端 - Opensearch (
OpensearchVectorStore) 将Opensearch作为向量数据库. 快速开始 - Pinecone (
PineconeVectorStore)。安装/快速开始。 - Qdrant (
QdrantVectorStore) 安装指南 Python客户端 - LanceDB (
LanceDBVectorStore) Installation/Quickstart - Redis (
RedisVectorStore). 安装. - Relyt (
RelytVectorStore). 快速入门. - Supabase(
SupabaseVectorStore)。快速入门。 - 表格存储 (
Tablestore)。表格存储概述。快速入门。Python客户端。 - TiDB (
TiDBVectorStore)。快速开始。安装指南。Python客户端。 - 时间刻度 (
TimescaleVectorStore). 安装. - Upstash (
UpstashVectorStore)。快速开始 - VectorX 数据库 (
VectorXVectorStore)。快速入门 - Vertex AI 向量搜索 (
VertexAIVectorStore)。快速入门 - Weaviate (
WeaviateVectorStore)。安装指南。Python客户端。 - WordLift (
WordliftVectorStore). 快速入门. Python客户端. - Zep (
ZepVectorStore)。安装指南。Python客户端。 - Zilliz (
MilvusVectorStore). 快速入门
详细的API参考文档请点击此处。
与LlamaIndex中的任何其他索引(树索引、关键词表索引、列表索引)类似,VectorStoreIndex可以在任何文档集合上构建。我们使用索引中的向量存储来存储输入文本块的嵌入向量。
构建完成后,该索引可用于查询。
默认向量存储索引构建/查询
默认情况下,VectorStoreIndex 使用内存中的 SimpleVectorStore,该存储作为默认存储上下文的一部分进行初始化。
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
# Load documents and build indexdocuments = SimpleDirectoryReader("../paul_graham_essay/data").load_data()index = VectorStoreIndex.from_documents(documents)
# Query indexquery_engine = index.as_query_engine()response = query_engine.query("What did the author do growing up?")自定义向量存储索引构建/查询
我们可以按如下方式查询自定义向量存储:
from llama_index.core import ( VectorStoreIndex, SimpleDirectoryReader, StorageContext,)from llama_index.vector_stores.deeplake import DeepLakeVectorStore
# construct vector store and customize storage contextstorage_context = StorageContext.from_defaults( vector_store=DeepLakeVectorStore(dataset_path="<dataset_path>"))
# Load documents and build indexdocuments = SimpleDirectoryReader("../paul_graham_essay/data").load_data()index = VectorStoreIndex.from_documents( documents, storage_context=storage_context)
# Query indexquery_engine = index.as_query_engine()response = query_engine.query("What did the author do growing up?")以下我们展示更多关于如何构建我们支持的各种向量存储的示例。
阿里云开放搜索
from llama_index.vector_stores.alibabacloud_opensearch import ( AlibabaCloudOpenSearchStore, AlibabaCloudOpenSearchConfig,)
config = AlibabaCloudOpenSearchConfig( endpoint="***", instance_id="***", username="your_username", password="your_password", table_name="llama",)
vector_store = AlibabaCloudOpenSearchStore(config)适用于 PostgreSQL 的 Google AlloyDB
pip install llama-indexpip install llama-index-alloydb-pgpip install llama-index-llms-vertexgcloud services enable aiplatform.googleapis.comfrom llama_index_alloydb_pg import AlloyDBEngine, AlloyDBVectorStorefrom llama_index.core import Settingsfrom llama_index.embeddings.vertex import VertexTextEmbeddingfrom llama_index.llms.vertex import Verteximport google.auth
# Replace with your own AlloyDB infoengine = AlloyDBEngine.from_instance( project_id=PROJECT_ID, region=REGION, cluster=CLUSTER, instance=INSTANCE, database=DATABASE, user=USER, password=PASSWORD,)
engine.init_vector_store_table( table_name=TABLE_NAME, vector_size=768, # Vector size for VertexAI model(textembedding-gecko@latest))
vector_store = AlloyDBVectorStore.create_sync( engine=engine, table_name=TABLE_NAME,)Amazon Neptune - Neptune 分析
from llama_index.vector_stores.neptune import NeptuneAnalyticsVectorStore
graph_identifier = ""embed_dim = 1536
neptune_vector_store = NeptuneAnalyticsVectorStore( graph_identifier=graph_identifier, embedding_dimension=1536)Apache Cassandra®
from llama_index.vector_stores.cassandra import CassandraVectorStoreimport cassio
# To use an Astra DB cloud instance through CQL:cassio.init(database_id="1234abcd-...", token="AstraCS:...")
# For a Cassandra cluster:from cassandra.cluster import Cluster
cluster = Cluster(["127.0.0.1"])cassio.init(session=cluster.connect(), keyspace="my_keyspace")
# After the above `cassio.init(...)`, create a vector store:vector_store = CassandraVectorStore( table="cass_v_table", embedding_dimension=1536)Astra DB
from llama_index.vector_stores.astra_db import AstraDBVectorStore
astra_db_store = AstraDBVectorStore( token="AstraCS:xY3b...", # Your Astra DB token api_endpoint="https://012...abc-us-east1.apps.astra.datastax.com", # Your Astra DB API endpoint collection_name="astra_v_table", # Table name of your choice embedding_dimension=1536, # Embedding dimension of the embeddings model used)Azure 认知搜索
from azure.core.credentials import AzureKeyCredentialfrom llama_index.vector_stores.azureaisearch import AzureAISearchVectorStore
search_service_api_key = "YOUR-AZURE-SEARCH-SERVICE-ADMIN-KEY"search_service_endpoint = "YOUR-AZURE-SEARCH-SERVICE-ENDPOINT"search_service_api_version = "2023-11-01"credential = AzureKeyCredential(search_service_api_key)
# Index name to useindex_name = "llamaindex-vector-demo"
client = SearchIndexClient( endpoint=search_service_endpoint, credential=credential,)
vector_store = AzureAISearchVectorStore( search_or_index_client=client, index_name=index_name, embedding_dimensionality=1536,)Azure CosmosDB Mongo vCore
import pymongoimport osfrom llama_index.vector_stores.azurecosmosmongo import ( AzureCosmosDBMongoDBVectorSearch,)
# Set up the connection string with your Azure CosmosDB MongoDB URIconnection_string = os.getenv("YOUR_AZURE_COSMOSDB_MONGODB_URI")mongodb_client = pymongo.MongoClient(connection_string)
# Create an instance of AzureCosmosDBMongoDBVectorSearchvector_store = AzureCosmosDBMongoDBVectorSearch( mongodb_client=mongodb_client, db_name="demo_vectordb", collection_name="paul_graham_essay",)Azure CosmosDB NoSQL
from azure.cosmos import CosmosClient, PartitionKeyimport osfrom llama_index.vector_stores.azurecosmosnosql import ( AzureCosmosDBNoSqlVectorSearch,)
URL = os.getenv("AZURE_COSMOSDB_URI")KEY = os.getenv("AZURE_COSMOSDB_KEY")database_name = "test_database"container_name = "test_container"test_client = CosmosClient(URL, credential=KEY)
indexing_policy = { "indexingMode": "consistent", "includedPaths": [{"path": "/*"}], "excludedPaths": [{"path": '/"_etag"/?'}], "vectorIndexes": [{"path": "/embedding", "type": "quantizedFlat"}],}
vector_embedding_policy = { "vectorEmbeddings": [ { "path": "/embedding", "dataType": "float32", "distanceFunction": "cosine", "dimensions": 1536, } ]}
partition_key = PartitionKey(path="/id")cosmos_container_properties_test = {"partition_key": partition_key}cosmos_database_properties_test = {}
vector_store = AzureCosmosDBNoSqlVectorSearch( cosmos_client=test_client, vector_embedding_policy=vector_embedding_policy, indexing_policy=indexing_policy, database_name=database_name, container_name=container_name, cosmos_database_properties=cosmos_database_properties_test, cosmos_container_properties=cosmos_container_properties_test,)Chroma
import chromadbfrom llama_index.vector_stores.chroma import ChromaVectorStore
# Creating a Chroma client# EphemeralClient operates purely in-memory, PersistentClient will also save to diskchroma_client = chromadb.EphemeralClient()chroma_collection = chroma_client.create_collection("quickstart")
# construct vector storevector_store = ChromaVectorStore( chroma_collection=chroma_collection,)ClickHouse
import clickhouse_connectfrom llama_index.vector_stores import ClickHouseVectorStore
# Creating a ClickHouse clientclient = clickhouse_connect.get_client( host="YOUR_CLUSTER_HOST", port=8123, username="YOUR_USERNAME", password="YOUR_CLUSTER_PASSWORD",)
# construct vector storevector_store = ClickHouseVectorStore(clickhouse_client=client)Couchbase
from datetime import timedelta
from couchbase.auth import PasswordAuthenticatorfrom couchbase.cluster import Clusterfrom couchbase.options import ClusterOptions
# Create a Couchbase Cluster objectauth = PasswordAuthenticator("DATABASE_USERNAME", "DATABASE_PASSWORD")options = ClusterOptions(auth)cluster = Cluster("CLUSTER_CONNECTION_STRING", options)
# Wait until the cluster is ready for use.cluster.wait_until_ready(timedelta(seconds=5))
# Create the Vector Storevector_store = CouchbaseSearchVectorStore( cluster=cluster, bucket_name="BUCKET_NAME", scope_name="SCOPE_NAME", collection_name="COLLECTION_NAME", index_name="SEARCH_INDEX_NAME",)DashVector
import dashvectorfrom llama_index.vector_stores.dashvector import DashVectorStore
# init dashvector clientclient = dashvector.Client( api_key="your-dashvector-api-key", endpoint="your-dashvector-cluster-endpoint",)
# creating a DashVector collectionclient.create("quickstart", dimension=1536)collection = client.get("quickstart")
# construct vector storevector_store = DashVectorStore(collection)DeepLake
import osimport getpathfrom llama_index.vector_stores.deeplake import DeepLakeVectorStore
os.environ["OPENAI_API_KEY"] = getpath.getpath("OPENAI_API_KEY: ")os.environ["ACTIVELOOP_TOKEN"] = getpath.getpath("ACTIVELOOP_TOKEN: ")dataset_path = "hub://adilkhan/paul_graham_essay"
# construct vector storevector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True)DocArray
from llama_index.vector_stores.docarray import ( DocArrayHnswVectorStore, DocArrayInMemoryVectorStore,)
# construct vector storevector_store = DocArrayHnswVectorStore(work_dir="hnsw_index")
# alternatively, construct the in-memory vector storevector_store = DocArrayInMemoryVectorStore()Elasticsearch
首先,您可以在本地或Elastic云上启动Elasticsearch。
要在本地使用 Docker 启动 Elasticsearch,请运行以下命令:
docker run -p 9200:9200 \ -e "discovery.type=single-node" \ -e "xpack.security.enabled=false" \ -e "xpack.security.http.ssl.enabled=false" \ -e "xpack.license.self_generated.type=trial" \ docker.elastic.co/elasticsearch/elasticsearch:8.9.0然后连接并使用 Elasticsearch 作为 LlamaIndex 的向量数据库
from llama_index.vector_stores.elasticsearch import ElasticsearchStore
vector_store = ElasticsearchStore( index_name="llm-project", es_url="http://localhost:9200", # Cloud connection options: # es_cloud_id="<cloud_id>", # es_user="elastic", # es_password="<password>",)这可以与 VectorStoreIndex 配合使用,为检索、查询、删除、持久化索引等操作提供查询接口。
Epsilla
from pyepsilla import vectordbfrom llama_index.vector_stores.epsilla import EpsillaVectorStore
# Creating an Epsilla clientepsilla_client = vectordb.Client()
# Construct vector storevector_store = EpsillaVectorStore(client=epsilla_client)注意: EpsillaVectorStore 依赖于 pyepsilla 库和一个正在运行的 Epsilla 向量数据库。
如果尚未安装,请使用 pip/pip3 install pyepsilla。
可以通过 Docker 镜像找到正在运行的 Epsilla 向量数据库。
完整说明请参阅以下文档:
https://epsilla-inc.gitbook.io/epsilladb/quick-start
Faiss
import faissfrom llama_index.vector_stores.faiss import FaissVectorStore
# create faiss indexd = 1536faiss_index = faiss.IndexFlatL2(d)
# construct vector storevector_store = FaissVectorStore(faiss_index)
# if update/delete functionality is needed you can leverage the FaissMapVectorStore
d = 1536faiss_index = faiss.IndexFlatL2(d)id_map_index = faiss.IndexIDMap2(faiss_index)vector_store = FaissMapVectorStore(id_map_index)
...
# NOTE: since faiss index is in-memory, we need to explicitly call# vector_store.persist() or storage_context.persist() to save it to disk.# persist() takes in optional arg persist_path. If none give, will use default paths.storage_context.persist()适用于 PostgreSQL 的 Google Cloud SQL
pip install llama-indexpip install llama-index-cloud-sql-pgpip install llama-index-llms-vertexgcloud services enable aiplatform.googleapis.comfrom llama_index_cloud_sql_pg import PostgresEngine, PostgresVectorStorefrom llama_index.core import Settingsfrom llama_index.embeddings.vertex import VertexTextEmbeddingfrom llama_index.llms.vertex import Verteximport google.auth
# Replace with your own Cloud SQL infoengine = PostgresEngine.from_instance( project_id=PROJECT_ID, region=REGION, instance=INSTANCE, database=DATABASE, user=USER, password=PASSWORD,)
engine.init_vector_store_table( table_name=TABLE_NAME, vector_size=768, # Vector size for VertexAI model(textembedding-gecko@latest))
vector_store = PostgresVectorStore.create_sync( engine=engine, table_name=TABLE_NAME,)txtai
import txtaifrom llama_index.vector_stores.txtai import TxtaiVectorStore
# create txtai indextxtai_index = txtai.ann.ANNFactory.create( {"backend": "numpy", "dimension": 512})
# construct vector storevector_store = TxtaiVectorStore(txtai_index)捷豹
from llama_index.core.schema import TextNodefrom llama_index.core.vector_stores import VectorStoreQueryfrom jaguardb_http_client.JaguarHttpClient import JaguarHttpClientfrom llama_index.vector_stores.jaguar import JaguarVectorStore
# construct vector store clienturl = "http://127.0.0.1:8080/fwww/"pod = "vdb"store = "llamaindex_rag_store"vector_index = "v"vector_type = "cosine_fraction_float"vector_dimension = 3
# require JAGUAR_API_KEY environment variable or file $HOME/.jagrc to hold the# jaguar API key to connect to jaguar store servervector_store = JaguarVectorStore( pod, store, vector_index, vector_type, vector_dimension, url)
# login to jaguar server for security authenticationvector_store.login()
# create a vector store on the back-end servermetadata_fields = "author char(32), category char(16)"text_size = 1024vector_store.create(metadata_fields, text_size)
# store some textnode = TextNode( text="Return of King Lear", metadata={"author": "William", "category": "Tragedy"}, embedding=[0.9, 0.1, 0.4],)vector_store.add(nodes=[node], use_node_metadata=True)
# make a queryqembedding = [0.4, 0.2, 0.8]vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=1)query_result = vector_store.query(vsquery)
# make a query with metadata filter (where condition)qembedding = [0.6, 0.1, 0.4]vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=3)where = "author='Eve' or (author='Adam' and category='History')"query_result = vector_store.query(vsquery, where=where)
# make a query ignoring old data (with time cutoff)qembedding = [0.3, 0.3, 0.8]vsquery = VectorStoreQuery(query_embedding=qembedding, similarity_top_k=3)args = "day_cutoff=180" # only search recent 180 days dataquery_result = vector_store.query(vsquery, args=args)
# check if a vector is anomaloustext = ("Gone With The Wind",)embed_of_text = [0.7, 0.1, 0.2]node = TextNode(text=text, embedding=embed_of_text)true_or_false = vector_store.is_anomalous(node)
# llama_index RAG applicationfrom llama_index.embeddings.openai import OpenAIEmbeddingfrom llama_index.core import StorageContextfrom llama_index.core import VectorStoreIndex
question = "What did the author do growing up?"
storage_context = StorageContext.from_defaults(vector_store=vector_store)embed_model = OpenAIEmbedding()embed_of_question = [0.7, 0.1, 0.2]
db_documents = vector_store.load_documents(embed_of_question, 10)index = VectorStoreIndex.from_documents( db_documents, embed_model=embed_model, storage_context=storage_context,)
query_engine = index.as_query_engine()print(f"Question: {question}")response = query_engine.query(question)print(f"Answer: {str(response)}")
# logout to clean up resourcesvector_store.logout()注意: 客户端(需要 jaguardb-http-client) <—> HTTP 网关 <—> JaguarDB 服务器 客户端需要运行: “pip install -U jaguardb-http-client”
MariaDB
from llama_index.vector_stores.mariadb import MariaDBVectorStore
vector_store = MariaDBVectorStore.from_params( host="localhost", port=3306, user="llamaindex", password="password", database="vectordb", table_name="llama_index_vectorstore", embed_dim=1536, # OpenAI embedding dimension)Milvus
- Milvus 索引提供存储文档及其嵌入向量的能力。
import pymilvusfrom llama_index.vector_stores.milvus import MilvusVectorStore
# construct vector storevector_store = MilvusVectorStore( uri="https://localhost:19530", overwrite="True")注意: MilvusVectorStore 依赖于 pymilvus 库。
如果尚未安装,请使用 pip install pymilvus。
如果在构建 grpcio 的wheel包时遇到问题,请检查是否正在使用Python 3.11
(存在一个已知问题:https://github.com/milvus-io/pymilvus/issues/1308)
并尝试降级版本。
MongoDBAtlas
# Provide URI to constructor, or use environment variableimport pymongofrom llama_index.vector_stores.mongodb import MongoDBAtlasVectorSearchfrom llama_index.core import VectorStoreIndexfrom llama_index.core import StorageContextfrom llama_index.core import SimpleDirectoryReader
# mongo_uri = os.environ["MONGO_URI"]mongo_uri = ( "mongodb+srv://<username>:<password>@<host>?retryWrites=true&w=majority")mongodb_client = pymongo.MongoClient(mongo_uri)
# construct storestore = MongoDBAtlasVectorSearch(mongodb_client)storage_context = StorageContext.from_defaults(vector_store=store)uber_docs = SimpleDirectoryReader( input_files=["../data/10k/uber_2021.pdf"]).load_data()
# construct indexindex = VectorStoreIndex.from_documents( uber_docs, storage_context=storage_context)MyScale
import clickhouse_connectfrom llama_index.vector_stores.myscale import MyScaleVectorStore
# Creating a MyScale clientclient = clickhouse_connect.get_client( host="YOUR_CLUSTER_HOST", port=8443, username="YOUR_USERNAME", password="YOUR_CLUSTER_PASSWORD",)
# construct vector storevector_store = MyScaleVectorStore(myscale_client=client)Neo4j
- Neo4j 存储文本、元数据和嵌入向量,并可定制以元数据形式返回图数据。
from llama_index.vector_stores.neo4jvector import Neo4jVectorStore
# construct vector storeneo4j_vector = Neo4jVectorStore( username="neo4j", password="pleaseletmein", url="bolt://localhost:7687", embed_dim=1536,)Pinecone
import pineconefrom llama_index.vector_stores.pinecone import PineconeVectorStore
# Creating a Pinecone indexapi_key = "api_key"pinecone.init(api_key=api_key, environment="us-west1-gcp")pinecone.create_index( "quickstart", dimension=1536, metric="euclidean", pod_type="p1")index = pinecone.Index("quickstart")
# construct vector storevector_store = PineconeVectorStore(pinecone_index=index)Qdrant
import qdrant_clientfrom llama_index.vector_stores.qdrant import QdrantVectorStore
# Creating a Qdrant vector storeclient = qdrant_client.QdrantClient( host="<qdrant-host>", api_key="<qdrant-api-key>", https=True)collection_name = "paul_graham"
# construct vector storevector_store = QdrantVectorStore( client=client, collection_name=collection_name,)Redis
首先,启动 Redis-Stack(或从 Redis 提供商获取 URL)
docker run --name redis-vecdb -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest然后连接并使用 Redis 作为 LlamaIndex 的向量数据库
from llama_index.vector_stores.redis import RedisVectorStore
vector_store = RedisVectorStore( index_name="llm-project", redis_url="redis://localhost:6379", overwrite=True,)这可以与 VectorStoreIndex 配合使用,为检索、查询、删除、持久化索引等操作提供查询接口。
SingleStore
from llama_index.vector_stores.singlestoredb import SingleStoreVectorStoreimport os
# can set the singlestore db url in env# or pass it in as an argument to the SingleStoreVectorStore constructoros.environ["SINGLESTOREDB_URL"] = "PLACEHOLDER URL"vector_store = SingleStoreVectorStore( table_name="embeddings", content_field="content", metadata_field="metadata", vector_field="vector", timeout=30,)表格存储
import tablestorefrom llama_index.vector_stores.tablestore import TablestoreVectorStore
# create a vector store that does not support filtering non-vector fieldssimple_vector_store = TablestoreVectorStore( endpoint="<end_point>", instance_name="<instance_name>", access_key_id="<access_key_id>", access_key_secret="<access_key_secret>", vector_dimension=512,)
# create a vector store that support filtering non-vector fieldsvector_store_with_meta_data = TablestoreVectorStore( endpoint="<end_point>", instance_name="<instance_name>", access_key_id="<access_key_id>", access_key_secret="<access_key_secret>", vector_dimension=512, # optional: custom metadata mapping is used to filter non-vector fields. metadata_mappings=[ tablestore.FieldSchema( "type", # non-vector fields tablestore.FieldType.KEYWORD, index=True, enable_sort_and_agg=True, ), tablestore.FieldSchema( "time", # non-vector fields tablestore.FieldType.LONG, index=True, enable_sort_and_agg=True, ), ],)TiDB
from llama_index.vector_stores.tidbvector import TiDBVectorStore
tidbvec = TiDBVectorStore( # connection url format # - mysql+pymysql://root@34.212.137.91:4000/test connection_string="PLACEHOLDER URL", table_name="llama_index_vectorstore", distance_strategy="cosine", vector_dimension=1536,)时间尺度
from llama_index.vector_stores.timescalevector import TimescaleVectorStore
vector_store = TimescaleVectorStore.from_params( service_url="YOUR TIMESCALE SERVICE URL", table_name="paul_graham_essay",)Upstash
from llama_index.vector_stores.upstash import UpstashVectorStore
vector_store = UpstashVectorStore(url="YOUR_URL", token="YOUR_TOKEN")VectorX 数据库
from vecx_llamaindex import VectorXVectorStorefrom llama_index.core import StorageContextimport time
# Create a unique index name with timestamp to avoid conflictstimestamp = int(time.time())index_name = f"llamaindex_demo_{timestamp}"
# Set up the embedding modelembed_model = OpenAIEmbedding()
# Get the embedding dimensiondimension = 1536 # OpenAI's default embedding dimension
# Initialize the VectorX vector storevector_store = VectorXVectorStore.from_params( api_token=vecx_api_token, encryption_key=encryption_key, index_name=index_name, dimension=dimension, space_type="cosine", # Can be "cosine", "l2", or "ip")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents( documents, storage_context=storage_context, embed_model=embed_model)
query_engine = index.as_query_engine()
# Ask a questionresponse = query_engine.query("What is Python?")
# Create a filtered retriever to only search within AI-related documentsai_filter = MetadataFilter( key="category", value="ai", operator=FilterOperator.EQ)ai_filters = MetadataFilters(filters=[ai_filter])
# Create a filtered query enginefiltered_query_engine = index.as_query_engine(filters=ai_filters)
# Ask a general question but only using AI documentsresponse = filtered_query_engine.query("What is learning from data?")Vertex AI 向量搜索
from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore
vector_store = VertexAIVectorStore( project_id="[your-google-cloud-project-id]", region="[your-google-cloud-region]", index_id="[your-index-resource-name]", endpoint_id="[your-index-endpoint-name]",)Weaviate
import weaviatefrom llama_index.vector_stores.weaviate import WeaviateVectorStore
# creating a Weaviate clientresource_owner_config = weaviate.AuthClientPassword( username="<username>", password="<password>",)client = weaviate.Client( "https://<cluster-id>.semi.network/", auth_client_secret=resource_owner_config,)
# construct vector storevector_store = WeaviateVectorStore(weaviate_client=client)泽普
Zep存储文本、元数据和嵌入向量。所有内容都会在搜索结果中返回。
from llama_index.vector_stores.zep import ZepVectorStore
vector_store = ZepVectorStore( api_url="<api_url>", api_key="<api_key>", collection_name="<unique_collection_name>", # Can either be an existing collection or a new one embedding_dimensions=1536, # Optional, required if creating a new collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents( documents, storage_context=storage_context)
# Query index using both a text query and metadata filtersfilters = MetadataFilters( filters=[ExactMatchFilter(key="theme", value="Mafia")])retriever = index.as_retriever(filters=filters)result = retriever.retrieve("What is inception about?")Zilliz
- Zilliz Cloud(Milvus的托管版本)使用Milvus索引并附带一些额外参数。
import pymilvusfrom llama_index.vector_stores.milvus import MilvusVectorStore
# construct vector storevector_store = MilvusVectorStore( uri="foo.vectordb.zillizcloud.com", token="your_token_here", overwrite="True",)LlamaIndex 支持从大量来源加载数据。有关更多详情和 API 文档,请参阅数据连接器。
AlloyDB同时存储文档和向量。 本教程演示同步接口。所有同步方法都有对应的异步方法。 以下是如何使用AlloyDB的示例:
pip install llama-indexpip install llama-index-alloydb-pgfrom llama_index.core import SummaryIndexfrom llama_index_alloydb_pg import AlloyDBEngine, AlloyDBReader
engine = AlloyDBEngine.from_instance( project_id=PROJECT_ID, region=REGION, cluster=CLUSTER, instance=INSTANCE, database=DATABASE, user=USER, password=PASSWORD,)reader = AlloyDBReader.create_sync( engine, table_name=TABLE_NAME,)documents = reader.load_data()
index = SummaryIndex.from_documents(documents)
query_engine = index.as_query_engine()response = query_engine.query("<query_text>")display(Markdown(f"<b>{response}</b>"))Google Cloud SQL for PostgreSQL 同时存储文档和向量。 本教程演示同步接口。所有同步方法都有对应的异步方法。 以下是如何使用 Cloud SQL for PostgreSQL 的示例:
pip install llama-indexpip install llama-index-cloud-sql-pgfrom llama_index.core import SummaryIndexfrom llama_index_cloud_sql_pg import PostgresEngine, PostgresReader
engine = PostgresEngine.from_instance( project_id=PROJECT_ID, region=REGION, instance=INSTANCE, database=DATABASE, user=USER, password=PASSWORD,)reader = PostgresReader.create_sync( engine, table_name=TABLE_NAME,)documents = reader.load_data()
index = SummaryIndex.from_documents(documents)
query_engine = index.as_query_engine()response = query_engine.query("<query_text>")display(Markdown(f"<b>{response}</b>"))Chroma 同时存储文档和向量。以下是如何使用 Chroma 的示例:
from llama_index.readers.chroma import ChromaReaderfrom llama_index.core import SummaryIndex
# The chroma reader loads data from a persisted Chroma collection.# This requires a collection name and a persist directory.reader = ChromaReader( collection_name="chroma_collection", persist_directory="examples/data_connectors/chroma_collection",)
query_vector = [n1, n2, n3, ...]
documents = reader.load_data( collection_name="demo", query_vector=query_vector, limit=5)index = SummaryIndex.from_documents(documents)
query_engine = index.as_query_engine()response = query_engine.query("<query_text>")display(Markdown(f"<b>{response}</b>"))Qdrant 同时存储文档和向量。以下是如何使用 Qdrant 的示例:
from llama_index.readers.qdrant import QdrantReader
reader = QdrantReader(host="localhost")
# the query_vector is an embedding representation of your query_vector# Example query_vector# query_vector = [0.3, 0.3, 0.3, 0.3, ...]
query_vector = [n1, n2, n3, ...]
# NOTE: Required args are collection_name, query_vector.# See the Python client: https;//github.com/qdrant/qdrant_client# for more details
documents = reader.load_data( collection_name="demo", query_vector=query_vector, limit=5)注意:由于 Weaviate 可以存储文档和向量对象的混合体,用户可以选择显式指定 class_name 和 properties 来查询文档,或者也可以选择指定原始 GraphQL 查询。具体用法请参见下文。
# option 1: specify class_name and properties
# 1) load data using class_name and propertiesdocuments = reader.load_data( class_name="<class_name>", properties=["property1", "property2", "..."], separate_documents=True,)
# 2) example GraphQL queryquery = """{ Get { <class_name> { <property1> <property2> } }}"""
documents = reader.load_data(graphql_query=query, separate_documents=True)注意:Pinecone和Faiss数据加载器均假设相应的数据源仅存储向量;文本内容存储在其他位置。因此,这两个数据加载器都要求用户在load_data调用中指定一个id_to_text_map。
例如,这是Pinecone数据加载器PineconeReader的示例用法:
from llama_index.readers.pinecone import PineconeReader
reader = PineconeReader(api_key=api_key, environment="us-west1-gcp")
id_to_text_map = { "id1": "text blob 1", "id2": "text blob 2",}
query_vector = [n1, n2, n3, ...]
documents = reader.load_data( index_name="quickstart", id_to_text_map=id_to_text_map, top_k=3, vector=query_vector, separate_documents=True,)- 阿里云开放搜索
- Amazon Neptune - Neptune 分析
- Astra DB
- 异步索引创建
- Azure AI 搜索
- Azure Cosmos DB
- 卡桑德拉
- Chroma数据库
- Couchbase
- 仪表板
- 深度湖
- DocArray HNSW
- DocArray 内存存储
- 埃斯皮拉
- 适用于 PostgreSQL 的 Google AlloyDB
- 适用于 PostgreSQL 的 Google Cloud SQL
- LanceDB
- 灯笼
- Milvus
- Milvus 异步 API
- Milvus 全文搜索
- Milvus 混合搜索
- MyScale
- ElsaticSearch
- FAISS
- MongoDB Atlas
- Neo4j
- OpenSearch
- Pinecone
- Pinecone 混合搜索
- PGvectoRS
- PostgreSQL
- Redis
- Qdrant
- Qdrant 混合搜索
- Rockset
- 简单
- Supabase
- 表格存储
- 阿里云Tair
- 腾讯
- 时间刻度
- Upstash
- VectorX 数据库
- Weaviate
- Weaviate 混合搜索
- WordLift
- 泽普