Embeddings
- Embedding is process of converting text into vectors which helps in semantic search.
- There are many embedding models
- OpenAI
- Vertex
- Huggingface embedding
- Each model will have dimensions
- For querying and indexing we should use the same embedding model.
| Platform | Default Dim | High-End Dim | Notes |
| —————- | ———– | ———— | —————- |
| Vertex AI | 768 | 1408 | Cost-efficient |
| OpenAI | 768 | 3072 | Best raw quality |
| Azure OpenAI | 768 | 3072 | Enterprise-ready |
| AWS Bedrock | 1024–1536 | 1536 | AWS-native |
- Refer Here for langchain embedding docs
Vector Databases
- These databases store vectors and metadata
- Popular Vector databases
- FIASS
- Chroma
- PGVector
- Pinecone
- Weaviate
- Refer Here for vector store integration with langchain
- Lets try FIASS and Chromadb
Retrievers
- Refer Here for official docs
- Refer Here for the notebook
Lets try an indexing pipeline
- Refer Here for example.
