Chunking or Splitting
- IN RAG we deal with two models
- large language model (llm)
- embedding model
- Embedding models convert text into vector which we store in vector database.
- For effective retrieval, it better if we chunk or split the document into multiple chunks.
-
chunk in langchain will be document.
-
For chunking Refer Here
-
Chunking stragies
- Chunking also uses overlap.
Experiment with chunking
- Prompt
Popular embeddings by different providers, their token limit and vector dimensions in a tabular form
- Lets add a package
uv add langchain-text-splitters
-
Chunking visualizer Refer Here
-
Note in reality we prefer loading and splitting as one operation
-
Exercise: While we load and split, i want to add additional metadata
- ingested_date = “Current date”
- project = “learning”
-
Refer Here for the notebook
