Section Aware chunking
-
NCERT book is organized as
- per chapter pdf
- sections span across pages
-
To solve this we need to split the entire chapter text into sections
- And lets have some chunk length (1000) with overlap of (150)
-
We need to Perform Recursive Splitting
- First split by sections (for this we need to use a Pattern)
- then by length with overlap
-
Exercise:
- Write code to read all text the pages content (page_n.txt) and combine into a single text
- Now perform recursive splitting by regex and then by length
-
Refer Here for the changes where we were able to chunk by section
-
Fix required is to respect the chunk_size (Max chunk size)
