Skip to content
Changelog

Chunking keeps line breaks and indentation

Contributorsicn5381Harsh Sinha

September 16, 20261 min read

Agno's FixedSizeChunking and RecursiveChunking clean each document before they split it, and cleaning now keeps a line break as a line break and a tab as a tab. It used to replace every newline and tab with a space, so code, lists and Markdown reached your vector database as a single line.

The same text after cleaning, before and now. Before, "Steps:", "- Open a ticket" and "- Attach the receipt" ran together on one line. Now the three lines stay separate, with the tab before the last item.

Cleaning still collapses repeated spaces, tabs and line breaks.

To give stored chunks this structure, re-insert the documents you chunked with these strategies.

See the cookbook, and learn more about fixed-size chunking and recursive chunking in the documentation.

Frequently asked questions

FixedSizeChunking and RecursiveChunking clean each document before they split it, and cleaning used to replace every newline and tab with a space. Cleaning now keeps line breaks and tabs, so code, lists and Markdown keep their structure.

Yes, if you want existing chunks to keep their structure. Re-insert the documents you chunked with FixedSizeChunking or RecursiveChunking.

Yes. Cleaning in FixedSizeChunking and RecursiveChunking still collapses repeated spaces, tabs and line breaks.

Shipped around the same time