Split PDF text into overlapping chunks for RAG and embeddings

How to use the PDF chunker

1. Drop a text-based PDF onto the box or press Choose a file.
2. Set Chunk size in approximate tokens (50 to 8,000; default 500).
3. Set Overlap in approximate tokens (0 to 2,000; default 50).
4. Choose the Format: JSON or JSONL.
5. Press Chunk text, check the counts, and download the file, for example handbook-chunks.jsonl.

What it does and when to use it

Retrieval-augmented generation (RAG) means storing a document as many small pieces, finding the pieces that match a question, and giving only those to a language model. Before you can create embeddings, the text has to be cut into pieces of a sensible size. Cutting at a fixed number of characters breaks sentences in half; this tool cuts at paragraph and sentence boundaries where it can and records which pages each chunk came from, so answers can cite a page.

Typical uses: a support bot over a product manual, search over company policies, or study notes from a textbook PDF. To see the text first, use the PDF text extractor. To keep headings and lists as Markdown, use PDF to Markdown.

How it works

pdf.js (pdfjs-dist) reads the text layer in your browser; nothing is uploaded, though pdf.js may download standard font or character-map data from cdn.jsdelivr.net if the PDF needs it. The text is then rebuilt into headings, paragraphs and list items (lists keep a - in front). After that:

1. The size limit is the chunk size × 4 characters. A paragraph longer than that, minus the overlap, is split between sentences, and a single over-long sentence between words.
2. Pieces are packed into a chunk, separated by blank lines, until the next piece would pass the limit.
3. Each new chunk starts with the tail of the previous one (overlap × 4 characters, at most half the chunk) and then continues with new text.
4. Every chunk gets an id, pageStart, pageEnd, an estimated tokens count (characters ÷ 4, rounded up) and its text.

Worked examples

A 2-page invoice with default settings. Size 500, overlap 50, JSON gives Chunks: 3, Largest chunk: 483 tokens and Total: 911 tokens. The first chunk runs from page 1 to page 2 because it ends with the “Terms” heading at the top of page 2.

The same file in small chunks. Size 100, overlap 20, JSONL gives Chunks: 11, Largest chunk: 95 tokens and Total: 1,013 tokens; the total is higher because the overlaps are counted again. The first line is:

{"id":1,"pageStart":1,"pageEnd":2,"tokens":93,"text":"Invoice 2041\n\nBilling summary\n\n…"}

Limits and tips

• Scanned PDFs need OCR PDF first.
• Headers, footers and page numbers are kept in the text; remove them afterwards if they add noise.
• Tables come through as plain text lines, not structured rows.