I paid three AI companies to split documents. A 50-line script beat them all.

Where this started

I built a legal research assistant for Serbian law. Before any AI can search a law, the text has to be cut into pieces small enough to index - the industry calls this chunking, and it quietly decides whether a search system works or embarrasses you. Serbian laws made this easy: every provision starts with "Član" (article), so a short script - a regex, a pattern-matching rule about fifty lines long - split 582 laws perfectly, instantly, at no cost. It shipped. It works today.

Then the questions started: why didn't I use LangChain or LlamaIndex? These are popular frameworks that orchestrate AI models for exactly this kind of document work, and not using them apparently requires an explanation. Even though it looked like overkill, I am a man who values knowledge above all, so I went ahead. The results were not better than what was already made with the script. The cost multiplied, because every document now passed through a paid AI model. And when something failed, the failure was buried under so many layers of framework that finding it took longer than the original work.

The third push was AWS. Amazon's cloud platform dominates job postings and architecture diagrams alike, and the pressure to run everything on it is constant. Fair enough - instead of arguing, I rebuilt the experiment properly on the stack everyone asks for: S3 (Amazon's file storage) holding the documents, Lambda (code that runs on demand, billed by the millisecond) fanning out one job per document per model, Secrets Manager holding the API keys, and hand-written IAM permission rules so each component can touch only what it needs. Three frontier models did the chunking - GPT-4.1 mini, Claude Haiku 4.5, Gemini 3.5 Flash - all given the same instructions at temperature zero, with my regex running as the fourth contestant.

Round one: the biggest law we have

The test document was Zakon o energetici - the Serbian energy law, 772,003 characters, 559 articles. Each model was asked to return chunk boundaries as character positions. Here is what happened:

Only one model finished cleanly. Claude produced sixteen times more output text than GPT for the identical task, ran over its output limit, and cut off its own answers mid-sentence on 15 of 34 segments - while the summary statistics still looked healthy, hiding a 91,721-character gap in the result. Gemini's own API answered 44% of requests with "service unavailable" and the job spent fifteen minutes retrying before the time limit killed it. And GPT, the best behaved of the three, still placed 84% of its boundaries where no boundary belonged:

The cause is fundamental, not a bug: language models cannot count characters. Ask one for exact positions in a long text and you get plausible-looking numbers that drift hundreds of characters from any real boundary. The models may well have chosen sensible places to split - they simply cannot express those places as arithmetic. A search index built on these chunks stores sentences cut in half, and nobody notices until the search results read strangely.

Round two: fix the ruler, re-measure everything

A fair experiment fixes its own measuring instrument. Version two changed the output format: instead of positions, each model returns the first few words of every chunk verbatim, and my code locates them by simple text search - no arithmetic required of the model. The Lambda function also got parallel processing, ten segments at once instead of a polite queue, which cut Claude's runtime from 670 seconds to 37 and brought Gemini back from the dead. Every conclusion from round one flipped:

Gemini went from failing completely to being the most disciplined model in the field. Claude found 98% of all 559 articles. GPT produced five times more chunks than before - same model, same instructions, opposite behavior, purely from changing how it was asked to answer. That is a finding in its own right: how you measure AI models changes their ranking more than the models themselves do. Any benchmark that doesn't disclose its output format is closer to a coin flip than a measurement. And through both rounds, the regex sat unchanged at 100% precision, five milliseconds, zero dollars.

Round three: does language matter?

Serbian law makes a poor demonstration for anyone who doesn't read Serbian, so the experiment needed a document everyone recognizes. GDPR - the EU's data-protection regulation - exists in 24 official, professionally aligned translations: the same 99 articles in every language. That makes it something rare: a controlled experiment on whether AI models treat languages equally, using a document your compliance department already quotes. I ran five versions - English, German, French, Spanish, Italian - through all three models. Fifteen cloud jobs, two minutes, zero errors.

The reassuring part: every model finds essentially every article in every language - recall stays between 93% and 100% across the board. The caution: boundary discipline is a different story. GPT places 82% of its boundaries correctly in English and only 38% in French - the same model, same instructions, noticeably different behavior outside its home language. Gemini barely registers the language changing. If your documents are multilingual and your chunking was only ever tested in English, you tested a different product than the one you're running.

Search it yourself, in your own language

The chunks from every model are searchable in the live demo. They were embedded once with BGE-M3, an open-source multilingual model that maps text from about 100 languages into a shared mathematical space by meaning rather than words - it runs on my own server, so your searches cost nothing and call no external AI. The practical consequence is worth trying firsthand: ask a question in Serbian and the English GDPR returns Article 17 on the right to erasure; ask in Russian and the German text answers. Pick a chunker, pick a language, and compare what each model's splitting decisions actually do to search quality - the differences described above stop being abstract the moment you see one model return an article with its heading and another return the same content decapitated.

The point

None of this says LangChain, LlamaIndex, Lambda or frontier models are bad tools. It says tools have territories. Where documents carry their own structure - laws, regulations, contracts with numbered clauses - a deterministic script beats any model on cost, speed, and correctness, and no amount of framework changes that. The models earn their price where structure dissolves: transcripts, free-form judgments, the genuinely messy documents. The same logic applies within AI itself - reaching for the largest, most expensive model by default buys you slower answers and sometimes worse ones, particularly when a model overthinks a task that needed no thinking at all, as one of the contestants here expensively demonstrated. Measuring before committing is cheaper than the alternative; this entire investigation cost less than one API call to Fable 5 asking what the weather will be like today.