Why Persian retrieval is a different problem
Two spellings of the same Persian word look identical to a reader and completely different to a tokenizer. That detail is the gap between a useful assistant and a toy.
When we first tested retrieval against a Persian knowledge base, accuracy came out at roughly half what the same content scored in English. Same embedding model, same chunk size, questions that were literal translations of each other. The problem was somewhere else entirely.
One word, several spellings
Persian text on the web writes ye and ke in two forms: the Persian characters (U+06CC, U+06A9) and the Arabic ones (U+064A, U+0643). A reader cannot tell them apart. A tokenizer can. The practical result is that a document typed on an Arabic keyboard and a question typed on a Persian one occupy two separate vocabularies as far as the model is concerned.
- Normalise Arabic ye and ke to their Persian equivalents before anything else
- Strip diacritics, which are applied inconsistently across web text
- Normalise the zero-width non-joiner (U+200C) — do not remove it, that changes meaning
- Fold Arabic-Indic and Persian digits into one representation for comparison
Chunking and sentence length
Persian sentences run longer than English ones and carry fewer terminal punctuation marks. Character-count chunking therefore splits mid-paragraph again and again, leaving the second half with no context. We moved to structure-aware chunking: boundaries land on headings, paragraphs and table cells, never on the thousandth character.
before: [ ...end of paragraph one and half of paragraph two ]
after: [ heading + whole paragraph ] [ heading + whole paragraph ]The result
Those two changes alone — normalise before embedding, chunk on structure — moved top-ten recall of the correct passage from about 61% to 89%, with no model change and no extra cost.