03 — Loading Persian documents¶
Persian lesson: بارگذاری اسناد فارسی · Module: parsrag.loaders
What you build¶
Loaders for TXT, Markdown, HTML, DOCX and PDF that return a Document with its text and
metadata — the source file, sections with character offsets, and pages for PDFs — plus
locate(), which tells you the section and page of any offset. The core of the lesson is a set
of repairs for text that looks fine but cannot be searched.
Key ideas¶
The fixtures simulate six real problems, each with its own fix:
- Old encodings. A Windows-1256 text file with Arabic letters: try BOM, then UTF-8, then cp1256, and normalise line endings.
- Web page boilerplate. Menus, breadcrumbs, cookie banners and footers are removed before
indexing;
‌entities and LRM marks are handled. - Presentation forms in PDFs. Many PDF producers store glyph shapes (initial, medial, final forms) instead of letters. NFKC applied to those ranges maps them back.
- Visual order. Some PDFs store each line in visual (reversed) order. The loader detects it by voting — glyph shapes that cannot occur in logical order, and common words that only appear reversed — and reverses runs while keeping Latin text and numbers intact.
- Lost ZWNJs. The joining shape of the letters on both sides tells where a half-space was.
- Invisible bidi controls. RLM/LRM and embedding marks around Latin model numbers are stripped.
repair_pdf_text chains them in the right order: controls → visual order → ZWNJ → NFKC.
Measured results¶
Word recall against the original text (python examples/pdf_repair_impact.py):
| Step | kahroba-catalog.pdf | mehregan-newsletter.pdf |
|---|---|---|
| raw pypdf text | 0.076 | 0.067 |
| + NFKC presentation forms | 0.938 | 0.080 |
| + visual-order fix | 0.938 | 0.893 |
+ ZWNJ recovery (= load_pdf) |
1.000 | 1.000 |
Raw PDF text is practically unsearchable (about 7% of the words, mostly numbers). NFKC alone recovers 94% of a logical-order PDF but only 8% of a visual-order one; the order fix brings it to 89%, and ZWNJ recovery makes both texts identical to the source.
Honest caveat: both PDFs were generated by the course's own simulator, so they follow exactly the model the repair code assumes. 1.000 means the code works, not that every real PDF will reach it. Real PDFs have columns, headers, fonts without ToUnicode tables, and scanned pages that need OCR, which is outside this lesson.
Takeaway¶
Check extracted text before you blame retrieval. A loader that keeps offsets, sections and pages also makes exact citations possible later (lesson 09).