ToolBlur Research · Corpus v1.0.0
Arabic Text Normalization Methodology
An open reference that states exactly what each rule changes—and what must not be mistaken for linguistic correction.
Scope and purpose
The rules create a working copy for search, matching and deduplication. They do not correct grammar or spelling, choose contextually correct words or replace the source text. Every transformation is deterministic: identical input and settings produce identical browser output.
Two normalization levels
Conservative — default
Removes optional marks and tatweel, folds Alef forms, converts digits and cleans spacing. Suitable for general matching when the original is retained.
Aggressive — opt in
Also folds Alef Maqsura and Hamza carriers. It increases recall while hiding orthographic distinctions, so it is not suitable for publication copies.
Reproducible examples
| Input | Expected | Rule |
|---|---|---|
| اللُّغَةُ العَرَبِيَّةُ | اللغة العربية | Remove diacritics |
| العــربية | العربية | Remove tatweel |
| إلى أحمد وآمال | الي احمد وامال | Normalize Alef |
| فتى | فتي | Optional: Alef Maqsura → Ya |
| عام ٢٠٢٦ | عام 2026 | Arabic-Indic digits → ASCII |
Important limitations
- Never replace authoritative Quranic, educational or quoted text with a stripped copy.
- Aggressive folding can merge distinct words into one comparison form.
- Version 1.0 does not automatically fold Persian or Urdu letter variants.
- Word counts rely on whitespace and do not perform morphological analysis.
- Test the rules on a representative sample before processing a full corpus.
License and citation
The test corpus is available under CC BY 4.0. When reusing it, cite: ToolBlur Arabic Text Normalization Test Corpus, version 1.0.0, with a link to this page.