Fruga
العربية

Arabic text toolkit

Prepare Arabic text for a search index or a training set: strip what carries no meaning, unify the spellings that a model would otherwise learn twice, and find the invisible characters that corrupt a dataset without ever appearing on screen.

  • Operations15, each separate
  • PresetsSearch, training, display
  • Your textNever leaves the tab

Your text

Paste it, or open a .txt file.

Characters
270
49 words
Arabic
68%
of the letters
Diacritics
19
5 tatweel
Invisible
1
you cannot see these

Invisible characters found

  • U+200F right-to-left mark · 1×

What to apply

Each one is separate on purpose - normalising Arabic is not a single decision.

Clean

Normalise

Numbers

Punctuation

Whitespace

Result

Normalises spelling and drops duplicates, but keeps ta marbuta and hamza - the model has to generate readable Arabic.

Tokens
≈ 57
was 88
Lines
3
3 unique
Invisible
0
was 1

7 operations changed the text:

  • Invisible characters removed−1 chars
  • Tatweel removed−5 chars
  • Diacritics removed−19 chars
  • Alef unified±0 chars
  • Ya unified±0 chars
  • Digits converted to ASCII±0 chars
  • Duplicate lines dropped−63 chars
182 characters · ≈ 57 tokens · 35% fewer than before

Why unifying alef matters

أ إ آ ا are one letter to a reader and four different tokens to a model. A corpus that keeps all four teaches the model to spend capacity on spelling variance instead of on language - and a search index that keeps them fails to match the same word typed on a different keyboard.

Why diacritics are not always noise

Removing tashkeel cuts the token count sharply and is right for ordinary prose. It is wrong for Qur'anic text, poetry and teaching material, where the marks carry meaning and pronunciation. That is why it is a switch and not a default.

The characters nobody can see

Direction marks and zero-width joiners survive a copy-paste out of a PDF or a chat app, then sit inside a JSONL file where no reviewer will ever spot them. They change the token count, break exact-match comparisons, and are reported here by name and count.

Frequently asked questions

Why normalise Arabic text before indexing or training?
أ, إ, آ and ا are one letter to a reader and four separate tokens to a model. A corpus that keeps all four spends capacity learning spelling variance instead of language, and a search index that keeps them fails to match the same word typed on a different keyboard.
Should I always remove Arabic diacritics?
No. Removing tashkeel cuts the token count sharply and is right for ordinary prose. It is wrong for Qur'anic text, poetry and teaching material, where the marks carry meaning and pronunciation. It is a switch here rather than a default for exactly that reason.
What are the invisible characters in my Arabic dataset?
Direction marks, zero-width joiners and non-joiners, and similar formatting characters. They survive a copy-paste out of a PDF or a chat app and then sit inside a JSONL file where no reviewer will spot them, changing token counts and breaking exact-match comparisons. This page reports each one by name and count.
Does it handle Arabic-Indic digits?
Yes. ١٢٣ and ١٢٣-style Eastern digits can be converted to 123 so that numbers compare and sort consistently with the rest of your data.