Arabic Word Frequency Counts

ARABIC WORD FREQUENCY COUNTS

When tokenizing a text for purposes of generating a word frequency count, I define an Arabic word as:

one or more consecutive Arabic characters [\xC1-\xD6\xD8-\xDB\xDD-\xDF\xE1\xE3-\xE6\xEC\xED]
including Persian characters [\x81\x8D\x8E\x90]
short vowels and diacritics [\xF0-\xF3\xF5\xF6\xF8\xFA]
and the lengthening character [\xDC]

Note: All hex values are those of the Arabic Windows (1256) code page.

When tokenizing Arabic input it's a good idea to make a preliminary pass to detect and fix punctuation anomalies, such as the Arabic character ra' (\xD1) used as a numeric comma or "decimal separator" (U+066B), and the Arabic lengthening character (\xDC) used as an em dash or numeric hyphen. Numbers are sometimes encoded visually instead of logically, and the digit zero occasionally functions as a period (full stop).

Aften tokenizing according to the above criteria I remove all short vowels, diacritics and the lengthening character, and count the remainder as a word. Null strings are discarded.

Here are the types/tokens figures from the last three word frequencies I have generated:

The table below shows the top 30 words and their frequencies from my three frequency counts. Note that starting with my Feb. 2001 count I added a "file count" figure and began to use it—instead of the frequency count—as the primary sort key. On Nov. 13, 2002 I wrote a script to get the Google frequency: it's an interesting statistic, especially if you look at the web as the Mother of All Corpora.

The above is not a lemmatized list. Although some word forms are easily merged (e.g., ال? ?=? إل?), most word forms require contextual analysis to be disambiguated (e.g. ان ?=? أَنْ or أَنّ? or إِنْ or إِنّ? or آن).

Before retrieving citations from my corpus I find it useful to go to the wordlist first and use it to test the regular expression that I will later use for searching the corpus. The wordlist also allows me to see the total number of hits that I will get when generating a concordance and thus anticipate the file size of the concordance. Some word forms are unambiguous and lend themselves to fairly simple regular expressions when searching for them in the wordlist. For example, the regular expression /[A><]stqlAb/ produced:

30 forms (Total Freq: 570 = 1 every 2,002,743 words) (Here is a page with my Transliteration).

Other lemmas require more complex regular expressions, such as the following, for the noun stem fltAn:

/^[wf]?([blk]?|[bk]?Al|[blk]?hAl|ll)fltAn/ (Note: this includes the colloquial prefix hAl-)

29 forms (Total Freq: 1,728 = 1 every 660,627 words) (Here is a page with my Transliteration).

The following regular expressions extract all the inflected forms of the preposition fy (both versions produce the same output):

/^[wf]?f[Yy](|h|hA|hmA|hm|hn|k|kmA|km|kn|nA)$/
/^[wf]?f[Yy]([hk]?|hA|[hk]mA?|[hk]n|nA)$/

48 forms (Total Freq: 42,810,237 = 1 every 26 words) (Here is a page with my Transliteration).

Some contamination from other lemmas does occur. Some of the homographs, such as وف?, are easy to spot, but others are quite unexpected. For example, فيكن is also the Armenian proper name Vigen (I thank my Arabic-L colleagues for pointing this out), and فينا is also the Belgian firm FINA and a rare spelling of Vienna. Unexpected contamination usually shows up when a concordance is generated (see CONCORDANCING). I can anticipate some of the extraneous lemmas by running the wordlist through my morphological parser (see MORPHOLOGY ANALYSIS) before generating the concordance.

---

HOME | CORPUS COMPILATION | WORD FREQUENCY COUNTS | CONCORDANCING | MORPHOLOGY ANALYSIS | ARABIC LEXICON

Copyright ? 2002 QAMUS LLC