[ class tree: Text_LanguageDetect ] [ index: Text_LanguageDetect ] [ all elements ]

Packages:
Text_LanguageDetect

Classes:

Text_LanguageDetect

Files:

example_clui.php
example_web.php
LanguageDetect.php

Class: Text_LanguageDetect

Source Location: /Text_LanguageDetect-0.1.1/Text/LanguageDetect.php

Class Overview

Language detection class

Author(s):

Nicholas Pisarro <infinityminusnine+pear at gmail.com>

Version:

Release: @package_version@

Copyright:

2005 Nicholas Pisarro

Inherited Variables

Inherited Methods

Class Details

[line 60]
Language detection class

Requires the langauge model database (lang.dat) that should have accompanied this class definition in order to be instantiated.

Example usage:

 require_once 'Text/LanguageDetect.php';


 $l = new Text_LanguageDetect;

 $stdin = fopen ('php://stdin', 'r');

 echo "Supported languages:\n";

 $langs = $l->getLanguages ();
 if (PEAR::isError($langs)) {
   die($langs->getMessage());
 }

 sort ($langs);
 echo join (', ', $langs);

 while ($line = fgets ($stdin)) {
   print_r ($l->detect ($line, 4));
 }

Author: Nicholas Pisarro <infinityminusnine+pear@gmail.com>
Version: Release: @package_version@
Copyright: 2005 Nicholas Pisarro
Todo: allow users to generate their own language models
License: BSD

[ Top ]

Method Detail

Text_LanguageDetect (Constructor) [line 136]

Text_LanguageDetect Text_LanguageDetect(
)

Constructor

Will attempt to load the language database. If it fails, you will get a PEAR_Error object returned when you try to use detect()

[ Top ]

clusteredSearch [line 1098]

array clusteredSearch(
string
$str)

Perform an intelligent detection based on clusterLanguages()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

This compares the sample text to top the top level of clusters. If the sample is similar to the cluster it will drop down and compare it to the languages in the cluster, and so on until it hits a leaf node.

this should find the language in considerably fewer compares (the equivalent of a binary search), however clusterLanguages() is costly and the loss of accuracy from this technique is significant.

This method may need to be 'fuzzier' in order to become more accurate.

This function could be more useful if the universe of possible languages was very large, however in such cases some method of Bayesian inference might be more helpful.

Return: language scores (only those compared)
See: Text_LanguageDetect::clusterLanguages()
Throws: PEAR_Error
Access: public

Parameters:

string $str — input string

[ Top ]

clusterLanguages [line 910]

array clusterLanguages(
)

Cluster known languages according to languageSimilarity()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

Uses a nearest neighbor technique to generate the maximum possible number of dendograms from the similarity data.

Return: language cluster data
See: Text_LanguageDetect::languageSimilarity()
Throws: PEAR_Error
Access: public

[ Top ]

detect [line 627]

mixed detect(
string
$sample, [int
$limit = 0])

Detects the closeness of a sample of text to the known languages

Calculates the statistical difference between the text and the trigrams for each language, normalizes the score then returns results for all languages in sorted order

If perl compatible, the score is 300-0, 0 being most similar. Otherwise, it's 0-1 with 1 being most similar.

The $sample text should be at least a few sentences in length; should be ascii-7 or utf8 encoded, if other and the mbstring extension is present it will try to detect and convert.

Return: sorted array of language scores, blank array if no useable text was found, or PEAR_Error if error with the object setup
See: _distance()
Throws: PEAR_Error
Access: public

Parameters:

string $sample — a sample of text to compare.

int $limit — if specified, return an array of the most likely $limit languages and their scores.

[ Top ]

detectConfidence [line 751]

array detectConfidence(
string
$sample)

Returns an array containing the most similar language and a confidence rating

Confidence is a simple measure calculated from the similarity score minus the similarity score from the next most similar language divided by the highest possible score. Languages that have closely related cousins (e.g. Norwegian and Danish) should generally have lower confidence scores.

The similarity score answers the question "How likely is the text the returned language regardless of the other languages considered?" The confidence score is one way of answering the question "how likely is the text the detected language relative to the rest of the language model set?"

To see how similar languages are a priori, see languageSimilarity()

Return: most similar language, score and confidence rating or null if no language is similar
See: Text_LanguageDetect::detect()
Throws: PEAR_Error
Access: public

Parameters:

string $sample — text for which language will be detected

[ Top ]

detectSimple [line 705]

string detectSimple(
string
$sample)

Returns only the most similar language to the text sample

Calls $this->detect() and returns only the top result

Return: the name of the most likely language or null if no language is similar
See: Text_LanguageDetect::detect()
Throws: PEAR_Error
Access: public

Parameters:

string $sample — text to detect the language of

[ Top ]

getLanguageCount [line 299]

int getLanguageCount(
)

Returns the number of languages that this object can detect

Return: the number of languages
Throws: PEAR_Error
Access: public

[ Top ]

getLanguages [line 350]

array getLanguages(
)

Returns the list of detectable languages

Return: the names of the languages known to this object
Throws: PEAR_Error
Access: public

[ Top ]

languageExists [line 318]

bool languageExists(
mixed
$lang)

Returns true if a given language exists

If passed an array of names, will return true only if all exist

Return: true if language model exists
Throws: PEAR_Error
Access: public

Parameters:

mixed $lang — language name or array of language names

[ Top ]

languageSimilarity [line 811]

array languageSimilarity(
[string
$lang1 = null], [string
$lang2 = null])

Calculate the similarities between the language models

Use this function to see how similar languages are to each other.

If passed 2 language names, will return just those languages compared. If passed 1 language name, will return that language compared to all others. If passed none, will return an array of every language model compared to every other one.

Return: scores of every language compared or the score of just the provided languages or null if one of the supplied languages does not exist
Throws: PEAR_Error
Access: public

Parameters:

string $lang1 — the name of the first language to be compared

string $lang2 — the name of the second language to be compared

[ Top ]

omitLanguages [line 236]

int omitLanguages(
mixed
$omit_list, [bool
$include_only = false])

Omits languages

Pass this function the name of or an array of names of languages that you don't want considered

If you're only expecting a limited set of languages, this can greatly speed up processing

Return: number of languages successfully deleted
Throws: PEAR_Error
Access: public

Parameters:

mixed $omit_list — language name or array of names to omit

bool $include_only — if true will include (rather than exclude) only those in the list

[ Top ]

setPerlCompatible [line 365]

void setPerlCompatible(
[bool
$setting = true])

Make this object behave like Language::Guess

Access: public

Parameters:

bool $setting — false to turn off perl compatibility

[ Top ]