Public Types

typedef std::list< std::pair
< std::string, std::string > > WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word. More...

Static Public Member Functions

static void tokenize (const std::string &s, WordTokenizerResults &results)

Tokenizes the passed string and stores the results. More...

Tokenizes a paragraph separating words from spaces.

typedef std::list<std::pair<std::string, std::string> > srchilite::WordTokenizer::WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

The two elements are mutually exclusive

void srchilite::WordTokenizer::tokenize ( const std::string & s,

WordTokenizerResults & results

)

static

Tokenizes the passed string and stores the results.

The documentation for this class was generated from the following files:

/home/bettini/work/source-highlight/src-highlite/lib/srchilite/wordtokenizer.h
/home/bettini/work/source-highlight/src-highlite/lib/srchilite/wordtokenizer.cpp