UGCNormal

 UGC-Normalizer
INPUT
|
| ----------------------------- ------------- -----------
---> | SentenceBoundaryDetection | ---> | tokenizer | ---> | speller | ----
 ----------------------------- ------------- ----------- |
 |
 |
 ----------------------------------------------------------------------
 |
 | -------------- ------------------ ----------
 ---> | siglas_map | ---> | internetes_map | ---> | np_map | ---> OUTPUT
 -------------- ------------------ ----------
>>> HOW TO USE:
Before anything else, run ./configure.sh script to check and solve all
dependencies. After that you can run the normalizer script.
Main script is ugc_norm.sh. Use it to apply the normalization pipeline. Just run and pass as
parameters INPUT_dir and OUTPUT_dir. The INPUT_dir must contain all text files
to be processed.
You can test the normalizer using the data in directory "test":
./ugc_norm.sh ./test/input/ ./test/output/
>>> MORE INFO:
******************************* test
Input and output directories to test the normalizer. The output directory tree
has the output produced by each step of this pipeline (sent -> tok -> checked
-> siglas -> internetes -> nomes). The deeper directory ('nomes') has the
result of the full pipeline (probably you are interested only in this result).
******************************* internetes_map.pl
perl script to translate web language using dictionary
******************************* np_map.pl
perl script to normalize NPs using (./resources/np_data.txt). It just
capitalizes the first letter
******************************* siglas_map.pl
Script to put all letters to upper case, if it is in ./resources/lexico_siglas.txt
******************************* upper_handler.py
It checks if a text file is totally in uppercase, if it is, only words after
punctuation are capitalized, all the others are set to lowercase
******************************* upper_periods.py
It capitalizes words after periods
******************************* README.txt
This file !
******************************* resources
Directory with dictionaries for NPs and web language
******************************* SentenceBoundaryDetection
Sentence boundary detection tool, it appends <S> tags at the end of each sentence
******************************* speller
Speller tool directory
******************************* tokenizer
Tokenizer tool directory, you can change lex rules in webtok.lex and run
Makefile using make tool
******************************* utils
- ./utils/extract.sh
This script extract all opinions (text files) in a corpus (many
subdirectories)

References