All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

EcmascriptTokenizer, FortranTokenizer, RubyTokenizer

public abstract class AbstractTokenizer
extends Object
implements Tokenizer 

Author:

Zev Blut zb@ubit.com, Romain PELISSE belaran@gmail.com

• Field Summary

  Fields

  Modifier and Type	Field and Description
  protected List<String>	ignorableCharacter
  protected List<String>	ignorableStmt
  protected char	oneLineCommentChar
  protected boolean	spanMultipleLinesString
  protected List<String>	stringToken

• Constructor Summary

  Constructors

  Constructor and Description
  AbstractTokenizer ()

• Method Summary

  Methods

  Modifier and Type	Method and Description
  void	tokenize (SourceCode tokens, Tokens tokenEntries)

  • Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• Field Detail

  • stringToken

    protected List<String> stringToken

  • ignorableCharacter

    protected List<String> ignorableCharacter

  • ignorableStmt

    protected List<String> ignorableStmt

  • oneLineCommentChar

    protected char oneLineCommentChar

  • spanMultipleLinesString

    protected boolean spanMultipleLinesString

• Constructor Detail

  • AbstractTokenizer

    public AbstractTokenizer()

• Method Detail

  • tokenize

    public void tokenize(SourceCode tokens,
     Tokens tokenEntries)

    Specified by:

    tokenize in interface Tokenizer