🚀 The feature, motivation and pitch

As a sequel to #1518 where we added an enum for tokenizer types to simplify TokenizerArgs __post_init__, we need to further improve it to simplify new tokenizer type onboarding:

Tasks

• Move TokenizerType to a centralized place
  • We now have two of them:

    torchchat/dist_run.py

    Lines 67 to 69 in 0299a37

    class TokenizerType(Enum):

    Tiktoken = auto()

    SentencePiece = auto()

    torchchat/torchchat/cli/builder.py

    Lines 241 to 245 in 0299a37

    class TokenizerType(Enum):

    NONE = 0

    TIKTOKEN = 1

    SENTENCEPIECE = 2

    HF_TOKENIZER = 3
• Check all getters of tokenizer types
  • It may be able to be simplified as inline

    torchchat/torchchat/generate.py

    Line 368 in 0299a37

    self.is_llama3_model = self.tokenizer_args.is_tiktoken()
• Add documentation for future tokenizer onboard.
  • We may need to point people to update the model validation logic:

    torchchat/torchchat/cli/builder.py

    Lines 290 to 322 in 0299a37

    def validate_model(

    self,

    model: Optional[Model],

    model_description: str = "model",

    ) -> None:

    if model is None:

    return

    if self.tokenizer_type == TokenizerType.NONE:

    raise RuntimeError(f"no tokenizer was found at {self.tokenizer_path}")

    is_tiktoken = self.is_tiktoken()

    is_sentencepiece = self.is_sentencepiece()

    is_hf_tokenizer = self.is_hf_tokenizer()

    use_tiktoken = model.config.use_tiktoken

    use_hf_tokenizer = model.config.use_hf_tokenizer

    use_sentencepiece = not (use_tiktoken or use_hf_tokenizer)

    if (

    (is_tiktoken and not use_tiktoken) or

    (is_hf_tokenizer and not use_hf_tokenizer) or

    (is_sentencepiece and not use_sentencepiece)

    ):

    raise RuntimeError(

    "model-specified tokenizer ({}) does not match provided tokenizer ({}) for {}".format(

    tokenizer_setting_to_name(use_tiktoken, use_hf_tokenizer),

    tokenizer_setting_to_name(is_tiktoken, is_hf_tokenizer),

    model_description,

    )

    )

    return

To test, run a model with each tokenizer type:

• python torchchat.py generate llama2
• python torchchat.py generate llama3
• python torchchat.py generate granite-code

cc @Jack-Khuu @byjlw