I like this direction to modularize the components and sharing them for different use-cases beyond those backed by ET.

It seems like what we could think on further is how to actually modularize it even more, ie. not just reuse the Tokenizer and Sampler, but also most of the runner logic, if possible.

Runner now does some things that are still common for ET or AOTI while preparing the inputs and processing the outputs, eg. the run loop, applying chat pattern with special tokens, etc. So in theory we could have a common Runner pipeline and inject specific minimal components to do the actual forward() call implemented differently for ET or AOTI.

So overall we could have a collection of abstract interfaces for Tokenizer, Sampler, Processor (ie. image processor), Runner (to generate a single next token), Pipeline (to utilize the Runner for completion or multi-turn generation), etc. and also provide some concrete implementations for them, eg. TikTokenTokenizer, ArgMaxSampler, ImageProcessor, LLaMA3Runner, LLaMAChatPipeline, etc. That way we provide some working solution out-the-box, and also allow clients implement their own components by inheriting the interfaces and injecting them instead, eg. a custom SentencePieceTokenizer injected into a custom LLaMA2Runner injected into LLaMAChatPipeline instead of the default LLaMA3Runner.

Such interfaces and default implementations can live in pytorch-labs, if that's a good place for everyone to depend on.

I also really like the idea of unifying these c++ layers! I've been working on extending the existing c++ tokenizer support to handle parts of the tokenizers library from huggingface. I have a draft PR up for discussion, but the code should be very self-contained and portable if the decision is to move this level of support elsewhere.

A couple of key elements of the PR to point out that may be seeds for this discussion:

• Addition of PreTokenizer and TokenDecoder abstract interfaces
• Basic config-based factory mechanisms for each (e.g. PreTokenizerConfig) with json parsing.
• Unit test framework using gtest and ctest

I'm not at all tied to the specific shapes of these implementations, but wanted to point them out as the discussion touches on how to wrangle these abstractions.

One note I did want to make as I was prototyping (I'll comment on the PR as well): I opted not to use enclosing namespaces since they were not used in the existing torchchat c++ layers. I think for a more reusable abstract layer, we would certainly want to use namespace encapsulation.

@shoumikhin @gabe-l-hart thanks for chiming in. I spawn up https://github.com/pytorch-labs/tokenizers as our first step to enforcing code sharing on tokenizers. I'm still waiting for legal approval to open this repo up but would love to collaborate.

In the short term I think this is a good direction, but long-term I do not like how ExecuTorch does not just "work" well with an LLM exported from torch.export.

ExecuTorch having its own export_llama_lib script and special source transformations is not a good user experience in my opinion. My one worry about the proposal here is that it further entrenches this flow.

In the short term I think this is a good direction, but long-term I do not like how ExecuTorch does not just "work" well with an LLM exported from torch.export.

ExecuTorch having its own export_llama_lib script and special source transformations is not a good user experience in my opinion. My one worry about the proposal here is that it further entrenches this flow.

I somewhat agree, I want to point out source transformation is "optional" meaning ET will also work without source transformation, just slower and consumes more memory. Having source transformation available gives ET user an easy way to optimize their LLM model.

It is a separate question on whether we should internalize these optimizations into the framework, e.g., find a smarter way to reduce unnecessary copies.

In the short term I think this is a good direction, but long-term I do not like how ExecuTorch does not just "work" well with an LLM exported from torch.export.
ExecuTorch having its own export_llama_lib script and special source transformations is not a good user experience in my opinion. My one worry about the proposal here is that it further entrenches this flow.

I somewhat agree, I want to point out source transformation is "optional" meaning ET will also work without source transformation, just slower and consumes more memory. Having source transformation available gives ET user an easy way to optimize their LLM model.

It is a separate question on whether we should internalize these optimizations into the framework, e.g., find a smarter way to reduce unnecessary copies.

Actually I think it is fine to apply optimizing transformation be that module swap or graph rewrites. to_backend within ET pipeline is one such optimization and that is explicit opt-in. Scripts that wrap all the (export, to_edge, to_backend etc.) provide convenience wrappers. So in general it is better to do a) building composable components and b) have wrappers or boxing logic that puts together these components, thus allowing custom wrappers. It is better compared to monolithic design by say, having a giant optimized operator lib that works like a charm when it does, but is extremely hard to extend.