Building applications with LLMs through composability, in Kotlin
kotlin scala ai functional-programming embeddings artificial-intelligence openai multiplatform agents tokenizers llm chatgpt-api
-
Updated
Oct 14, 2024 - Kotlin
#
tokenizers
Here are 42 public repositories matching this topic...
Language: All
Filter by language
Sort: Most stars
Sort options
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
nlp machine-learning natural-language-processing ai deep-learning transformers artificial-intelligence gpt language-model natural-language-inference natural tokenization natural-language-understanding attention-is-all-you-need attention-mechanisms transformer-architecture natural-language-procressing tokenizers llm
-
Updated
Jul 20, 2024 - Python
This repository is part of a course on Elasticsearch in Python. It includes notebooks that demonstrate its usage, along with a YouTube series to guide you through the material.
search-engine elasticsearch embeddings elastic semantic-search knn-algorithm hybrid-search tokenizers
-
Updated
Sep 13, 2025 - Jupyter Notebook
Develop DL models using Pytorch and Hugging Face
-
Updated
Sep 5, 2025
the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly
distributed-systems distributed-computing nats embeddings gemini nats-streaming mistral mlx nats-messaging claude fine-tuning sentence-splitting huggingface milvus inference-api apple-silicon tokenizers langchain llm-training finetuning-llms
-
Updated
Oct 19, 2024 - Python
This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.
-
Updated
Oct 20, 2022 - Python
Python script for manipulating the existing tokenizer.
-
Updated
Jun 11, 2025 - Python
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust
python rust openai pypi-package bpe byte-pair-encoding huggingface tokenizers llm tiktoken bpe-tokenizer byte-pair-tokenizer
-
Updated
Mar 19, 2025 - Python
Use custom tokenizers in spacy-transformers
-
Updated
Aug 9, 2022 - Python
[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library
-
Updated
Sep 30, 2025 - C#
Package to align tokens from different tokenizations.
-
Updated
Mar 25, 2024 - Python
Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels
nlp natural-language-processing tensorflow tokenizer transformers bert tensorflow-hub tokenizers sentencepie
-
Updated
Mar 29, 2022 - Python
Small library that provides functions to tokenize a string into an array of words with or without punctuation
-
Updated
Aug 9, 2023 - JavaScript
Visualize some important concepts related to LLM architectures.
transformers attention-mechanism huggingface huggingface-transformers tokenizers llm llm-inference llm-architecture
-
Updated
Oct 16, 2023 - Jupyter Notebook
A graphical user interface for the Elasticsearch Analyze API
-
Updated
Apr 1, 2025 - JavaScript
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
-
Updated
Nov 16, 2023 - Python
A high-performance tokenizer built to rival GPT-4, trained on the C4 dataset.
-
Updated
Mar 25, 2025 - Jupyter Notebook
Byte Pair Encoding (BPE) tokenizer tailored for the Turkish language
-
Updated
Dec 10, 2024 - Python
-
Updated
Jan 17, 2025 - Crystal
Improve this page
Add a description, image, and links to the tokenizers topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the tokenizers topic, visit your repo's landing page and select "manage topics."