Skip to content

#

byte-pair-encoding

Here are 61 public repositories matching this topic...

Language: All

Filter by language

All 61 Python 30 Jupyter Notebook 10 C++ 8 Go 3 Rust 3 CSS 1 HTML 1 Haskell 1 JavaScript 1 Kotlin 1

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

samber / go-gpt-3-encoder

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

go encoder decoder tokenizer transformer openai token codex bpe byte-pair-encoding gpt-2 gpt-3

Updated Dec 2, 2024
Go

aallam / ktoken

Kotlin multiplatform BPE tokenizer library for OpenAI models

kotlin tokenizer openai gpt bpe byte-pair-encoding tiktoken binary-p

Updated Jan 27, 2025
Kotlin

gweidart / rs-bpe

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

python rust openai pypi-package bpe byte-pair-encoding huggingface tokenizers llm tiktoken bpe-tokenizer byte-pair-tokenizer

Updated Mar 19, 2025
Python

ankane / youtokentome-ruby

High performance unsupervised text tokenization for Ruby

unsupervised-learning word-segmentation tokenization npl bpe byte-pair-encoding

Updated Dec 27, 2023
Ruby

DolbyUUU / byte_pair_encoding_BPE_subword_tokenization_implementation_python

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

python nlp natural-language-processing tokenizer data-preprocessing data-cleaning bpe byte-pair-encoding subword-tokenization

Updated Jan 30, 2023
Python

BobMcDear / minbpe-hs

Byte-level byte pair encoding (BPE) in Haskell

haskell tokenizer bpe byte-pair-encoding llm

Updated May 27, 2024
Haskell

bnosac / tokenizers.bpe

R package for Byte Pair Encoding based on YouTokenToMe

text-mining tokenization bpe byte-pair-encoding

Updated Sep 5, 2025
C++

akhvorov / vgram

Feature extraction from sequential data

natural-language-processing text-classification feature-extraction word-segmentation sequential-data vgram byte-pair-encoding

Updated Jul 4, 2019
C++

theskyinflames / word2png

This is a tool that encrypts a sequence of words (or pieces of texts) using the AES-256 algorithm and encodes the encrypted result into a PNG image by linking each byte value to a specific color. It also decodes the before image to get back the original sequence of words

go golang encryption tdd clean-code wasm image-processing bit-manipulation aes-256 hexagonal-architecture byte-array bip39 png-decoder solid-principles encryption-decryption cold-wallet golang-wasm byte-pair-encoding pterm go-aes-256

Updated Sep 23, 2023
Go

Ascend-Research / AutoGO

Code repo for the paper "AutoGO: Automated Computation Graph Optimization for Neural Network Evolution", accepted to NeurIPS 2023.

deep-neural-networks deep-learning tensorflow tokenizer pytorch classification human-pose-estimation super-resolution semantic-segmentation neural-architecture-search pytorch-geometric byte-pair-encoding neurips-2023

Updated Jun 7, 2024
Python

jmaczan / bpe-tokenizer

Byte-Pair Encoding tokenizer for training large language models on huge datasets

python machine-learning deep-learning tokenizer chunking from-scratch bpe byte-pair-encoding large-language-models llm bpe-tokenizer

Updated Jun 4, 2024
Python

zouharvi / tokenization-principle

tokenization bpe byte-pair-encoding

Updated Jan 30, 2025
Python

AndreyKolomiets / News_Headline_Generation

Генерация новостных заголовков

pytorch summarization reinforce abstractive-summarization pointer-generator byte-pair-encoding summatization

Updated Nov 21, 2022
Python

SeonbeomKim / Python-Byte_Pair_Encoding

Byte Pair Encoding (BPE)

multiprocessing bpe byte-pair-encoding wordpiece

Updated Feb 25, 2019
Python

rraghavkaushik / smol-bpe-tokenizer

A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.

natural-language-processing tokenizer bpe byte-pair-encoding bytepairencoding llms

Updated Jul 8, 2025
Python

mdabir1203 / BPE_Tokenizer_Visualizer

A Visualizer to check how BPE Tokenizer in an LLM Works

react javascript ai byte-pair-encoding tokenizer-nlp 42wolfsburg llm ai-club

Updated Feb 6, 2025
JavaScript

crodriguez1a / bpe-summarizer

Auto summarization from BPE tokenization

python nlu bart summarization bpe byte-pair-encoding huggingface gpt2tokenizer

Updated Aug 20, 2020
Jupyter Notebook

iamlxb3 / PMTC

Code for the publication of WWW'22

mmorpg representation-learning byte-pair-encoding player-behavioral-sequences user-behavioral-sequences masked-language-modeling

Updated May 31, 2022
Python

AndreiMoraru123 / Neural-Machine-Translation

Modern Eager TensorFlow implementation of Attention Is All You Need

nlp language translation deep-learning tensorflow keras transformers embeddings attention beam-search language-model tokenization bleu-score encoder-decoder label-smoothing self-attention embedding-projector byte-pair-encoding dot-product-attention einops

Updated Oct 7, 2024
Python

Panjete / iidsearch

an efficient ranked retrieval system for English corpora, optimised with VBE and BPE.

inverted-index trec variable-byte-encoding byte-pair-encoding

Updated Nov 10, 2023
Python

Improve this page

Add a description, image, and links to the byte-pair-encoding topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the byte-pair-encoding topic, visit your repo's landing page and select "manage topics."