TFDS now supports the Croissant 🥐 format! Read the documentation to know more.
Module: tfds.deprecated.text
Stay organized with collections
Save and categorize content based on your preferences.
Text utilities.
tfds includes a set of TextEncoders as well as a Tokenizer to enable
expressive, performant, and reproducible natural language research.
Classes
class ByteTextEncoder: Byte-encodes text.
class SubwordTextEncoder: Invertible TextEncoder using word pieces with a byte-level fallback.
class TextEncoder: Abstract base class for converting between text and integers.
class TextEncoderConfig: Configuration for tfds.features.Text.
class TokenTextEncoder: TextEncoder backed by a list of tokens.
class Tokenizer: Splits a string into tokens, and joins them back.