TensorFlow text processing guide

The TensorFlow text processing guide documents libraries and workflows for natural language processing (NLP) and introduces important concepts for working with text.

KerasNLP

KerasNLP is a high-level natural language processing (NLP) library that includes all the latest Transformer-based models as well as lower-level tokenization utilities. It's the recommended solution for most NLP use cases.

• Getting Started with KerasNLP: Learn KerasNLP by performing sentiment analysis at progressive levels of complexity, from using a pre-trained model to building your own Transformer from scratch.

tf.strings

The tf.strings module provides operations for working with string Tensors.

• Unicode strings: Represent Unicode strings in TensorFlow and manipulate them using Unicode equivalents of standard string ops.

TensorFlow Text

If you need access to lower-level text processing tools, you can use TensorFlow Text. TensorFlow Text provides a collection of ops and libraries to help you work with input in text form such as raw text strings or documents.

• Introduction to TensorFlow Text: Learn how to install TensorFlow Text or build it from source.
• Converting TensorFlow Text operators to TensorFlow Lite: Convert a TensorFlow Text model to TensorFlow Lite for deployment to mobile, embedded, and IoT devices.

Pre-processing

• BERT Preprocessing with TF Text: Use TensorFlow Text preprocessing ops to transform text data into inputs for BERT.
• Tokenizing with TF Text: Understand the tokenization options provided by TensorFlow Text. Learn when you might want to use one option over another, and how these tokenizers are called from within your model.
• Subword tokenizers: Generate a subword vocabulary from a dataset, and use it to build a text.BertTokenizer from the vocabulary.

TensorFlow models – NLP

The TensorFlow Models - NLP library provides Keras primitives that can be assembled into Transformer-based models, and scaffold classes that enable easy experimentation with novel architectures.

• Introduction to the TensorFlow Models NLP library: Build Transformer-based models for common NLP tasks including pre-training, span labelling, and classification using building blocks from the NLP modeling library.
• Customizing a Transformer Encoder: Customize tfm.nlp.networks.EncoderScaffold, a bi-directional Transformer-based encoder network scaffold, to employ new network architectures.