is an option for Read and related functions which gives a list of token words to be used to delimit words.
TokenWords
is an option for Read and related functions which gives a list of token words to be used to delimit words.
Details
- The setting for TokenWords is a list of strings which are used as delimiters for words to be read.
- The delimiters specified by TokenWords are themselves returned as words.
Examples
Basic Examples (1)
A text file:
Read the file:
Read the file using token words to delimit words in the text:
See Also
Tech Notes
Related Guides
History
Introduced in 1991 (2.0)
Text
Wolfram Research (1991), TokenWords, Wolfram Language function, https://reference.wolfram.com/language/ref/TokenWords.html.
CMS
Wolfram Language. 1991. "TokenWords." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TokenWords.html.
APA
Wolfram Language. (1991). TokenWords. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TokenWords.html
BibTeX
@misc{reference.wolfram_2025_tokenwords, author="Wolfram Research", title="{TokenWords}", year="1991", howpublished="\url{https://reference.wolfram.com/language/ref/TokenWords.html}", note=[Accessed: 04-January-2026]}
BibLaTeX
@online{reference.wolfram_2025_tokenwords, organization={Wolfram Research}, title={TokenWords}, year={1991}, url={https://reference.wolfram.com/language/ref/TokenWords.html}, note=[Accessed: 04-January-2026]}