Shypd.aiThe largest open index of AI toolsSentencePiece is an unsupervised text tokenizer for neural network-based text generation. It supports subword units like BPE and unigram models, allowing direct training from raw sentences without language-specific pre-processing.
Trending
About
SentencePiece is an unsupervised text tokenizer and detokenizer primarily designed for Neural Network-based text generation systems where the vocabulary size is predetermined. It implements subword units such as byte-pair-encoding (BPE) and unigram language models, uniquely allowing direct training from raw sentences. This eliminates the need for language-specific pre-tokenization tools like Moses or MeCab, making it purely data-driven and language-independent. SentencePiece treats sentences as sequences of Unicode characters, including whitespace as a basic symbol, which ensures reversible tokenization and detokenization. It also supports subword regularization and BPE-dropout to enhance the robustness and accuracy of NMT models, and offers fast, lightweight segmentation with direct vocabulary ID generation.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
