Shypd.aiThe largest open index of AI toolsRedPajama-Data is an open-source Data & Analytics tool that provides code for preparing large datasets for training large language models. It includes tools for data processing, quality signal computation, and deduplication.
Trending
Also listed in
About
RedPajama-Data is an open-source repository containing code designed to prepare extensive datasets for training large language models. This tool facilitates the creation and management of high-quality training data through a multi-step pipeline. Key functionalities include preparing artifacts like quality classifiers and generative models, computing various quality signals such as perplexity scores and importance weights, and performing both exact and fuzzy deduplication to refine the dataset. It supports multiple languages including English, German, French, Italian, and Spanish, and offers a robust framework for researchers and developers working with large-scale language model training.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
