Shypd.aiThe largest open index of AI toolsData Prep Kit is an open-source data preparation tool that accelerates unstructured data preparation for GenAI applications. It helps developers cleanse, transform, and enrich data for LLM pre-training, fine-tuning, and RAG applications.
Trending
Also listed in
About
Data Prep Kit is an open-source project designed to accelerate unstructured data preparation specifically for Generative AI applications. It provides a comprehensive set of modules and transforms that enable developers to cleanse, transform, and enrich various types of unstructured data, including natural language, code, and images. This kit supports use cases such as pre-training Large Language Models (LLMs), fine-tuning LLMs, instruct-tuning LLMs, and building Retrieval Augmented Generation (RAG) applications. It is built on common frameworks for Python and Ray runtimes, allowing it to scale from a commodity laptop to data center-scale processing. The kit also offers a framework for developing custom transforms and provides examples for deploying transforms on Kubernetes clusters using Python or Ray jobs, and for orchestrating multiple transforms with Tekton pipelines.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
