Prm800k

Visit Tool

prm800k is an Open Source data labeling and annotation tool that provides 800,000 step-level correctness labels for LLM solutions to MATH problems. It is designed to improve mathematical reasoning in AI models through process supervision.

Claim this tool

3Views

At a glance

Pricing

Open Source

Free tier

Yes

API

Skill level

Technical

About

What is prm800k?

prm800k is an open-source dataset and accompanying tools, released by OpenAI, that provides 800,000 step-level correctness labels for large language model (LLM) solutions to mathematical problems from the MATH dataset. This resource is crucial for researchers and developers aiming to enhance the mathematical reasoning capabilities of AI models through process supervision. The repository includes raw labels, instructions for labelers, Python grading logic for answer correctness, and non-standard MATH train/test splits. It also contains scored samples used to evaluate large-scale ORM and PRM models, making it a comprehensive resource for advancing AI in mathematics.

Best used for

Ideal for researchers and developers who need to train AI models for advanced mathematical reasoning, evaluate the performance of LLMs on complex math problems, and develop process supervision techniques. Especially valuable for those working with the MATH dataset and seeking detailed, step-level correctness labels.

Common actions

label data

annotate data

train AI models

evaluate LLM performance

research AI reasoning

face swappinggithub copilot"AI Agents"automated workflowdeepfakelow-code/no-codecollaborationopen-sourceworkflows

Capabilities

Key features

800,000 step-level labels
LLM math solutions
Process supervision data
Python grading logic
Non-standard MATH splits
Scored model samples

Target Audience

professorstudent

Integrations

Not yet documented

Pricing & Plans

Open Source

Free

FAQs

What kind of data does prm800k provide?

prm800k provides 800,000 step-level correctness labels for solutions generated by large language models (LLMs) to problems from the MATH dataset. This data is designed to facilitate process supervision in AI models, helping them improve mathematical reasoning.

How can I access the prm800k dataset?

The prm800k dataset is available on GitHub. You will need to install Git LFS (Large File Storage) to properly clone the repository and access the data files, which are formatted as newline-delimited JSON.

Does prm800k include tools for evaluating model answers?

Yes, prm800k includes Python grading logic to determine if a model-outputted answer correctly matches the ground truth answer in the Hendrycks' MATH dataset. This logic builds on MATH's normalization and uses sympy for expression equality checks.

Trending

Subcategories trending in Data & Analytics

Business Intelligence Predictive Analytics Real-Time Analytics Market Research Data Cleaning & Prep Data Pipelines & Integration

Trending

Also listed in

This tool also appears in

Research & Education › Academic Research Research & Education › Math & Problem Solving Coding & Development › Open Source & Models

Explore

Browse AI tools by category

Content & Design Productivity & Business Coding & Development AI Agents & Automation Research & Education Wellness & Lifestyle Career Development Marketing & Growth Data & Analytics Customer Support & CX Finance E-commerce