Shypd.aiThe largest open index of AI toolsArena-Hard-Auto is an AI Testing & QA tool that provides an automatic LLM benchmark. It evaluates instruction-tuned LLMs with high correlation to human preference using automatic judges like GPT-4.1 and Gemini-2.5.
Trending
About
Arena-Hard-Auto is an automatic evaluation tool designed for instruction-tuned Large Language Models (LLMs). It offers a robust benchmark that demonstrates high correlation and separability to LMArena (Chatbot Arena) compared to other open-ended LLM benchmarks. The tool utilizes automatic judges, such as GPT-4.1 and Gemini-2.5, to provide a faster and more cost-effective approximation of human preference. It includes a new evaluation set, Arena-Hard-v2.0-Preview, featuring 500 challenging real-world queries and 250 creative writing queries. Arena-Hard-Auto also supports Style Control for evaluations and provides metrics like Separability with Confidence and Agreement with Confidence to assess benchmark quality. It integrates with various API endpoints, including OpenAI compatible servers, Anthropic, and Amazon Bedrock.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
