Shypd.aiThe largest open index of AI toolsLLMTest_NeedleInAHaystack is a Coding & Development tool that tests the in-context retrieval ability of Large Language Models. It supports OpenAI, Anthropic, and Cohere models for performance measurement.
Trending
About
LLMTest_NeedleInAHaystack is an open-source tool designed to evaluate the in-context retrieval capabilities of Large Language Models (LLMs). It operates by embedding a specific 'needle' (a random fact or statement) within a lengthy 'haystack' (a long context window) and then prompts the LLM to retrieve this statement. The tool allows for iteration over various document depths and context lengths to comprehensively measure model performance. It supports major LLM providers including OpenAI, Anthropic, and Cohere, offering flexibility in testing different models. The package can be easily installed via PyPi and executed from the command line, with options to customize test parameters such as provider, model name, context lengths, and document depths. It also includes features for multi-needle evaluation and integration with LangSmith for orchestrating and storing evaluation results.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
