Coding & Development

Auto-evaluator is a lightweight, open-source evaluation tool designed for question-answering systems utilizing Langchain. It streamlines the process of assessing LLM QA chains by allowing users to input documents, then automatically generating question-answer pairs using GPT-3.5-turbo. The tool then uses a specified QA chain to generate responses to these questions and employs GPT-3.5-turbo again to score the responses against the generated answers. This enables users to explore and compare scoring across various chain configurations, making it an invaluable resource for developers and researchers working on improving the accuracy and performance of their LLM-powered QA applications. It can be run as a Streamlit app and offers configurable inputs for evaluation parameters.

LongBench is an open-source evaluation tool designed to rigorously assess the capabilities of Large Language Models (LLMs) in processing and reasoning over extensive contexts. LongBench v2, the latest iteration, features context lengths ranging from 8k to 2M words, presenting a significant challenge even for human experts. It covers six major task categories including single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repo understanding, and long structured data understanding. The benchmark consists of 503 challenging multiple-choice questions, ensuring reliable evaluation. Data is collected from nearly 100 highly educated individuals, undergoing both automated and manual review to maintain high quality and difficulty. LongBench aims to provide a reliable standard for developing future superhuman long-context AI systems.

Melting Pot is an open-source suite of test scenarios specifically designed for multi-agent reinforcement learning (MARL). Developed by Google DeepMind, it offers researchers a robust platform to train and evaluate AI agents in complex social situations. The tool includes over 50 multi-agent games (substrates) and more than 256 unique test scenarios, allowing for the assessment of generalization to novel social interactions like cooperation, competition, and trust. It is built on DeepMind Lab2D and provides tools for interactive play, evaluation of trained models, and example training scripts using frameworks like RLlib. Melting Pot aims to become a standard benchmark for MARL research, with ongoing development to expand its coverage of social interactions and generalization scenarios.

Markdown Validator is an AI-powered tool built on the CrewAI framework, designed to automate the process of reviewing Markdown files for syntax issues. It integrates a custom tool to identify linting errors within Markdown documents. The system then summarizes these errors into a clear list of recommended changes, helping to maintain consistency and quality in documentation. This tool is particularly useful for developers and content creators who frequently work with Markdown and need to ensure their files adhere to established formatting standards. It can be configured to use various models, including locally hosted solutions or the OpenAI API, offering flexibility in deployment. The project also supports agent training, allowing for iterative improvements based on user feedback.

PromptBench is a PyTorch-based Python package designed as a unified evaluation framework for large language models (LLMs). It offers user-friendly APIs for researchers and developers to conduct comprehensive evaluations of LLMs, including quick performance assessments, prompt engineering method testing (like Chain-of-Thought, Emotion Prompt, and Expert Prompting), and adversarial prompt robustness analysis. The framework integrates dynamic evaluation techniques such as DyVal to mitigate test data contamination and efficient multi-prompt evaluation with PromptEval. It supports a wide range of language and multi-modal datasets and models, both open-source and proprietary, making it a versatile tool for understanding and benchmarking LLM capabilities.

Weighted-Boxes-Fusion is a comprehensive Python library designed for advanced object detection tasks, specifically focusing on ensembling bounding boxes from multiple models. It offers implementations of several key methods, including Non-maximum Suppression (NMS), Soft-NMS, Non-maximum weighted (NMW), and its namesake, Weighted Boxes Fusion (WBF). The WBF method is highlighted for providing superior results compared to other ensembling techniques. The library supports various dimensions, with specific functions for 3D boxes and 1D line segments, the latter being particularly useful for Natural Language Processing (NLP) tasks like Named-entity recognition (NER). It is built with Python 3.*, Numpy, and Numba, ensuring efficient processing. Usage examples are provided for both multiple and single model predictions, making it accessible for developers looking to enhance their object detection pipelines.

Visualizer is a specialized tool designed to simplify the process of visualizing attention maps within deep learning models, particularly those based on Transformer architectures. It addresses common challenges faced by developers, such as the difficulty of extracting deeply nested attention maps without modifying model code or encountering out-of-memory errors. The tool provides a non-intrusive method using Python decorators and PyTorch hooks, allowing users to precisely retrieve intermediate variables like attention maps. This ensures consistency between training and testing phases, as no code changes are required for visualization. It's particularly useful for analyzing complex models like Vision Transformers, enabling the extraction of all attention maps across multiple layers with minimal effort.

Prompt Refine was a dedicated tool for enhancing prompt engineering workflows, allowing users to methodically improve their Large Language Model (LLM) prompts. It integrated with various AI models, including OpenAI, Anthropic, Together, and Cohere, providing a versatile environment for prompt development. Key functionalities included comprehensive history tracking to analyze and compare different prompt runs, enabling users to refine their approaches based on past results. The platform also supported the creation and reuse of variables within prompts, streamlining the experimentation process. Users could export their experiments to CSV for further analysis, making it a valuable asset for data-driven prompt optimization. However, the tool has since been shut down.

ChatGPT-API-Faucet is an open-source project designed to support AI ecosystem developers by providing free ChatGPT API tokens. Inspired by cryptocurrency faucets, this platform allows users to claim one token every 24 hours, which can be used for developing and testing AI products. The project's frontend is built using Next.js and React, making it a suitable resource for developers looking to experiment with AI APIs without immediate cost. It offers a practical solution for those needing small amounts of API credit for initial development, prototyping, or educational purposes, fostering innovation within the AI community.

MOVEdot.ai offers AI agents designed to accelerate hardware engineering tasks, particularly in the analysis of test data. The platform helps engineering teams, especially in automotive, motorsports, manufacturing, aerospace, and robotics, to analyze complex data sets, identify anomalies, and make faster decisions. MOVEdot agents can process 100% of data, reducing analysis time by up to 80% and accelerating iteration cycles by 3x. It connects to various data sources like telemetry, sensor logs, and test standards, providing detailed reports and answers to complex questions in plain English. Proven in demanding environments like motorsports, MOVEdot aims to bring this efficiency to all hardware engineering teams.

Foolbox is a Python library designed to facilitate the creation of adversarial examples that can fool neural networks. Built on EagerPy, it offers native performance across PyTorch, TensorFlow, and JAX, allowing for a unified codebase without duplication. The toolbox provides a comprehensive collection of state-of-the-art gradient-based and decision-based adversarial attacks. It emphasizes type checking to catch bugs early and includes extensive documentation, guides, and tutorials for ease of use. Foolbox is ideal for machine learning researchers and security engineers focused on evaluating and improving the robustness of their models against adversarial attacks.

QA Sphere is an AI-powered test management platform designed to streamline QA processes for software testing. It enables QA teams to create, organize, and track tests with greater speed and efficiency. The platform features AI test case generation, transforming manual writing into intelligent automation, and comprehensive test case management for organizing and scaling test libraries. Users can build advanced test runs, integrate with popular issue trackers like Jira and GitHub, and leverage real-time reporting and analytics. QA Sphere also offers guided migration support to transfer existing test cases and attachments from other systems, ensuring a smooth transition without data loss.

Vispera offers image recognition-based retail execution and tracking services designed for grocery retailers and suppliers. The platform addresses key pain points in retail by improving the speed, accuracy, and precision of information from the selling floor. Vispera's solutions help businesses maximize visibility, minimize out-of-stock situations, and ensure compliance with planograms. Powered by sophisticated deep learning architectures and AI know-how, Vispera provides an end-to-end solution from data collection to reporting, with rich content and flexible integrations. It includes a proprietary retail KPI engine framework, customized and maintained per customer, emphasizing customer-centric onboarding and agile project management.

seqeval is a Python framework designed for the evaluation of sequence labeling tasks, including named-entity recognition (NER), part-of-speech (POS) tagging, and semantic role labeling. It provides robust evaluation capabilities, tested against the industry-standard Perl script `conlleval` for compatibility with CoNLL-2000 shared task data. The framework supports multiple common annotation schemes such as IOB1, IOB2, IOE1, IOE2, IOBES, and BILOU, with strict mode evaluation available for IOBES and BILOU. Users can compute standard metrics like accuracy, precision, recall, and F1 score, and generate comprehensive classification reports to assess model performance effectively. Its flexibility makes it a valuable tool for researchers and developers working on natural language processing tasks.

tau2-bench is a comprehensive simulation framework designed for evaluating customer service agents across various real-world domains. It offers robust support for both text-based, turn-by-turn (half-duplex) evaluation and voice-based, simultaneous (full-duplex) evaluation, leveraging real-time audio APIs. The framework allows users to define a policy for the agent to follow, specify a set of tools the agent can use, and establish tasks to assess the agent's performance. With domains like airline, retail, telecom, and banking knowledge, tau2-bench provides a versatile environment for benchmarking AI performance and testing collaborative workflows. It also includes features for knowledge retrieval with configurable RAG pipelines and an updated leaderboard for comparing model performance.

Awesome-LLM-Eval is a comprehensive, curated list designed for the evaluation of Large Language Models (LLMs) and the exploration of Generative AI's capabilities and limitations. This open-source GitHub project compiles a wide array of resources, including evaluation tools, diverse datasets and benchmarks, practical demos, competitive leaderboards, relevant academic papers, and various LLM models. It serves as an official project for the survey "Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap," offering continuous updates that may not be reflected in the arXiv paper. The repository is actively maintained, welcoming community contributions through pull requests and issues, ensuring it remains a dynamic and up-to-date resource for researchers and developers in the LLM evaluation space.

Keploy is an open-source, AI-powered testing agent and sandboxing platform designed to automate test case generation, dependency mocking, and the creation of production-like sandboxes. Utilizing eBPF, Keploy captures real user traffic to generate comprehensive test cases, helping developers achieve up to 90% test coverage rapidly and with zero code changes. It supports unit, integration, and API testing across multiple programming languages including Go, Java, TypeScript, JavaScript, and Python. Keploy simplifies testing workflows by eliminating flaky tests through AI noise detection, enabling continuous validation in CI/CD pipelines, and supporting legacy application testing without modifications. Its features include infrastructure mocking for databases, APIs, and message queues, making it a robust solution for enhancing test reliability and accelerating development cycles.

QAEverest revolutionizes software testing by leveraging AI to generate comprehensive test cases, significantly improving test coverage and accelerating software development cycles. The platform streamlines workflows through seamless integration with popular project management tools like Jira and ClickUp. Users can download generated test cases in multiple formats, ensuring flexibility and compatibility with existing testing frameworks. QAEverest is designed to enhance test case accuracy and maximize coverage, making it an essential tool for teams looking to optimize their software quality assurance processes with AI-enabled automation.

AI-Red-Teaming-Playground-Labs provides a comprehensive set of training labs and challenges designed for AI Red Teaming. This open-source repository, developed by Microsoft, enables security professionals to run AI Red Teaming exercises, complete with necessary infrastructure. The challenges, originally taught at Black Hat USA 2024, focus on identifying potential issues before AI systems are deployed, covering novel adversarial machine learning and Responsible AI (RAI) failures. It includes labs for direct and indirect prompt injection, metaprompt extraction, and multi-turn attacks, with varying difficulty levels. The playground environment is based on Chat Copilot and integrates with tools like PyRIT for automating attack scenarios, making it an invaluable resource for practical AI security training.

GhostEye is a human vulnerability management platform designed to test and improve an organization's resilience against social engineering attacks. It goes beyond traditional security awareness training by simulating realistic, personalized attacks, including deepfake voice phishing and executive impersonation. The platform maps employee online presence and relationships to build targeted attacks, mirroring real-world attacker methodologies. GhostEye continuously runs tests based on current attack campaigns, providing insights into who might fall for specific tactics. Built by offensive security professionals, it focuses on identifying whether attackers can move money, reset access, or bypass identity workflows through people, vendors, and help desks, offering immediate, attack-specific remediation.

Nullify offers an AI workforce designed for product security automation, acting as an autonomous security engineer. It identifies, triages, and resolves vulnerabilities around the clock, aiming to replace over four traditional security tools and the human effort required to operate them. The platform excels at finding complex bug classes, including business logic flaws, and provides context-rich triage with proof-of-exploit and impact assessment. Nullify automates the resolution process by generating merge-ready fixes, assigning them to the correct developers, and escalating unmerged fixes to ensure SLA compliance. It continuously learns from feedback, adapting its reasoning and actions to the user's environment.

Roborabbit is a powerful no-code web scraping and robotic process automation (RPA) tool designed for data extraction and browser automation. It leverages AI to help users find and capture the data they need with ease. The platform features a task builder for creating custom automations, supporting web scraping, automated testing, and integrations with popular tools like Zapier and Make.com, as well as a REST API. Users can perform various browser actions, capture data, save it to sheets, and even take screenshots. Roborabbit is cloud-based, allowing for simultaneous task execution without limits, and offers video tutorials to guide users through its features. It's ideal for businesses and individuals looking to automate repetitive web tasks and extract valuable data without writing any code.

Katalon True Platform is an AI-driven software quality platform designed to unify the entire testing lifecycle. It enables teams to plan, author, execute, and analyze quality across web, mobile, API, and desktop applications. Key features include AI-powered test generation, autonomous test execution, automated defect reporting, and root cause analysis. The platform supports no-code, low-code, and full-code automation with Katalon Studio, alongside comprehensive test management and real-time reporting and analytics. It also offers production insights to monitor quality post-release, using AI to observe user behavior and generate missing tests automatically. Trusted by over 120,000 testers worldwide, Katalon aims to accelerate releases and deliver exceptional software quality.

Reqops is an AI-powered platform designed to revolutionize requirement management, accelerating innovation and reliability for product teams. It bridges the gap between design and development by instantly converting UX designs into detailed, actionable requirements. The tool eliminates the need for manual requirement building and tedious documentation, boosting productivity and clarity across teams. Reqops facilitates visual mapping and alignment, ensuring creative visions are accurately translated. It also supports testing and automation, improving quality and speed by enabling faster feedback. With features like process flow diagram generation, user story creation, and test case generation, Reqops ensures continuous alignment and enhanced team collaboration.