Coding & Development

Score Jacobian Chaining is a technique designed for analyzing the sensitivity of machine learning models. This tool is invaluable for AI researchers and machine learning engineers seeking to understand the intricate relationship between model inputs and outputs. By providing insights into how changes in input data propagate through a model, it facilitates effective debugging and optimization. This understanding is crucial for improving model performance, ensuring robustness, and gaining deeper insights into model behavior. While the current live website indicates a runtime error, the underlying concept is highly relevant for academic research and practical application in machine learning development.

Ray 3.0 is a comprehensive debugging tool designed to streamline the development process by organizing all debug output in a dedicated desktop application. It eliminates the need for debug output to clutter your application or browser, providing a clean and interactive interface. Ray supports a wide range of languages and frameworks, including PHP, Laravel, JavaScript, Node.js, Vue.js, React, WordPress, and more, allowing developers to use the same debugging syntax across different environments. Key features include remote debugging over SSH, message archiving for later reference, and powerful tools to pause and measure code execution. The latest version, Ray 3.0, introduces enhanced AI integration, enabling users to interact with AI-generated HTML components, Mermaid, and ERD diagrams directly within the app, making it an invaluable tool for modern development workflows.

evalite is an open-source tool designed for developers to evaluate their LLM-powered applications using TypeScript. It provides a robust framework for testing and assessing the performance of AI applications, ensuring quality and reliability. Developers can use evalite to build, run, and analyze tests for their language model integrations. The tool supports a development workflow that includes building, running tests, and a UI dev server for real-time evaluation. It is particularly useful for identifying and fixing issues in LLM-based projects before deployment, contributing to more stable and effective AI solutions.

ttach is an open-source PyTorch library designed for Test Time Augmentation (TTA) in image processing tasks. Similar to data augmentation during training, TTA involves applying random modifications like flips, rotations, and scaling to test images. Instead of feeding a model a single 'clean' image, ttach allows users to show augmented versions multiple times, then averages the predictions from each augmented image to produce a more robust final output. The library provides wrappers for segmentation, classification, and keypoint detection models, along with a flexible `Compose` function for custom transform pipelines. It supports various merge modes for predictions, including mean, geometric mean, sum, max, and min, making it a versatile tool for enhancing model accuracy and stability during inference.

MLJ.jl (Machine Learning in Julia) is an open-source machine learning framework designed for the Julia programming language. It offers a unified interface and a collection of meta-algorithms for various machine learning tasks, including model selection, hyperparameter tuning, evaluation, composition, and comparison. The framework integrates over 200 machine learning models, encompassing those developed in Julia and other languages, providing a comprehensive ecosystem for machine learning workflows. It serves as an umbrella package, distributing components across several other specialized packages, making it a versatile tool for developers and data scientists working with Julia.

LongVideoBench is an AI tool designed for evaluating and benchmarking long video models. It provides a platform to view and sort leaderboard data based on different criteria, including accuracy by duration groups and question categories. This allows researchers and developers to compare the performance of various AI models in understanding and analyzing long-form video content. The tool is particularly useful for those working on video analysis and understanding, offering a structured way to assess model capabilities and identify areas for improvement. Hosted on Hugging Face Spaces, it leverages a robust infrastructure for data display and sorting.

BlinqIO is the first AI Test Engineer, offering an autonomous testing platform designed to understand test requirements and autonomously generate and maintain automation code. It combines AI-powered capabilities with human supervision to ensure limitless scalability and efficiency in software testing. Key features include autonomous test generation, AI-powered test maintenance, multi-language support, enterprise-grade security, seamless integrations, real-time test execution, and intelligent test optimization. BlinqIO aims to revolutionize QA automation by providing a comprehensive solution for developers and QA engineers to streamline their testing processes and deliver high-quality software faster.

RewardBench is an open-source benchmark and evaluation tool specifically designed for assessing the capabilities and safety of reward models, including those utilizing Direct Preference Optimization (DPO). The repository offers common inference code compatible with various reward models such as Starling, PairRM, OpenAssistant, and DPO. It ensures fair evaluation through standardized dataset formatting and testing procedures. Additionally, RewardBench includes robust analysis and visualization tools to help researchers and developers interpret results effectively. It supports quick evaluation of any reward model on any preference set, with features for logging model outputs and accuracy scores, and options for generative models (LLM-as-judge) and DPO models. The platform also facilitates contributing models to a public leaderboard and offers offline ensemble testing.

Shapash is a Python library designed to make machine learning models interpretable and comprehensible for everyone. It offers various visualizations with clear and explicit labels, simplifying the understanding of interactions between a model's features. A key feature is its ability to generate a Webapp, allowing users to easily navigate between local and global explainability. This Webapp helps Data Scientists understand their models and share results with non-data experts. Shapash also contributes to data science auditing by providing comprehensive reports about models and data. It supports Regression, Binary Classification, and Multiclass problems and is compatible with numerous models like Catboost, Xgboost, LightGBM, Sklearn Ensemble, Linear models, and SVM, with options to integrate other models.

Awesome-seml is a comprehensive, curated list of articles dedicated to software engineering best practices for developing machine learning applications. This resource goes beyond core ML algorithms, focusing instead on the crucial surrounding activities such as data ingestion, coding standards, rigorous testing, version control, seamless deployment, quality assurance, and effective team collaboration. It serves as an invaluable guide for ML engineers and software engineers aiming to build robust, reliable, and production-ready machine learning systems. The list is categorized into broad overviews, data management, model training, deployment and operation, social aspects, governance, and tooling, offering a structured approach to understanding and implementing best practices.

deepframeworks offers a comprehensive evaluation of popular deep learning toolkits, including Caffe, CNTK, TensorFlow, Theano, and Torch. This resource, though last updated in early 2016, provides detailed insights into each framework's modeling capability, interfaces, model deployment, performance, architecture, and ecosystem. It highlights strengths and weaknesses, such as Caffe's strong computer vision support versus poor recurrent network capabilities, or TensorFlow's clean architecture but lack of Windows support at the time. The evaluation also covers cross-platform compatibility and performance benchmarks, making it a valuable historical reference for understanding the evolution of deep learning frameworks.

VisualDL is a powerful visualization analysis tool specifically designed for the PaddlePaddle deep learning platform. It offers comprehensive features to help users gain insights into their model training processes and structures. Key capabilities include displaying parameter trends through various charts, visualizing complex model architectures, and examining data samples. By providing a clear and intuitive representation of these critical aspects, VisualDL enables developers and data scientists to efficiently monitor, debug, and optimize their deep learning models, ultimately leading to improved performance and understanding.

VLMEvalKit is an open-source evaluation toolkit designed for large vision-language models (LVLMs), supporting over 220 LMMs and 80+ benchmarks. It simplifies the evaluation process by allowing one-command evaluation without extensive data preparation across multiple repositories. The toolkit uses generation-based evaluation for all LVLMs, offering results with both exact matching and LLM-based answer extraction. Recent updates include improved handling for models with thinking mode and long responses, as well as multi-node distributed inference support for faster evaluations. It aims to provide an easy-to-use, reproducible evaluation environment for researchers and developers.

Will AI do This? is an online gaming platform that offers a comprehensive selection of casino games, including baccarat, slots, roulette, blackjack, and more, from over 50 leading providers. The platform emphasizes direct API connections to game developers, ensuring authenticity and fairness without intermediaries. It features an auto deposit and withdrawal system with no minimum limits, making transactions easy and accessible. The service is available 24/7, supports multiple languages, and is accessible across various operating systems like Android, iOS, Windows, and macOS. The platform also highlights its high customer satisfaction scores and experienced management team in the online gaming industry.

XAI is a comprehensive Machine Learning library focused on AI explainability, maintained by The Institute for Ethical AI & ML. It provides various tools for analyzing and evaluating both data and models, adhering to the 8 principles for Responsible Machine Learning. The library supports a 3-step approach to explainable machine learning: data analysis, model evaluation, and production monitoring. Key functionalities include identifying data imbalances, visualizing correlations, performing balanced train-test splits, evaluating model performance through permutation feature importance, and visualizing metric imbalances across different data columns. It also offers tools for confusion matrix plots, ROC curve analysis, and understanding accuracy grouped by probability buckets, making it invaluable for machine learning engineers and domain experts.

auto-attack is an open-source Python library designed for the reliable evaluation of adversarial robustness in machine learning models. It employs an ensemble of four diverse, parameter-free attacks: APGD-CE, APGD-DLR, FAB, and Square Attack. This approach ensures a comprehensive assessment of model vulnerabilities without requiring extensive hyperparameter tuning. The tool supports both PyTorch and TensorFlow models, providing adapters for seamless integration. It offers standard and more expensive evaluation versions, as well as options for randomized defenses and custom attack configurations. auto-attack is widely used as a standard evaluation benchmark in research, including the RobustBench leaderboard, and provides access to a Model Zoo of robust classifiers.

Vexyl serves as a comprehensive signal console specifically designed for Cloudflare Workers and Pages environments. It enables users to monitor status, metrics, change correlation, incidents, and Service Level Objectives (SLOs) from a unified command-line interface (CLI) and web hub. The tool aims to help developers and operations teams quickly identify and resolve issues such as errors, latency spikes, and deployment regressions. By consolidating critical operational data, Vexyl streamlines the process of moving from issue detection to resolution, reducing the need to switch between multiple dashboards and tools. This focus on Cloudflare's serverless platforms makes it a specialized solution for managing the health and performance of applications deployed on Workers and Pages.

Cybord offers an AI-driven software solution for real-time component inspection during electronics manufacturing. Integrated with pick-and-place and AOI machines, it visually analyzes every component placed on a PCB to enforce Approved Vendor Lists (AVL), provide accurate visually verified traceability, and meet industry regulations like IPC-A-610, AS9100, and ISO26262. The platform inspects both top and bottom sides of components, detecting issues such as body defects, tampering, authenticity, contaminations, bent leads, and setup failures. It also collects critical data like country of origin, lot code, date code, and MPNs, storing it in a secure data repository accessible via API for integration with existing MES or BI systems. Cybord's solution helps prevent defective components from being assembled, significantly reducing quality failures, recalls, and warranty claims.

LogicStar AI is an advanced Coding & Development tool designed to help engineering teams identify and prioritize bugs based on their potential revenue and customer impact. It integrates with existing tools across your stack, such as Sentry, Datadog, Jira, and Git repositories, to connect weak signals and trace issues to their root causes. The platform provides a daily priority queue of bugs, complete with validated fixes and tests to reproduce the bug and confirm its resolution. LogicStar AI aims to reduce the time engineers spend investigating noisy bugs, allowing them to focus on high-impact issues and feature development. It leverages static and dynamic analysis, production signals, and customer usage patterns to build a system-level understanding of your codebase.

Notto is a visual bug reporting tool designed to streamline the QA process by allowing users to annotate directly on any webpage. It eliminates the need for screenshots and lengthy descriptions by enabling users to draw rectangles, arrows, and add text comments on staging or production sites. The tool offers instant synchronization, turning annotations into actionable tickets with a single click, integrating seamlessly with platforms like Linear, Jira, and Asana through webhooks. Notto is particularly beneficial for non-tech-savvy individuals and teams, offering a faster and more efficient way to report visual bugs and provide feedback.

Gitalyze offers an instant code quality assessment for GitHub profiles, allowing users to analyze any developer's GitHub activity, languages, and expertise. This tool is designed to provide insights into code quality, activity, and overall developer expertise. It helps users discover their developer score and uncover hidden strengths within their GitHub contributions. Recruiters and collaborators can leverage Gitalyze to quickly understand a developer's technical background and proficiency based on their public GitHub data. The platform aims to streamline the evaluation process for technical talent and foster better collaboration by providing a clear overview of coding habits and language usage.

Resaro is an independent, third-party AI assurance provider co-headquartered in Singapore and Germany. The company specializes in building AI testing tools to evaluate the performance, safety, and security of dual-use AI systems. Their flagship offering, the Approved Intelligence Platform (AIP), is designed for mission-critical AI validation, bridging the gap between lab performance and real-world deployment. The AIP enables users to run Testing, Evaluation, Validation, and Verification (TEVV) workflows, produce ASQI-structured deployment evidence, and generate clear signals for leadership before fielding AI at scale. It supports operational AI assurance across critical civil and defense sectors, providing granular evidence for engineering teams and continuous evaluation as systems operate.

Deepseek-ai-DeepSeek-R1-0528 Demo offers a direct way to interact with and evaluate the DeepSeek-R1-0528 AI model. This Hugging Face Space allows users to input text prompts and observe the model's natural language generation capabilities. It serves as a practical showcase for the R1 version of the DeepSeek AI model, enabling developers, researchers, and AI enthusiasts to quickly understand its performance and response quality. To access and utilize this demo, users are required to sign in with a Hugging Face account, ensuring a controlled environment for interaction with the model.

CrowdTest offers a human-focused approach to app testing, connecting indie developers with real testers to identify UI/UX issues, unclear flows, spelling mistakes, and other bugs. By leveraging a community of passionate testers, CrowdTest helps ensure products are polished before launch, catching critical issues that automated tools might overlook. The process is straightforward: users sign up, submit their app or website URL, and testers interact with the project to report bugs and suggest improvements. This results in actionable feedback and clear bug reports, enabling developers to launch better, more reliable products with confidence.