Shypd.aiThe largest open index of AI toolsGPU-Benchmarks-on-LLM-Inference is an Open Source & Models tool that benchmarks GPU performance for Large Language Model inference. It compares LLaMA models' inference speed on NVIDIA GPUs and Apple Silicon, providing detailed performance metrics.
Trending
About
GPU-Benchmarks-on-LLM-Inference is an open-source project designed to evaluate and compare the inference speed of Large Language Models (LLMs) on various GPUs. Utilizing llama.cpp, it provides comprehensive benchmarks for LLaMA models across different NVIDIA GPUs and Apple Silicon devices, including MacBooks and Mac Studio. The project details average speed in tokens/s for both text generation and prompt processing, offering insights into performance across various model sizes and quantization levels. It also includes information on total VRAM requirements and perplexity tables, making it a valuable resource for developers and researchers optimizing LLM deployments.
Capabilities
Pricing & Plans
Open Source · Free
Free
FAQs
Trending
