Shypd.aiThe largest open index of AI toolstensor_parallel automatically splits PyTorch models across multiple GPUs for training and inference, enabling large models to run with potentially linear speedup. It reduces memory load by only loading parts of the model at once.
Trending
Also listed in
About
tensor_parallel is a Python library designed to automatically split PyTorch models across multiple GPUs, facilitating both training and inference for large language models (LLMs). This tool allows users to run models that would otherwise exceed the memory capacity of a single GPU, offering potentially linear speedups. It simplifies the process with a single line of code integration and supports memory-efficient dispatch by converting state_dicts. Key features include options for custom parallelism strategies, distributed training with `torch.distributed`, and sharding parameters using the ZeRO-3 algorithm to avoid duplicate parameters. It is particularly useful for quick prototyping on a single machine with multiple GPUs, offering an easier setup compared to more complex distributed training frameworks.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
