Shypd.aiThe largest open index of AI toolsGPT-NeoX is an open-source library for training large-scale language models on GPUs, leveraging techniques from Megatron and DeepSpeed. It supports distributed training across various systems and hardware, including AWS and Slurm.
Trending
About
GPT-NeoX is EleutherAI's open-source library designed for training large-scale language models on GPUs. It builds upon NVIDIA's Megatron Language Model and integrates techniques from DeepSpeed, along with novel optimizations. The project aims to be a centralized and accessible resource for advanced techniques in large-scale autoregressive language model training, accelerating research in the field. It uniquely supports a wide array of systems and hardware, including Slurm, MPI, and IBM Job Step Manager, and has been successfully deployed on platforms like AWS, CoreWeave, and ORNL Summit. Key features include distributed training with ZeRO and 3D parallelism, cutting-edge architectural innovations like rotary and alibi positional embeddings, and easy integration with the open-source ecosystem, such as Hugging Face libraries and experiment monitoring tools like WandB and TensorBoard. It is primarily intended for users looking to train models with billions of parameters from scratch.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
