Shypd.aiThe largest open index of AI toolsLongNet is an open-source research tool that implements a novel attention mechanism for Transformers. It allows scaling sequence length up to 1 billion tokens without performance loss on shorter sequences.
Trending
Also listed in
About
LongNet is an open-source implementation of the plug-in and play attention mechanism described in the paper "LongNet: Scaling Transformers to 1,000,000,000 Tokens." This Transformer variant is designed to significantly extend the sequence length that models can handle, reaching up to 1 billion tokens, while maintaining strong performance on shorter sequences. Its core innovation is dilated attention, which expands the attentive field exponentially as the distance between tokens grows. LongNet offers linear computational complexity and a logarithmic dependency between tokens, making it suitable for distributed training of extremely long sequences. Its dilated attention can be seamlessly integrated into existing Transformer-based optimization methods, providing a drop-in replacement for standard attention.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
