Shypd.aiThe largest open index of AI toolspytorch-memonger is an open-source coding & development tool that optimizes memory usage for deep learning models in PyTorch. It reduces memory footprint from O(N) to O(sqrt(N)) for sequential models, enabling training of larger models or batch sizes.
Trending
Also listed in
About
pytorch-memonger is an open-source Python library designed for sublinear memory optimization in deep learning models built with PyTorch. It re-implements the technique described in "Training Deep Nets with Sublinear Memory Cost," significantly reducing the memory required for backward passes from O(N) to O(sqrt(N)). This optimization is particularly beneficial for training large sequential models or using larger batch sizes that would otherwise exceed available GPU memory. By replacing `nn.Sequential` with `memonger.SublinearSequential`, developers can easily integrate this memory-saving technique. The tool also addresses challenges with non-deterministic layers like BatchNorm and Dropout by re-scaling momentum and memorizing random number generator states, respectively, ensuring reliable performance.
Capabilities
Pricing & Plans
Open Source
Free
FAQs
Trending
