Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware https://ift.tt/ctHyE0F
Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware https://ift.tt/ctHyE0F
Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is. It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one tensor at a time. I describe the rationale and technical details of the library's design in the blogpost: https://ift.tt/Hr8wXmN https://ift.tt/621sCWg March 4, 2023 at 12:59AM
via Blogger https://ift.tt/RhYU0uD
March 04, 2023 at 01:32AM
Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is. It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one tensor at a time. I describe the rationale and technical details of the library's design in the blogpost: https://ift.tt/Hr8wXmN https://ift.tt/621sCWg March 4, 2023 at 12:59AM
via Blogger https://ift.tt/RhYU0uD
March 04, 2023 at 01:32AM
No comments