Reducing Memory Footprint While Creating Archive in Django

Python built-in zip library is commonly used to create archive. However, there is a concern when creating zip files using built-in library. Consider the case which we are zipping files larger than our available memory, we would easily run out of memory.

I was building a feature that requires zipping of files and upload to our Django backend storage. After digging around the internet, I summarize the logic I used to support this feature.

Use NamedTemporaryFile Instead Of Memory

A NamedTemporaryFile resides in secondary memory instead of main memory, thus using it does not consume extra memory.

Pseudocode

1. Create a NamedTempFile
2. Create a ZipFile with NamedTempFile as file output
3. Write files into ZipFile
4. Move the cursor of the NamedTempFile back to the beginning
5. Wrap it with Django File 
6. Inject a file name
7. Upload it to storage

Sample Code

Assume we have a model as in models.py, the logic of using secondary memory to create zip files lies in zipping.py.

I hope this helps. Please leave me a comment if you have better idea on reducing memory load when zipping files.

Your clap will definitely drive me further. Give me a clap if you like this post.

Facebook Comments

More Stuff

Top 5 Courses to Learn Python in 2018 — Best of Lo... How to become a Python Programmer If you have just started learning to code a Computer Science graduate and thinking to learn Python in 2018 then...
Making web apps with Jupyter notebook This article will explain how to make Jupiter notebook as a GUI app on the web. What is Jupiter notebook Jupiter notebook is browser-based REPL. REPL ...
Performance Monitoring for the Frontend — An intro... Over the course of this year, I have seen a lot of talks given at different conferences around the globe. Quite a few of them had the focus on perfor...
Writing a Basic Keylogger for macOS in Python A brief look at how to covertly log user activity on macOS Photo by Christin Hume on Unsplash??This post is for educational purposes only?? A keylo...
Spread the love

Posted by News Monkey