I recently encountered a series of memory errors when using pandas in the LSE high powered computing environment. While detailing the problem, isn’t going to be of particular interest to any one I thought that a quick run down of how to monitor memory usage in a jupyter notebook may be of use to a few people out there.
While there are a few different ways of doing this I found that using the package memory profiler was by far the easiest option. This package doesn’t come with python or anaconda by default, but can be easily installed via pip:
pip install memory_profiler
With the package installed we can load it neatly into a jupyter notebook as an intergrated magic extension:
With the extension loaded we can simply add the magic command ‘%memit’ in any notebook cell to see how much memory is currently in use.
And to quickly make some random data frames to increase memory usage:
Outside of jupyter memory_profiler can also be used when running a script to give you a nice little plot of the memory used by simply running:
mprof run test.py mprof plot
Which will output: