Monitoring memory usage in jupyter notebooks

I recently encountered a series of memory errors when using pandas in the LSE high powered computing environment. While detailing the problem, isn’t going to be of particular interest to any one I thought that a quick run down of how to monitor memory usage in a jupyter notebook may be of use to a few people out there.

While there are a few different ways of doing this I found that using the package memory profiler was by far the easiest option. This package doesn’t come with python or anaconda by default, but can be easily installed via pip:

pip install memory_profiler

With the package installed we can load it neatly into a jupyter notebook as an intergrated magic extension:

%load_ext memory_profiler

With the extension loaded we can simply add the magic command ‘%memit’ in any notebook cell to see how much memory is currently in use.

For example:

And to quickly make some random data frames to increase memory usage:

Outside of jupyter memory_profiler can also be used when running a script to give you a nice little plot of the memory used by simply running:

mprof run test.py
mprof plot

Which will output: