Following up on my task to make it easier to benchmark memory usage in Python, I updated Fabian’s [memory_profiler][] to include a couple of useful IPython magics. While in my last post, I used the new IPython 0.13 syntax for defining magics, this time I used the backwards-compatible one from the previous version.
You can find this work-in-progress as a [pull request on memory_profiler][] from where you can trace it to my GitHub repo. Here’s what you can do with it:
%mprun¶
Copying the spirit of %lprun, since imitation is the most sincere form
of flattery, you can use %mprun to easily view line-by-line memory usage
reports, without having to go in and add the @profile decorator.
For example:
[sourcecode lang=”python”]
In [1]: import numpy as np
In [2]: from sklearn.linear_model import ridge_regression
In [3]: X, y = np.array([[1, 2], [3, 4], [5, 6]]), np.array([2, 4, 6])
In [4]: %mprun -f ridge_regression ridge_regression(X, y, 1.0)
(…)
109 41.6406 MB 0.0000 MB if n_features > n_samples or \
110 41.6406 MB 0.0000 MB isinstance(sample_weight, np.ndarray) or \
111 41.6406 MB 0.0000 MB sample_weight != 1.0:
112
113 # kernel ridge
114 # w = X.T * inv(X X\^t + alpha*Id) y
115 A = np.dot(X, X.T)
116 A.flat[::n_samples + 1] += alpha * sample_weight
117 coef = np.dot(X.T, _solve(A, y, solver, tol))
118 else:
119 # ridge
120 # w = inv(X\^t X + alpha*Id) * X.T y
121 41.6484 MB 0.0078 MB A = np.dot(X.T, X)
122 41.6875 MB 0.0391 MB A.flat[::n_features + 1] += alpha
123 41.7344 MB 0.0469 MB coef = _solve(A, np.dot(X.T, y), solver,
tol)
124
125 41.7344 MB 0.0000 MB return coef.T
[/sourcecode]
%memit¶
As described in my previous post, this is a %timeit-like magic for
quickly seeing how much memory a Python command uses.
Unlike %timeit, however, the command needs to be executed in a fresh
process. I have to dig in some more to debug this, but if the command is
run in the current process, very often the difference in memory usage
will be insignificant, I assume because preallocated memory is used. The
problem is that when running in a new process, some functions that I
tried to bench crash with SIGSEGV. For a lot of stuff, though,
%memit is currently usable:
[sourcecode lang=”python”]
In [1]: import numpy as np
In [2]: X = np.ones((1000, 1000))
In [3]: %memit X.T
worst of 3: 0.242188 MB per loop
In [4]: %memit np.asfortranarray(X)
worst of 3: 15.687500 MB per loop
In [5]: Y = X.copy(‘F’)
In [6]: %memit np.asfortranarray(Y)
worst of 3: 0.324219 MB per loop
[/sourcecode]
It is very easy, using this small tool, to see what forces memory copying and what does not.
Installation instructions¶
First, you have to get the source code of this version of
memory_profiler. Then, it depends on your version of IPython. If you
have 0.10, you have to edit ~/.ipython/ipy_user_conf.py like this:
(once again, instructions borrowed from [line_profiler][])
[sourcecode lang=”python”]
# These two lines are standard and probably already there.
import IPython.ipapi
ip = IPython.ipapi.get()
# These two are the important ones.
import memory_profiler
ip.expose_magic(‘mprun’, memory_profiler.magic_mprun)
ip.expose_magic(‘memit’, memory_profiler.magic_memit)
[/sourcecode]
If you’re using IPython 0.11 or newer, the steps are different. First
create a configuration profile:
[sourcecode lang=”bash”]
$ ipython profile create
[/sourcecode]
Then create a file named ~/.ipython/extensions/memory_profiler_ext.py
with the following content:
[sourcecode lang=”python”]
import memory_profiler
def load_ipython_extension(ip):
ip.define_magic(‘mprun’, memory_profiler.magic_mprun)
ip.define_magic(‘memit’, memory_profiler.magic_memit)
[/sourcecode]
Then register it in ~/.ipython/profile_default/ipython_config.py, like
this. Of course, if you already have other extensions such as
line_profiler_ext, just add the new one to the list.
[sourcecode lang=”python”]
c.TerminalIPythonApp.extensions = [
‘memory_profiler_ext’,
]
c.InteractiveShellApp.extensions = [
‘memory_profiler_ext’,
]
[/sourcecode]
Now launch IPython and you can use the new magics like in the examples above.
Comments !