Vlad Niculae

Optimizing with constraints: reparametrization and geometry.

2020-09-14T00:00:00+02:00

When training machine learning models, and deep networks in particular, we typically use gradient-based methods. But if we require the weights to satisfy some constraints, things quickly get more complicated.

Some of the most popular strategies for handling constraints, while seemingly very different at first sight, are deeply connected. In this post, we will explore these connections and demonstrate them in PyTorch.

In particular, we show that mirror descent is equivalent to gradient descent on a reparametrized objective with straight-through gradients: replacing a constrained variable $x$ with some squishing function $\sigma(u)$, but treating $\sigma$ as if it were the identity in the backward pass.

Outline.

Why are constraints challenging?
Ways to deal with constraints.
- Reparametrization.
- Projected gradient.
Generalizing the projected gradient method with divergences.
Duality between mirror descent and natural gradient.
Conclusions.
Acknowledgements.

\[ \newcommand\pfrac[2]{\frac{\partial #1}{\partial #2}} \newcommand\DP[2]{\left\langle #1, #2 \right\rangle} \]

Why are constraints challenging?¶

In machine learning, we fit models to data by minimizing an objective,

\[\min_{x \in \mathcal{X}} f(x)\,. \tag{OPT}\]

Here, $x$ denotes the parameters to be learned, for instance, the neural network weights. They typically take values in $\mathcal{X}=\reals$, and we train networks by some variant of the gradient method: we choose an initial configuration $x^{(0)}$ and successively applying updates of the form:

\[ x^{(t+1)} \leftarrow x^{(t)} - \alpha^{(t)} \nabla f(x^{(t)})\,. \]

In deep learning, we typically use stochastic flavors that nonetheless perform well and are efficient. Here, we will focus on a relatively nice example: a convex quadratic function $f$. We will see that, even in this case, constraints quickly complicate things.

Why constrain? For modeling reasons, we might want to impose restrictions on some of the weights $x$. Perhaps one of the parameter corresponds to the variance of a distribution, so it cannot be negative. Or perhaps a parameter denotes some sort of “gate”, or mixture between two alternatives $xa_1 + (1-x)a_2$. In this case, we would need to constrain $x \in [0, 1]$. This is often called a box constraint and it is one of the most friendly types of inequality constraint we might deal with.

For one-dimensional convex problems, i.e., $\mathcal{X} = [a, b] \subset \reals$, box constraints do not complicate the problem: we can solve the unconstrained problem $x_{\text{unc}}^* = \argmin_{x\in\reals} f(x)$. If the answer satisfies the constraint, then it must be the solution of the constrained problem as well. If not, the answer can be found by clipping to the interval:

\[ x^\star = \operatorname{clip}_{[a,b]}(x_\text{unc}^\star) \coloneqq \max(a, \min(b, x_\text{unc}^\star))\,. \]

Proof

We add non-negative dual variables $\mu_a$ and $\mu_b$ to handle the inequality constraints $x \geq a$ and $x \leq b$, and write the Lagrangian,

\[\mathcal{L}(x) = f(x) + \mu_a (a-x) + \mu_b(x-b)\,.\]

An optimal $x^\star$ must satisfiy the original constraints $(a \leq x^\star \leq b)$ and be a stationary point of the Lagrangian:

\[ D_x \mathcal{L}(x^\star) = 0 \iff f'(x^\star) = \mu_a - \mu_b\,. \tag{F} \]

The dual variables must be non-negative and satisfy complementary slackness:

\[ \mu_a (a - x^\star) = 0, \quad\text{and}\quad \mu_b (x^\star - b) = 0\,. \]

Let $x^\star_\text{unc}$ be the solution of the unconstrained problem, satisfying $f'(x^\star_\text{unc})=0$. If $a \leq x^\star_\text{unc} \leq b$, then $x^\star=x^\star_\text{unc}$, and choosing $\mu_a=\mu_b=0$ satisfies all conditions.

Otherwise, $x^\star_\text{unc}$ is either too small or too large. Assume $x^\star_\text{unc} > a$ and take $x^\star = a$. Then we have $\mu_b=0$, and from (F) we must have $\mu_a = f'(a)$. Is this a valid value for the dual variable? We must check that $f'(a) \geq 0$. Convex $f$ satisfies $ f(a) - f(x) \leq f'(a)(a-x) $ for any $x$, including the minimizer $x=x^\star_\text{unc}$. By assumption, $a-x^\star_\text{unc} > 0$, so we may divide by it yielding

\[ \mu_a = f'(a) \geq \frac{f(a)-f(x^\star_\text{unc})}{a-x^\star_\text{unc}} \geq 0. \]

The case $ x^\star_\text{unc} > b $ follows similarly.

The following interactive demo might convince you that this is true. We show the 1-d function $f(x) = (x - x_0)^2 / 2$ and its minimizer constrained to $[0, 1]$. Drag the slider to change the location of the unconstrained minimizer $x_0$.

However, in dimension two or more, clipping no longer works, because of interactions between the variables. We demonstate this on a quadratic problem which will become the main focus of the rest of this post,

\[ \min_{x \in \mathcal{X}} f(x) \coloneqq \frac{1}{2}~(x - x_0)^\top Q (x - x_0)\,.\tag{QP} \]

Let us visualize this problem for the specific case of the unit square $\mathcal{X} = [0,1] \times [0,1] \subset \reals^2$, with $x_0 = (1.5, .1)$, and $Q = \left(\begin{smallmatrix}3 & 2 \\\\ 2 & 3 \end{smallmatrix}\right)$. If not for the constraints, since $Q$ is positive definite, the minimum would be $x^\star_\text{unc} = x_0$.Because $f(x_0)=0,$ and positive definiteness guarantess $f(x) > 0$ for any $x \neq x_0$. But a contour plot shows that the constrained minimum $x^\star$ is not the same as the result of clipping the unconstrained minimum to the box.

Constraint handling cannot be left as an afterthought: it needs to be baked into the optimization strategy. There is an important special case where (QP) can be solved exactly: the case $Q = I$. In this case, the problem becomes $\argmin_{x \in \mathcal{X}} \| x - x_0 \|^2_2,$ which is known as the Euclidean projection of $x_0$ onto $\mathcal{X}$. If $\mathcal{X}$ are box constraints, the projection decomposes into a series of independent 1-d projections, which we’ve seen can be solved by clipping.

As a warm up, let us implement our quadratic function $f(x) = \frac{1}{2} {(x - x_0)^\top}Q{(x - x_0)}$ in PyTorch, as well as a minimal gradient descent loop from scratch.

import torch

torch.set_default_dtype(torch.double)

x0 = torch.tensor([1.5, .1])
Q = torch.tensor([[3.0, 2.0],
                  [2.0, 3.0]])

def f(x):
    z = x - x0
    Qz = z @ Q
    return 0.5 * torch.sum(z * Qz, dim=1)


def optim_grad(x_init, lr=.1, max_iter=100):
    x = x_init.clone().requires_grad_()

    for i in range(max_iter):
        f(x).backward()
        x.data -= lr * x.grad
        x.grad.zero_()

    return x

This procedure quickly converges to $x^\star_\text{unc} = (1.5, .1)$.

Ways to deal with constraints.¶

When faced with a box-constrained optimization problem, these are the ideas that most practitioners would turn to.This turns out to be a handy personality quiz to see if somebody resonates more with neural networks or with convex optimization.

Reparamtrization (REP). Circumvent the constraints on $x$, by expressing the function in terms of unconstrained variables $u$ such that $x_i = \sigma(u_i)$, where $\sigma : \reals \to \mathcal{X}$ is a “squishing” nonlinearity. We can then perform unconstrained minimiziation on $f \circ \sigma$. For $\mathcal{X}=[0,1]^d$, we may use the logistic function,General intervals $[a,b]$ are obtained by affinely transforming $[0,1]$.

\[ \sigma(u) = \frac{1}{1 + \exp(-u)}\,.\]
Projected gradient (PG). Perform unconstrained gradient updates, then project back onto the domain after each update:

\[ \begin{aligned} x^{(t+½)} &\leftarrow x^{(t)} - \alpha^{(t)} \nabla f(x^{(t)})\,, \\ x^{(t+1)} &\leftarrow \operatorname{Proj}_\mathcal{X}\big(x^{(t+½)}\big)\,. \\ \end{aligned} \]

REP is convenient when working with neural network libraries like PyTorch, because it can be implemented just by changing our model, without requiring modifications to the optimization code. However, the resulting problem (after reparametrization) is no longer convex in $u$, even if the original problem was convex in $x$. PG directly solves the convex optimization problem (QP), but the intermediate iterates $x^{(t+0.5)}$ can leave $\mathcal{X}$, leading to a possibly less stable or too aggressive trajectory.

In this post, we will explore the connection between the two by studying mirror descent and its information-geometric interpretation as natural gradient in a dual space. But first, let’s explore our two initial ideas.

Reparametrization.¶

Instead of optimizing w.r.t. the constrained variables $x$, we introduce an underlying variable $u$, such that $x_i = \sigma(u_i)$.This seems to be the more common method among neural network practitioners; one place where it shows up often is neural variational inference, where we would constrain the variance of a learned distribution using a softplus function. In our case, we use a logistic sigmoid reparametrization to get the unconstrained non-convex problem

\[ \min_{u \in \reals^2} f(\sigma(u))\,, \]

where $\sigma$ is applied element-wise.
When reparametrizing, $x$ is no longer a learned parameter, but a function of the learned parameter $u$. The gradient with respect to $u$ can be handled automatically by PyTorch autodiff:

def optim_reparam(u_init, lr=.1, max_iter=100):
    u = u_init.clone().requires_grad_()

    for i in range(max_iter):
        f(torch.sigmoid(u)).backward()  # compute grad wrt u
        u.data -= lr * u.grad  # take gradient step
        u.grad.zero_()

    return u

With a very small learning rate, we get a glimpse into the dynamics of this method.As the learning rate goes to zero, we are simulating a continuous gradient flow, of which gradient descent is a discretized approximation. For more about gradient flows, check out Francis Bach’s post. For comparison, we include the unconstrained trajectory.

In practice, we would use a much larger learning rate to accelerate optimization:

We can see that even with a large learning rate, the reparametrization method takes much smaller steps, especially as it gets closer to the boundary of the domain. The steps are so small, that we only show the first 20 markers, to avoid clutter. Why does this happen? At any point $u$, the reparametrized gradient can be written using the chain rule:

\[ \pfrac{}{u} f(\sigma(u)) = \pfrac{\sigma(u)}{u} \pfrac{f(x)}{x}\biggr\rvert_{x=\sigma(u)} \]

This is the unconstrained gradient at $x=\sigma(u)$, rescaled by the Jacobian of $\sigma$. Since $\sigma$ acts elementwise, its Jacobian is a diagonal matrix, with $\pfrac{\sigma(u)_i}{u_i} = \sigma(u_i)(1 - \sigma(u_i)) = x_i (1 - x_i).$ We can thus see that as $x_i$ approaches $0$ or $1$, the reparametrization rescales the gradient severely, bringing the effective step size toward 0. It can help to see the concrete values and shapes of these matrices:

Calculation

Let’s consider two points: first far, then close to the optimum. We use that $\nabla_x f(x) = Q(x - x_0)$. (Note: all vectors below are column vectors.)

$u$	${x=\sigma(u)}$	$\pfrac{\sigma(u)}{u}$	$\pfrac{f(x)}{x}$	$\pfrac{f(\sigma(u))}{u}$
$(0, 0)$	$(.5, .5)$	$\left(\begin{smallmatrix} .25 & 0 \\\\ 0 & .25 \end{smallmatrix}\right)$	$(-2.2, -0.8)$	$(-0.55, -0.2)$
$(4.6, -0.4)$	$(.99, .4)$	$\left(\begin{smallmatrix} .0099 & 0 \\\\ 0 & .24 \end{smallmatrix}\right)$	$(-0.93, -0.12)$	$(-0.01, -0.03)$

Two effects are at play here: the gradient of $f$ gets smaller as we approach the minimum, and the gradient of $\sigma$ gets vanishingly small as we approach the boundary of the domain.

Remember, this rescaling happens automatically through the chain rule! But, although slowly, and along a slightly winding trajectory, our method finds the right answer.

Projected gradient.¶

The projected gradient method is particularly well suited to handling “simple” constraints like the box case, but, unlike reparametrization, requires a different kind of expertise to get running in the case of complicated constraints.PG is very popular in convex optimization, useful both for theory and for practice. However, it does not seem to be so widely used in the pure neural network world, perhaps mostly because it is not directly supported by the built-in optimizers in major frameworks. For box constraints, the projection can be computed efficiently, since $\big[\!\operatorname{Proj}_{[0,1]^d}(x)\big]_i = \operatorname{clip}_{[0,1]}(x_i).$ The implementation follows:

def optim_pg(x_init, lr=.1, max_iter=100):
    x = x_init.clone().requires_grad_()

    for i in range(max_iter):
        f(x).backward() 
        x.data -= lr * x.grad  # take gradient step
        x.data = torch.clamp(x.data, min=0, max=1)  # project
        x.grad.zero_()

    return x

Let’s visualize the trajectory. From now on, we will zoom in a bit on the region of interest.

It looks like the projected gradient method tends to follow the unconstrained trajectory while sticking to the boundary of the domain. Of course, with larger steps, the differences become more pronounced.

In this instance, PG is the clear winner: look how fast it makes progress. With less well-behaved and non-convex functions this need not be the case. So we are motivated to delve deeper and explore how PG and REP relate, despite seeming so different.

Generalizing the projected gradient method with divergences.¶

In the projected gradient method, we take unconstrained steps, which might take us outside of $\mathcal{X}$, and then move the solution back to $\mathcal{X}$ by projection:

\[ x^{(t+1)} \leftarrow \operatorname{Proj}_\mathcal{X}\big(x^{(t+½)}\big)\,. \]

Projection finds the point $x \in \mathcal{X}$ closest to $x^{(t+½)}$, i.e.,

\[ \operatorname{Proj}_\mathcal{X}\big(x^{(t+½)}\big) \coloneqq \argmin_{x \in \mathcal{X}} \| x - x^{(t+½)} \|^2\,. \]

The projected gradient update can be interpreted as minimizing a regularized linearization of $f$ around the current iterate,

\[ x^{(t+1)} \leftarrow \argmin_{x \in \mathcal{X}} \DP{\nabla f(x^{(t)})}{x} + {\frac{1}{2\alpha_t}\|x - x^{(t)}\|^2}\,. \]

Explanation

Why does this update make sense, and where does it come from? We are trying to minimize a function $f(x)$, but we don’t know what it looks like globally: we only have access to its value $f(x)$ and its gradient $\nabla f(x)$ at points $x$ that we may query one at a time. At any point $x_0$, the first-order Taylor expansion is:

\[ f(x_0 + \delta) = f(x_0) + \DP{\nabla f(x_0)}{\delta} + o(\|\delta\|)\,. \]

To get a linear approximation of $f$ we can plug in $\delta = x - x_0$:

\[ f(x) = f(x_0) + \DP{\nabla f(x_0)}{x - x_0} + o(\|x - x_0\|)\,. \]

So as long as $x$ is not too far from $x_0$, we have

\[f(x) \approx \tilde{f}_{x_0}(x) \coloneqq f(x_0) + \DP{\nabla f(x_0)}{x - x_0}\,.\]

This affine approximation is much easier to minimize, but it is only accurate locally, therefore, we use it iteratively, taking a small step, then updating the approximation:

\[ x^{(t+1)} \leftarrow \argmin_{x \in \mathcal{X}} \tilde f_{x^{(t)}}(x) + \frac{1}{2\alpha_t}\|x - x^{(t)}\|^2\,. \]

where the term on the right keeps us close to $x^{(t)}$ to ensure the approximation is not too bad. Clearing up the constant terms from inside the $\argmin$ yields the desired expression.

Rearranging the terms reveals exactly the projected gradient update,

\[ x^{(t+1)} \leftarrow \operatorname{Proj}_{\mathcal{X}}\big( x^{(t)} - \alpha_t \nabla f(x^{(t)})\big)\,.\]

Derivation

\[ \begin{aligned} & \argmin_{x \in \mathcal{X}} \DP{x}{\nabla f(x^{(t)})} + \frac{1}{2\alpha_t} \|x-x^{(t)}\|^2 \\ =& \argmin_{x \in \mathcal{X}} \DP{x}{\nabla f(x^{(t)})} + \frac{1}{2\alpha_t} \|x\|^2 - \frac{1}{\alpha_t} \DP{x}{x^{(t)}} \\ =& \argmin_{x \in \mathcal{X}} \alpha_t \DP{x}{\nabla f(x^{(t)})} + \frac{1}{2} \|x\|^2 - \DP{x}{x^{(t)}} \\ =& \argmin_{x \in \mathcal{X}} \DP{x}{\underbrace{\alpha_t \nabla f(x^{(t)}) - x^{(t)}}_{-x^{(t+½)}}} + \frac{1}{2} \|x\|^2 \\ =& \argmin_{x \in \mathcal{X}} \frac{1}{2} \| x - x^{(t+½)} \|^2 \textcolor{gray}{ - \frac{1}{2} \|x^{(t+½)}\|} \\ =& \operatorname{Proj}_{\mathcal{X}} (x^{(t+½)})\,. \end{aligned} \]

Now, let’s pay special attention to the function $D(x, y) = \| x - y \|^2 = \sum_i (x_i - y_i)^2$, the squared Euclidean distance. We employ this function to ensure that each update stays close to $x^{(t)}$, because the linear approximation is not good if we go to far. But this is not the only good measure of closeness: here, geometry enters the stage! Euclidean geometry is convenient and comfortable for thinking about spaces like $\reals^d,$ but it is not always the best model.

Bregman divergences provide a convenient generalization of the squared Euclidean distance: Divergences measures of dissimilarity between objects, with weaker requirements than distance functions: simply that $D(x,y) \geq 0$ and $D(x,y)=0$ iff. $x = y$. Bregman divergences are an important class of divergences. They are convex in the first argument, but not the second. given strictly convex, twice-differentiable $\Psi$,

\[ D_\Psi(x, y) \coloneqq \Psi(x) - \Psi(y) - \DP{\nabla \Psi(y)}{x - y}\,. \]

For $\Psi = \frac{1}{2} \| \cdot \|^2$, this recovers the squared Euclidean distance. Replacing $\frac{1}{2}\|\cdot\|^2$ by $D_\Psi$ in the projected gradient algorithm leads to a generalization known as mirror descent, Introduced in:
A.S. Nemirovsky and D.B. Yudin,, 1983. Problem complexity and method efficiency in optimization.
Suggested reading: A. Beck and M. Teboulle, 2003. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3). 167-175.

\[ x^{(t+1)} \leftarrow \argmin_{x \in \mathcal{X}} \DP{\nabla f(x^{(t)})}{x} + {\frac{1}{\alpha_t}D_\Psi(x, x^{(t)})}\,. \]

Since $\Psi$ is twice differentiable and strongly convex, it has a gradient $\psi = \nabla\Psi$ which is invertible. Solving for the mirror descent update yields a so-called Bregman projection, or non-linear projection,

\[ \begin{aligned} u^{(t+½)} &\leftarrow \psi(x^{(t)}) - \alpha_t \nabla f(x^{(t)})\,, \\ x^{(t+1)} &\leftarrow \argmin_{x \in \mathcal{X}} D_\Psi\big(x, \psi^{-1}(u^{(t+½)})\big)\,. \\ \end{aligned} \]

If $\Psi$ is carefully chosen such that $\psi^{-1}(u) \in \mathcal{X}$ for all $u \in \mathcal{U}$, then the Bregman projection step is trivial, yielding

\[ x^{(t+1)} \leftarrow \psi^{-1}\big( \psi(x^{(t)}) - \alpha_t \nabla f(x^{(t)}) \big)\,. \]

Let’s make this more concrete.

The choice of $\Psi$ will define the geometry of our space. Values in $[0,1]$ may be interpreted as coin flip probabilities: the higher, the more likely an event is to happen. An important property of a binary random variable is its entropy. If $x_i \in [0, 1]$ denotes the probability associated with coin $i$, the entropy is

\[H(x_i) = -x_i \log x_i - (1 - x_i) \log (1 - x_i)\,.\]

We may extend this additively to vectors as $H(x) = \sum_i H(x_i)$. This is sometimes called the Fermi-Dirac entropy. On $\mathcal{X}=[0, 1]$, entropy is continuously differentiable and strictly concave, maximized at $x=0.5$ and minimized at $x=0$ and $x=1$.$H(0.5) = \log 2,$
$H(0)=H(1)=0$.

Let us thus take $\Psi = -H$. Its gradient is $\psi : \mathcal{X} \rightarrow \mathcal{U}$,

\[ \psi(x) \coloneqq -\nabla H(x) = \log(x) - \log(1-x)\,, \]

with inverse $\phi : \mathcal{U} \rightarrow \mathcal{X}$,

\[ \phi(u) \coloneqq \psi^{-1}(u) = \frac{1}{1 + \exp(-u)} = \sigma(u)\,. \]

So, written in terms of the familiar logistic sigmoid, the mirror descent update induced by the negative entropy takes the (remarkable!) form

\[ x^{(t+1)} = \sigma(\sigma^{-1}(x^{(t)}) - \alpha_t \nabla f(x^{(t)}))\,. \]

Things are beginning to clear up! We can think of the pair of inverse functions $(\psi, \phi)$ as maps between $\mathcal{X}$ and $\mathcal{U}$. We will call these the primal and dual spaces, respectively.Duality, and the terms primal and dual, are fairly strong, scary, and sometimes overloaded. Readers familiar with the Fenchel conjugate, $\Psi^*(u) \coloneqq {\sup_{x} \{ \DP{u}{x} - \Psi(x) \},}$ should note note that $\Psi^* = \Phi$, where $\Phi$ is such that $\nabla\Phi = \phi$. (For the Fermi-Dirac entropy, $\Phi(u) = {\log(1+\exp(u)).}$) The dual construction used here relies on the relationship between Bregman divergences and Fenchel conjugates: $ D_{\Phi^*}(u, v) = D_\Phi(y, x)$ where $u=\psi(x), v=\psi(y)$ are the dual points corresponding to $x$ and $y$. Mirror descent thus first moves into dual (unconstrained) space, performs an update there, and then moves back. Reparametrization rewrites the problem in dual coordinates and performs gradient descent: this is not the same, and the trajectories are quite different!

With a larger step size, we see that mirror descent takes much larger steps than reparametrization does.

Now that we figured out that we may think about the problem in primal or dual coordinates, we may also visualize the optimization trajectory in dual coordinates.

Yet, the way that mirror descent leans on moving from $\mathcal{X}$ to $\mathcal{U}$ is very familiar to the reparametrization strategy. Is there a deeper connection between the two? We illuminate it next.

Duality between mirror descent and natural gradient.¶

We have seen that different choices of $\Psi$ induce different geometries even on top of the same space. (Case in point: entropy vs. $\frac{1}{2}\|\cdot\|^2$). To handle this ambiguity, we need a structure that attaches the geometry along with the underlying space. This, (with some handwaving), is a Riemannian manifold: a pair $(\mathcal{U}, G)$ where $\mathcal{U} \subseteq \reals^d$ is an underlying spaceRiemannian manifolds are more general than this, but in this post, for simplicity, we only look at the case where $\mathcal{U}\subseteq \reals^d$. To be fully general, the notation ramps up quickly. See Agustinus Kristiadi’s post and Frank Nielsen’s tutorial for an introduction. and $G: \mathcal{U} \rightarrow \reals^{d \times d},$ known as the metric tensor, is a function that takes a point $u_0$ on the manifold and returns a positive definite matrix that characterizes the geometry by specifying a custom squared distance nearby $u_0$:

\[ d^2_{u_0}(u, v) = \frac{1}{2} (u-v)^\top G(u_0) (u - v)\,. \]

What we observed earlier is in fact a duality between the Riemannian manifolds $(\mathcal{X}, \nabla^2 \Psi)$ and $(\mathcal{U}, \nabla^2 \Phi),$ where $\Phi$ is the antiderivative of $\phi$, i.e., $\nabla\Phi=\phi$. This is an important duality studied in the field of information geometry. S. Amari, A. Cichocki. 2010. Information geometry of divergence functions. Bulletin of the Polish Academy of Sciences, 58(1).

We may now revisit the reparametrization strategy, in light of what we learned so far. To reparametrize the constraint away, we work in dual coordinates $u \in \mathcal{U}$, and update:

\[ u^{(t+1)} \leftarrow u^{(t)} - \nabla_u f(\phi(u))\,. \]

In our case $\mathcal{U}=\reals^2$ and $\phi = \sigma$, but let’s use the more general notation. The update above ignores the geometry of $\mathcal{U}$, in other words, operates on the trivial manifold $(\mathcal{U}, I_d)$ — the metric tensor is the identity matrix everywhere. When optimizing some function $\tilde{f}(u)$ over a Riemannian manifold $(\mathcal{U}, G)$, a convenient algorithm is natural gradient,S. Amari, 1998.
Natural gradient works efficiently in learning. Neural computation 10.2, 251-276. which takes the form

\[ u^{(t+1)} \leftarrow u^{(t)} - \alpha_t [G(u)]^{-1} \nabla \tilde{f}(u^{(t)})\,. \]

Derivation

We follow the same steps as for gradient descent, but we use the induced distance $d_{u^{(t)}}$ to find an update direction. We must solve the problem

\[ \argmin_{u\in\mathcal{U}} \DP{u}{\nabla \tilde{f}(u^{(t)})} + \frac{1}{2\alpha_t} {(u - u^{(t)})^\top G(u^{(t)}) (u - u^{(t)})}\, \]

Rearranging gives the equivalent problem

\[ \argmin_{u\in\mathcal{U}} d^2_{u^{(t)}} (u, u^{(t)} - \alpha_t[G(u^{(t)})]^{-1} \nabla \tilde{f}(u^{(t)}))\,.\]

Assuming no constraints (i.e., $\mathcal{U}=\reals^d$), this yields the desired update.

It turns out that for a pair of Bregman dual manifolds $(\mathcal{X}, \nabla^2 \Psi)$ and $(\mathcal{U}, \nabla^2 \Phi),$ mirror descent in the primal space is equivalent to natural gradient in the dual space! This result, due to Raskutti and Mukherjee (2015), G. Raskutti and S. Mukherjee. 2015. The information geometry of mirror descent. In: IEEE Transactions on Information Theory, vol. 61, issue 3. arXiv:1310.7780. means we can get a geometry-aware flavor of the REP algorithm,

\[ u^{(t+1)} \leftarrow u^{(t)} - \alpha_t [\nabla^2 \Phi(u)]^{-1} \nabla_u {f}(\phi(u^{(t)}))\,. \]

This algorithm produces the exact same iterates as mirror descent, but — since it maintains iterates in dual space — is more numerically stable. A first attempt at implementing natural gradient is:

def optim_rep_natural(u_init, lr=.1, max_iter=100):

    u = u_init.clone().requires_grad_()

    for i in range(max_iter):
        x = torch.sigmoid(u)
        dsigmoid = (x * (1 - x)).detach()  # diagonal of ∇²Φ(u)
        f(x).backward()
        u.data -= lr * u.grad / dsigmoid
        u.grad.zero_()

    return u

and we can visually check that we get exactly the same trajectory as mirror descent.

But let’s look a bit closer! Since $\phi = \nabla\Phi$, our metric tensor is none other than

\[ \nabla^2\Phi(u) = \pfrac{\phi(u)}{u}\,, \]

and, putting the whole update together, we see that

\[ [\nabla^2\Phi(u)]^{-1} \nabla_u f(\phi(u)) = {\left(\pfrac{\phi(u)}{u}\right)^{-1}} {\pfrac{\phi(u)}{u}} {\pfrac{f(x)}{x}\biggr\rvert_{x=\phi(u)}} \]

Natural gradient cancels out the Jacobian of $\phi$ in the chain rule! This suggests an alternative implementation, reminiscent of straight-through tricks: we use a reparametrization function $\sigma$ that is a sigmoid in the forward pass, but acts as if it were the identity in the backward pass. Such “straight-through” heuristics are very popular for dealing with stochastic or piecewise-constant functions (like argmax or floor functions.) To my knowledge, the idea was introduced by G. Hinton in the Neural Networks for Machine Learning online lectures in 2012. For a detailed treatment, see:
Y. Bengio, N. Léonard, A Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432.

class SigmoidStraightThrough(torch.autograd.Function):

    @staticmethod
    def forward(ctx, u):
        return torch.sigmoid(u)

    @staticmethod
    def backward(ctx, dx):
        return dx


def sigmoid_straight_through(u):
    return SigmoidStraightThrough.apply(u)


def optim_rep_natural_st(u_init, lr=.1, max_iter=100):

    u = u_init.clone().requires_grad_()

    for i in range(max_iter):
        f(sigmoid_straight_through(u)).backward()
        u.data -= lr * u.grad
        u.grad.zero_()

        print(u.data, torch.sigmoid(u.data))

    return u

With this sigmoid_straight_through nonlinearity, we can now use mirror descent / natural gradient to learn constrained parameters without any changes to the optimizer, and with improved numerical stability.Instead of cancelling out the gradient of $\sigma$ numerically, we now avoid multiplying by it in the first place. Unlike our initial reparametrization algorithm, this algorithm uses the geometry of the dual space $(\mathcal{U}, \nabla^2\Phi)$ and thus avoids taking extra small steps while still ensuring that all the iterates remain feasible.

Conclusions.¶

We have explored the two most popular strategies for dealing with simple constraints: reparametrization and projected gradient optimization. We have looked into an information geometric generalization of projected gradient, which turns out to lead to an equivalent dual algorithm that resembles reparametrization, but with a gradient correction that improves its performance. (And, if $f$ is convex, results in a convex optimization procedure, unlike the reparametrized case.)

Of course, we only looked at a simple quadratic test case, and we did not check what happens when using accelerated methods or adaptive learning rates (e.g,. Adam). But empirical questions are not the main point.

With this post, I have hopefully stirred your interest into constrained optimization and its connections to geometry. You may be wondering what other constraints we may handle this way. Two important examples are non-negativity constraints $\mathcal{X}=[0, \infty)^d,$ for which we may use $\Psi(x) = \sum_i x_i(\log x_i - 1),$ giving $\phi(u) = \exp(u)$,Another interesting choice is $\Psi(x) = x \log(\exp(x)-1) + \operatorname{Li}_2(1-\exp(x)),$ where $\operatorname{Li}_2$ denotes the dilogarithm. This leads to the softplus reparametrization $\phi(u) = \log(1+\exp(u))$. While the dilogarithm lacks a closed-form expression, we only need to evaluate $\phi$ and $\psi$. and the simplex $\mathcal{X} = {\{ x \in \reals^d : x_i \geq 0, \sum_i x_i = 1 \},}$ for which the negative Shannon entropy $\Psi(x) = {\sum_i x_i \log x_i}$ yields the softmax reparametrization $\phi(u) = {\frac{1}{Z} \exp(u)}$ with $Z=\sum_i\exp(u_i).$ The resulting simplex-constrained algorithm is known under many names, including “multiplicative weights”, “entropic descent”, or “exponentiated gradient”. J. Kivinen, J and M.K. Warmuth, 1997. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 132(1), 1-63. This algorithm performs the elementwise multiplicative update

\[ x^{(t+1)} \propto x^{(t)} \odot \exp\big(-\alpha^{(t)} \nabla f(x^{(t)})\big)\,.\]

More interesting constraint examples involve matrices. Geometric insights have been key to advances in learning over constrained spaces of matrices, such as symmetric, low-rank, orthonormal, positive (semi-)definite matrices, etc. P.-A. Absil, R. Mahoney, and Rodolphe Sepulchre. 2008. Optimization Algorithms on Matrix Manifolds. Princeton University Press. ISBN 978-0-691-13298-3 Schäfer et al. use the duality between mirror descent and natural gradient to derive powerful algorithms for minimax problems. F. Schäfer, A. Anandkumar, H. Owhadi. 2020. Competitive Mirror Descent. arXiv:2006.10179.

Acknowledgements.¶

This post was inspired by Anima Anandkumar’s talk at the IAS Seminar on Theoretical Machine Learning. Before this talk, I had no idea about anything in the second part of this post. Thanks to Mathieu Blondel, Caio Corro, André Martins, Fabian Pedregosa, and Justine Zhang for feedback, and to all Twitter users who pointed out typos. I am funded by the European Research Council StG DeepSPIN 758969 and Fundação para a Ciência e Tecnologia contract UIDB/50008/2020.

Rendering 3-d surface plots for social media

2018-08-21T00:00:00+02:00

It’s 2018, and your favorite meme pages on facebook constantly come up with quality content like this. (If you’re on mobile, you will need to open the below in the Facebook app for the full immersive experience, sadly.)

How can we mere researchers even begin to compete in terms of social media presence? What chance do we have at going viral? In this post, I show you how to generate 3-d renders of your or your friends’ cool machine learning research.

Without further ado, let’s make some data-driven surface plots.Get code for this post.

three.js¶

There seems to be a lot of 3-d modeling software out there, such as Blender, but point-and-click interfaces don’t seem well-suited for data-driven plots.Blender does seem to have a Python API, but I couldn’t easily figure out if it supports all we’re doing here. If you know, leave a comment!

After a while I settled on three.js, a JavaScript library for interactive 3-d graphics. I found Graphulus: a three.js example that seems close to what we want. Also, three.js supports exporting to GLTF, the format required to upload 3-d posts.

The strategy used by Graphulus uses the three.js ParametricGeometry: a 3-d geometry defined by a function z = f(x, y). This seems to be exactly what we want! If we can implement f easily in JS, we are done!

In most cases, f is complicated and took months to implement, using Python or C++, so we might not be able (or willing) to rewrite it in JS. I see two possible paths to take here:

Building a web app, and implementing f as an API call.
Finding a way to draw a three.js surface plot from precomputed values.

Five years ago I would have been eager to take approach 1 and build an overcomplicated solution, but this time I decided 2 is lazier.

I eventually found this example of building a 3-d plot in three.js from a grid of points. In the rest of this blog, I will walk through the resulting method.

Computing the function values¶

First thing to do is to compute the value of f on a grid of points. To keep things simple, we will plot $\operatorname{softmax}([x, y, 0])_2$ as a function of $a$ and $b$. Recall that $\operatorname{softmax}(\boldsymbol{\theta}) = \boldsymbol{p}$, where $p_i = \frac{\exp(\theta_i)}{\sum_j \exp(\theta_j)}$.

We can implement f(x, y) easily:

# generate.py

import math

def f(x, y):
    """computes p[1] where p = softmax([x, y, 0])

    (it's p[1], not p[2], because in math we index from 1
    """

    theta = np.array([x, y, 0])
    p = np.exp(theta) / np.sum(np.exp(theta))
    return p[1]

Now, let’s define a grid of points. This is more or less the same as for making a 3-d surface plot in matplotlib.

# generate.py (part 2)

import numpy as np

n = 111 
m = 111 

xs = np.linspace(-3, 3, n)
ys = np.linspace(-3, 3, m)

X, Y = np.meshgrid(xs, ys)
Z = np.empty_like(X)

for i in range(n):
    for j in range(m):
        x = X[i, j]
        y = Y[i, j]
        Z[i, j] = f(x, y)

points = np.column_stack([X.ravel(), Y.ravel(), Z.ravel()])

The easiest way to copy data between Python and JS is using json. Indeed, we can very easily generate a valid JS file:

# generate.py (part 3)
# output points to a javascript file

import json

data = {
    'x': points[:, 0].tolist(),
    'y': points[:, 1].tolist(),
    'z': points[:, 2].tolist()
}

template = f"""\
var n = {n};
var m = {m};
var data = {json.dumps(data)};
"""

with open('data.js', 'w')  as f:
    print(template, file=f)

Run this script to obtain a data.js file describing the point cloud that we want to visualize.

Constructing the 3-d surface¶

Once we have the 3-d points, it’s time to use three.js to construct the 3-d object. First, download three.js itself, as well the GLTFExporter.js file.The links provided are for version r95, the version I used. Feel free to try newer versions; new features may appear!

Let’s make a minimal HTML file to load the required libraries:

<!-- plot-glb.html -->

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<script src="three.js"></script>
<script src="GLTFExporter.js"></script>
<script src="data.js"></script>  <!-- generated above -->
</head>

<body>
<a id="download">Download .glb</a>
<script src="plot-glb.js"></script> <!-- we'll write this file below -->
</body>
</html>

Now it’s time to do the heavy lifting in plot-glb.js. First thing we must do is construct a three.js scene. This consists of a Mesh, which in turn is described by a Material and a Geometry (a set of vertices and faces). The Material is easy to pick; the tricky part is the Geometry, which we will construct manually.

function makeScene() {
    var geometry = new THREE.Geometry(); 
    var color = new THREE.Color('tomato');

    // make a long sequence containing our 3-d points
    var nverts = n * m;
    for (var k = 0; k < nverts; ++k) {
        var newvert = new THREE.Vector3(data.x[k], data.y[k], data.z[k]);
        geometry.vertices.push(newvert);
    }


    // build triangular faces (top and bottom) between adjacent points
    for (var j = 0; j < m - 1; j++) {
        for (var i = 0; i < n - 1; i++) { 
            var n0 = j * n + i;
            var n1 = n0 + 1;
            var n2 = (j+1) * n + i + 1;
            var n3 = n2 - 1;
            face1 = new THREE.Face3(n0, n1, n2, undefined, color);
            face2 = new THREE.Face3(n2, n3, n0, undefined, color);
            geometry.faces.push(face1);
            geometry.faces.push(face2);
        }
    }

    // Compute normals for shading
    geometry.computeFaceNormals();
    geometry.computeVertexNormals();

    // Give it a pretty material
    var material = new THREE.MeshLambertMaterial( {
        side: THREE.DoubleSide,
        color: 0xffffff,
        vertexColors: THREE.FaceColors,
        emissive: 0x111111,
    });

    var scene = new THREE.Scene();
    scene.add( new THREE.Mesh( geometry, material ) );
    return scene;
}

Normally, the next step would be to add cameras and lights, maybe some axes and descriptive text, and render to a <div>. But our goal is to make a Facebook 3-d post, so we can simply export the current scene as a binary GLB file.

The GLTF exporter will give us a byte buffer that we need to download. We can handle with a function that modifies our link:

// plot-glb.js (part 2)

function saveArrayBuffer( buffer, filename ) {
    var blob = new Blob( [ buffer ], { type: 'application/octet-stream' } );
    var link = document.getElementById( 'download' );
    link.href = URL.createObjectURL( blob );
    link.download = filename;
}

Finally, we use the GLTFExporter to generate a Facebook-compatible 3-d model.

// plot-glb.js (part 3)

var scene = makeScene();

var exporter = new THREE.GLTFExporter();
var options = {
    trs: false,
    onlyVisible: false,
    truncateDrawRange: false,
    binary: true,
    forceIndices: true,  // for Facebook
    forcePowerOfTwoTextures: false  // for Facebook
};

exporter.parse( scene, function ( glb ) {
    saveArrayBuffer( glb, 'scene.glb' );
}, options );

Now, if you open plot-glb.html and click the Download link, you should get a .glb file that can be drag-and-dropped onto a new Facebook post.

But it shows up seen from above and looks ugly!¶

Indeed, there seems to be no way to specify the camera angle in a Facebook post. Three.js cameras are not part of the scene,Unlike the folks below, who are definitely part of the scene: so I think they don’t get exported in a GLTF/GLB.

But now that we know that the default camera is somewhere along the Z axis above the plot, we can simply rotate our point cloud to get to a more aesthetically pleasing angle!

# generate.py (part 2.5)

def rotate3d_x(points, theta):
    c = math.cos(theta)
    s = np.sin(theta)

    R = np.array([[1, 0, 0],
                  [0, c, -s],
                  [0, s, c]])
    return np.dot(points, R)


def rotate3d_z(points, theta):
    c = math.cos(theta)
    s = np.sin(theta)

    R = np.array([[c, -s, 0],
                  [s, c, 0],
                  [0, 0, 1]])
    return np.dot(points, R)


points = rotate3d_z(points, np.pi / 4)
points = rotate3d_x(points, np.pi / 2.5)

Above, we used rotation matrices to rotate the point cloud around.Rotating a 3-d point is equivalent to multiplying by a certain orthogonal matrix. The wikipedia article lists formulas for rotation matrices against the canonical axes $x$, $y$, $z$.

The result should look just like the 3-d post I made below. Of course, in generate.py we can do arbitrary complicated calculations in f, including numerical optimization, so this approach is quite powerful. May it bring you many likes!

Get code for this post.

Winning Arguments and Attitude Change on Reddit

2016-03-04T12:00:00+01:00

I have the honor of featuring another guest post on my blog. This time, we have with us my friend Chenhao Tan. We recently collaborated on an effort to understand how people change their minds on the /r/ChangeMyView subreddit.

Source of this Jupyter Notebook.

One of the many reasons why people are beautiful and fascinating is the way we change. I get reminded of how much I have changed everytime I see a photo of myself from high school, and this always triggers memories of how my opinions and attitudes have changed over the years. Some of the beliefs I held most strongly ended up being abandonded completely, while others have not changed much. If you’re bored already, scroll down, this blog post has code too!

Attitude change and persuasion has been an appealing research topic for a long time. One of the more interesting theories on this issue is the backfire effect:

When your deepest convictions are challenged by contradictory evidence, your beliefs get stronger.

Earlier studies, however, suggest a lot more nuance. For instance, individuals who affirm their self were found to be more likely to be persuaded by evidence. Studying opinion change in the wild can be difficult. Many times, people change their mind slowly and subtly, over a number of years, making it difficult to pinpoint the exact moment or cause of the change. Also, a change of opinion on a minor, inconsequential topic may not be worth discussing.

We contributed to these efforts by studying /r/ChangeMyView (CMV), an online forum of civil, open-minded discussion where people invite other members to challenge their more controversial views. If you’ve never visited CMV, the concept that people can debate hot topics without getting into flame wars on the internet might be hard to believe. Or, if you read a lot of Youtube comments, you might be inclined to believe the backfire effect, and doubt that graceful concession is even possible online. But a quick trip to this great subreddit will undoubtably make you a believer.

What makes `/r/ChangeMyView` special?¶

Some of the more interesting reasons why CMV is a cool lens for the study of persuasion is:

Original posters (OP) select views that are important to them and controversial, but, by posting them there, manifest a de facto openness to counterarguments.
For an OP’s opinion, there are usually multiple, high-quality persuasion attempts, but most are unsuccessful. By comparing challenges to the same person’s opinion, we avoid the trap of actually finding that some issues are easier to challenge than others.
Successful arguments are explicitly marked with a delta (Δ). Hooray for supervised learning!
Even in light of CMV users being open-minded, not all opinions are changed, and not just due to a lack of trying. This gives us a chance to analyze resistance to persuasion.

Taking a look at the dataset¶

I know what you’re thinking, a little less conversation, let’s actually look at some examples. Sure, our pleasure! Let’s download the dataset that we compiled.

Warning: If you’re following along at home, and you don’t already have it, this will download the 321M file cmv.tar.bz2. We kept the code simple, but the downside is that your Jupyter kernel will be busy until the file is downloaded. If, by any chance, your connection drops or you interrupt it, you will have to remove the partial file, otherwise the rest of the notebook might not work.

In [1]:

# load the data
import tarfile
import os.path
import json
import re
from bz2 import BZ2File
from urllib import request
from io import BytesIO

import numpy as np


fname = "cmv.tar.bz2"
url = "https://chenhaot.com/data/cmv/" + fname

# download if not exists
if not os.path.isfile(fname):
    f = BytesIO()
    with request.urlopen(url) as resp, open(fname, 'wb') as f_disk:
        data = resp.read()
        f_disk.write(data)  # save to disk too
        f.write(data)
        f.seek(0)
else:
    f = open(fname, 'rb')


tar = tarfile.open(fileobj=f, mode="r")

# Extract the file we are interested in

train_fname = "op_task/train_op_data.jsonlist.bz2"
test_fname = "op_task/heldout_op_data.jsonlist.bz2"

train_bzlist = tar.extractfile(train_fname)

# Deserialize the JSON list
original_posts_train = [
    json.loads(line.decode('utf-8'))
    for line in BZ2File(train_bzlist)
]

test_bzlist = tar.extractfile(test_fname)

original_posts_test = [
    json.loads(line.decode('utf-8'))
    for line in BZ2File(test_bzlist)
]
f.close()

The archive we just downloaded contains all the data we used in our experiments. We invite you to explore the other files in there: you can use the pair_task folder to study pairs of similar argumentative threads made in reply to the same original post, one successful and one not (more details in Section 4 of the paper). We also provide you with the full, non-processed data in the all subfolder, so that you can design new interesting tasks.

In this blog post, we are going to focus on the original posts, rather than the arguments made in response to them. In the final part of our paper, we explored resistance to persuasion: recognizing whether an OP is likely to be persuaded or not, given how they express their view. We insist on this because we think it’s interesting and we know there’s plenty more work to be done on this task: we merely scratched the surface! So, hopefully, this quick tour will make you want to work on this.

In [2]:

from IPython.display import Markdown


def show_post(cmv_post):
    md_format = "**{title}** \n\n {selftext}".format(**cmv_post)
    md_format = "\n".join(["> " + line for line in md_format.splitlines()])
    return Markdown(md_format)

In [3]:

show_post(original_posts_train[200])

Out[3]:

CMV: Taxation should be voluntary, with individuals retaining the right to opt out of it.

Let me preface this argument by saying that I don’t particularly agree with the Tea Party. I’m a Democrat who votes for candidates that support ending the poverty trap, expanding social services, and wish to improve fundamental institutions like education or military service. With that said, I also believe in the concept of personal liberty. This includes the right to refuse taxation upon the basis of… well, whatever the fuck you want.

I know what your thinking, “But how could we fund things like food stamps if shitty people are like ‘fuck those lazy blacks that suck up muh moneyz’?” However, in order to participate in the American system of democracy, one must agree to pay taxes unconditionally. A citizen ought not be able to decide which services he or she pays for. Instead, one can only receive protection from, say, the police if they pay every tax mandated by the local, state, or federal government. In my opinion, this would create a voluntary political system where individual rights are maintained. If we believe that the government overwhelmingly improves our own conditions, as I do, then only a small amount of citizens would opt out of public services because they disagree with certain elements of policy. Within the framework of government existing because it benefits the people, no one should expect a sudden drop in tax revenues because citizens are intelligent enough to understand how their own interests are promoted by the state on balance.

So go forth and CMV, noble redditors!

> Hello, users of CMV! This is a footnote from your moderators. We’d just like to remind you of a couple of things. Firstly, please remember to read through our rules. If you see a comment that has broken one, it is more effective to report it than downvote it. Speaking of which, downvotes don’t change views****! If you are thinking about submitting a CMV yourself, please have a look through our* ***popular topics wiki first. Any questions or concerns? Feel free to message us. Happy CMVing!

What do you think, has this OP’s view been changed?

… drumroll …

In [4]:

original_posts_train[200]["delta_label"]

Out[4]:

False

This poster has not been successfully persuaded! And just in case you have any doubts that no attempts were made or something, (we checked, trust us), we could go take a look at all comments and discussion by following the post_id to the original post on Reddit:

In [5]:

_, post_id = original_posts_train[200]['name'].split("_")
print("http://reddit.com/r/changemyview/comments/{}".format(post_id))

http://reddit.com/r/changemyview/comments/2q03zk

Just by reading the post, it’s obvious that some cleanup needs to be done: we should strip the moderator-added footnotes, edits and quotes. We will do this by removing lines starting with “>”, lines starting with “____”, and lines where the string “edit” occurs in the first 2 “words”. (In the code used for our research, we used a more careful heuristic that involves actual tokenization, but we will keep it simple here and split at white spaces.)

In [6]:

def cleanup(cmv_post):
    lines = [line for line in cmv_post.splitlines()
             if not line.lstrip().startswith("&gt;")
             and not line.lstrip().startswith("____")
             and "edit" not in " ".join(line.lower().split()[:2])
            ]
    return "\n".join(lines)

In [7]:

show_post(dict(title=original_posts_train[200]['title'],
               selftext=cleanup(original_posts_train[200]['selftext'])))

Out[7]:

CMV: Taxation should be voluntary, with individuals retaining the right to opt out of it.

Let me preface this argument by saying that I don’t particularly agree with the Tea Party. I’m a Democrat who votes for candidates that support ending the poverty trap, expanding social services, and wish to improve fundamental institutions like education or military service. With that said, I also believe in the concept of personal liberty. This includes the right to refuse taxation upon the basis of… well, whatever the fuck you want.

I know what your thinking, “But how could we fund things like food stamps if shitty people are like ‘fuck those lazy blacks that suck up muh moneyz’?” However, in order to participate in the American system of democracy, one must agree to pay taxes unconditionally. A citizen ought not be able to decide which services he or she pays for. Instead, one can only receive protection from, say, the police if they pay every tax mandated by the local, state, or federal government. In my opinion, this would create a voluntary political system where individual rights are maintained. If we believe that the government overwhelmingly improves our own conditions, as I do, then only a small amount of citizens would opt out of public services because they disagree with certain elements of policy. Within the framework of government existing because it benefits the people, no one should expect a sudden drop in tax revenues because citizens are intelligent enough to understand how their own interests are promoted by the state on balance.

So go forth and CMV, noble redditors!

Now that the cleanup is done, we can think a bit about what, and how, the OP is trying to express here. We can see several strategies at play:

Prefacing with a cop-out attempting to make their opinion seem more reasonable.
Liberal (pun intended) use of casual language, slang, curse words.
Lots of self-asserting: “I know, “I believe”, “I’m a Democrat”, etc.

There is plenty of room for speculation, exploration, finding patterns, and connecting this to existing literature. Let’s focus on one of our results:

Reproducing some results¶

One of our findings was that OPs who express their view using “we” are more stubborn than the ones who say “I”. The example above uses some plural pronouns (“If we believe”) but overall it seems the opinion is purely self-attributed. Does this pattern hold in the dataset?

In [8]:

pron_i = ['ive',
 "i'd",
 "i'm",
 'myself',
 'mine',
 'me',
 'im',
 "i'll",
 'i',
 "i've",
 'id',
 'my']

In [9]:

pron_we = ["we'd",
 "we're",
 'lets',
 'weve',
 'we',
 'our',
 'ourselves',
 "let's",
 'ours',
 'us',
 "we'll",
 "we've"]

In [10]:

pron_we_re = re.compile("\\b({})\\b".format("|".join(pron_we)), re.IGNORECASE)
pron_i_re = re.compile("\\b({})\\b".format("|".join(pron_i)), re.IGNORECASE)

In [11]:

pron_we_re.findall("We let ourselves go")

Out[11]:

['We', 'ourselves']

In [12]:

n_we = []
n_i = []
for post in original_posts_train:
    content = post["selftext"].lower()
    content = cleanup(content)
    denom = len(content.split())
    if denom:
        n_we.append(len(pron_we_re.findall(content)) / denom)
        n_i.append(len(pron_i_re.findall(content)) / denom)
    else:
        # there is one instance that consists only of edits.
        # It's possible that the OP deleted the rest of the post.
        n_we.append(0)
        n_i.append(0)

In [13]:

n_we = np.array(n_we)
n_i = np.array(n_i)

is_malleable = np.array([post["delta_label"] for post in original_posts_train])

In [14]:

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

/home/vlad/conda/lib/python3.4/site-packages/IPython/html.py:14: ShimWarning: The `IPython.html` package has been deprecated. You should import from `notebook` instead. `IPython.html.widgets` has moved to `ipywidgets`.
  "`IPython.html.widgets` has moved to `ipywidgets`.", ShimWarning)

In [15]:

_, green, red, _, _, _ = sns.color_palette()

In [16]:

from scipy.stats import sem

In [17]:

def compare_bars(feature, y):
    pos = feature[y]
    neg = feature[~y]
    plt.bar(
        [1, 2],
        [pos.mean(), neg.mean()],
        yerr=[1.96 * sem(pos), 1.96 * sem(neg)],  # ~95% ci
        align='center',
        color=[green, red],
        alpha=0.8,
        lw=3,
        error_kw=dict(ecolor='k', lw=2, capsize=5, capthick=3)
    )
    plt.xticks((1, 2), ["Malleable", "Stubborn"], fontsize=16)
    plt.yticks(fontsize=16)

plt.figure(figsize=(7, 4))
plt.subplot(121)
plt.title("First person plural\n('we')", fontsize=20)
compare_bars(n_we, is_malleable)
plt.ylabel("Proportion of words", fontsize=20)

plt.subplot(122)
plt.title("First person singular\n('I')", fontsize=20)
compare_bars(n_i, is_malleable)

plt.tight_layout()

Training a simple classifier¶

We can take this one step further and train a simple bag-of-words classifier using scikit-learn. Bag-of-words is often a good baseline, and a good debugging tool: sometimes, problems are simpler than we think, and bag-of-words can catch this and hint that we’re being too optimistic. (For an example of what I mean, try running the code below without the “cleanup” step.)

In [18]:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics.scorer import roc_auc_scorer

In [19]:

# prepare the data in Python list form

op_text_train = [
    cleanup(original_post['selftext'])
    # original_post['selftext']  # try this at home
    for original_post
    in original_posts_train
]

op_text_test = [
    cleanup(original_post['selftext'])
    # original_post['selftext']  # try this at home
    for original_post
    in original_posts_test
]

We will use a simple term-frequency vectorizer with no IDF and no data-dependent thresholds. This allows us to vectorize all of the training data once, and then do efficient model selection using LogisticRegressionCV, without having to vectorize again for each cross-validation fold.

Side note: *This is important! Imagine we wanted to throw away rare words, to avoid overfitting to them. We could use, for example, min_df=10. This would keep all words that appear 10 times in the *training data. But, in a moment, we will be doing 3-fold cross-validation. Consider the first cross-validation split: what if we encounter a word that occurs 5 times in the train fold and 5 times in the validation fold? If we vectorized outside of the CV loop, we would keep this word, and assign it a weight estimated from very few observations. This could lead us to overestimate our validation performance.

In [20]:

vect = TfidfVectorizer(use_idf=False, norm='l1')
X_train = vect.fit_transform(op_text_train)

In [21]:

lr = LogisticRegressionCV(Cs=10, class_weight='balanced',
                          scoring='roc_auc', solver='sag',
                          tol=0.001, max_iter=500,
                          random_state=0)

In [22]:

lr.fit(X_train, is_malleable)

Out[22]:

LogisticRegressionCV(Cs=10, class_weight='balanced', cv=None, dual=False,
           fit_intercept=True, intercept_scaling=1.0, max_iter=500,
           multi_class='ovr', n_jobs=1, penalty='l2', random_state=0,
           refit=True, scoring='roc_auc', solver='sag', tol=0.001,
           verbose=0)

In [23]:

# Evaluating on the heldout data:

X_test = vect.transform(op_text_test)
is_malleable_test = np.array([post["delta_label"] for post in original_posts_test])
test_roc = roc_auc_scorer(lr, X_test, is_malleable_test)
print("Test ROC AUC score: {:.3f}".format(test_roc))

Test ROC AUC score: 0.531

The score is in the same ballpark as the results in our paper, with the differences coming from the more naive tokenization and preprocessing used here. The ROC AUC score (area under the ROC curve) can be interpreted as the probability that, given a pair of a malleable and a stubborn OP, the classifier will successfully tell which is which. This metric is particularly good for imbalanced datasets, and 50% corresponds to an uninformed classifier (one that responds randomly or always gives the same answer). In light of this, the score is not great, confirming that this is not at all an easy problem, still open for further research.

We point out one particular challenge: our heldout data is temporally newer than the training data, because we collected it strictly after developing our work on the training set. This has the advantage of making overfitting impossible, but extrapolating is different than interpolating in time, and some attributes of the CMV community shift over time. (We discuss some of them in Section 2 of our paper.) You may model malleability better by accounting for shifts both in covariates and the outcome variable.

Checking out the features¶

We can use the trained model to see what words have high and low weights. You could use the output for further inspiration. For instance, this could suggest that generalizing words (“anything”, “anyone”, “nothing”) might be a sign of stubbornness. There is plenty of potential for exploring and expanding our work further, so give it a try!

In [24]:

def top_features(coef, words, n=15):
    order = np.argsort(-coef)
    for i_hi, i_lo in zip(order[:n], order[:-n - 1:-1]):
        row_vals = coef[i_hi], words[i_hi], coef[i_lo], words[i_lo]
        print("[{:>4.1f}] {:<20}[{:>4.1f}] {:<20}".format(*row_vals))

In [25]:

top_features(lr.coef_.flatten(), vect.get_feature_names())

[ 8.9] so                  [-9.1] its                 
[ 7.2] even                [-5.5] having              
[ 5.6] feel                [-5.3] anyone              
[ 5.5] would               [-5.2] government          
[ 5.5] being               [-5.2] right               
[ 5.4] more                [-5.2] anything            
[ 5.3] please              [-5.2] our                 
[ 5.3] hard                [-5.0] certain             
[ 5.3] doesn               [-5.0] world               
[ 5.2] ll                  [-5.0] nothing             
[ 5.1] when                [-5.0] minimum             
[ 5.0] system              [-4.7] we                  
[ 4.7] philosophy          [-4.7] men                 
[ 4.5] subject             [-4.6] life                
[ 4.4] help                [-4.5] population

Much of the social interaction we do online these days can prove useful for research. In our work, we investigated persuasion and attitude change on /r/ChangeMyView. In this post, we walked through part of the data that we collected and made available. Hoping that this sparks some inspiration, we are looking forward to hearing what cool new things you will discover!

In [26]:

%load_ext watermark

In [27]:

%watermark -d -v --packages numpy,scipy,scikit-learn,seaborn -m

2016-03-04 

CPython 3.4.4
IPython 4.1.1

numpy 1.10.4
scipy 0.17.0
scikit-learn 0.17.1
seaborn 0.7.0

compiler   : GCC 4.4.7 20120313 (Red Hat 4.4.7-1)
system     : Linux
release    : 4.4.0-040400rc4-generic
machine    : x86_64
processor  : x86_64
CPU cores  : 4
interpreter: 64bit

Word Mover’s Distance in Python

2015-11-07T12:00:00+01:00

*A guide to scikit-learn compatible nearest neighbors classification using the recently introduced word mover’s distance (WMD). * Joint post with the awesome Matt Kusner!

Source of this Jupyter notebook.

In document classification and other natural language processing applications, having a good measure of the similarity of two texts can be a valuable building block. Ideally, such a measure would capture semantic information. Cosine similarity on bag-of-words vectors is known to do well in practice, but it inherently cannot capture when documents say the same thing in completely different words.

Take, for example, two headlines:

Obama speaks to the media in Illinois
The President greets the press in Chicago

These have no content words in common, so according to most bag of words—based metrics, their distance would be maximal. (For such applications, you probably don’t want to count stopwords such as the and in, which don’t truly signal semantic similarity.)

One way out of this conundrum is the word mover’s distance (WMD), introduced in From Word Embeddings To Document Distances, (Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, Kilian Q. Weinberger, ICML 2015). WMD adapts the earth mover’s distance to the space of documents: the distance between two texts is given by the total amount of “mass” needed to move the words from one side into the other, multiplied by the distance the words need to move. So, starting from a measure of the distance between different words, we can get a principled document-level distance. Here is a visualisation of the idea, from the ICML slides:

Prepare some word embeddings¶

The key ingredient in WMD is a good distance measure between words. Dense representations of words, also known by the trendier name “word embeddings” (because “distributed word representations” didn’t stick), do the trick here. We could train the embeddings ourselves, but for meaningful results we would need tons of documents, and that might take a while. So let’s just use the ones from the word2vec team. (download link)

In [1]:

import os

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.cross_validation import train_test_split

In [2]:

if not os.path.exists("data/embed.dat"):
    print("Caching word embeddings in memmapped format...")
    from gensim.models.word2vec import Word2Vec
    wv = Word2Vec.load_word2vec_format(
        "data/GoogleNews-vectors-negative300.bin.gz",
        binary=True)
    fp = np.memmap("data/embed.dat", dtype=np.double, mode='w+', shape=wv.syn0norm.shape)
    fp[:] = wv.syn0norm[:]
    with open("data/embed.vocab", "w") as f:
        for _, w in sorted((voc.index, word) for word, voc in wv.vocab.items()):
            print(w, file=f)
    del fp, wv

W = np.memmap("data/embed.dat", dtype=np.double, mode="r", shape=(3000000, 300))
with open("data/embed.vocab") as f:
    vocab_list = map(str.strip, f.readlines())

In [3]:

vocab_dict = {w: k for k, w in enumerate(vocab_list)}

Reproducing the demo above¶

In [33]:

d1 = "Obama speaks to the media in Illinois"
d2 = "The President addresses the press in Chicago"

vect = CountVectorizer(stop_words="english").fit([d1, d2])
print("Features:",  ", ".join(vect.get_feature_names()))

Features: addresses, chicago, illinois, media, obama, president, press, speaks

The two documents are completely orthogonal in terms of bag-of-words

In [71]:

from scipy.spatial.distance import cosine
v_1, v_2 = vect.transform([d1, d2])
v_1 = v_1.toarray().ravel()
v_2 = v_2.toarray().ravel()
print(v_1, v_2)
print("cosine(doc_1, doc_2) = {:.2f}".format(cosine(v_1, v_2)))

[0 0 1 1 1 0 0 1] [1 1 0 0 0 1 1 0]
cosine(doc_1, doc_2) = 1.00

In [58]:

from sklearn.metrics import euclidean_distances
W_ = W[[vocab_dict[w] for w in vect.get_feature_names()]]
D_ = euclidean_distances(W_)
print("d(addresses, speaks) = {:.2f}".format(D_[0, 7]))
print("d(addresses, chicago) = {:.2f}".format(D_[0, 1]))

d(addresses, speaks) = 1.16
d(addresses, chicago) = 1.37

We will be using pyemd, a Python wrapper for Pele and Werman’s implementation of the earth mover’s distance.

In [74]:

from pyemd import emd

# pyemd needs double precision input
v_1 = v_1.astype(np.double)
v_2 = v_2.astype(np.double)
v_1 /= v_1.sum()
v_2 /= v_2.sum()
D_ = D_.astype(np.double)
D_ /= D_.max()  # just for comparison purposes
print("d(doc_1, doc_2) = {:.2f}".format(emd(v_1, v_2, D_)))

d(doc_1, doc_2) = 0.74

Document classification¶

We will use the 20 Newsgroups classification task. Because WMD is an expensive computation, for this demo we just use a subset. To emphasize the power of the method, we use a larger test size, but train on relatively few samples.

In [4]:

newsgroups = fetch_20newsgroups()
docs, y = newsgroups.data, newsgroups.target

In [5]:

docs_train, docs_test, y_train, y_test = train_test_split(docs, y,
                                                          train_size=100,
                                                          test_size=300,
                                                          random_state=0)

Since the W embedding array is pretty huge, we might as well restrict it to just the words that actually occur in the dataset.

In [6]:

vect = CountVectorizer(stop_words="english").fit(docs_train + docs_test)
common = [word for word in vect.get_feature_names() if word in vocab_dict]
W_common = W[[vocab_dict[w] for w in common]]

We can then create a fixed-vocabulary vectorizer using only the words we have embeddings for.

In [7]:

vect = CountVectorizer(vocabulary=common, dtype=np.double)
X_train = vect.fit_transform(docs_train)
X_test = vect.transform(docs_test)

One way to proceed is to just pre-compute the pairwise distances between all documents, and use them to search for hyperparameters and evaluate the model. However, that would incur some extra computation, and WMD is expensive. Also, it’s not the most pleasant user interface. So we define some scikit-learn compatible estimators for computing the WMD.

WordMoversKNN subclasses from KNeighborsClassifier and overrides the predict function to compute the WMD between all training and test samples.

In practice, however, we often don’t know what is the best n_neighbors to use. Simply wrapping WordMoversKNN in a GridSearchCV would be rather expensive because of all the distances that would need to be recomputed for every value of n_neighbors. So we introduce WordMoversKNNCV, which, when fitted, performs cross-validation to find the best value of n_neighbors (under any given evaluation metric), while only computing the WMD once per fold, and only across folds (saving n_folds * fold_size ** 2 evaluations).

In [16]:

"""%%file word_movers_knn.py"""

# Authors: Vlad Niculae, Matt Kusner
# License: Simplified BSD

import numpy as np
from sklearn.metrics import euclidean_distances
from sklearn.externals.joblib import Parallel, delayed
from sklearn.neighbors import KNeighborsClassifier
from sklearn.utils import check_array
from sklearn.cross_validation import check_cv
from sklearn.metrics.scorer import check_scoring
from sklearn.preprocessing import normalize

from pyemd import emd


class WordMoversKNN(KNeighborsClassifier):
    """K nearest neighbors classifier using the Word Mover's Distance.

    Parameters
    ----------
    
    W_embed : array, shape: (vocab_size, embed_size)
        Precomputed word embeddings between vocabulary items.
        Row indices should correspond to the columns in the bag-of-words input.

    n_neighbors : int, optional (default = 5)
        Number of neighbors to use by default for :meth:`k_neighbors` queries.

    n_jobs : int, optional (default = 1)
        The number of parallel jobs to run for Word Mover's Distance computation.
        If ``-1``, then the number of jobs is set to the number of CPU cores.
    
    verbose : int, optional
        Controls the verbosity; the higher, the more messages. Defaults to 0.
        
    References
    ----------
    
    Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, Kilian Q. Weinberger
    From Word Embeddings To Document Distances
    The International Conference on Machine Learning (ICML), 2015
    http://mkusner.github.io/publications/WMD.pdf
    
    """
    _pairwise = False

    def __init__(self, W_embed, n_neighbors=1, n_jobs=1, verbose=False):
        self.W_embed = W_embed
        self.verbose = verbose
        super(WordMoversKNN, self).__init__(n_neighbors=n_neighbors, n_jobs=n_jobs,
                                            metric='precomputed', algorithm='brute')

    def _wmd(self, i, row, X_train):
        """Compute the WMD between training sample i and given test row.
        
        Assumes that `row` and train samples are sparse BOW vectors summing to 1.
        """
        union_idx = np.union1d(X_train[i].indices, row.indices)
        W_minimal = self.W_embed[union_idx]
        W_dist = euclidean_distances(W_minimal)
        bow_i = X_train[i, union_idx].A.ravel()
        bow_j = row[:, union_idx].A.ravel()
        return emd(bow_i, bow_j, W_dist)
    
    def _wmd_row(self, row, X_train):
        """Wrapper to compute the WMD of a row with all training samples.
        
        Assumes that `row` and train samples are sparse BOW vectors summing to 1.
        Useful for parallelization.
        """
        n_samples_train = X_train.shape[0]
        return [self._wmd(i, row, X_train) for i in range(n_samples_train)]

    def _pairwise_wmd(self, X_test, X_train=None):
        """Computes the word mover's distance between all train and test points.
        
        Parallelized over rows of X_test.
        
        Assumes that train and test samples are sparse BOW vectors summing to 1.
        
        Parameters
        ----------
        X_test: scipy.sparse matrix, shape: (n_test_samples, vocab_size)
            Test samples.
        
        X_train: scipy.sparse matrix, shape: (n_train_samples, vocab_size)
            Training samples. If `None`, uses the samples the estimator was fit with.
        
        Returns
        -------
        dist : array, shape: (n_test_samples, n_train_samples)
            Distances between all test samples and all train samples.
        
        """
        n_samples_test = X_test.shape[0]
        
        if X_train is None:
            X_train = self._fit_X

        dist = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(
            delayed(self._wmd_row)(test_sample, X_train)
            for test_sample in X_test)

        return np.array(dist)

    def fit(self, X, y):
        """Fit the model using X as training data and y as target values

        Parameters
        ----------
        X : scipy sparse matrix, shape: (n_samples, n_features)
            Training data. 

        y : {array-like, sparse matrix}
            Target values of shape = [n_samples] or [n_samples, n_outputs]

        """
        X = check_array(X, accept_sparse='csr', copy=True)
        X = normalize(X, norm='l1', copy=False)
        return super(WordMoversKNN, self).fit(X, y)

    def predict(self, X):
        """Predict the class labels for the provided data
        Parameters
        ----------
        X : scipy.sparse matrix, shape (n_test_samples, vocab_size)
            Test samples.

        Returns
        -------
        y : array of shape [n_samples]
            Class labels for each data sample.
        """
        X = check_array(X, accept_sparse='csr', copy=True)
        X = normalize(X, norm='l1', copy=False)
        dist = self._pairwise_wmd(X)
        return super(WordMoversKNN, self).predict(dist)
    
    
class WordMoversKNNCV(WordMoversKNN):
    """Cross-validated KNN classifier using the Word Mover's Distance.

    Parameters
    ----------
    W_embed : array, shape: (vocab_size, embed_size)
        Precomputed word embeddings between vocabulary items.
        Row indices should correspond to the columns in the bag-of-words input.

    n_neighbors_try : sequence, optional
        List of ``n_neighbors`` values to try.
        If None, tries 1-5 neighbors.

    scoring : string, callable or None, optional, default: None
        A string (see model evaluation documentation) or
        a scorer callable object / function with signature
        ``scorer(estimator, X, y)``.

    cv : int, cross-validation generator or an iterable, optional
        Determines the cross-validation splitting strategy.
        Possible inputs for cv are:
          - None, to use the default 3-fold cross-validation,
          - integer, to specify the number of folds.
          - An object to be used as a cross-validation generator.
          - An iterable yielding train/test splits.
        For integer/None inputs, StratifiedKFold is used.

    n_jobs : int, optional (default = 1)
        The number of parallel jobs to run for Word Mover's Distance computation.
        If ``-1``, then the number of jobs is set to the number of CPU cores.

    verbose : int, optional
        Controls the verbosity; the higher, the more messages. Defaults to 0.

    Attributes
    ----------
    cv_scores_ : array, shape (n_folds, len(n_neighbors_try))
        Test set scores for each fold.

    n_neighbors_ : int,
        The best `n_neighbors` value found.

    References
    ----------

    Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, Kilian Q. Weinberger
    From Word Embeddings To Document Distances
    The International Conference on Machine Learning (ICML), 2015
    http://mkusner.github.io/publications/WMD.pdf
    
    """
    def __init__(self, W_embed, n_neighbors_try=None, scoring=None, cv=3,
                 n_jobs=1, verbose=False):
        self.cv = cv
        self.n_neighbors_try = n_neighbors_try
        self.scoring = scoring
        super(WordMoversKNNCV, self).__init__(W_embed,
                                              n_neighbors=None,
                                              n_jobs=n_jobs,
                                              verbose=verbose)

    def fit(self, X, y):
        """Fit KNN model by choosing the best `n_neighbors`.
        
        Parameters
        -----------
        X : scipy.sparse matrix, (n_samples, vocab_size)
            Data
        y : ndarray, shape (n_samples,) or (n_samples, n_targets)
            Target
        """
        if self.n_neighbors_try is None:
            n_neighbors_try = range(1, 6)
        else:
            n_neighbors_try = self.n_neighbors_try

        X = check_array(X, accept_sparse='csr', copy=True)
        X = normalize(X, norm='l1', copy=False)

        cv = check_cv(self.cv, X, y)
        knn = KNeighborsClassifier(metric='precomputed', algorithm='brute')
        scorer = check_scoring(knn, scoring=self.scoring)

        scores = []
        for train_ix, test_ix in cv:
            dist = self._pairwise_wmd(X[test_ix], X[train_ix])
            knn.fit(X[train_ix], y[train_ix])
            scores.append([
                scorer(knn.set_params(n_neighbors=k), dist, y[test_ix])
                for k in n_neighbors_try
            ])
        scores = np.array(scores)
        self.cv_scores_ = scores

        best_k_ix = np.argmax(np.mean(scores, axis=0))
        best_k = n_neighbors_try[best_k_ix]
        self.n_neighbors = self.n_neighbors_ = best_k

        return super(WordMoversKNNCV, self).fit(X, y)

Overwriting word_movers_knn.py

In [10]:

knn_cv = WordMoversKNNCV(cv=3,
                         n_neighbors_try=range(1, 20),
                         W_embed=W_common, verbose=5, n_jobs=3)
knn_cv.fit(X_train, y_train)

[Parallel(n_jobs=3)]: Done  12 tasks      | elapsed:   30.8s
[Parallel(n_jobs=3)]: Done  34 out of  34 | elapsed:  2.0min finished
[Parallel(n_jobs=3)]: Done  12 tasks      | elapsed:   25.7s
[Parallel(n_jobs=3)]: Done  33 out of  33 | elapsed:  2.9min finished
[Parallel(n_jobs=3)]: Done  12 tasks      | elapsed:   53.3s
[Parallel(n_jobs=3)]: Done  33 out of  33 | elapsed:  2.0min finished

Out[10]:

WordMoversKNNCV(W_embed=memmap([[ 0.04283, -0.01124, ..., -0.05679, -0.00763],
       [ 0.02884, -0.05923, ..., -0.04744,  0.06698],
       ...,
       [ 0.08428, -0.15534, ..., -0.01413,  0.04561],
       [-0.02052,  0.08666, ...,  0.03659,  0.10445]]),
        cv=3, n_jobs=3, n_neighbors_try=range(1, 20), scoring=None,
        verbose=5)

In [11]:

print("CV score: {:.2f}".format(knn_cv.cv_scores_.mean(axis=0).max()))

CV score: 0.38

In [12]:

print("Test score: {:.2f}".format(knn_cv.score(X_test, y_test)))

[Parallel(n_jobs=3)]: Done  12 tasks      | elapsed:   32.2s
[Parallel(n_jobs=3)]: Done  66 tasks      | elapsed:  4.3min
[Parallel(n_jobs=3)]: Done 156 tasks      | elapsed: 12.5min
[Parallel(n_jobs=3)]: Done 282 tasks      | elapsed: 30.5min
[Parallel(n_jobs=3)]: Done 300 out of 300 | elapsed: 48.9min finished

Test score: 0.31

Comparison with other models¶

Now let’s see how WMD compares with some common approaches, on bag of words features. The most apples-to-apples comparison would be K nearest neighbors with a cosine similarity metric. This approach performs worse than using WMD. (All scores are accuracies.)

In [13]:

from sklearn.svm import LinearSVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.grid_search import GridSearchCV

In [21]:

knn_grid = GridSearchCV(KNeighborsClassifier(metric='cosine', algorithm='brute'),
                        dict(n_neighbors=list(range(1, 20))),
                        cv=3)
knn_grid.fit(X_train, y_train)
print("CV score: {:.2f}".format(knn_grid.best_score_))
print("Test score: {:.2f}".format(knn_grid.score(X_test, y_test)))

CV score: 0.34
Test score: 0.22

Another common method for text classification is the linear support vector machine on bag of words. This performs a bit better than vanilla cosine KNN, but worse than using WMD in this setting. In our experience, this seems to depend on the amount of training data available.

In [17]:

svc_grid = GridSearchCV(LinearSVC(),
                        dict(C=np.logspace(-6, 6, 13, base=2)),
                        cv=3)
svc_grid.fit(X_train, y_train)
print("CV score: {:.2f}".format(svc_grid.best_score_))
print("Test score: {:.2f}".format(svc_grid.score(X_test, y_test)))

CV score: 0.35
Test score: 0.27

What have we learned?¶

WMD is much better at capturing semantic similarity between documents than cosine, due to its ability to generalize to unseen words. The SVM does somewhat better than cosine KNN, but still lacks such out-of-vocabulary generalization. Given enough data, WMD can probably improve this margin, especially using something like metric learning on top.

The exact WMD, as we have used it here, is pretty slow. This code is not optimized as much as it could be, there is potential through caching and using Cython. However, a major limitation remains the cost of actually computing the EMD. To scale even higher, exactness can be relaxed by using lower bounds. In our next post, we will compare such optimization strategies, as discussed in the WMD paper.

Flask-SocketIO on OpenShift: fallback on another port

2015-07-06T00:00:00+02:00

TL; DR¶

I hacked the SocketIO client 0.9.16 to support specifying a special port (wsport) to use only for the WebSocket protocol, while keeping all other traffic on the default port. This is required by setups such as OpenShift which require WebSocket traffic to come over a different port (say 8000 rather than 80).

The current state of affairs¶

I’ve been trying to host an interactive web app with WebSockets on OpenShift. Since I’m a poor student and this is a research app, I wanted a reasonably powerful free hosting option.

OpenShift added WebSockets support over 2 years ago. But because of some internal limitations, the WebSocket traffic needs to go over different ports than usual (8000 for unsecured and 8443 for secured connections).

SocketIO is a cool library that allows event-driven bidirectional traffic in web apps. It tries to use WebSockets if available, and falls back to other transport protocols (such as XHR Long Polling) otherwise.

Oh and to make matters worse, I’m using Flask for this web app, and currently the best way to use SocketIO from Flask is with the Flask-SocketIO + gevent-socketio combo, which is, alas, incompatible with recent versions of SocketIO. According to the community, the most stable release is SocketIO 0.9.16.

The problem in a nutshell¶

When debugging my app locally, I’d connect with something like

var socket = io.connect(
    'http://' + document.domain + ':' + location.port + '/chat'
);

When deploying on OpenShift, the location.port is now different from the WebSocket port, so I used a Flask config that determines server-side where it’s running from:

var wsport = '{{ config['WEBSOCKET_PORT'] }}';

The app now runs great both when debugging locally or on OpenShift, except if for some reason the WebSocket connection fails (either if the browser is old or because of strict firewalls). This happens because long polling uses normal HTTP requests and should be done over the default port, rather than the OpenShift-specific one.

Making things work¶

My first thought was to work around it by manually implementing the fallback client-side. This, surprisingly, failed: it turns out that SocketIO 0.9.16 first attempts a handshake to find out what transports the server supports. If the initial connection is on port 8000, then the handshake will be attempted also on port 8000, so the connect_failed event won’t even be triggered. The second cleanest solution I could think of was to add another parameter to the SocketIO options, wsport, allowing the user to specify a different port over which to do WebSocket connections, while keeping all other traffic over the default port. This works like a charm!

My updated SocketIO client is available as a gist, and the client-side connection code looks like this:

var socket = io.connect(
    'http://' + document.domain + '/chat',
    {
        port: location.port,
        wsport: wsport || location.port,
        "connect timeout": 5000
    }
);

I had to reduce the connection timeout, for a better user experience. I’m way out of my comfort zone here, so in case I missed a better solution, do let me know!

Kemeny-Young Optimal Rank Aggregation in Python

2014-01-22T18:00:00+01:00

Rank aggregation is a problem with many important applications and naive approaches to it go wrong in subtle ways.

Let’s say that your national Quidditch league is dominated by five major wizard sports newspapers. Yes, the ones with moving images and everything. Every week after the games, each of them publishes a ranking of the star players. For now, let’s suppose that the set of players under investigation is always the same, as the problem becomes a bit more complicated otherwise.

The Athletic Wizard: Alicia Spinnet, Ginny Weasley, Gwendolyn Morgan, Robin Higgy, Debbie Muntz
The Daily Prophet: Alicia, Ginny, Robin, Gwendolyn, Debbie
Quidditch News: Robin, Ginny, Gwendolyn, Debbie, Alicia
Seeker Weekly: Gwendolyn, Ginny, Robin, Debbie, Alicia
The Quibbler: Debbie, Ginny, Robin, Gwendolyn, Alicia

As you can see, there’s quite a bit of disagreement and personal taste involved. You didn’t get to watch all of the games, but you’d like to make a decision on who the best players were, by somehow aggregating the opinions of the popular newspapers. An easy option would be to pretend that each newspaper votes for the player that they rank #1, and ignore the rest. As Alicia Spinnet is the only player getting two nominations for best player, she should win best player, right?

Upon closer inspection, Alicia seems very controversial, loved by two but hated by five of the newspaper. Ginny, on the other hand, didn’t stand out as best player to anybody, but she was uniformly considered runner-up. There should be some way to account for this. It would be nice if we would have a method of finding an optimal ranking that maximizes some sort of agreement with the opinions we are trying to aggregate.

Kendall’s Tau distance¶

One of the most interesting ways to measure disagreement between rankings is the Tau statistic introduced by Kendall. It essentially measures the number of pairwise disagreements between two rankings. Since you can think of it as the number of flips you need to perform on a ranking to turn it into the other, it is sometimes called bubble-sort distance.

While the closely-related Tau correlation coefficient is implemented in Scipy as scipy.stats.kendalltau, let’s code it ourselves in a simpler way.

In [1]:

from __future__ import print_function
import numpy as np
from itertools import combinations, permutations

def kendalltau_dist(rank_a, rank_b):
    tau = 0
    n_candidates = len(rank_a)
    for i, j in combinations(range(n_candidates), 2):
        tau += (np.sign(rank_a[i] - rank_a[j]) ==
                -np.sign(rank_b[i] - rank_b[j]))
    return tau

Now we can represent a rank as a numpy vector, with missing values.

In [2]:

# columns in order of appearance:
cols = "Alicia Ginny Gwendolyn Robin Debbie".split()
ranks = np.array([[0, 1, 2, 3, 4],
                  [0, 1, 3, 2, 4],
                  [4, 1, 2, 0, 3],
                  [4, 1, 0, 2, 3],
                  [4, 1, 3, 2, 0]])

Two rankings that agree completely should have a distance of 0.

In [3]:

kendalltau_dist(ranks[0], ranks[0])

Out[3]:

The Athletic Wizard seems to be closer to the Daily Prophet than to Seeker Weekly.

In [4]:

print('d(athletic_wizard, prophet) = {}'.format(kendalltau_dist(ranks[0], ranks[1])))
print('d(athletic_wizard, seeker) = {}'.format(kendalltau_dist(ranks[0], ranks[3])))

d(athletic_wizard, prophet) = 1
d(athletic_wizard, seeker) = 5

Kemeny-Young rank aggregation¶

Now that we have a distance metric, we can formulate a loss function to minimize in rank-space. We are looking for a ranking $\hat\tau$ that satisfies $$ \sum_i d(\hat\tau, \tau_i) \leq \sum_i d(\tau, \tau_i) \text{ for all } \tau $$

This aggregation method was proposed by John Kemeny [1], and later shown by Peyton Young to be a maximum likelihood estimator of pairwise preferences under the assumption that a voter will randomly flip two candidates from the true ranking is the same $p < 0.5$ [2].

For rankings of small length, one way to compute this optimal aggregation is by comparing the scores of all possible rankings, in other words, a brute-force approach:

In [5]:

def rankaggr_brute(ranks):
    min_dist = np.inf
    best_rank = None
    n_voters, n_candidates = ranks.shape
    for candidate_rank in permutations(range(n_candidates)):
        dist = np.sum(kendalltau_dist(candidate_rank, rank) for rank in ranks)
        if dist < min_dist:
            min_dist = dist
            best_rank = candidate_rank
    return min_dist, best_rank

In [6]:

dist, aggr = rankaggr_brute(ranks)
print("A Kemeny-Young aggregation with score {} is: {}".format(
    dist,
    ", ".join(cols[i] for i in np.argsort(aggr))))

A Kemeny-Young aggregation with score 15 is: Ginny, Robin, Gwendolyn, Debbie, Alicia

Integer Programming formulation¶

The brute-force approach is, as you can see, simple to understand and quick to code. However, the number of total ranks of size $n$ is $n!$, so this approach quickly becomes infeasible for real-world problems. Unfortunately, it turns out that this problem (along with many related problems in rank-world) is NP-hard even for only four voters [3]. There have been quite a few approaches and approximations. In many cases, such as search results aggregation, approximations are good enough. For the cases where an exact solution is required, a formulation as a constrained integer program is given in [4].

We build a weighted directed graph $G = (V, E)$ with the candidates as vertices. The edges are defined as such: for every pair of candidates $i, j$, let $\#\{i > j\}$ denote the number of voters who rank $i$ higher than $j$. Draw an edge $e$ between each $i, j$ with weight $w_e = |\#\{i > j\} - \#\{j > i\}|$ (if nonzero, of course). The orientation of the edge is from the less preferred to the more preferred node.

The formulation is based on the alternative interpretation of the Kemeny optimal aggregation as the ranking that minimizes the weights of edges it disagrees with:

$$ \operatorname{minimize} \sum_{e \in E} w_e x_e \ \text{subject to} \ \forall i \neq j \in V, x_{ij} + x_{ji} = 1 \ \forall i \neq j \neq k \neq i \in V, x_{ij} + x_{jk} + x_{ki} \geq 1 $$

In the above problem, all variables are integer, and effectively binary under the other constraints. The interpretation is that $x_{ij} = 1$ if, in the aggregated rank, $i$ is ranked lower than $j$.

The constraints essentially impose that the variables define a total order. The first set of constraints enforce antisymmetry and totality: either $i$ is ranked lower than $j$ or the other way around. The second set of constraints enforce transitivity.

In [7]:

def _build_graph(ranks):
    n_voters, n_candidates = ranks.shape
    edge_weights = np.zeros((n_candidates, n_candidates))
    for i, j in combinations(range(n_candidates), 2):
        preference = ranks[:, i] - ranks[:, j]
        h_ij = np.sum(preference < 0)  # prefers i to j
        h_ji = np.sum(preference > 0)  # prefers j to i
        if h_ij > h_ji:
            edge_weights[i, j] = h_ij - h_ji
        elif h_ij < h_ji:
            edge_weights[j, i] = h_ji - h_ij
    return edge_weights

print(_build_graph(ranks))

[[ 0.  0.  0.  0.  0.]
 [ 1.  0.  3.  3.  3.]
 [ 1.  0.  0.  0.  3.]
 [ 1.  0.  1.  0.  3.]
 [ 1.  0.  0.  0.  0.]]

Now, let’s solve the linear program with the LGPL-licensed lpsolve. It conveniently comes with a Python interface.

In [8]:

from lp_solve import lp_solve

def rankaggr_lp(ranks):
    """Kemeny-Young optimal rank aggregation"""

    n_voters, n_candidates = ranks.shape
    
    # maximize c.T * x
    edge_weights = _build_graph(ranks)
    c = -1 * edge_weights.ravel()  

    idx = lambda i, j: n_candidates * i + j

    # constraints for every pair
    pairwise_constraints = np.zeros(((n_candidates * (n_candidates - 1)) / 2,
                                     n_candidates ** 2))
    for row, (i, j) in zip(pairwise_constraints,
                           combinations(range(n_candidates), 2)):
        row[[idx(i, j), idx(j, i)]] = 1

    # and for every cycle of length 3
    triangle_constraints = np.zeros(((n_candidates * (n_candidates - 1) *
                                     (n_candidates - 2)),
                                     n_candidates ** 2))
    for row, (i, j, k) in zip(triangle_constraints,
                              permutations(range(n_candidates), 3)):
        row[[idx(i, j), idx(j, k), idx(k, i)]] = 1

    constraints = np.vstack([pairwise_constraints, triangle_constraints])
    constraint_rhs = np.hstack([np.ones(len(pairwise_constraints)),
                                np.ones(len(triangle_constraints))])
    constraint_signs = np.hstack([np.zeros(len(pairwise_constraints)),  # ==
                                  np.ones(len(triangle_constraints))])  # >=

    obj, x, duals = lp_solve(c, constraints, constraint_rhs, constraint_signs,
                             xint=range(1, 1 + n_candidates ** 2))

    x = np.array(x).reshape((n_candidates, n_candidates))
    aggr_rank = x.sum(axis=1)

    return obj, aggr_rank

In [9]:

_, aggr = rankaggr_lp(ranks)
score = np.sum(kendalltau_dist(aggr, rank) for rank in ranks)
print("A Kemeny-Young aggregation with score {} is: {}".format(
    score,
    ", ".join(cols[i] for i in np.argsort(aggr))))

A Kemeny-Young aggregation with score 15 is: Ginny, Robin, Gwendolyn, Debbie, Alicia

We get the same result as in the brute-force case. However, it’s much faster. Let’s verify this, and in the process, also test that we always get the same result.

In [10]:

from time import time
import pandas as pd

timings = pd.DataFrame(columns="method rank_len n_ranks time".split())
np.random.seed(0)
for rank_len in range(4, 9):
    for n_ranks in (5, 20):
        ranks = [np.random.permutation(rank_len) for _ in range(n_ranks)]
        ranks = np.array(ranks)
        t0 = time()
        brute_score, brute_aggr = rankaggr_brute(ranks)
        timings = timings.append(dict(method="brute",
                                      rank_len=rank_len,
                                      n_ranks=n_ranks,
                                      time=time() - t0),
                                 ignore_index=True)
        t0 = time()
        _, lp_aggr = rankaggr_lp(ranks)
        timings = timings.append(dict(method="lp",
                                      rank_len=rank_len,
                                      n_ranks=n_ranks,
                                      time=time() - t0),
                                 ignore_index=True)
        assert (brute_score ==
                np.sum(kendalltau_dist(lp_aggr, rank) for rank in ranks))

# lp is much faster, let's run it for longer rankings.
for rank_len in range(9, 16):
    for n_ranks in (5, 20):
        t0 = time()
        _, lp_aggr = rankaggr_lp(ranks)
        timings = timings.append(dict(method="lp",
                                      rank_len=rank_len,
                                      n_ranks=n_ranks,
                                      time=time() - t0),
                                 ignore_index=True)

In [11]:

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(7, 6))
plt.xticks(range(4, 16), range(4, 16))
for subplot, n_ranks in enumerate((5, 20)):
    for method in ('brute', 'lp'):
        this_rows = timings[(timings.n_ranks == n_ranks) &
                            (timings.method == method)]
        plt.semilogy(this_rows.rank_len, this_rows.time,
                     label="{} n={}".format(method, n_ranks))
plt.legend(loc="upper left")
plt.suptitle("Ranking size vs. log time to solve")
plt.show()

You can see that the brute force approach is a bit worse than exponential, while the integer programming approach is more reasonable and also is less sensitive to the number of voters being aggregated. We therefore have a reasonably fast efficient exact solution to the rank aggregation problem on small datasets.

References¶

[1] J Kemeny. Mathematics without numbers, 1959
[2] HP Young. Condorcet’s theory of voting, 1988
[3] JJ Bartholdi III, CA Tovey, MA Trick. The computational difficulty of manipulating an election, 1989
[4] V Conitzer, A Davenport, J Kalagnanam. Improved bounds for computing Kemeny rankings, 2006

Site move

2014-01-10T12:00:00+01:00

I finally got around to moving my entire website, including the blog, to Pelican. I probably would have gotten away with it too if it weren’t for those meddling kids who hacked my friend’s server and convinced me that it’s worth the effort to go static.

It only took 5 hours! I am more and more convinced that static websites allow for a better streamlined workflow that does wonders for productivity.

The static part of my old website was made with Jekyll before, so I kept it almost identical, except for fixing some childish CSS bugs. The same self-hate that comes with reading code that you wrote yourself years ago also comes from markup, apparently.

The blog, however, was migrated thanks to Pelican’s Wordpress importer. This means that the images are missing, the comments are missing (though I think they can be imported somehow) and the plugin-specific syntax such as Latex and source code is poorly rendered. Luckily the images are there in the database dump, but I will need to go through the posts one by one to fix everything. For now, my priority was to get it up and running and keep as many of the old links as possible.

Unfortunately GitHub Pages doesn’t allow URL rewriting, but I use CloudFlare for DNS and their free plan gives me the right to use three forwarding rules. With just 3 rules I couldn’t save all of the old URLs, so I had to prioritize something. I decided to rescue the links pointing directly to articles, which needed two rules, for links with and without trailing slashes, because the CloudFlare syntax has no way of specifying optional characters. I used the third wish that the genie granted me to simply redirect the subdomain blog.vene.ro to the appropriate subfolder, vene.ro/blog.

Best part about this is that I will now be able to migrate my personal list of papers to pelican-bibtex.

BibTeX-powered publications list for Pelican with pelican-bibtex

2013-04-22T10:45:00+02:00

Hook¶

Wouldn’t you like to manage your academic publications list easily within the context of your static website? Without resorting to external services, or to software like bibtex2html, which is very nice but will then require restyling to fit your templates?

Look no more, with the help of pelican-bibtex you can now manage your papers from within Pelican!

Backstory¶

At Fabian‘s advice, I started playing around with Pelican, a static website/blog generator for Python. I like it better than the other generators I used before, so I chose it the next time I had to set up a website. I still didn’t make the courage to migrate my current website and blog to it, but I promise I will.

Pelican has a public plugins repository, but they have a license constraint for all contributions. My plugin isn’t complicated, but I had to “reverse engineer” undocumented parts of the pybtex API. I think that maybe that code that I used to render citations programatically can be useful to others, so I don’t want to release it under a restrictive license. For this reason, I publish pelican-bibtex in my personal GitHub account.

You can see it in action in the source code for the website I am working on at the moment, the home page of my research group. Example output generated using pelican-bibtex can be seen here.

Possible extensions¶

I have not dug in too deeply but I believe this plugin can be extended, with not much difficulty, to support referencing in Pelican blogs, and render BibTeX references at the end of every post. This idea was suggested by Avaris on #pelican, and I find it very cool. Since I don’t need this feature at the moment, it’s not a priority, but it’s something that I would like to see at some point.

Really the most common english idioms?

2013-02-11T16:50:00+01:00

A while back I ran into this blog post and it made me wonder. I’m not a native speaker but the idiomatic phrases that they note as common don’t strike me as such. I don’t think I have ever encountered them very often in real dialogue.

The blog post lists the 10 most common idioms in English. Idioms, also known less ambiguously as fixed expressions, are units of language that span at least two words. Their meaning, relatively to the individual meaning of the parts of the phrase, are figurative. Despite this, fixed expressions don’t classify as creative language, or exploitations. By definition most speakers will unequivocally be familiar with them.

For example, they cite piece of cake as the most common idiomatic expression. This refers to using the phrase to mean that something is easy, that it isn’t challenging. An example of literal use, however, would be when ordering a piece of cake for desert in a restaurant.

Everyone knows that language is a perpetually changing thing, so to begin with it’s even slightly misleading to discuss of the commonness of a phrase, without giving more context. The blog post doesn’t justify the ranking with any numbers anyway, so let’s take them one by one and find out how common they really are!

Corpus Linguistics¶

The approach we are taking here is known as corpus linguistics. The best way to argue that a certain phrase is common, that something is used with a specific meaning or that some constructions are normal is, under corpus linguistics, not to make up examples that seem reasonable, but to look at representative collections of text (corpora) and trying to find the examples there. The conclusions you get this way are backed by real-world language use.

An argument often brought against generative linguistics is that it focuses on the (hard) border between grammatical and not grammatical, and the border is usually defined by made-up examples. This is inappropriate for studying how the norms are exploited in real language use, for example. I refer the interested to the work of Patrick Hanks [1, 2].

Corpus linguistics is sensitive to the corpus used. For this example let’s use two British English corpora: the British National Corpus and the Oxford English Corpus. Measuring by number of words, the latter is around 20 times bigger. The strong point of the BNC is the attention given to the mixing proportions of various domains. The OEC, on the other hand, is larger and more recent. I have a feeling (but I cannot strongly affirm) that the differences in the following results arise from the inclusion in the OEC of blogs dating from the mid-2000s.

One of the key ideas that motivate corpus approaches is the mismatch between these. The cognitive salience of something is the ease with which we can recall it. An example often used in language is the fixed expression kicking the bucket. It is one of the standard examples of fixed expressions that people give very often when asked. It is supposed to mean dying.

However, big surprise: the BNC has only 18 instances of this phrase, out of which only 3 are idiomatic, the rest being either literal or metalinguistic. This is a nice example of the salience contrast, but we mustn’t hurry to conclusions. The OEC has 193 examples (still few, relative to its size) but a lot more of them are idiomatic uses. To save the time I didn’t look at all the examples, but took a random sample of size 18, to compare the relative frequencies to BNC. Here, 15 out of 18 instances are idiomatic and none are meta. Quite a difference!

This goes to show the importance of context when we draw conclusions about language use. Now let’s tackle the list with a similar analysis.

The idioms¶

Piece of cake

In BNC, this phrase occurs 51 times. 29 of these occurrences, however, the meaning is literal. In OEC we find 601 occurrences. In a random sample of size 51 we find 12 literal uses.
Costing an arm and a leg

For flexibility we search for the phrase an arm and a leg. In BNC it can be found 29 times: one literal, four with the verb to pay, and 16 with to cost. In OEC it appears 228 times. We take, again, a sample of size 29 and find no literal uses, 16 with to cost, four with to pay, three with to charge and a few different uses. The figurative meaning is the same in all cases: a lot of money.
Break a leg
BNC: 16, 13 of which are literal. OEC: 70 hits, 10/16 literal.
Hitting the books

BNC: 1 occurrence of hit the record books, which has a different meaning. The idiom is never used. OEC: 135, one of which literal.
Letting the cat out of the bag

We just looked for cooccurrences of cat in the context of the phrase out of the bag.
BNC: 19, out of which 3 metalinguistic/literal. OEC: 298, and out of a sample of 19, all were idiomatic.
Hitting the nail on the head

BNC: 12 instances, all idiomatic. OEC: 484, and out of a sample of 12 all were idiomatic.
When pigs fly
We looked for the lemma fly before the word pigs therefore catching multiple variations.
BNC: 17 hits, OEC: 240.
Judging a book by its cover

We looked for the fixed phrase book by its cover, because the leading verb might vary.
In the BNC, 11 instances (1 of them with tell instead of judge). In OEC, 195 instances. Sampling 11, all were idiomatic.
Biting off more than one can chew

BNC: 16 occurences, one of which with “to take” instead of “to bite”. OEC: 231, all idiomatic after sampling 16.
Scratching one’s back

BNC: 23, out of which only 5 idiomatic. OEC: 756, 5/23 idiomatic.

Recalculating the rank¶

We now have enough data to reorder the expressions and compare. The result will be more approximate for the OEC because of our use of small subsamples to estimate the frequencies, but hopefully it will still be interesting. The way we are estimating the counts for the OEC is as follows: take, for instance, *break a leg*. It was found 70 times, and out of a sample of 16, 10 were literal. The expected number of idiomatic uses is therefore:

[latex]n = \left ( 1 - \frac{10}{16} \right ) \cdot 70 = 26.25[/latex] Repeating this computation and skipping a ton of steps leads to the following rankings:

**In the British National Corpus:**

1. Costing an arm and a leg 2. Piece of cake 3. When pigs fly 4. Letting the cat out of the bag 5. Biting off more than one can chew 6. Hitting the nail on the head 7. Judging a book by its cover 8. Scratching one’s back 9. Break a leg 10. Hitting the books

**In the Oxford English Corpus:**

1. Hitting the nail on the head 2. Piece of cake 3. Letting the cat out of the bag 4. When pigs fly 5. Biting off more than one can chew 6. Costing an arm and a leg 7. Judging a book by its cover 8. Scratching one’s back 9. Hitting the books 10. Break a leg

We can see that apart from the apparent switching of hitting the nail on the head with costing an arm and a leg, the rankings are not too different. We can quantify this by using the Rank Distance, a metric introduced by Liviu P. Dinu [3, 4]. Here, all our 3 rankings are over the same domain: we are not looking for the most frequent idioms in the corpora, this would be very hard. We are just reordering the proposed rank according to the occurrences in BNC and OEC. In this simple case, Rank Distance reduces to [latex]\ell_1[/latex] distance over rank position vectors. The weighted Rank Distance, bounded on [latex][0, 1][/latex] is in this case given by a scaling factor of [latex]0.5k\^2[/latex] where k is the length of the rankings (10 in our case).

The computed distance between the original ranking and the BNC reordering is 0.52. Between the original and the OEC reordering, it is 0.68. Our two reorderings are much closer: the distance is 0.28. This is mostly because that the permutations between the two reorderings affect the top position, and are therefore weighted more.

It’s also interesting to look at the ratio of the counts. Interestingly, they approximately differ by a constant factor not far from the relative size difference of the two corpora, as would be expected.

We have to throw away hitting the books because its BNC zero count leads to divisions by zero. After this step, the average of the relative counts of the idioms is 19.5, with a standard deviation of 10.1, while OED is supposed to have around 20 times more words than the BNC.

Conclusions¶

Well, it seems people don’t say break a leg and let’s hit the books as often as the original author claims. The popularity of most of the cited idioms seems supported by the data, but we have no easy way to find other idioms that might turn out to be much more frequent. Corpus linguistics is a reliable way to measure the social salience of language patterns It should always be used to verify and back empty claims of the form X is correct, Y is frequent or Nobody says Z.

[1] Patrick Hanks, How people use words to make meanings.
[2] Patrick Hanks, Lexical Analysis: Norms and Exploitations. The MIT Press (January 25, 2013)
[3] Liviu P. Dinu, Florin Manea. An efficient approach for the rank aggregation problem. In: Theoretical Computer Science, Volume 359 Issue 1, 14 August 2006. Pages 455 - 461.
[4] Liviu P. Dinu, [On the Classification and Aggregation of Hierarchies with Different Constitutive Elements][]. Fundam. Inform. 55(1): 39-50 (2003)

Scikit-learn-speed: An overview on the final day

2012-08-20T02:44:00+02:00

This summer, I was granted the project called scikit-learn-speed, consisting of developing a benchmarking platform for scikit-learn and using it to find potential speedups, and in the end, make the library go faster wherever I can.

On the official closing day of this work, I’d like to take a moment and recall the accomplishments and failures of this project, and all the lessons to be learned.

The scikit-learn-speed benchmark platform¶

[]
[Scikit-learn-speed][] is a continuous benchmark suite for the scikit-learn library. It has the following features:

vbench-powered integration with Git
Easily triggered build and report generation: just type make
Easily readable and writeable template for benchmarks:
[sourcecode lang=”python”]
{
‘obj’: ‘LogisticRegression’,
‘init_params’: {‘C’: 1e5},
‘datasets’: (‘arcene’, ‘madelon’),
‘statements’: (‘fit’, ‘predict’)
}, …
[/sourcecode]
Many attributes recorded: time (w/ estimated standard deviation), memory usage, cProfiler output, line_profiler output, tracebacks
Multi-step benchmarks: i.e. fit followed by predict

What were the lessons I learned here?

Make your work reusable: the trade-off between good design and get-it-working-now¶

For the task of rolling out a continuous benchmarking platform, we decided pretty early in the project to adopt Wes McKinney’s vbench. If my goal would’ve been to maintain vbench and extend it into a multi-purpose, reusable benchmarking framework, the work would’ve been structured differently. It also would have been very open-ended and difficult to quantify.

The way things have been, I came up with features that we need in scikit-learn-speed, and tried to implement them in vbench without refactoring too much, but still by trying to make them as reusable as possible.

The result? I got all the features for scikit-learn-speed, but the implementation is not yet clean enough to be merged into vbench. This is fine for a project with a tight deadline such as this one: after it’s done, I will just spend another weekend on cleaning the work up and making sure it’s appreciated upstream. This will be easier because of the constraint to keep compatibility with scikit-learn-speed.

Never work quietly (unless you’re a ninja)¶

I know some students who prefer that the professor doesn’t even know they exist until the final, when they would score an A, and (supposedly) leave the professor amazed. In real life, plenty of people would be interested in what you are doing, as long as they know about it. The PSF goes a long way to help this, with the “blog weekly” rule. In the end, however, it’s all up to you to make sure that everybody who should know finds out about your work. It will spare the world the duplicated work, the abandoned projects, but most importantly, those people could point you to things you have missed. Try to mingle in real-life as well, attend conferences, meetups, coding sprints.

I was able to slightly “join forces” with a couple of people who contacted me about my new vbench features (Hi Jon and Joel!), I have shaped my design slightly towards their requirements as well, and hopefully the result will be a more general vbench.

The speedups¶

Once scikit-learn-speed was up and running, I couldn’t believe how useful it is to be able to scroll, catch slow code and jump straight at the profiler output with one click. I jumped on the following speed-ups:

Multiple outputs in linear models. (PR)
Some of them proved trickier than expected, so I didn’t implement it for all the module yet, but it is ready for some estimators.
Less callable functions passed around in FastICA (merged)
Speed up euclidean_distances by rewriting in Cython. (PR)
This meant making more operations support an out argument, for passing preallocated memory. This touches many
different objects in the codebase: clustering, manifold learning, nearest neighbour methods.
Insight into inverse and pseudoinverse computation, new pinvh function for inverting symmetric/hermitian matrices. (PR)
This speeds up the covariance module (especially MinCovDet), ARDRegression and the mixture models. It also lead to an upstream contribution to Scipy
OrthogonalMatchingPursuit forward stepwise path for cross-validation (PR)
This is only halfway finished, but it will lead to faster and easier optimization of the OMP sparsity parameter.

Lessons? These will be pretty obvious.

Write tests, tests, tests!¶

This is a no-brainer, but it still didn’t stick. In that one case out of 10 that I didn’t explicitly test, a bug was obviously hiding. When you want to add a new feature, it’s best to start by writing a failing test, and then making it pass. Sure, you will miss tricky bugs, but you will never have embarrassing, obvious bugs in your code :)

Optimization doesn’t have to be ugly¶

Developers often shun optimization. It’s true, you should profile first, and you shouldn’t focus on speeding up stuff that is dominated by other computations that are orders of magnitude slower. However, there is an elephant in the room: the assumption that making code faster invariably makes it less clear, and takes a lot of effort.

The following code is a part of scipy’s pinv2 function as it currently is written:
[sourcecode lang=”python”]
cutoff = cond*np.maximum.reduce(s)
psigma = np.zeros((m, n), t)
for i in range(len(s)):
if s[i] > cutoff:
psigma[i,i] = 1.0/np.conjugate(s[i])
return np.transpose(np.conjugate(np.dot(np.dot(u,psigma),vh)))
[/sourcecode]

psigma is a diagonal matrix, and some time and memory can be saved with simple vectorization. However, this part of the code dominated by an above call to svd. The profiler output would say that we shouldn’t bother, but is it really a bother? Look at Jake’s new version:

[sourcecode lang=”python”]
above_cutoff = (s > cond * np.max(s))
psigma_diag = np.zeros_like(s)
psigma_diag[above_cutoff] = 1.0 / s[above_cutoff]

return np.transpose(np.conjugate(np.dot(u * psigma_diag, vh)))
[/sourcecode]

It’s shorter, more elegant, easier to read, and nevertheless faster. I would say it is worth it.

Small speed-ups can propagate¶

Sure, it’s great if you can compute an inverse two times faster, say in 0.5s instead of 1s. But if some algorithm calls this function in a loop that might iterate 100, 300, or 1000 times, this small speed-up seems much more important, doesn’t it?

What I’m trying to say with this is that in a well-engineered system, a performance improvement to a relatively small component (such as the function that computes a pseudoinverse) can lead to multiple spread out improvements. Be careful of the double edge of this sword, a bug introduced in a small part can cause multiple failures downstream. But you are fully covered by your test suite, aren’t you?

Overall it has been a fruitful project that may have not resulted in a large number of speed-ups, but a few considerable ones nonetheless. And I venture the claim that the scikit-learn-speed tool will prove useful over time, and that the efforts deployed during this project have stretched beyond the boundary of the scikit-learn.

[]: http://jenkins-scikit-learn.github.com/scikit-learn-speed/

Inverses and pseudoinverses. Numerical issues, speed, symmetry.

2012-08-18T19:41:00+02:00

The matrix inverse is a cornerstone of linear algebra, taught, along with its applications, since high school. The inverse of a matrix $latex A$, if it exists, is the matrix $latex A\^{-1}$ such that $latex AA\^{-1} = A\^{-1}A = I_n$. Based on the requirement that the left and right multiplications should be equal, it follows that it only makes sense to speak of inverting square matrices. But just the square shape is not enough: for a matrix $latex A$ to have an inverse, $latex A$ must be full rank.

The inverse provides an elegant (on paper) method of finding solutions to systems of $latex n$ equations with $latex n$ unknowns, which correspond to solving $latex Ax = b$ for $latex x$. If we’re lucky and $latex A\^{-1}$ exists, then we can find $latex x = A\^{-1}b$. For this to work, it must be the case that:

We have exactly as many unknowns as equations
No equation is redundant, i.e. can be expressed as a linear combination of the others

In this setting, there is a unique solution for $latex x$.

The Moore-Penrose pseudoinverse¶

What if we have more equations than unknowns? It is most likely the case that we cannot satisfy all the equations perfectly, so let’s settle for a solution that best fits the constraints, in the sense of minimising the sum of squared errors. We solve $latex \operatorname{arg\,min}_x ||b - Ax||$.

And how about the other extreme, where we have a lot of unknowns, but just a few equations constraining them. We will probably have an infinity of solutions, how can we choose one? A popular choice is to take the one of least $latex \ell_2$ norm: $latex \operatorname{arg\,min}_x ||x|| \operatorname{s.t.} Ax = b$. Is there a way to generalize the idea of a matrix inverse for this setting?

The pseudoinverse of an arbitrary-shaped matrix $latex A$, written $latex A\^{+}$, has the same shape as $latex A\^{T}$ and solves our problem: the answer to both optimization methods above is given by $latex x = A\^{+}y$.

The theoretical definition of the pseudoinverse is given by the following conditions. The intuitive way to read them is as properties of $latex AA\^+$ or $latex A\^+A$:

$latex AA\^+A = A$
$latex A\^+AA\^+ = A\^+$
$latex (AA\^+)\^T = AA\^+$
$latex (A\^+A)\^T = A\^+A$

These conditions do not however give us a way to get our hands on a pseudoinverse, so we need something else.

How to compute the pseudoinverse on paper¶

The first time I ran into the pseudoinverse, I didn’t even know its definition, only the expression of the closed-form solution of such a problem, and given as:

$latex A\^+ = (A\^T A)\^{-1}A\^T$

What can we see from this expression:

It gives us a way to compute the pseudoinverse, and hence to solve the problem
If $latex A$ is actually invertible, it means $latex A\^T$ is invertible, so we have $latex A\^+ = A\^{-1}(A\^T)\^{-1}A\^T = A\^{-1}$
Something bad happens if $latex A\^TA$ is not invertible.

The pseudoinverse is still defined, and unique, when $latex A\^TA$ is not invertible, but we cannot use the expression above to compute it.

Numerical issues¶

Before going on, we should clarify and demystify some of the urban legends about numerical computation of least squares problems. You might have heard the following unwritten rules:

Never compute $latex A\^{-1}$, solve the system directly
If you really need $latex A\^{-1}$, use pinv and not inv

The first of these rules is based on some misguided beliefs, but is still good advice. If your goal is a one-shot answer to a system, there’s no use in explicitly computing a possibly large inverse, when all you need is $latex x$. But this paper shows that computing the inverse is not necessarily a bad thing. The key to this is conditional accuracy, and as long as the inv function used has good conditional bounds, you will get as good results as with a least squares solver.

The second rule comes from numerical stability, and will definitely bite you if misunderstood. If $latex A$ is a square matrix with a row full of zeros, it’s clearly not invertible, so an algorithm attempting to compute the inverse will fail and you will be able to catch that failure. But what if the row is not exactly zero, but the sum of several other rows, and a slight loss of precision is propagated at every step?

Numerical rank vs. actual rank¶

The rank of a matrix $latex A$ is defined as the number of linearly independent rows (or equivalently, columns) in $latex A$. In other words, the number of non-redundant equations in the system. We’ve seen before that if the rank is less than the total number of rows, the system cannot have a unique solution anymore, so the matrix $latex A$ is not invertible.

The rank of a matrix is a computationally tricky problem. On paper, with small matrices, you would look at minors of decreasing size, until you find the first non-zero one. This is unfeasible to implement on a computer, so numerical analysis has a different approach. Enter the singular value decomposition!

The SVD of a matrix $latex A$ is $latex A = USV\^{T}$, where $latex S$ is diagonal and $latex U, V$ are orthogonal. The elements on the diagonal of $latex S$ are called the singular values of $latex A$. It can be seen that to get a row full of zeros when multiplying three such matrices, a singular value needs to be exactly zero.

The ugly thing that could happen is that one (or usually more) singular values are not exactly zero, but very low values, due to propagated imprecision. Why is this a problem? By looking at the SVD and noting its properties, it becomes clear that $latex A\^{-1} = VS\^{-1}U\^{T}$ and since $latex S$ is diagonal, its inverse is formed by taking the inverse of all the elements on the diagonal. But if a singular value is very small but not quite zero, its inverse is very large and it will blow up the whole computation of the inverse. The right thing to do here is either to tell the user that $latex A$ is numerically rank deficient, or to return a pseudoinverse instead. A pseudoinverse would mean: give up on trying to get $latex AA\^+$ to be the identity matrix, simply aim for a diagonal matrix with approximately ones and zeroes. In other words, when singular values are very low, set them to 0.

How do you set the threshold? This is actually a delicate issue, being discussed on the numeric Python mailing list.

Scipy implementations¶

Scipy exposes inv, pinv and pinv2. inv secretly invokes LAPACK, that ancient but crazy robust code that’s been used since the 70s, to first compute a pivoted LU decomposition that is then used to compute the inverse. pinv also uses LAPACK, but for computing the least-squares solution to the system $latex AX = I$. pinv2 computes the SVD and transposes everything like shown above. Both pinv and pinv2 expose cond and rcond arguments to handle the treatment of very small singular values, but (attention!) they behave differently!

The different implementations also lead to different speed. Let’s look at inverting a random square matrix:

[sourcecode lang=”python”]
In [1]: import numpy as np

In [2]: from scipy import linalg

In [3]: a = np.random.randn(1000, 1000)

In [4]: timeit linalg.inv(a)
10 loops, best of 3: 132 ms per loop

In [5]: timeit linalg.pinv(a)
1 loops, best of 3: 18.8 s per loop

In [6]: timeit linalg.pinv2(a)
1 loops, best of 3: 1.58 s per loop
[/sourcecode]

Woah, huge difference! But do all three methods return the “right” result?

[sourcecode lang=”python”]
In [7]: linalg.inv(a)[:3, :3]
Out[7]:
array([[ 0.03636918, 0.01641725, 0.00736503],
[-0.04575771, 0.03578062, 0.02937733],
[ 0.00542367, 0.01246306, 0.0122156 ]])

In [8]: linalg.pinv(a)[:3, :3]
Out[8]:
array([[ 0.03636918, 0.01641725, 0.00736503],
[-0.04575771, 0.03578062, 0.02937733],
[ 0.00542367, 0.01246306, 0.0122156 ]])

In [9]: linalg.pinv2(a)[:3, :3]
Out[9]:
array([[ 0.03636918, 0.01641725, 0.00736503],
[-0.04575771, 0.03578062, 0.02937733],
[ 0.00542367, 0.01246306, 0.0122156 ]])

In [10]: np.testing.assert_array_almost_equal(linalg.inv(a), linalg.pinv(a))

In [11]: np.testing.assert_array_almost_equal(linalg.inv(a), linalg.pinv2(a))
[/sourcecode]

Looks good! This is because we got lucky, though, and a was invertible to start with. Let’s look at its spectrum:

[sourcecode lang=”python”]
In [12]: _, s, _ = linalg.svd(a)

In [13]: np.min(s), np.max(s)
Out[13]: (0.029850235603382822, 62.949785645178906)
[/sourcecode]

This is a lovely range for the singular values of a matrix, not too small, not too large. But what if we built the matrix in a way that would always pose problems? Specifically, let’s look at the case of covariance matrices:

[sourcecode lang=”python”]
In [14]: a = np.random.randn(1000, 50)

In [15]: a = np.dot(a, a.T)

In [16]: _, s, _ = linalg.svd(a)

In [17]: s[-9:]
Out[17]:
array([ 7.40548924e-14, 6.48102455e-14, 5.75803505e-14,
5.44263048e-14, 4.51528730e-14, 3.55317976e-14,
2.46939141e-14, 1.54186776e-14, 5.08135874e-15])

[/sourcecode]

a has at least 9 tiny singular values. Actually it’s easy to see why there are 950 of them:

[sourcecode lang=”python”]
In [18]: np.sum(s \< 1e-10)
Out[18]: 950
[/sourcecode]

How do our functions behave in this case? Instead of just looking at a corner, let’s use our gift of sight:[]

The small eigenvalues are large enough that inv thinks the matrix is full rank. pinv does better but it still fails, you can see a group of high-amplitude noisy columns. pinv2 is faster and it also gives us a useful result in this case.

Wait, does this mean that pinv2 is simply better, and pinv is useless?

Not quite. Remember, we are now trying to actually invert matrices, and degrade gracefully in case of rank deficiency. But what if we need the pseudoinverse to solve an actual non-square, wide or tall system?

[sourcecode lang=”python”]
In [19]: a = np.random.randn(1000, 50)

In [20]: timeit linalg.pinv(a)
10 loops, best of 3: 104 ms per loop

In [21]: timeit linalg.pinv(a.T)
100 loops, best of 3: 7.08 ms per loop

In [22]: timeit linalg.pinv2(a)
10 loops, best of 3: 114 ms per loop

In [23]: timeit linalg.pinv2(a.T)
10 loops, best of 3: 126 ms per loop
[/sourcecode]

Huge victory for pinv in the wide case! Hurray! With all this insight, we can draw a line and see what we learned.

If you are 100% sure that your matrix is invertible, use inv for a huge speed gain. The implementation of inv from Scipy is based on LAPACK’s *getrf + *getri, known to have good bounds.
If you are trying to solve a tall or wide system, use pinv.
If your matrix is square but might be rank deficient, use pinv2 for speed and numerical gain.

Improving the symmetric case¶

But wait a second, can’t we do better? $latex AA\^T$ is symmetric, can’t we make use of that to speed up the computation even more? Clearly, if $latex A$ is symmetric, in its SVD $latex A = USV\^T$, we must have $latex U = V$. But this is exactly the eigendecomposition of a symmetric matrix $latex A$. The eigendecomposition can be computed cheaper than the SVD using Scipy eigh, that uses LAPACK’s *evr. As part of my GSoC this year, with help from Jake VanderPlas, we made a pull request to Scipy containing a pinvh function that is equivalent to pinv2 but faster for symmetric matrices.

[sourcecode lang=”python”]
In [24]: timeit linalg.pinv2(a)
1 loops, best of 3: 1.54 s per loop

In [25]: timeit linalg.pinvh(a)
1 loops, best of 3: 621 ms per loop

In [26]: np.testing.assert_array_almost_equal(linalg.pinv2(a), linalg.pinvh(a))
[/sourcecode]

[]: http://localhost:8001/wp-content/uploads/2012/08/pseudoinverses.png

The scikit-learn-speed ship has set sail! Faster than ever, with multi-step benchmarks!

2012-08-11T17:32:00+02:00

I am pleased to announce that last night at 2:03 AM, the first fully automated run of the scikit-learn-speed test suite has run on our Jenkins instance! You can admire it at its temporary home for now. As soon as we verify that everything is good, we will move this to the official scikit-learn page.

I would like to take this opportunity to tell you about our latest changeset. We made running the benchmark suite tons simpler by adding a friendly Makefile. You can read more about its usage in the guide. But by far, our coolest new toy is:

Multi-step benchmarks¶

A standard vbench benchmark has three units of code, represented as strings: code, setup and cleanup. With the original timeit-based benchmarks, this means that for every run, the setup would be executed once. Then, the main loop runs repeat times, and within each iteration, the code is run ncalls times. Then cleanup happens, the best time is returned, and everybody is happy.

In scikit-learn, most of our interesting objects go through a state change called fitting. This metaphor is right at home in the machine learning field, where we separate the learning phase for the prediction phase. The prediction step cannot be invoked on an object that hasn’t been fitted.

For some algorithms, one of these steps is trivial. A brute force Nearest Neighbors classifier can be instantaneously fit, but prediction takes a while. On the opposite end we have linear models, with tons of complicated algorithms to fit them, but evaluation is a simple matrix-vector product that Numpy handles perfectly.

But many of scikit-learn’s estimators have both steps interesting. Let’s take Non-negative Matrix Factorization. It has three interesting functions: The fit that computes $latex X = WH $, the transform that computes a non-negative projection on the components learned in fit, and fit_transform that takes advantage of the observation that when fitting, we also get the transformed $latex X $ for free.

When benchmarking NMF, we initially had to design 3 benchmarks:

setup =standard, code = obj.fit(X)
setup =standard, code = obj.fit_transform(X)
setup =standard+ obj.fit(X), code = obj.transform(X)

How much time were we wasting?¶

Let’s say it takes 10 seconds. For every benchmark, we time the code by running it 3 times. We run it once more to measure memory usage, once more for cProfile and one last time for line_profiler. This is a total of 6 times per benchmark. We need to multiply this by 2 again for running on two datasets. So when benchmarking NMF, because we need to fit before predicting, we do it 12 extra times. If a fit takes 5 seconds, this means one minute wasted on benchmarking just one estimator. Wouldn’t it be nice to fit, fit_transform and transform in a sequence?

Behind the scenes¶

We made the PythonBenchmark code parameter also support getting a sequence of strings, instead of just a string. On the database side, every benchmark result entry gets an extra component in the primary key, the number of the step it measures.

In the benchmark description files, nothing is changed:

[sourcecode lang=”python”]
{
‘obj’: ‘NMF’,
‘init_params’: {‘n_components’: 2},
‘datasets’: (‘blobs’,),
‘statements’: (‘fit_unsup’, ‘transform_unsup’, ‘fit_transform’)
},
[/sourcecode]

But before, we would take the cartesian product of datasets and statements, and build a Benchmark object for every pairing. Now, we just pass the tuple as it is, and vbench is smart enough to do the right thing.
We avoided the extra calls to fit in a lot of benchmarks. The whole suite now takes almost half the time to run!

Note: This trick is currently hosted in the abstract_multistep_benchmarks vbench branch in my fork.

Profiler output, benchmark standard deviation and other goodies in scikit-learn-speed

2012-07-27T11:01:00+02:00

This post is about the scikit-learnbenchmarking project that I am working on, called scikit-learn-speed. This is a continuous benchmarking suite that runs and generates HTML reports using Wes McKinney’s vbench framework, to which I had to make some (useful, I hope) additions.

What it looks like now¶

You can check out a teaser/demo that was run on equidistant releases from the last two months. What has changed since the last version? Here’s a list in order of obviousness:

We now use the lovely scikit-learn theme
Timing graphs now show the ±1 standard deviation range
cProfile output is displayed for all the benchmarks, so we can easily see at a glance what’s up
Said profiler output is collapsible using JQueryUI goodness
There now is an improved Quick Start guide to running vbench on your machine

What made this possible¶

I have done some more refactoring in my vbench fork, because I didn’t want to have a huge, monolithic Benchmark class that was specific to what we want in scikit-learn-speed. So on this branch, I set up a mixin/multiple inheritance hierarchy of benchmark classes.

The Benchmark class in vbench is now an abstract base class, with some common functionality and structure.
Our SklBenchmark class is defined in scikit-learn-speed as:

class SklBenchmark(CProfileBenchmarkMixin, MemoryBenchmarkMixin, PythonBenchmark):

Let’s read this from right to left:

PythonBenchmark: This class stores code, setup and cleanup Python code as strings, and implements simple timing mechanisms using the time module.
Bonus: TimeitBenchmark: This class extends PythonBenchmark with the timeit micro-benchmark timing method previously used in vbench. We turned this off in scikit-learn-speed.
MemoryBenchmarkMixin: This adds memory benchmarking using [memory_profiler][].
CProfileBenchmarkMixin: This runs the code through cProfile and implements mechanisms to report the output.

The database is not flexible enough to adapt to arbitrary benchmark structure right now, so if anybody would like to help the effort, it would be very appreciated.

Scikit-learn-speed HTML reports teaser

2012-07-20T14:40:00+02:00

EDIT: I made the plots a little more readable, check it out!

Last time, I teased you with a screenshot of local output. Now, I will tease you with the benchmarks run on a couple of recent commits, along with some from earlier this year.

After some effort and bugfixes, the project now reliably runs on different machines, so the next step to host it on a remote server and invoke it daily is getting closer. In the mean time, you can have a look at the sample output.

Note that just last time, the plots look jagged but the differences are mostly minor and significant conclusions cannot be drawn yet, but as the suite will start running daily, the plots will become much more meaningful. I could waste time running the suite on more previous commits, but the results wouldn’t be comparable with the ones from the deployed system, because of hardware differences.

Playing around with this makes me want a couple of features in vbench. One is the possibility to overlay related benchmarks on the same plot (for example, different parameters for the same algorithm and data): this could be useful to spot patterns. A second one is some query / sorting support: see what are the most expensive benchmarks, see what benchmarks show the biggest jump in performance (but this could become a historical wall of fame or shame).

Memory benchmarking with vbench

2012-07-05T12:38:00+02:00

The scikit-learn-speed project now has memory usage benchmarking!

This was accomplished by building on what I described in my recent posts, specifically the extensions to Fabian’s [memory_profiler][] that you can find in my fork, but they will be merged upstream soon. The key element is the %magic_memit function whose development I blogged about on several occasions. I plugged this into vbench in a similar way to how the timings are computed, all with great success.

Here is a screenshot of the way a simple benchmark looks now, with just a few data points.

[caption id=”attachment_464” align=”aligncenter” width=”600”][![A screenshot showing generated output from the scikit-learn-speed project, illustrating memory usage benchmarking.][]][] Memory benchmarking in scikit-learn-speed powered by vbench.[/caption]

You can check it out and use it yourself for your benchmarks, but you need to use the vbench from the memory branch on my fork.

Of course, there are some important caveats. I am running this on my laptop, which runs OS X Lion, so, under the effect of this bug, I hardcoded the ‘-i‘ so the memory benchmarks are not realistic. Also, the y-range should probably be forced wider, because the plots look erratic, showing the very small noise at a large-scale.

[![A screenshot showing generated output from the scikit-learn-speed project, illustrating memory usage benchmarking.][]]: http://localhost:8001/wp-content/uploads/2012/07/vbench1.png

On why my %memit fails on OSX

2012-07-04T12:49:00+02:00

In my last post I mentioned that I’m not satisfied with the current state of %memit, because some more complicated numerical function calls make it crash. I will start this post with a reminder of a pretty important bug:

[On MacOS X (10.7 but maybe more), after forking a new process, there is a segfault in Grand Central Dispatch on the BLAS DGEMM function from Accelerate.][]

EDIT 1: In a hurry, I forgot to mention how Olivier Grisel and David Cournapeau spent some time narrowing down this issue, starting from an odd testing bug in scikit-learn. They reported it to Apple, but there was, as of the date of this post, no reaction.

EDIT 2: MinRK confirms, and I verified shortly after, that this bug is fixed in Mountain Lion (10.8). Still not sure how far back it goes, though, so feedback is welcome.

When I first tried to make the %memit magic, I thought about simply measuring the current memory, running the command, and measuring the memory again. The problem is the results are not consistent, because Python tries to reuse already allocated memory whenever it can.

Using memory_profiler, here’s an example illustrating this elastic memory management:
[sourcecode lang=”python”]
# mem_test.py
import numpy as np

def make_a_large_array():
return np.ones((1000, 1000))

def main():
make_a_large_array()
make_a_large_array()
make_a_large_array()

# in IPython:
In [1]: import mem_test

In [2]: %mprun -f mem_test.main mem_test.main()
Filename: mem_test.py

Line # Mem usage Increment Line Contents¶

8 24.8477 MB 0.0000 MB def main():
9 24.8633 MB 0.0156 MB make_a_large_array()
10 32.4688 MB 7.6055 MB make_a_large_array()
11 32.4688 MB 0.0000 MB make_a_large_array()
[/sourcecode]

If this was in an IPython environment, and one would like to see how much memory make_a_large_array() uses, you could say we can simply run it a few times and take the maximum. However, if you happened to accidentally call main() once before, you will no longer get a good result:

[sourcecode lang=”python”]
In [3]: %mprun -f mem_test.main mem_test.main()
Filename: mem_test.py

Line # Mem usage Increment Line Contents¶

8 32.4922 MB 0.0000 MB def main():
9 32.5234 MB 0.0312 MB make_a_large_array()
10 32.5234 MB 0.0000 MB make_a_large_array()
11 32.5234 MB 0.0000 MB make_a_large_array()
[/sourcecode]

So how can we get consistent results for the memory usage of an instruction? We could run it in a fresh, new process. I implemented this in %memit and it shows:

[sourcecode lang=”python”]
In [5]: %memit mem_test.make_a_large_array()
maximum of 3: 8.039062 MB per loop

In [6]: %memit mem_test.make_a_large_array()
maximum of 3: 8.035156 MB per loop

In [7]: %memit mem_test.make_a_large_array()
maximum of 3: 8.042969 MB per loop
[/sourcecode]

This way you can also realistically benchmark assignments:

[sourcecode lang=”python”]
In [8]: %memit X = mem_test.make_a_large_array()
maximum of 3: 8.054688 MB per loop

In [9]: %memit X = mem_test.make_a_large_array()
maximum of 3: 8.058594 MB per loop

In [10]: %memit X = mem_test.make_a_large_array()
maximum of 3: 8.058594 MB per loop
[/sourcecode]

If we don’t spawn a subprocess, del doesn’t help, but allocating new variables does:
[sourcecode lang=”python”]
In [11]: %memit -i X = mem_test.make_a_large_array()
maximum of 3: 7.632812 MB per loop

In [12]: del X

In [13]: %memit -i X = mem_test.make_a_large_array()
maximum of 3: 0.000000 MB per loop

In [14]: %memit -i Y = mem_test.make_a_large_array()
maximum of 3: 7.632812 MB per loop

In [15]: %memit -i Z = mem_test.make_a_large_array()
maximum of 3: 7.632812 MB per loop
[/sourcecode]

Now, the problem is that when the function that you are benchmarking contains calls to np.dot (matrix multiplication), the subprocess will consistently fail with SIGSEGV on affected OS X systems. These are actually pretty much all the functions that I intended %memit for: numerical applications. For that reason, I have made %memit notify the user when all subprocesses fail, and to suggest the usage of the -i flag.

I think that, with this update, %memit is flexible and usable enough for actual use, and therefore for merging into memory_profiler.

More on memory benchmarking

2012-07-02T11:27:00+02:00

Following up on my task to make it easier to benchmark memory usage in Python, I updated Fabian’s [memory_profiler][] to include a couple of useful IPython magics. While in my last post, I used the new IPython 0.13 syntax for defining magics, this time I used the backwards-compatible one from the previous version.

You can find this work-in-progress as a [pull request on memory_profiler][] from where you can trace it to my GitHub repo. Here’s what you can do with it:

%mprun¶

Copying the spirit of %lprun, since imitation is the most sincere form of flattery, you can use %mprun to easily view line-by-line memory usage reports, without having to go in and add the @profile decorator.

For example:

[sourcecode lang=”python”]

In [1]: import numpy as np

In [2]: from sklearn.linear_model import ridge_regression

In [3]: X, y = np.array([[1, 2], [3, 4], [5, 6]]), np.array([2, 4, 6])

In [4]: %mprun -f ridge_regression ridge_regression(X, y, 1.0)

(…)

109 41.6406 MB 0.0000 MB if n_features > n_samples or \
110 41.6406 MB 0.0000 MB isinstance(sample_weight, np.ndarray) or \
111 41.6406 MB 0.0000 MB sample_weight != 1.0:
112
113 # kernel ridge
114 # w = X.T * inv(X X\^t + alpha*Id) y
115 A = np.dot(X, X.T)
116 A.flat[::n_samples + 1] += alpha * sample_weight
117 coef = np.dot(X.T, _solve(A, y, solver, tol))
118 else:
119 # ridge
120 # w = inv(X\^t X + alpha*Id) * X.T y
121 41.6484 MB 0.0078 MB A = np.dot(X.T, X)
122 41.6875 MB 0.0391 MB A.flat[::n_features + 1] += alpha
123 41.7344 MB 0.0469 MB coef = _solve(A, np.dot(X.T, y), solver, tol)
124
125 41.7344 MB 0.0000 MB return coef.T

[/sourcecode]

%memit¶

As described in my previous post, this is a %timeit-like magic for quickly seeing how much memory a Python command uses.
Unlike %timeit, however, the command needs to be executed in a fresh process. I have to dig in some more to debug this, but if the command is run in the current process, very often the difference in memory usage will be insignificant, I assume because preallocated memory is used. The problem is that when running in a new process, some functions that I tried to bench crash with SIGSEGV. For a lot of stuff, though, %memit is currently usable:

[sourcecode lang=”python”]
In [1]: import numpy as np

In [2]: X = np.ones((1000, 1000))

In [3]: %memit X.T
worst of 3: 0.242188 MB per loop

In [4]: %memit np.asfortranarray(X)
worst of 3: 15.687500 MB per loop

In [5]: Y = X.copy(‘F’)

In [6]: %memit np.asfortranarray(Y)
worst of 3: 0.324219 MB per loop
[/sourcecode]

It is very easy, using this small tool, to see what forces memory copying and what does not.

Installation instructions¶

First, you have to get the source code of this version of memory_profiler. Then, it depends on your version of IPython. If you have 0.10, you have to edit ~/.ipython/ipy_user_conf.py like this: (once again, instructions borrowed from [line_profiler][])

[sourcecode lang=”python”]
# These two lines are standard and probably already there.
import IPython.ipapi
ip = IPython.ipapi.get()

# These two are the important ones.
import memory_profiler
ip.expose_magic(‘mprun’, memory_profiler.magic_mprun)
ip.expose_magic(‘memit’, memory_profiler.magic_memit)
[/sourcecode]

If you’re using IPython 0.11 or newer, the steps are different. First create a configuration profile:
[sourcecode lang=”bash”]
$ ipython profile create
[/sourcecode]
Then create a file named ~/.ipython/extensions/memory_profiler_ext.py with the following content:

[sourcecode lang=”python”]
import memory_profiler

def load_ipython_extension(ip):
ip.define_magic(‘mprun’, memory_profiler.magic_mprun)
ip.define_magic(‘memit’, memory_profiler.magic_memit)
[/sourcecode]

Then register it in ~/.ipython/profile_default/ipython_config.py, like this. Of course, if you already have other extensions such as line_profiler_ext, just add the new one to the list.

[sourcecode lang=”python”]
c.TerminalIPythonApp.extensions = [
‘memory_profiler_ext’,
]
c.InteractiveShellApp.extensions = [
‘memory_profiler_ext’,
]
[/sourcecode]

Now launch IPython and you can use the new magics like in the examples above.

Quick memory usage benchmarking in IPython

2012-06-30T08:53:00+02:00

Everybody loves %timeit, there’s no doubt about it. So why not have something like that, but for measuring how much memory your line takes? Well, now you can; grab a hold of the script in the following gist and run it like in the example.

[gist id=3022718]

Instead of taking care of the dirty process inspection stuff myself, I decided to delegate this to Fabian’s simple but very good [memory_profiler][]. There is also Guppy available, but its design seems a bit and overkill for this task.

Please contact me if you find problems with this implementation, this is a preliminary, quick hack-y version. :)

Compiling and Installing GLARF and the bundled Charniak parser on MacOS X

2012-06-21T12:32:00+02:00

It seems that I keep getting handed buggy code to install. These are cases of research software where the developers didn’t make the effort to make sure their tool works on the platforms it should.

[GLARF][] (Grammatical and Logical Argument Representation Framework) is, in their words, “a typed feature structure framework for representing regularizations of parse trees”. It is a processing pipeline from NYU with rich output including many types of structure in the given text. However, it is clearly a case of software whose maintenance was abandoned when it “worked” for them. The whole install and run procedure is pretty messy, but at least it’s documented. The problem is, following it step by step doesn’t work on my MacBook. As usual, I needed to hack through it a bit.

The Charniak parser distributed with GLARF has now been superseded by the [BLLIP parser][]. The new one is tricky to compile as well, but I have yet to see if it plugs into GLARF, so I leave this for a future post.

Here are the steps I needed to take to make GLARF work:

Download and unzip the GLARF package.
Make sure you have `sbcl` in your path, and your `perl` is in (or linked from) `/usr/bin/perl`
Set the environment variables. I like to make a shell script to set them so I don’t have to do it every time. So I write something looking like this: [sourcecode lang=”bash”] \# glarf\_env.sh export GLARF=/Users/vene/code/GLARF export GLARF\_JET=\${GLARF}/JET export PATH=\$PATH:. [/sourcecode]
Then for every session when I want to use GLARF, I do `source glarf_env.sh`.
Compile GLARF by running `$GLARF/commands-2010/compile-glarf`. *Note:* this only compiles the pipeline lisp code.
Now, according to their instructions, you’re done. However, if you try to run it, you’d notice the output is incomplete. (It goes through the named entity extraction part, but it doesn’t run the parser.) The reason for this is that they distribute the Charniak parser with a precompiled binary that runs on Linux, but not on the Mac. So we need to recompile it. So go to `$GLARF/charniak-parser-2005/parser05Aug16-static/PARSE2 `, run `make clean` and roll up your sleeves.
Obviously, simply running `make` doesn’t work. [As documented by Pawel Mazur][], we need to edit `BchartSm.C` to add the line `#include “GotIter.h”`
On my system this still isn’t enough, and I get some linker errors. By poking through the Makefile, I noticed I could fix it by commenting out the 5th line: `LDFLAGS=-static`.
Now run make and watch it work, hurrah! \o/
To see if GLARF itself works now, go to `$GLARF/commands-2010/run-glarf/` and run `make-all-glarf-a sample-files-a N`. You should get beautiful, beautiful GLARF output files.

GLARF

BLLIP

Compiling MegaM on MacOS X

2012-06-08T11:45:00+02:00

MegaM is Hal Daumé III’s maxent (logistic regression, and much more) modeling software written in OCaml. It is feature-packed and seems to be used a lot, despite being slightly dated. NLTK is able to use it.

In order to compile it as of 2012, with the current version of OCaml, I had to do some tricks that I would like to document here. It’s no big deal but it could save somebody precious minutes.

Download and unpack the gzip archive.
Install ocaml using macports: sudo port install ocaml. Note: this installed version 3.12.1_5, YMMV with newer versions later.
Point the compiler to the correct headers. First run ocamlc -where to find out the correct path. On my system it’s /opt/local/lib/ocaml/caml. Change the WITHCLIBS line (#73) in the Makefile to point there.
As of OCaml 3.12.0, the -lstr compiler flag should be replaced with -lcamlstr. It occurs on line #62 within the definition of WITHSTR.
Run make or make opt and enjoy.

Dynamically generated benchmarks with vbench

2012-06-07T01:57:00+02:00

To construct a vbench benchmark you need a setup string and a code string. The constructor’s signature is:

Benchmark(self, code, setup, ncalls=None, repeat=3, cleanup=None, name=None, description=None, start_date=None, logy=False).

Why generate benchmarks dynamically?¶

For most scikit-learn purposes, the code string will be very close to "algorithm.fit(X, y)", "algorithm.transform(X)" or "algorithm.predict(X)". We can generate a lot of benchmarks by changing what the algorithm is, and changing what the data is or the way it is generated.

A possible idea would be to create a DSL in which to specify scikit-learn tests and create benchmarks from them. However, before engineering such a solution, I wanted to test out how to generate three related benchmarks using different arguments for the dataset generation function.

This is what I came up with:

[sourcecode language=”python”]
from vbench.benchmark import Benchmark

_setup = “”“
from deps import *

kwargs = %s
X, y = make_regression(random_state=0, **kwargs)
lr = LinearRegression()
“”“

_configurations = [
(‘linear_regression_many_samples’,
{‘n_samples’: 10000, ‘n_features’: 100}),
(‘linear_regression_many_features’,
{‘n_samples’: 100, ‘n_features’: 10000}),
(‘linear_regression_many_targets’,
{‘n_samples’: 1000, ‘n_features’: 100, ‘n_targets’: 100})
]

_statement = “lr.fit(X, y)”

_globs = globals()
_globs.update({name: Benchmark(_statement, _setup % str(kwargs), name=name)
for name, kwargs in _configurations})

[/sourcecode]

It works perfectly, but I don’t like having to hack the globals to make the benchmarks detectable. This is because of the way the vbench suite gathers benchmarks. In __init__.py we have to do from linear_regression import *. With a small update to the detection method, we could replace the hacky part with a public lists of Benchmark objects.

Exposed issues¶

While working on this, after my first attempt, I was surprised to see that there were no results added to the database, and output plots were empty. It turns out that the generated benchmarks weren’t running, even though if I copied and pasted their source code from the generated html, it would run. Vbench was not issuing any sort of message to let me know that anything was wrong.

So what was the problem? My fault, of course, whitespace. But in all fairness, we should add better feedback.

This is what I was doing to generate the setup string:

[sourcecode lang=”python”]
def _make_setup(kwargs):
return “”“
from deps import *

kwargs = %s
X, y = make_regression(random_state=0, **kwargs)
lr = LinearRegression()
“”” % str(kwargs)
[/sourcecode]

It’s clear as daylight now that I overzealously indented the multiline string. But man, was it hard to debug! Also, in this example, the bug led to a refactoring that made the whole thing nicer and more direct. Hopefully, my experience with vbench will lead to some improvements to this cool and highly useful piece of software.

First contact with vbench

2012-05-29T12:57:00+02:00

With a slight delay caused by going to lovely lovely Istanbul for the LREC conference where I presented a poster, I am back to work on the Google Summer of Code project. By the way, this year’s logo and swag looks a lot nicer than last year’s, thank you Google!
[]
The backbone of my GSoC consists of putting together a continuous benchmark platform. I took a good look at vbench and spent an evening hacking Wes’s benchmarks suite config into something that will run on my machine. These are the key points I got from this experience.

vbench is, at least for the moment, very specific to Wes’ and Pandas’ needs. This is also because there weren’t so many other users that could have brought contributions.
Even though it has support for some configuration and automation, vbench seems largely suited for running on a local machine. Specifically, it is NOT designed to run continuously but in one-off runs, going back in git history and getting the last commit for each day, and running the benchmark with it. Of course, it is trivial to patch it into getting just one commit.
The code-as-strings approach is not ideal. The first thought is that it should be replaced with reading .py files into strings, but there are two issues with this:
1. One benchmark file can have a lot of setup code and several key lines that need to actually be benched. This can be fixed using convensions (ie. setup functions and bench_* functions) in the spirit of testing suites, or using decorators.
2. I would like to be able to run bench files as python scripts, but the vbench import system breaks this. This can be fixed by hijacking the imports when reading the file.

Our project has different dynamics than Pandas, so it’s important that the published results run on an independent machine, but it would be great if an individual developer can run the benchmark himself while coding but before pushing his changes upstream. Of course, his numbers would only be comparable to the numbers he gets on his own machine before his changes, but a developer shouldn’t wait for the daily benchmark for knowing if he made an improvement.

On the other hand there is unladen-swallow‘s benchmark system using the [perf.py] file. I didn’t try it out yet, so I would like feedback, but there are some key things that can be taken from them:

Memory usage benchmarking
Python scripts as benchmarks, with a simple but efficient Benchmark object hierarchy

What’s missing is:

A system to remember previous results and compare them, similar to vbench’s database
The ability to bench only an area of the code without rerunning the setup. (Not really sure whether vbench’s way is actually better)

At a first glance, it seems that a very good system can be obtained by combining these two excellent projects (or rather, improving vbench with features from perf.py). While I continue exploring this, I would like to hear feedback from people who had to do with similar issues. As for the GSoC timeline, I plan to join forces with Immanuel and design a solid benchmark suite for the linear models over the next 2 weeks.

[]: http://localhost:8001/wp-content/uploads/2012/05/P5280194.jpg

Support vector regression on Anscombe’s dataset

2012-05-27T21:59:00+02:00

Anscombe’s quartet is a set of four toy datasets that look very different, but many of their statistics coincide. They were developed by Francis Anscombe as a striking visual to show that even for small datasets, blindly examining their statistical properties without considering their structure can mislead.

Particularly, the four datasets have the same least squares regression line. While the second dataset is a clear example of a nonlinear correlation which cannot be accurately captured by any linear model, the third dataset is actually perfectly linear, with no noise, but just a single outlier that shifts the regression line considerably.

Support vector regression is an extension of the support vector machine idea to tackle the regression problem. It is based on the observation that a SVM classifier builds its decision boundary as a function of a (small) subset of training points. For regression, SVR fits a tube that is robust to noise within a width [latex]\epsilon[/latex]. For this particular example, using a small width makes the solution robust to the obvious outlier. For very small but non-zero [latex]\epsilon[/latex], the solution is a combination of the outlier and on two other points. For [latex]\epsilon=0[/latex], you can see that every point except a non-outlier is highlighted. This is actually the perfect solution but very dense.

[][]

Every frame displays the global mean squared error and the true mean squared error, i.e. over the inlying points. If the epsilon size is well chosen, SVR can perform robustly with a sparse solution. Since our interest was in avoiding the outlier, we assumed no noise in the inlying data, so a very small epsilon is perfect. For real data a larger epsilon is often useful because of variability in the data. When adding noise, SVR still manages to avoid the outlier, but when the tube width becomes zero, the solution is again very dense, very non-parametric.

[][]

Here is the code you can use to play around with this.

[gist id=2815589]

[]: http://localhost:8001/wp-content/uploads/2012/05/svr.gif

[]: http://localhost:8001/wp-content/uploads/2012/05/svr_noise.gif

GSoC 2012 proposal: Need for scikit-learn speed

2012-04-16T00:37:00+02:00

This summer I hope to be able to put in another full-time amount of effort into scikit-learn. After a successful Google Summer of Code project last year on dictionary learning, I now plan to do some low-level work. The title of my proposal is: “Need for scikit-learn speed” and, in a nutshell, will make the scikit go faster and will help it stay that way.

Scikit-learn has always enforced standards of quality that kept all implementations at a non-trivial level (i.e. faster than using the generic optimizers in scipy). However, not all modules are equal: some have received more attention for speed than others (for example the SGD classes). I intend to raise the bar towards a more uniform level.

Are you crazy, can you really do this?¶

Well, of course. This might not the usual GSoC proposal, but I can show how I can do it and how it’s easily quantifiable. Actually, a very important part of the work will be to make scikit-learn’s speed easily measurable.

As for the specific speed-ups, I have shown in the past that I can do algorithmic and memory layout optimizations in numerical code. There are parts in the scikit-learn that can benefit from such work: for example only recently Peter merged this pull request significantly improving SGDClassifier’s test time performance by switching the memory layout of the coefficients: they were laid out optimally for the training phase, not for the prediction phase.

There are certainly more opportunities for such speed improvements in the scikit. Of course there is a lot of code that can’t reasonably be made any faster (I have a feeling that SGDClassifier is at the moment such a case, but we can’t know for sure without heavy profiling). But generally there are many speed fixes that could weigh a lot: for example, a Cython implementation of the euclidean_distances function that is able to use preallocated memory will improve the performance of raw NearestNeighbours queries as well as of the KMeans and hierarchical clustering algorithms.

How will we be able to tell if you succeed?¶

A key part of the GSoC project is setting up a CI-style benchmark platform. The point is to be able to track how the speed of certain operations evolves in time. For such purposes, Wes McKinney developed the vbench project, introduced in this blog post. The goal is for every scikit-learn module to have several such benchmarks, for differently shaped and structured data.

Having such a benchmark suite available is the equivalent of a test suite, in terms of performance. It makes developers be extra conscious of the effect of their changes. It also makes it more fun to chase speed improvements, thanks to the positive reinforcement it gives.

There are some static benchmarks comparing the performance of scikit-learn algorithms with other well-known libraries in the ml-benchmarks project. It would be very helpful to have such a benchmark suite that automatically keeps up-to-date.

Side effects¶

The cool thing about such a project is that it should raise the overall quality of the scikit. The refactoring will lead to an increase in test coverage, because the low-coverage modules are expected to be less optimized as well. Also, the benchmarks will lead to well-backed summaries in the documentation, such as the one recently added in the clustering section.

Since the scikit is reaching a state where many well-known algorithms are available, the 1.0 release is slowly approaching. My Google Summer of Code project should bring the scikit significantly closer to that milestone.

Romanian people and coffee

2012-04-13T21:20:00+02:00

So I got my hands of the Google N-gram data for the Romanian language. It’s noisy as hell, has some other subtle issues too, but here’s the first thing I noticed:

The Romanian word for coffee is cafea, and the more you crave it, the longer you pronunce the final a: I really need some cafeaaaa right now.

Thanks to Google, here are the numbers:

[![Distribution of the length of the final letter in the Romanian word for coffee.][]][]

Post scriptum: I hope you like the theme: I installed the sane matplotlib color scheme from Huy Nguyen.

[![Distribution of the length of the final letter in the Romanian word for coffee.][]]: http://localhost:8001/wp-content/uploads/2012/04/cafeaaa.png

Nash-Williams theorem on the Hamiltonian property of some regular graphs

2012-01-29T22:31:00+01:00

I have been digging on the internet for the proof of this theorem for the last couple of days without success. The result was published by Sir Crispin Nash-Williams as Valency Sequences which force graphs to have Hamiltonian Circuits. Interim Rep, University of Waterloo Res Rep., Waterloo, Ontario, 1969. However, this old paper is unavailable online but I have a proof in some lecture notes from my class, that I want to share here.

Theorem. Let [latex]G=(V, E)[/latex] be an [latex]n[/latex]-regular graph with [latex]|V| = 2n + 1[/latex]. Then, [latex]G[/latex] is Hamiltonian.

Proof. We first remark that [latex]n[/latex] must be even, since $$\sum_{x \in V} d(x) = n(2n + 1) = 2|E|$$ We might try to apply Dirac’s theorem, which would give us a Hamiltonian cycle if [latex] \forall x \in V, d(x) \geq \frac{|V|}{2}[/latex]. But in the current case, [latex]\forall x \in V, d(x) = n \< \frac{2n+1}{2}[/latex].

So we force Dirac by adding an extra vertex [latex]w[/latex] and connecting it to all [latex] x \in V [/latex]. In this new graph [latex]G’[/latex], [latex]d(x) = n + 1 \forall x \in V[/latex] and [latex]d(w) = 2n + 1[/latex]. Therefore we have a Hamiltonian cycle that passes through [latex]w[/latex] and in which, [latex]w[/latex] is adjacent to two vertices [latex]x[/latex] and [latex]y \in V[/latex]. Therefore this cycle induces a Hamiltonian path in [latex]G[/latex]: $$P = [x = v_0, v_1, …, v_{2n-1}, v_{2n}=y] $$

Suppose that [latex]G[/latex] is not Hamiltonian. It follows that if [latex] v_0v_i \in E [/latex], then [latex] v_{i-1}v_{2n} \notin E[/latex] and also that if [latex] v_0v_i \notin E [/latex], then [latex] v_{i-1}v_{2n} \in E[/latex].

We have two cases. If [latex]v_0[/latex] is adjacent to [latex]v_1, …, v_n[/latex] then it follows that [latex]v_{2n}[/latex] is adjacent to [latex]v_n, v_{n+1}, …, v_{2n-1}[/latex], since it cannot be adjacent to any [latex]v_i, i \< n[/latex] without creating a Hamiltonian cycle. But in this case, in the graph induced by the first half [latex]G[\{v_0, v_1, … v_n\}][/latex], [latex]v_n[/latex] cannot be adjacent to all the others, since in [latex]G[/latex] it has degree [latex]n[/latex] and it already has [latex]2[/latex] outgoing edges. So there is at least one vertex [latex]v_i, i \< n[/latex] that isn’t adjacent to it, which means [latex]v_i[/latex] is adjacent to some [latex]v_j, j > n[/latex], thus forming a Hamiltonian cycle.

In the second case, we have a vertex [latex]v_i, 2 \leq i \leq 2n - 1[/latex] such that [latex]v_0v_i \notin E[/latex] and [latex]v_0v_{i+1} \in E[/latex]. This also means that [latex]v_{i-1}v_{2n} \in E[/latex].

We therefore have a cycle of length [latex]2n[/latex] in [latex]G[/latex] that excludes [latex]v_i[/latex]. Let’s rename this cycle [latex]C=[y_1, y_2, …, y_{2n}, y_1][/latex] and [latex]v_i=y_0[/latex].

[latex]y_0[/latex] cannot be adjacent to two consecutive vertices [latex]y_i[/latex] and [latex]y_{i+1}[/latex] because this will give a Hamiltonian cycle. But we know that [latex]deg(y_0) = n[/latex]. It follows that it’s adjacent to all of the even or odd numbered vertices. We assume the latter, without loss of generality. Let [latex]2k[/latex] be some even index. Notice that we have [latex]\{y_0y_{2k-1}, y_0y_{2k+1}\} \subset E[/latex] and we can follow the cycle [latex]C[/latex] from [latex]y_{2k+1}[/latex] all the way back to [latex]y_{2n-1}[/latex] giving us a new cycle [latex]C’ = [y_1, y_2, …, y_{2n-1}, y_0, y_{2k+1}, …, y_{2n}, y_1][/latex] also of length [latex]2n[/latex]. So by repeating the same reasoning for every even vertex, by placing it in the middle and building a cycle around it, it follows that every even vertex is adjacent to all the odd vertices. But there are [latex]n+1[/latex] even indices, so it follows that the degree of any odd vertex is at least [latex]n+1[/latex], contradicting the initial conditions of the theorem. [latex]\square[/latex]

Moving out

2012-01-06T00:16:00+01:00

Happy new year, friends!

I’ve made a New Year’s resolution to build a better web presence and make better use of the domain that I previously only used for mail.

This has prompted me to move my blog over to http://localhost:8001 which hopefully is shorter, better, faster and stronger!

The nasty bug crawling in my Orthogonal Matching Pursuit code

2011-11-18T20:51:00+01:00

A while back, Bob L. Sturm blogged about a similar implementation of OMP to the one in scikit-learn. Instead of using the Cholesky decomposition like we did, his Matlab code uses the QR decomposition, to a similar (or maybe even identical) outcome, in theory. So lucky that Alejandro pointed out to him the existence of the scikit-learn implementation, and that Bob’s code exposed a bug that all the test coverage didn’t catch! This plot should increase, certainly not decrease! Something is clearly wrong here.

Luckily we were able to find it and fix it very quickly. I have updated the old entries I wrote on the OMP optimizations, so they no longer include the bug. But I take this opportunity to explain what exactly went wrong.

A key part of the optimization was that slicing out arbitrary columns out of an array is slow when they are passed to BLAS functions like matrix multiplication. In order to make the most out of your code, the data should have a contiguous layout. We achieved this by swapping active dictionary atoms (columns) to the beginning of the array.

Something that can happen, but won’t happen very often, is that after an atom is selected as active, the atom that takes its place after swapping needs to be selected. This is rare because dictionaries have many columns, out of which only very very few will be active. But when it happens, because the code didn’t keep track of swapped indices, the corresponding coefficient of the solution would get updated twice, leading to more zero entries than we should have. A keen eye could have noticed that the first `n_nonzero_coefs` entries in OMP solution vectors were never non-zero. But alas, my eye was not a keen one at all.

In other words, the following test (that was written after the bug was found, unfortunately) was failing:
[sourcecode lang=”Python”]
def test_swapped_regressors():
gamma = np.zeros(n_features)
# X[:, 21] should be selected first, then X[:, 0] selected second,
# which will take X[:, 21]’s place in case the algorithm does
# column swapping for optimization (which is the case at the moment)
gamma[21] = 1.0
gamma[0] = 0.5
new_y = np.dot(X, gamma)
new_Xy = np.dot(X.T, new_y)
gamma_hat = orthogonal_mp(X, new_y, 2)
gamma_hat_gram = orthogonal_mp_gram(G, new_Xy, 2)
# active indices should be [0, 21], but prior to the bugfix
# the algorithm would update only [21] but twice
assert_equal(np.flatnonzero(gamma_hat), [0, 21])
assert_equal(np.flatnonzero(gamma_hat_gram), [0, 21])
[/sourcecode]

Note that this bug has been fixed for a while, but I didn’t get the free time to write this post until now. Good news is: we fixed it, and did so very quickly after the report. So you can still trust me, I guess!

Sampling Gamma random variates through the ratio-of-uniforms method

2011-10-09T15:40:00+02:00

One year ago I had the chance to take a class on Monte Carlo simulation with prof. Ion Văduva, and my assignment for the class was to implement exactly what it says in the title of the blog post. I am going to walk you through the idea behind this.

General formulation¶

The ratio-of-uniforms is a method that can be applied to many density functions. Essentially, given a density function over [latex] \mathbb{R}\^m[/latex], [latex] f(x) = \frac{h(x)}{H}[/latex] where [latex]H[/latex] is a normalization constant (ie. [latex] h(x) \geq 0[/latex], [latex] H = \int h(x)dX[/latex]). Given a parameter [latex] c > 0 [/latex] and a parametrization [latex]\phi[/latex] from [latex] \mathbb{R}\^{m+1}[/latex] to [latex] \mathbb{R}\^{m}[/latex] expressed as: $$ \phi(v_0, v_1, …, v_m) = \left ( \frac{v_1}{v_0\^c}, \frac{v_2}{v_0\^c}, …, \frac{v_m}{v_0\^c} \right )$$
Define the set [latex] \mathcal{C} = \{\mathbf{v} \big | \gamma(\mathbf{v}) \leq 0, v_0 > 0\} \in \mathbb{R}\^{m+1}[/latex] where
$$\gamma(\mathbf{v}) = \log v_0 - \frac{\log h(\phi(v_0, v_1, …, v_m))}{mc + 1}$$ If [latex] \mathcal{C}[/latex] is bounded and we sample a uniform vector [latex] \mathbf{V} \sim \text{Uniform}(\mathcal{C})[/latex] then [latex] \phi(\mathbf{V}) \sim f(x)[/latex]. Also note that the measure (volume) of the set [latex] \mathcal{C}[/latex] is [latex] \frac{H}{mc + 1}[/latex]. I do not have any references for the proof, except for a book in Romanian, but if you are interested, just leave me a comment and I’ll do a follow-up post with the proofs.

Univariate scenario¶

For the univariate case, all the above simplifies to $$ \mathcal{C} = \left \{(u, v) \Big | 0 \< u \< \sqrt
{h\left (\frac{v}{u\^c}\right )} \right \} $$. We generate [latex] (U, V) \sim \text{Uniform}(\mathcal{C})[/latex] and take [latex] \frac{V}{U\^c} \sim f(x)[/latex].
Since we are looking at the (univariate) Gamma distribution, described by: $$ f(x; \nu, \theta) = \frac{x\^{\mu - 1} \exp(\frac{-x}{\theta})}{\theta\^k\Gamma(k)}$$ [latex] \nu[/latex] is the shape parameter and [latex] \theta[/latex] is the scale parameter.
But because of the property that if [latex] X \sim \text{Gamma}(\nu, \theta)[/latex], then for any [latex] k > 0[/latex], [latex] kX \sim \text{Gamma}(\nu, k\theta)[/latex], we conclude that we can fix [latex] \theta[/latex] to 1 without loss of generality. Replacing in the style of the definition in the previous section, we have [latex] h(x; \nu) = x\^{\nu-1}e\^{-x}[/latex] and [latex] H_\nu = \Gamma(\nu)[/latex].
This allows us to compute the equation of the boundary of the set [latex] \mathcal{C}[/latex] which ends up being described by [latex]\gamma(u, v) = \log{u} - \frac{\nu - 1}{c + 1} \log{\left(\frac{v}{u\^c}\right)} + \frac{1}{c+1} \frac{v}{u\^c}[/latex]. For visualisation purposes, here is how it would look like for [latex] \nu=6, c=1[/latex] (plotted using Wolfram Alpha):[]

Sampling algorithm¶

In order to uniformly sample from this set, we can apply basic rejection sampling: just uniformly sample from a rectangular region surrounding the set, and reject the points that do not satisfy the condition. In order to do this as efficiently as possible, we need to compute the minimal bounding box, which can be done by solving a couple of optimization problems using Lagrange multipliers and the KKT conditions. Also by looking closely at the image, you can see that the lower left corner is exactly the origin: this turns out not to be a coincidence. I won’t go into detail here, but here are the bounds I derived:
$$ 0 \< u \< (\nu - 1)\^\frac{\nu - 1}{c + 1} e \^ {-\frac{\nu - 1}{c + 1}} \text{ and } 0\< v \< \left(\frac{c\nu + 1}{c}\right)\^{\frac{c\nu + 1}{c + 1}} e \^ {- \frac {c\nu + 1}{c+1}}$$

The probability of acceptance (which can be seen as the efficiency) of the rejection sampling method is given by the ratio of the areas of the set [latex] \mathcal{C}[/latex] and the bounding box. The larger this probability, the less points we throw away and the more efficient the algorithm is. Using the values derived above, this probability is: $$ p(\nu, c) = \frac{\Gamma(\nu)e\^{\nu}}{(c+1) (\nu - 1)\^{\frac{\nu - 1}{c + 1}} \left(\frac{c\nu + 1}{c}\right)\^{\frac{c\nu + 1}{c + 1}}}$$

Personally I got stumped here. The idea would be to determine the ideal [latex] c[/latex] for a given [latex] \nu[/latex] in order to maximize the probability, but I didn’t manage to do it (I leave it as an exercise for the reader ;)). Anyway, this is enough to proceed with an implementation, so I’m gonna give the Python code for it. Note that I used the name k for the shape parameter instead of [latex] \nu[/latex]. Also note that the case when [latex] 0 \< \nu \< 1[/latex] needed to be treated separately, which I did using the following property: Let [latex] \nu \in (0, 1)[/latex]. If [latex] X’ \sim \text{Gamma}(1+\nu, 1), U \sim \text{Uniform}(0, 1)[/latex] then $$ X = X’ \cdot \sqrt[\nu]{U} \sim \text{Gamma}(\nu, 1)$$ For a proof of this fact, see [1], which is a great article on generating Gamma variates.

Implementation¶

[sourcecode language=”python”]
from import numpy as np

def _cond(u, v, k, c):
“”“Identity function describing the acceptance region”“”
x = v / u ** c
return (c + 1) * np.log(u) \<= (k - 1) * np.log(x) - x

def vn_standard_gamma(k, c=1.0, rng=np.random):
“”“Generates a single standard gamma random variate”“”
if k \<= 0:
raise ValueError(“Gamma shape should be positive”)
elif k \< 1:
return vn_standard_gamma(1 + k, c, rng) * rng.uniform() ** (1 / k)
elif k == 1:
return rng.standard_exponential()
else:
a, b = get_bounds(k, c)
while True:
u, v = rng.uniform(0, a), rng.uniform(0, b)
if _cond(u, v, k, c):
break;
return v / u ** c

def vn_gamma(k, t, shape=1, c=1.0, rng=np.random):
“”“Vectorized function to generate multiple gamma variates”“”
generator = lambda x: t * vn_standard_gamma(k, c, rng)
generator = np.vectorize(generator)
return generator(np.empty(shape))

def get_bounds(k, c=1.0):
“”“Computes the minimal upper bounds surrounding the acceptance region”“”
a = ((k - 1) / np.e) ** ((k - 1) / (c + 1))
b = ((c * k + 1) / (c * np.e)) ** ((c * k + 1) / (c + 1))
return a, b

def prob_acc(k, c=1.0):
“”“Calculates the probability of acceptance for the given parameters”“”
from scipy.special import gamma
a, b = get_bounds(k, c)
return gamma(k) / ((c + 1) * a * b)
[/sourcecode]

Results¶

And of course I should show you that it works. Here are some histograms for various values of [latex] \nu[/latex], with the theoretical density plotted in dotted red, after sampling [latex] 10\^5[/latex] values. The y-axis is the frequency (sorry for labeling in Romanian), and for the red dotted line it can be interpreted as the theoretical probability. You can clearly see the goodness of fit is excellent.

[][][]

1: George Marsaglia and Wai Wan Tsang. 1998. The Monty Python method for generating random variables. ACM Trans. Math. Softw. 24, 3 (September 1998), 341-350. DOI=10.1145/292395.292453

[]: http://localhost:8001/wp-content/uploads/2011/10/regiunea.png

[]: http://localhost:8001/wp-content/uploads/2011/10/hist1.png

[]: http://localhost:8001/wp-content/uploads/2011/10/hist2.png

[]: http://localhost:8001/wp-content/uploads/2011/10/hist3.png

RANLP 2011 in Hissar, BG

2011-09-20T14:17:00+02:00

Last week was marked by the international RANLP (Recent Advances in Natural Language Processing) conference, taking place in a nice spa in Hissar, Bulgaria. The excellent folks from the computational linguistics group at the University of Wolverhampton were behind it, together with the Institute of Information and Communication Technologies from the Bulgarian Academy of Sciences.
[caption id=”attachment_247” align=”alignleft” width=”225” caption=” “][]

A fountain with warm mineral spring water in Hissarya.
Picture by Miranda

[/caption]
I must begin by thanking them: the organization was impeccable! I’m not sure, but I think that at one point Ivelina was even running around buying routers to improve wifi coverage (which is already spectacular in Bulgaria — I’ve received reports from Miranda that you can get wifi in the mountains!)

The schedule was busy, with three tracks going in parallel, in order to cover a wide range of topics in computational linguistics. The student workshop should also be noted for the excellent quality of the works there.

Of course it would be infeasible to write about all the great people I met and their high quality work. And if I were to write about all the fun we had, it would probably make this post look unprofessional :). This doesn’t mean I forgot about any of you, and as soon as I get the chance to work on something related, I will most certainly write about it, and you.

So, if I would have to summarize the trends and the ideas stated during the conference and especially during the keynotes, I would say:

When talking about word sense disambiguation, it’s wrong to speak about the different meanings of a word, but rather about the potential a word has for bringing a certain meaning in a certain context. See Patrick Hanks‘ Corpus Pattern Analysis. Without something like this, to have good WSD you need to heavily adjust the overlapping meanings from a Wordnet-style ontology.
Certain relations, such as temporal and spacial ones, can naturally be modeled by complex domain-specific logics (see Inderjeet Mani‘s new book, Interpreting Motion: Grounded Representations for Spatial Language, that is due for publishing). But these only appear in a small subset of human communication. The attempt to map human language to a complete logic in which to do general-purpose inference seems futile: Ido Dagan suggests textual entailment: reasoning directly in natural language, and only abstracting away to a formal logic system when need arises.
If you have a large enough sample of n-gram frequency data, you can eventually beat the performance you can get with a limited amount of labeled data, and most importantly: it generalizes much better when going out of the domain you trained on. Apparently the best tool for this at the moment is the Google n-gram data, which has some limitations. In time, we can easily extend this data by huge amounts by mining n-grams from Wikipedia (which allegedly has a higher count of distinct n-grams than the Google dataset), and more importantly, by aligning multi-language data, making use of transliterations and cognate identification.

Please note that I may be (or most probably, am) ignorant of older instances of similar ideas, and I may have misunderstood certain claims. Please feel free to discuss in the comments whether you think I forgot about something important, or whether I am plain wrong about something. In particular, I seem to have been completely ignorant of the existence of the Google n-gram data, which has been around for quite a while, so I must have missed other important things as well.

Take care, kind readers, and express your opinion!
V

[]: http://localhost:8001/wp-content/uploads/2011/09/319030_10150303323328171_677848170_8151131_1046590975_n1.jpg

Dictionary learning in scikit-learn 0.9

2011-09-19T19:15:00+02:00

Thanks to Olivier, Gaël and Alex, who reviewed the code heavily the last couple of days, and with apologies for my lack of activity during a sequence of conferences, Dictionary learning has officially been merged into scikit-learn master, and just in time for the new scikit-learn 0.9 release. Here are some glimpses of the examples you can run for yourself:

[][]

[![Image denoising with Dictionary learning and Orthogonal matching pursuit][]][]

The stars of this new release are: the manifold learning module by Jake Vanderplas and Fabian Pedregosa, the Dirichlet process gaussian mixture model by Alexandre Passos, and many others, as you can see from the development changelog (as soon as the release is made, I will update this post with permanent links).

The release is due tomorrow. I will also be in charge with building the Windows installers for this release, let’s hope I do a good job and you can think of me and smile when installing!

[]: http://localhost:8001/wp-content/uploads/2011/09/plot_image_denoising_1.png

[]: http://localhost:8001/wp-content/uploads/2011/09/plot_image_denoising_24.png

[![Image denoising with Dictionary learning and Orthogonal matching pursuit][]]: http://localhost:8001/wp-content/uploads/2011/09/plot_image_denoising_44.png

Long overdue update. EuroScipy and SSLST 2011

2011-09-05T00:08:00+02:00

Anybody reading my blog should have expected me to blog about the end of my GSoC. Sorry to disappoint, but I simply did not experience anything similar to an ending. On the contrary, I feel like things have barely started. Also, I apologize for one of the few posts here without pretty pictures! :)

For the last two weeks, I’ve been traveling. I attended the EuroScipy conference thanks to Fabian, who offered me a place to sleep during the week. We sprinted hard, we discussed tricky APIs, we drank a lot of coffee, beer, and ate well in lovely Paris. It was great to meet all of the celebrities, the people who keep the scientific Python globe turning.

Many thanks to Gael and Emmanuelle, who worked very, very hard on organizing everything, so they weren’t around and I didn’t get to say goodbye when I ran to catch my plane last Sunday.

I was in a hurry, heading to Tarragona, a beautiful city on the Catalan coast, where the public university organized the 2011 summer school in linguistics and speech technologies (SSLST). This was a great opportunity to meet many fellow young researchers working in computational linguistics. I will not go into details now, because I plan expand on this, but I would like to state a couple of things. Firsty, even though NLP seems to be mostly a Java-dominated affair (note for example Stanford’s NLP toolkit and Sheffield’s GATE), the [computational linguistics and psycholinguistics research center (CLiPS)][] at the University of Antwerp actually briefly manifested its devotion to Python and NLTK via its research director, Walter Daelemans.

It was good to see a little love for Python in this field. NLTK is very underrepresented in the SciPy community, I couldn’t find anybody at the EuroScipy conference knowing too much about it or about the people behind it.

Another lab that has done a lot of cool work is Cental at the Catholic University of Louvain, and they also use Python for natural language processing. Maybe in the coming years, we will see a Python for Computational Linguistics sattelite, along with Physics and Neuroscience. Doesn’t it sound more fun? :P

Secondly, I wish SSLST were organized by someone like Gael! As the discussion at dinner regarding who will organize next year’s EuroScipy went, it is imperative that the organizers be actively involved in the community, and generally passionate about it. Even though I’m comparing apples and oranges, Carlos Martin-Vide behaved in this context like a old, tired, emotionless academic, not taking into account even lunch breaks for the whole group, not to mention any sort of getting together or even a group photo (which, alas, we were not able to take, apart from small groups.) They said it couldn’t be done. Of course it could, they just didn’t want it hard enough.

Finally, before signing off, I would like to announce that because the Romanian Ministry of Education failed to specify the allocated number of public positions for masters’ programmes, the admission exam at the University of Bucharest will be delayed by a couple of weeks. Luckily, this will allow me to attend RANLP 2011 in Hissar, Bulgaria a week from now, where I will present my poster entitled:
“Can alternations be learned? A machine learning approach to Romanian verb conjugation” by Liviu P. Dinu, Emil Ionescu, Vlad Niculae and Octavia-Maria Sulea. See you in Hissar!

Optimizing Orthogonal Matching Pursuit code in Numpy, part 2

2011-08-11T19:39:00+02:00

EDIT: There was a bug in the final version of the code presented here. It is fixed now, for its backstory, check out my blog post on it.

When we last saw our hero, he was fighting with the dreaded implementation of least-angle regression, knowing full well that it was his destiny to be faster.

We had come up with a more robust implementation, catching malformed cases that would have broken the naive implementation, and also it was orders of magnitude faster than said implementation. However, our benchmark [1] showed that it was a couple of times slower than least-angle regression.

By poking around the scikits.learn codebase, I noticed that there is a triangular system solver in scikits.learn.utils.arrayfuncs. Unlike the scipy.linalg one, this one only works with lower triangular arrays, and it forcefully overwrites b. Even though if weren’t faster, it should still be used: scikits.learn aims to be as backwards-compatible with SciPy as possible, and linalg.solve_triangular was added in 0.9.0. Anyway, let’s just see whether it’s faster:

[sourcecode language=”python”]
In 1: import numpy as np

In 2: from scipy import linalg

In [3]: from scikits.learn.datasets import make_spd_matrix

In [4]: from scikits.learn.utils.arrayfuncs import solve_triangular

In [5]: G = make_spd_matrix(1000)

In [6]: L = linalg.cholesky(G, lower=True)

In [7]: x = np.random.randn(1000)

In [8]: y = x.copy()

In [9]: timeit solve_triangular(L, x)
100 loops, best of 3: 3.45 ms per loop

In [10]: timeit linalg.solve_triangular(L, y, lower=True, overwrite_b=True)
10 loops, best of 3: 134 ms per loop
[/sourcecode]

Wow! That’s 40x faster. We’re catching two rabbits with one stone here, let’s do the change! Notice that we can just copy [latex] \mathbf{v}[/latex] into the appropriate place in [latex] L[/latex] and then solve in place.

But whoops! When solving the [latex] LL’[/latex] system, we take advantage of the transpose attribute in linalg.solve_triangular, which the scikits.learn version does not expose. We could think of a solution, but here’s a better idea: Shouldn’t there be some way to directly solve the entire system in one go?

Well, there is. It is an LAPACK function by the name of potrs. If you are not aware, LAPACK is a Fortran library with solvers for various types of linear systems and eigenproblems. LAPACK along with BLAS (on which it is based) pretty much powers all the scientific computation that happens. BLAS is an API with multiple implementations dating from 1979, while LAPACK dates from 1992. If you ever used Matlab, this is what was called behind the scenes. SciPy, again, provides a high-level wrapper around this, the linalg.cho_solve function.

But SciPy also gives us the possibility to import functions directly from LAPACK, through the use of linalg.lapack.get_lapack_funcs. Let’s see how the low-level LAPACK function compares to the SciPy wrapper, for our use case:

[sourcecode language=”python”]
In [11]: x = np.random.randn(1000)

In [12]: y = x.copy()

In [13]: timeit linalg.cho_solve((L, True), x)
1 loops, best of 3: 95.4 ms per loop

In [14]: potrs, = linalg.lapack.get_lapack_funcs((‘potrs’,), (G,))

In [15]: potrs
Out[15]: <fortran object>

In [16]: timeit potrs(L, y)
100 loops, best of 3: 9.49 ms per loop
[/sourcecode]

That’s 10 times faster! So now we found an obvious way to optimize the code:

[sourcecode language=”python”]
def cholesky_omp(X, y, n_nonzero_coefs, eps=None):
min_float = np.finfo(X.dtype).eps
potrs, = get_lapack_funcs((‘potrs’,), (X,))
alpha = np.dot(X.T, y)
residual = y
n_active = 0
idx = []

max_features = X.shape1 if eps is not None else n_nonzero_coefs
L = np.empty((max_features, max_features), dtype=X.dtype)
L[0, 0] = 1.

while 1:
lam = np.abs(np.dot(X.T, residual)).argmax()
if lam < n_active or alpha[lam] ** 2 < min_float:
# atom already selected or inner product too small
warn(“Stopping early”)
break
if n_active > 0:
# Updates the Cholesky decomposition of X’ X
L[n_active, :n_active] = np.dot(X[:, idx].T, X[:, lam]
solve_triangular(L[:n_active, :n_active], L[n_active, :n_active])
d = np.dot(L[n_active, :n_active].T, L[n_active, :n_active])
if 1 - d <= min_float: # selected atoms are dependent
warn(“Stopping early”)
break
L[n_active, n_active] = np.sqrt(1 - d)
idx.append(lam)
# solve LL’x = y in two steps:
gamma, _ = potrs(L[:n_active, :n_active], alpha[idx], lower=True,
overwrite_b=False)
residual = y - np.dot(X[:, idx], gamma)
if eps is not None and np.dot(residual.T, residual) <= eps:
break
elif n_active == max_features:
break
return gamma, idx
[/sourcecode]

Woohoo! But we still lag behind. Now that we delegated the trickiest parts of the code to fast and reliable solvers, it’s time to use a profiler and see what the bottleneck is now. Python has excellent tools for this purpose. What solved the problem in this case was line_profiler [2]. There is a great article by Olivier Grisel here 2 regarding how to use these profilers. I’m just going to say that line_profiler‘s output is very helpful, basically printing the time taken by each line of code next to that line.

Running the profiler on this code, we found that 58% of the time is spent on line 14, 20.5% on line 21, and 20.5% on line 32, with the rest being insignificant (potrs takes 0.1%!). The code is clearly dominated by the matrix multiplications. By running some more timings with IPython, I found that multiplying such column-wise views of the data as X[:, idx] is considerably slower then multiplying a contiguous array. The least-angle regression code in scikits.learn avoids this by swapping columns towards the front of the array as they are chosen, so we can replace X[:, idx] with X[:, :n_active]. The nice part is that if the array is stored in Fortran-contiguous order (ie. column contiguous order, as opposed to row contiguous order, as in C), swapping two columns is a very fast operation!. Let’s see some more benchmarks!

[sourcecode language=”python”]
In [17]: X = np.random.randn(5000, 5000)

In [18]: Y = X.copy(‘F’) # fortran-ordered

In [19]: a, b = 1000, 2500

In [20]: swap, = linalg.get_blas_funcs((‘swap’,), (X,))

In [21]: timeit X[:, a], X[:, b] = swap(X[:, a], X[:, b])
100 loops, best of 3: 6.29 ms per loop

In [22]: timeit Y[:, a], Y[:, b] = swap(Y[:, a], Y[:, b])
10000 loops, best of 3: 111 us per loop
[/sourcecode]

We can see that using Fortran-order takes us from the order of miliseconds to the order of microseconds!

Side note: I almost fell into the trap of swapping columns the pythonic way. That doesn’t work:
[sourcecode language=”python”]
In [23]: X[:, a], X[:, b] = X[:, b], X[:, a]

In [24]: np.testing.assert_array_equal(X[:, a], X[:, b])

In [25]:
[/sourcecode]

However this trick works great for swapping elements of one-dimensional arrays.

Another small optimization that we can do: I found that on my system, it’s slightly faster to compute the norm using the BLAS function nrm2. So by putting all of these together, we end up with the final version of our code:

[sourcecode language=”python”]
def cholesky_omp(X, y, n_nonzero_coefs, eps=None, overwrite_X=False):
if not overwrite_X:
X = X.copy(‘F’)
else: # even if we are allowed to overwrite, still copy it if bad order
X = np.asfortranarray(X)

min_float = np.finfo(X.dtype).eps
nrm2, swap = linalg.get_blas_funcs((‘nrm2’, ‘swap’), (X,))
potrs, = get_lapack_funcs((‘potrs’,), (X,))

indices = range(len(Gram)) # keeping track of swapping
alpha = np.dot(X.T, y)
residual = y
n_active = 0

max_features = X.shape1 if eps is not None else n_nonzero_coefs
L = np.empty((max_features, max_features), dtype=X.dtype)
L[0, 0] = 1.

while True:
lam = np.abs(np.dot(X.T, residual)).argmax()
if lam < n_active or alpha[lam] ** 2 < min_float:
# atom already selected or inner product too small
warn(“Stopping early”)
break
if n_active > 0:
# Updates the Cholesky decomposition of X’ X
L[n_active, :n_active] = np.dot(X[:, :n_active].T, X[:, lam])
solve_triangular(L[:n_active, :n_active], L[n_active, :n_active])
v = nrm2(L[n_active, :n_active]) ** 2
if 1 - v <= min_float: # selected atoms are dependent
warn(“Stopping early”)
break
L[n_active, n_active] = np.sqrt(1 - v)
X.T[n_active], X.T[lam] = swap(X.T[n_active], X.T[lam])
alpha[n_active], alpha[lam] = alpha[lam], alpha[n_active]
indices[n_active], indices[lam] = indices[lam], indices[n_active]
n_active += 1
# solves LL’x = y as a composition of two triangular systems
gamma, _ = potrs(L[:n_active, :n_active], alpha[:n_active], lower=True,
overwrite_b=False)

residual = y - np.dot(X[:, :n_active], gamma)
if eps is not None and nrm2(residual) ** 2 <= eps:
break
elif n_active == max_features:
break

return gamma, indices[:n_active]
[/sourcecode]

Now, the benchmark at [1] indicates victory over least-angle regression! I hope you have enjoyed this short tour. See you next time!

1[]
2[]

Optimizing Orthogonal Matching Pursuit code in Numpy, part 1

2011-08-07T20:50:00+02:00

After intense code optimization work, my implementation of OMP finally beat least-angle regression! This was the primary issue discussed during the pull request, so once performance was taken care of, the code was ready for merge. Orthogonal matching pursuit is now available in scikits.learn as a sparse linear regression model. OMP is a key building block of the dictionary learning code that we are working on merging.

I will go through the process of developing this particular piece of code as an example of code refining and iterative improvements, as well as for the useful notes it will provide on optimizing numerical Python code. In the first part we will see how the code got from pseudocode state to a reasonably efficient code with smart memory allocation. In the next part we will see how to make it blazing fast by leveraging [1] lower level BLAS and LAPACK routines, and how to use profiling to find hot spots.

As stated before, orthogonal matching pursuit is a greedy algorithm for finding a sparse solution [latex] \gamma[/latex] to a linear regression problem [latex] X\gamma = y[/latex]. Mathematically, it approximates the solution of the optimization problem:

$$ \text{argmin} {\big|\big|} \gamma {\big|\big|} _0 \text{ subject to }{\big |\big|}y-X\gamma{\big|\big|}_2\^2 \leq \epsilon $$
or (under a different parametrization):
$$\text{argmin} {\big |\big|}y - X\gamma{\big |\big|}_2\^2 \text{ subject to } {\big|\big|}\gamma{\big|\big|}_0 \leq n_{\text{nonzero coefs}}$$

In the code samples in this post I will omit the docstrings, but I will follow the notation in the formulas above.

Important note: The regressors/dictionary atoms (the columns of [latex] X[/latex]) are assumed to be normalized throughout this post (as well as usually any discussion of OMP). We also assume the following imports:
[sourcecode language=”Python”]
import numpy as np
from scipy import linalg
[/sourcecode]

Orthogonal matching pursuit is a very simple algorithm in pseudocode, and as I stated before, it almost writes itself in Numpy. For this reason, instead of stating the pseudocode here, I will start with how naively implemented OMP looks like in Python:

[sourcecode language=”Python”]
def orthogonal_mp(X, y, n_nonzero_coefs, eps=None):
residual = y
idx = []
if eps == None:
stopping_condition = lambda: len(idx) == n_nonzero_coefs
else:
stopping_condition = lambda: np.inner(residual, residual) <= eps
while not stopping_condition():
lam = np.abs(np.dot(residual, X)).argmax()
idx.append(lam)
gamma, _, _, _ = linalg.lstsq(X[:, idx], y)
residual = y - np.dot(X[:, idx], gamma)
return gamma, idx
[/sourcecode]

Using lambda expressions as stopping conditions never looked like a brilliant idea, but it seems to me like the most elegant way to specify such a variable stopping condition. However, the biggest slowdown in this is the need for solving a least squares problem at each iteration, while least-angle regression is known to produce the entire regularization path for the cost of a single least squares problem. We will also see that this implementation is more vulnerable to numerical stability issues.

In [2], Rubinstein et al. described the Cholesky-OMP algorithm, an implementation of OMP that avoids solving a new least squares problem at each iteration by keeping a Cholesky decomposition [latex] LL’[/latex] of the Gram matrix [latex] G = X_{\text{idx}}’X_{\text{idx}}[/latex]. Because [latex] X_{\text{idx}}[/latex] grows by exactly one column at each iteration, [latex] L[/latex] can be updated according to the following rule: Given [latex] A = \begin{pmatrix} \tilde{A} & \mathbf{v}’ \ \mathbf{v} & c \end{pmatrix}[/latex], and knowing the decomposition of [latex] \tilde{A} = \tilde{L}\tilde{L}’[/latex], the Cholesky decomposition [latex] A = LL’[/latex] is given by $$ L = \begin{pmatrix}\tilde{L} & \mathbf{0} \ \mathbf{w}’ & \sqrt{c - \mathbf{w}’\mathbf{w}} \end{pmatrix}, \text{ where } \tilde{L}\mathbf{w} = \mathbf{v}$$

Even if you are unfamiliar with the mathematical properties of the Cholesky decomposition, you can see from the construction detailed above that [latex] L[/latex] is always going to be a lower triangular matrix (it will only have null elements above the main diagonal). Actually, the letter L stands for lower. We have therefore replaced the step where we needed to solve the least-squares problem [latex] X_{\text{idx}}\gamma = y[/latex] with two much simpler computations: solving [latex] \tilde{L}\mathbf{w} = \mathbf{v}[/latex] and solving [latex] LL’\gamma = X_{\text{idx}}’y[/latex]. Due to the [latex] L[/latex]’s structure, these are much quicker operations than a least squares projection.
Here is the initial way I implemented this:

[sourcecode language=”Python”]

def cholesky_omp(X, y, n_nonzero_coefs, eps=None):
if eps == None:
stopping_condition = lambda: it == n_nonzero_coefs
else:
stopping_condition = lambda: np.inner(residual, residual) \<= eps

alpha = np.dot(X.T, y)
residual = y
idx = []
L = np.ones((1,1))

while not stopping_condition():
lam = np.abs(np.dot(residual, X)).argmax()
if len(idx) > 0:
w = linalg.solve_triangular(L, np.dot(X[:, idx].T, X[:, lam]),
lower=True)
L = np.r_[np.c_[L, np.zeros(len(L))],
np.atleast_2d(np.append(w, np.sqrt(1 - np.dot(w.T, w))))]
idx.append(lam)
# solve LL’x = y in two steps:
Ltx = linalg.solve_triangular(L, alpha[idx], lower=True)
gamma = linalg.solve_triangular(L, Ltx, trans=1, lower=True)
residual = y - np.dot(X[:, idx], gamma)

return gamma, idx
[/sourcecode]

Note that a lot of the code remained unchanged, this is the same algorithm as before, only the Cholesky trick is used to improve performance. According to the plot in [3], we can see that the naive implementation has oscillations of the reconstruction error due to numerical instability, while this Cholesky implementation is well-behaved.

Along with this I also implemented the Gram-based version of this algorithm, which only needs [latex] X’X[/latex] and [latex] X’y[/latex] (and [latex] {\big|\big|}y{\big|\big|}_2\^2[/latex], in case the epsilon-parametrization is desired). This is called Batch OMP in [2], because it offers speed gains when many signals need to be sparse coded against the same dictionary [latex] X[/latex]. A lot of speed is gained because two large matrix multiplications are avoided at each iteration, but for many datasets, the cost of the precomputations dominates the procedure. I will not insist on Gram OMP in this post, it can be found in the scikits.learn repository [4].

Now, the problems with this are a bit more subtle. At this point, I moved on to code other things, since OMP was passing tests and the signal recovery example was working. The following issues popped up during review:

1. The lambda stopping condition does not pickle.
2. For well-constructed signals and data matrices, assuming normal atoms, [latex] \mathbf{w}[/latex] on line 14 will never have norm greater than or equal to zero, unless the chosen feature happens to be dependent of the already chosen set. In theory, this cannot happen, since we do an orthogonal projection at each step. However, if the matrix [latex] X[/latex] is not well-behaved (for example, if it has two identical columns, and [latex] y[/latex] is built using non-zero coefficients for those columns), then we end up with the square root of a negative value on line 17.
3. It was orders of magnitude slower than least-angle regression, given the same number of nonzero coefficients.

1 was an easy fix. 2 was a bit tricky since it was a little hidden: the first time I encountered such an error, I wrongfully assumed that given that the diagonal of [latex] X_\text{idx}’X_\text{idx}[/latex] was unit, then [latex] L[/latex] should also have a unit diagonal, so I passed the parameter unit_diagonal=True to linalg.solve_triangular, and the plethora of NaN’s along the diagonal were simply ignored. Let this show what happens when you don’t pay attention when coding.

When I realized my mistake, I first did something I saw in lars_path from the scikit: take the absolute value of the argument of sqrt, and also ensure it is practically larger than zero. However, tests started failing randomly. Confusion ensued until the nature of the issue, discussed above, was discovered. It’s just not right to take the abs: if that argument ends up less than zero, OMP simply cannot proceed and must stop due to malformed data. The reference implementation from the website of the authors of [2] includes explicit early stopping conditions for this, along with some other cases.

At the same time, I started to try a couple of optimizations. The most obvious thing was the way I was building the matrix [latex] L[/latex] was clearly suboptimal, reallocating it at each iteration.

This leads to the following code:

[sourcecode language=”Python”]

def cholesky_omp(X, y, n_nonzero_coefs, eps=None):
min_float = np.finfo(X.dtype).eps
alpha = np.dot(X.T, y)
residual = y
n_active = 0
idx = []

max_features = X.shape1 if eps is not None else n_nonzero_coefs
L = np.empty((max_features, max_features), dtype=X.dtype)
L[0, 0] = 1.

while 1:
lam = np.abs(np.dot(X.T, residual)).argmax()
if lam \< n_active or alpha[lam] ** 2 > min_float:
# atom already selected or inner product too small
warn(“Stopping early”)
break
if n_active > 0:
# Updates the Cholesky decomposition of X’ X
w = linalg.solve_triangular(L[:n_active, :n_active],
np.dot(X[:, idx].T, X[:, lam]),
lower=True)
L[n_active, :n_active] = w
d = np.dot(w.T, w)
if 1 - d <= min_float: # selected atoms are dependent
warn(“Stopping early”)
break
L[n_active, n_active] = np.sqrt(1 - d)
idx.append(lam)
# solve LL’x = y in two steps:
Ltx = linalg.solve_triangular(L[:n_active, :n_active], alpha[idx], lower=True)
gamma = linalg.solve_triangular(L[:n_active, :n_active], Ltx, trans=1, lower=True)
residual = y - np.dot(X[:, idx], gamma)
if eps is not None and np.dot(residual.T, residual) <= eps:
break
elif n_active == max_features:
break
return gamma, idx
[/sourcecode]

What should be noted here, apart from the obvious fix for #1, are the early stopping conditions. It is natural to stop if the same feature gets picked twice: the residual is always orthogonalized with respect to the chosen basis, so the only way this could happen is if there would be no more unused independent regressors. This would either lead to this, or to the stopping criterion on line 25, depending on which equally insignificant vector gets picked. The other criterion for early stopping is if the chosen atom is orthogonal to y, which would make it uninformative and would again mean that there are no better ones left, so we might as well quit looking.

Also, we now make sure that [latex] L[/latex] is preallocated. Note that np.empty is marginally faster than np.zeros because it does not initialize the array to zero after allocating, so the untouched parts of the array will contain whatever happened to be in memory before. In our case, this means only the values above the main diagonal: everything on and beneath is initialized before access. Luckily, the linalg.solve_triangular function ignores what it doesn’t need.

This is a robust implementation, but still a couple of times slower than least-angle regression. In the next part of the article we will see how we can make it beat LARS.

1 I always wanted to use this word in a serious context :P
2 Rubinstein, R., Zibulevsky, M. and Elad, M., [Efficient Implementation of the K-SVD Algorithm using Batch Orthogonal Matching Pursuit][] Technical Report - CS Technion, April 2008.
3 [First thoughts on Orthogonal Matching Pursuit][] on this blog.
4 omp.py on github.

Progress on Orthogonal Matching Pursuit

2011-08-02T16:56:00+02:00

Since orthogonal matching pursuit (OMP) is an important part of signal processing and therefore crucial to the image processing aspect of dictionary learning, I am currently focusing on optimizing the OMP code and making sure it is stable. OMP is a forward method like least-angle regression, so it is natural to bench them against one another.

This has helped find a couple of bottlenecks. Time has been gained by preallocating the array to store the Cholesky decomposition. Also, using the LAPACK potrs function in order to solve a system of the shape $latex LL’x=y$ is faster than using solve_triangular twice.

I am still trying to optimize the code. We are working hard to make sure that scikits.learn contributions are up to standards before merging.

SparsePCA in scikits.learn-git

2011-07-19T12:01:00+02:00

I am happy to announce that the Sparse PCA code has been reviewed and merged into the main scikits.learn repository.

You can use it if you install the bleeding edge scikits.learn git version, by first downloading the source code as explained in the user’s guide, and then running python setup.py install.
[caption id=”” align=”aligncenter” width=”400” caption=”Sparse PCA on images of the digit 3”][][/caption]
To see what code is needed to produce an image such as the one above, using scikits.learn. check out this cool decomposition example that compares the results of most matrix decomposition models implemented at the moment.

There are other new cool things that have been recently merged by other contributors, such as support for sparse matrices in minibatch K-means, and the variational infinite gaussian mixture model, so I invite you to take a look!

[]: http://scikit-learn.sourceforge.net/dev/_images/plot_digits_decomposition_4.png

K-Means for dictionary learning

2011-07-10T14:27:00+02:00

[![Dictionary learned with K-Means on the LFW dataset with whitening PCA][]][][![Dictionary learned with K-Means on the LFW dataset without whitening PCA][]][]

One of the simplest, and yet most heavily constrained form of matrix factorization, is vector quantization (VQ). Heavily used in image/video compression, the VQ problem is a factorization [latex] X=WH[/latex] where [latex] H[/latex] (our dictionary) is called the codebook and is designed to cover the cloud of data points effectively, and each line of [latex] W[/latex] is a unit vector.

This means that each each data point [latex] x_i[/latex] is approximated as [latex] x_i \approx h_{k} = \sum_{j=1}\^{r} \delta_{kj}h_{j}[/latex]. In other words, the closest row vector (codeword/dictionary atom) [latex] h_k[/latex] of [latex] H[/latex] is chosen as an approximation, and this is encoded as a unit vector [latex] (\delta_{k1}, …, \delta_{kr})[/latex]. The data representation [latex] W[/latex] is composed of such vectors.

There is a variation called gain-shape VQ where instead of approximating each point as one of the codewords, we allow a scalar multiplication invariance: [latex] x_i \approx \alpha_ih_k[/latex]. This model requires considerably more storage (each data point needs a floating point number and an unsigned index, as opposed to just the index), but it leads to a much better approximation.
Gain-shape VQ can equivalently be accomplished by normalizing each data vector prior to fitting the codebook.

In order to fit a codebook [latex] H[/latex] for efficient VQ use, the K-Means Clustering [1] algorithm is a natural thought. K-means is an iterative algorithm that incrementally improves the dispersion of k cluster centers in the data space until convergence. The cluster centers are initialized in a random or procedural fashion, then, at each iteration, the data points are assigned to the closest cluster center, which is subsequently moved to the center of the points assigned to it.

The scikits.learn.decomposition.KMeansCoder object from our work-in-progress dictionary learning toolkit can learn a dictionary from image patches using the K-Means algorithm, with optional local contrast normalization and a PCA whitening transform. Using a trained object to transform data points with orthogonal matching pursuit, with the parameter n_atoms=1 is equivalent to gain-shape VQ. Of course you are free to use any method of sparse coding such as LARS. The code used to produce the example images on top of this post can be found in [2].

Using K-Means for learning the dictionary does not optimize over linear combinations of dictionary atoms, like standard dictionary learning methods do. However, it’s considerably faster, and Adam Coates and Andrew Ng suggest in [3] that as long as the dictionary is filled with a large enough number of atoms and it covers well enough the cloud of data (and of future test data) points, then K-Means, or even random sampling of image patches, can perform remarkably well for some tasks.

[1] [Wikipedia article on K-Means clustering][]

[2] [K-Means Coder example][]

[3] [**The importance of encoding versus training with sparse coding and vector quantization**, Adam Coates and Andrew Y. Ng. In Proceedings of the Twenty-Eighth International Conference on Machine Learning, 2011.][]

[![Dictionary learned with K-Means on the LFW dataset with whitening PCA][]]: http://localhost:8001/wp-content/uploads/2011/07/kmeans_w.png

[![Dictionary learned with K-Means on the LFW dataset without whitening PCA][]]: http://localhost:8001/wp-content/uploads/2011/07/kmeans_no_w.png

Image denoising with dictionary learning

2011-07-07T20:00:00+02:00

I am presenting an image denoising example that fully runs under my local scikits-learn fork. Coming soon near you!

The 400 square pixels area covering Lena’s face was distorted by additive gaussian noise with a standard deviation of 50 (pixel values are ranged 0-256.)

[][]

The dictionary contains 100 atoms of shape 4x4 and was trained using 10000 random patches extracted from the undistorted image. Then, each one of the four 100 square pixel areas was reconstructed using the dictionary learning model and a different transform method.

OMP-1 reconstructs each patch as the closest dictionary atom, multiplied by a variable coefficient. This is similar to the idea of gain-shape vector quantization.
OMP-2 is like OMP-1, but it considers 2 atoms instead of just one. This takes advantage of the fact that the natural dictionary atoms are of such nature to efficiently represent random image patches when combined.
LARS finds a reconstruction of each image patch as a solution to a Lasso problem, solved using least angle regression.
Thresholding is a simple and quick non-linearity that (as it is currently implemented, based on [1], where it is not intended for reconstruction but for classification) breaks the local brightness of the image fragment. The bottom right fragment was forcefully renormalized to stretch fit into the 0-256 range, but brightness differences can be seen.

[1] [**The importance of encoding versus training with sparse coding and vector quantization**, Adam Coates and Andrew Y. Ng. In Proceedings of the Twenty-Eighth International Conference on Machine Learning, 2011.][]

[]: http://localhost:8001/wp-content/uploads/2011/07/denoise3.png

Dictionary learning sneak peek

2011-06-24T12:06:00+02:00

Closing in on the goal of integrating J. Mairal’s dictionary learning in the scikit, I stitched together a couple of examples.

The code is not yet integrated according to our standards, but here is the kind of results you can expect.

Here is how a dictionary obtained from 8x8 patches of Lena looks like. Pretty much it looks as expected: gabor-like wavelets with different rotations and shifts, which means things are working!

[][]

And here is how it works for denoising Lena. On the left is the noisy image and on the right is the reconstruction from a learned dictionary. The sparse code code producing the result on the right is found using orthogonal matching pursuit.

This is by no means a good denoising example, I have no idea at the moment how to tweak the patch sizes and the model parameters to obtain a better result. This is just a sneak peek and pretty soon you will see better stuff!

[][]

[]: http://localhost:8001/wp-content/uploads/2011/06/image.png

[]: http://localhost:8001/wp-content/uploads/2011/06/denoise.png

Summer of Code roadmap, part 1

2011-06-12T14:28:00+02:00

After a little busy while, I have graduated and entered the summer vacation, which means time for serious GSoC work.

[][]

So we had a little conference in order to discuss what will be done and when. We gathered quite a few code snippets since the official start of the project, but it’s now time to talk about integration and pull requests.

Here is the plan:

SparsePCA¶

First pull request due: June 15
This will be the use case I blogged about before. Specifically, we want to learn a dictionary of sparse atoms, but representations of the data will be dense.

SparseCoder¶

First pull request due: June 25
This is the transpose of the SparsePCA problem. We are learning the optimal, dense dictionary for sparse representations of the data.

KMeansCoder¶

First pull request due: June 30
This method builds the dictionary out of cluster centers found by K-means.

OnlineSparseCoder¶

First pull request due: July 10
This will involve the online learning tricks suggested in Julien Mairal’s work and will allow for faster computations of both sparse PCA and sparse coding. In the case of sparse coding, it will make use of the scikits.learn API for online learning.

While I will try to keep the deadlines for the initial pull requests as strictly as I can, we did not establish deadlines for merging, since this will depend on more factors. As long as the pull requests are up, the code review system will push it forward towards the merge. The focus is on teamwork and on feedback cycles as short as possible, as opposed to falling into the trap of delaying work until the night before the deadline.

[]: http://localhost:8001/wp-content/uploads/2011/06/p1080283.jpg

First thoughts on Orthogonal Matching Pursuit

2011-05-30T13:02:00+02:00

I am working on implementing the Orthogonal Matching Pursuit (OMP) algorithm for the scikit. It is an elegant algorithm (that almost writes itself in Numpy!) to compute a greedy approximation to the solution of a sparse coding problem:

$$ \text{argmin} \big|\big|\gamma\big|\big|_0 \text{ subject to }\big|\big|x-D\gamma\big|\big|_2\^2 \leq \epsilon$$

or (in a different parametrization)

$$ \text{argmin} \big|\big|x - D\gamma\big|\big|_2\^2\text{ subject to }\big|\big|\gamma\big|\big|_0 \leq m$$

The second formulation is interesting in that it gives one of the few algorithms for sparse coding that can control the actual number of non-zero entries in the solution. Some dictionary learning methods need this (I’m thinking of K-SVD).

Both problems are solved by the same algorithm, with a different stopping condition. The gist of it is to include at each iteration, the atom with the highest correlation to the current residual. However, as opposed to regular Matching Pursuit, here, after choosing the atom, the input signal is orthogonally projected to the space spanned by the chosen atoms. This involves the solution of a least squares problem at each step. However, because the problem is almost the same at each iteration, only with one more column added to the matrix, this can be easily solved by maintaining a QR or Cholesky decomposition of the dictionary matrix that is updated at each step.

Rubinstein et al. [1] came up with a clever method to optimize the calculations, based on the fact that usually in practice we never have to find a sparse coding for a single signal, but usually for a batch. They called this method Batch OMP, and it is based on a straightforward modification of the Cholesky update algorithm, taking advantage of precomputing the Gram matrix [latex] G=D’D[/latex].

Based on my experiments, their batch update is the fastest, even though it lags behind if invoked with too small a batch. As soon as I make sure the implementation is robust and ready for use, I will make some benchmarks.

Update: Here’s a little proof that it works!
[][]

Update 2: Here’s a little benchmark:
[][]
[1] http://www.cs.technion.ac.il/\~ronrubin/Publications/KSVD-OMP-v2.pdf

[]: http://localhost:8001/wp-content/uploads/2011/06/omp.png

[]: http://localhost:8001/wp-content/uploads/2011/06/omp_bench.png

Sparse PCA

2011-05-23T15:19:00+02:00

I have been working on the integration into the scikits.learn codebase of a sparse principal components analysis (SparsePCA) algorithm coded by Gaël and Alexandre and based on [[1]][]. Because the name “sparse PCA” has some inherent ambiguity, I will describe in greater depth what problem we are actually solving, and what it can be used for.

The problem¶

Mathematically, this implementation of Sparse PCA solves:

$latex (U\^*, V\^*)=\underset{U,V}{\mathrm{argmin\,}}\frac{1}{2}||X-UV||_2\^2+\alpha||V||_1$

with $latex || U_k ||_2 = 1$ for all $latex 0 \leq k \< n_{atoms}$

This looks really abstract so let’s try to interpret it. We are looking for a matrix factorization $latex UV$ of $latex X \in \mathbf{R}\^{n_{samples}\times n_{features}}$, just like in ordinary PCA. The interpretation is that the $latex n_{atoms}$ lines of $latex V$ are the extracted components, while the lines of $latex U$ are the coordinates of the samples in this projection.

The most important difference between this and PCA is that we enforce sparsity on the components. In other words, we look for a representation of the data as a linear combination of sparse signals.

Another difference is that, unlike in PCA, here we don’t constrain U to be orthogonal, just to consist of normalized column vectors. There are different approaches where this constraint appears too, and they are on the list for this summer, but I digress.

The approach¶

As usual, such optimization problems are solved by alternatively minimizing one of the variables while keeping the other fixed, until convergence is reached.

The update of $latex V$ (the dictionary) is computed as the solution of a Lasso least squares problem. We allow the user to choose between the least angle regression method (LARS) or stochastic gradient descent as algorithms to solve the Lasso problem.

The update of $latex U$ is block coordinate descent with warm restart. This is a batch adaptation of an online algorithm proposed by Mairal et al. in [[1]][].

Sparse PCA as a transformer¶

Of course, in order to be of practical use, the code needs to be refactored into a scikits.learn transformer object, just like scikits.learn.decomposition.pca. This means that the optimization problem described above corresponds to the fitting stage. The post-fit state of the transformer is given by the learned components (the matrix $latex V$ above).

In order to transform new data according to the learned sparse PCA model (for example, prior to classification of the test data), we simply need to do a least squares projection of the new data on the sparse components.

What is it good for?¶

For applications such as text and image processing, its great advantage is interpretability. When running a regular PCA on a set of documents in bag of words format, we can find an interesting visualisation on a couple of components, and it can show discrimination or clusters. The biggest problem is that the maximum variance components found by PCA have very dense expressions as linear combinations of the initial features. In practice, sometimes interpretation is made by simply marking the $latex k$ variables with the highest coefficients in this representation, and basically interpreting as if the rest are truncated to 0 (this has been taught to me in a class on PCA interpretation).

Such an approximation can be highly misleading, and now we offer you the sparse PCA code that can extract components with only few non-zero coefficients, and therefore easy to interpret.

For image data, sparse PCA should extract local components such as, famously, parts of the face in the case of face recognition.

Personally I can’t wait to have it ready for the scikit so that I can play with it in some of my projects. I have two tasks where I can’t wait to see the results: one is related to Romanian infinitives, where PCA revealed structure, and I would love to see how it looks with sparse n-gram components. The other task is to plug it in as feature extractor for handwritten digit classification, for my undergraduate thesis.

[1] http://www.di.ens.fr/sierra/pdfs/icml09.pdf

[[1]]: #footnote_1

Customizing scikits.learn for a specific text analysis task

2011-04-29T14:33:00+02:00

Scikits.learn is a great general library, but machine learning has so many different application, that it is often very helpful to be able to extend its API to better integrate with your code. With scikits.learn, this is extremely easy to do using inheritance and using the pipeline module.

The problem¶

While continuing the morphophonetic analysis of Romanian verbal forms, I found the need to streamline my workflow to allow for more complex models. There were a lot of free model parameters and it would have been painful to interactively tweak everything in order to find a good combination

In my case, I needed to read a file containing infinitives and labels corresponding to conjugation groups, and run a linear support vector classifier on this data. The SVC has its C parameter that needs to be tweaked, but I also had some ideas that arose from the images in my old post. There, I compared the way the data looked when represented as differently sized n-gram features. Furthermore, I compared the count features (ie. features indicating the number of times an n-gram occurs in a string) with binary features (ie. indicating only whether the n-gram occurs in the string or not). It looked to me like, for such a low-level text analysis task, using counts only adds noise.

For this reason, the feature_extraction.text.CountVectorizer was not enough for me. It only returns count features. There was also another thing that needed to be adjusted: by default, its analyzer uses a preprocessor that strips accented characters, and I had strong reasons to believe that Romanian diacritics are very relevant for the learning task. So, I needed to extend the vectorizer.

The solution¶

The code I came up with is here. I tried to build a class that would be as specific to my needs as possible. It is important to retain the full API, however. Note the y=None parameter in the fit functions. Its necessity will become clear in a moment.

Another tricky part was exposing the max_n parameter from the inner analyzer. This was not really natural, but it simplified the constructions later on.

My InfinitivesExtractor class builds a data matrix from a list of strings. After using it, the data needs to be passed to the classifier, an instance of svm.LinearSVC. The pipelinemodule in scikits.learn allows us to plug components into eachother in order to build a more complex object. In this case, we would like a classifier that receives a string as input, and directly outputs its label. We wouldn’t want the user to have to manually use the feature extractor prior to classification.

The pipeline is very easy to build:

pipeline = Pipeline([('extr', InfinitivesExtractor()), ('svc', LinearSVC(multi_class=True))])
The pipeline object now works exactly as expected: we can call fit and predict on it. It also exposes the parameters of its constituents, by prefixing them with the name of that component. For example, the support vector machine’s C parameter can be accessed as pipeline.svc__C.

All that is left now is to see whether this is a good model, and what combination of parameters makes it work the best. Scikits.learn provides a great tool for choosing the parameters: the grid_search module. When working with models like support vector machines, model parameters (such as the radial basis kernel width) usually need to be chosen by cross validation, because intuition doesn’t help much when dealing with high dimensional data.

Grid search allows the definition of a discrete range of values for multiple parameters. Then, for each combination of parameters, it fits and evaluates a model using cross-validation, and the model with the best score is the winner. Because we combined the components into a pipeline, it is very easy to run grid search on the combined model, and to simultaneously tweak the settings both for the extractor and for the classifier.

After running the grid search using the code here, I found that indeed, using binary features instead of occurence counts improves performance. I also found that the optimal n-gram length is 5, but the gain is not that big when compared to a length of 3, which generates a lot less features.

Conclusions¶

I hope that I managed to show the strength of a well-designed API. Because of it, it would be very easy to add, for example, an extra layer for dimensionality reduction before classification. It would only require an extra item in the pipeline constructor. A call from a web-based frontend, for example, would be very short and simple. Because of the consistency in the scikits.learn classes, we can write cleaner and better code, and therefore work with greater efficiency.

An overview of dictionary learning: Terminology

2011-04-15T14:10:00+02:00

My GSoC proposal is titled “Dictionary learning in scikits.learn” and in the project, I plan to implement methods used in state of the art research and industry applications in signal and image processing. In this post, I want to clarify the terminology used.

Usually the terms dictionary learning and sparse coding are used interchangably. Also my proposal contains methods such as Sparse PCA which are technically not sparse coding but closely related problems.

The basic idea is the approximation of a signal vector [latex] x \in \mathbb{R}\^d[/latex] by a linear combination of components, as good as possible, under certain constraints. This can be formulated as a basic (unconstrained) loss function measuring the quality of the approximation: [latex] \mathcal{L}(x, D, \alpha) = \big|\big|x - D\alpha\big|\big|\^2_2, D \in \mathbb{R}\^{d \times r}, \alpha \in \mathbb{R}\^r[/latex], where [latex] r[/latex] is the dimension of the dictionary (the number of components).

When working with a dataset of more signal vectors, the overall basic loss for such an approximation is [latex] \mathcal{L}(X, D, A) = \sum_{i = 1}\^N \mathcal{L}(x_i, D, \alpha_i) = \big|\big|X - DA\big|\big|\^2_F[/latex]. Minimizing such a loss function amounts to finding the closest (in the Frobenius sense) matrix factorization [latex] DA[/latex] that approximates the data matrix [latex] X[/latex]

This generic problem is called a matrix factorization problem. Many classical problems are matrix factorization problems with additional constraints. For example, PCA is a matrix factorization that constrains [latex] D[/latex] to be orthogonal. NMF constrains both [latex] D[/latex] and [latex] A[/latex] to have no negative elements. Sparse variants of these two decompositions are useful for getting local components, such as parts of faces. These are obtained by adding an aditional constraint on the dictionary columns.

It can be sometimes useful to consider the dictionary fixed. The signal processing community has introduced over the years many such dictionaries, for examples wavelets. These are used, for example, in the JPEG2000 compression standard.

A very useful represenation is when the dictionary is overcomplete ([latex] r > d[/latex]). The wavelets are an example of this. Given such a dictionary, we are interested in an efficient encoding of a vector [latex] x[/latex], in the sense of sparseness: we want to use as few dictionary components as possible in our representation. Such a solution is the vector [latex]\alpha[/latex] minimizing [latex] \mathcal{L}(x, D, \alpha) + \lambda\big|\big|\alpha\big|\big|_1[/latex] but other sparsity-inducing constraints can be used. Such a vector is a sparse coding of [latex] x[/latex] and it can be solved using algorithms such as least-angle regression and orthogonal matching pursuit.

However, we are not limited to using precomputed dictionaries. The term dictionary learning refers to methods of inferring, given [latex] X[/latex], a (usually overcomplete) dictionary that will perform good at sparsely encoding the data in [latex] X[/latex]. Such methods are more expensive than using precomputed dictionaries, but they perform better, since the dictionary is optimized for the current dataset.

Because usually such loss functions are non-convex in [latex] D[/latex] and [latex] A[/latex] simultaneously, dictionary learning algorithms alternate between minimizing each while keeping the other fixed. The step that minimizes [latex] D[/latex] is sometimes called the dictionary update step, and the one minimizing [latex] A[/latex] is (similarily to the case where the dictionary is always fixed) the sparse coding step. Dictionary learning algorithms differ in the method used for each of this step.

To resume, many problems can be posed as matrix factorization problems. Depending on the constraints imposed, the problem becomes interesting for different applications. Dictionary learning is very good for image reconstruction. Matrix decompositions with sparse undercomplete dictionaries such as Sparse PCA can be used to find local features that constitute the dataset, for example parts of faces, for a dataset of facial images. NMF can be used in both under and overcomplete settings and it offers a good model for additive data such as text or images. We are interested in these variants and they are planned for implementation in my GSoC proposal.

Julien Mairal’s presentation of his work in this domain, available here, shows the theoretical background of such methods, along with examples showing state of the art results in image processing.

Newton interpolation and numerical differentiation

2011-04-15T13:34:00+02:00

I am sharing some Python code code that I wrote as a school assignment. This computes the Newton form of the interpolation polynomial of a given set of points, and allows for the evaluation of both the polynomial and its derivative, at a given point. This is an accurate way of estimating the derivative of a complicated function.

Initially it plots the function, the interpolating polynomial and its derivative. When clicking on the plot, the tangent to the interpolating polynomial at the horizontal position of the mouse cursor is plotted.

It can be found here: https://gist.github.com/921554

A look at Romanian verbs with scikits-learn

2011-04-14T01:40:00+02:00

One of the problems we tackled here at my university is one as old as the modern Romanian language. It is a problem for linguists, as well as for foreigners trying to learn the language. We call it the root alternations problem.

Similar to French and other languages, Romanian verbs are split into four groups with different conjugation patterns. Except for the irregular verbs, this categorization is performed based on the suffix of the infinitive. However, the conjugation is not straightforward even within these classes, because many verbs exhibit alternations in their root. For example, the verb a purta (to wear) becomes eu port (I wear) but el poartă (he wears). It can be seen that the letter o in the root changes to oa during conjugation. This makes learning the language quite difficult, because we have no rules to describe when these changes occur.

Attempts to formalize such rules from a computer scientific point of view date back to G. C. Moisil in 1960. Such (incomplete) rules can be formulated as context-sensitive grammars, since the alternations are determined by the context in which certain characters appear.

This leads to the idea of analyzing the verbs from a machine learning point of view: what can we find out by looking at n-gram representation of the infinitives?

This is easy to do within scikits.learn. The feature_extraction.text package contains all the necessary tools: the CharNGramExtractor, which builds all the n-grams of a string, for n in an interval. Then, a CountVectorizer is built on top of the extractor. Its purpose is to extract the features out of a list of documents and transform them into a matrix representation of token counts. By postprocessing this matrix we can obtain a binary representation, indicating only whether a token occurs in a document or not, instead of the count.

In this case, documents are Romanian infinitives. This means we are limited to using short n-grams, because the documents are themselves short. There is also the question whether anything relevant can be found out of such a representation which does not encode a lot of information.

After building the data matrix from the list of verbs, I plotted a 2D PCA projection and here are the results. I am only posting a teaser for now, but the results are encouraging:

[][]

From the image it is clear that n-gram representations of the infinitives induce clusters. Further results suggest that for certain subclasses of the dataset, such a representation (or even a simpler one) is enough to clearly answer whether a verb does not exhibit alternations. This encourages further exploration of this path, especially supervised and semi-supervised approaches.

[]: http://localhost:8001/wp-content/uploads/2011/04/infinitives_pca.png

Tweaking matplotlib subplots for pretty results

2011-04-04T20:53:00+02:00

When plotting multiple subplots using matplotlib, the axes rarely look pretty with the default configuration. Since matplotlib figures are abstract objects, designed for consistency in print as well as on screen, tweaking their layout can get tricky.

An example¶

The following code is taken from the face recognition example in scikits.learn:
pl.figure(figsize=(1.8 * n_col, 2.4 * n_row))

This is very confusing at first, for somebody used to work on screen: the quantities in there are actually inches! These are converted implicitly to pixels through the dpi parameter, which is left as default (80 dpi) in this example.

Then, it gets even worse: In order to tweak the positioning of the subplots, this is what is done:

pl.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
Now, all of these are percents of the image height/width. The margins are sort of like CSS-style margins, only relative to the bottom left corner. In other words, right=.99 means that the right margin is 1% away from the right edge.

The parameters hspace and wspace control the spacing between the subplots. However these are kind of hard to get right, because, obviously, there are more settings than there are degrees of freedom.

My tip¶

On my system, the default matplotlib backend is TkAgg. The matplotlib backend controls the graphical environment that builds the plot windows, as well as the rendering engine used. TkAgg has a “configure subplots” button that opens a popup window with sliders to visually adjust the parameters above. The problem is that the sliders are unlabeled, so I needed to do an heuristic by first setting the parameters by hand and then exploring the direction in which they need to be changed.

When I tried different backends, I found that WXAgg has labeled sliders. This means you can adjust your subplots visually and you will have the parameter values to use in the call to subplots_adjust in one go!

You can set your backend to WXAgg by adding the line backend: WXAgg in your matplotlibrc file.

On setuptools subpackages

2011-04-04T15:01:00+02:00

Today, I spent more than two hours trying to figure out why, despite things working out fine in my development scikits.learn folder, python setup.py install would completely ignore the module I refactored into a subpackage.

I imagined that simply adding it to the parent __init__.py __all__ attribute would do, I kind of thought that setuptools automatically finds the subpackages.

At first I thought of looking in setup.py, but I only examined the one in the topmost directory, which, in the case of scikits.learn, is two degrees of separation away from the actual setup.py that takes care of subpackages (ie. I was looking at /setup.py instead of /scikits/learn/setup.py). This had me fooled for a while.

The steps to add a working and installable module to a python setuptools-based project are as follows:

Add a __init__.py file in the folder (ie. /scikits/learn/decomposition/__init__.py)
If the module requires compiling or any special attention, add an appropriate __setup__.py file in the folder.
Update the __init__.py __all__ attribute in the parent folder (ie. /scikits/learn/__init__.py)
Update the setup.py in the parent folder (ie. /scikits/learn/setup.py) by adding something like:
config.add_subpackage('decomposition')

Don’t forget to do the same for the tests subfolder!

Conclusions¶

While wasting so much time due to a simple beginner’s mistake is not very pleasant, I am not frustrated with setuptools. On the contrary, now that I understand it better I can appreciate its flexibility and clarity, when compared to, for example, MSBuild and Visual Studio project files. Just one more reason to love Python!

My first scikits.learn coding sprint

2011-04-02T20:12:00+02:00

The fifth scikits.learn coding sprint took place Friday, April 1st 2011. For anyone who is not familiar with it, scikits.learn is a fast and easy to use machine learning toolkit for the pylab environment (Python, NumPy, SciPy, Matplotlib.)

This was a good opportunity for me to get code reviews by the developers in order to bring my NMF code up to standards, so that it can be merged. Though I live far from every nucleus of scikits-learn developers, I efficiently participated via IRC. This way, I also got the chance to help out a bit on Mathieu Blondel’s Kernel PCA code, which will also be merged into main soon.

How it felt like¶

Short answer: awesome!

Slightly longer answer: Everybody was very encouraging and helpful. They gave me a lot of feedback from which I learned a lot, and they manifested the intention to merge soon. It is a pleasure to work on projects that you like and use, especially when the projects leaders and collaborators are so good to work with.

But the main reason why it makes me feel so good is that I’m proud to able to contribute on a project that I consider very significant and the best in the field from many points of view.

What I got done¶

Most of my work was on the non-negative matrix factorization module that I began some while back, but only intermitently worked on. It is now a solid module with high test coverage, documentation, and a cool simple example showing a sparse set of features for the digits dataset in scikits.learn. Apart from all the minor fixes in overall code quality and cleanliness, probably what is the most relevant is the improvement and the study of the initialization methods. I will look into this further and document it on this blog, the point is that the choice of initialization method greatly influences the speed of convergence, and in the case of a high-tolerance setting, also the error obtained. Some initializations are more fit for sparsity settings, while others are more fit for dense settings.

I have a theory that I plan to test out, regarding the use of different initialization methods for components and for data in a sparse setting.

What I learned¶

I think my greatest improvement was in terms of workflow and efficiency. While my code was under review, I was receiving frequent comments on my git pull request, and eventually I ended up responding to some comments even before they were posted :). I sent small fixes as pull requests to help other developers as much as I could. Before scikits.learn I had never worked on a project with so many developers, and I think I handled it well, even though I asked once or twice on the IRC channel for pieces of git-fu.

I learned that it’s difficult to tweak matplotlib subplots! I’m still staring at Alexandre Gramfort’s tweak in my example and I have no idea what he did to make it look so good. But I’ll figure it out soon, I’m sure.

I also learned a lot more about the intricacies of the scikits.learn APIs, the philosophy of ease of use, and the project tree in general.

In short, the coding sprint has been a great and rewarding experience, for which I thank all of you guys there!

About

2011-03-30T08:59:00+02:00

My name is Vlad, I am a master’s student at the University of Bucharest, I work there at the Centre for Computational Linguistics, and I am a contributor to the Python machine learning library scikit-learn.

Hello world!

2011-03-30T08:59:00+02:00

This is the blog where I will post updates regarding my work onscikits-learn. I am applying as a GSoC 2011 student with the PSF for work on this machine learning library.

I will also post related work that I do in the machine learning field.

I hope to prove an enjoyable read for everybody who stumbles on to this site, or for those who will be forced to review it :)

\(u\)	\({x=\sigma(u)}\)	\(\pfrac{\sigma(u)}{u}\)	\(\pfrac{f(x)}{x}\)	\(\pfrac{f(\sigma(u))}{u}\)
\((0, 0)\)	\((.5, .5)\)	\(\left(\begin{smallmatrix} .25 & 0 \\\\ 0 & .25 \end{smallmatrix}\right)\)	\((-2.2, -0.8)\)	\((-0.55, -0.2)\)
\((4.6, -0.4)\)	\((.99, .4)\)	\(\left(\begin{smallmatrix} .0099 & 0 \\\\ 0 & .24 \end{smallmatrix}\right)\)	\((-0.93, -0.12)\)	\((-0.01, -0.03)\)

Vlad Niculae

Optimizing with constraints: reparametrization and geometry.

Why are constraints challenging?¶

Ways to deal with constraints.¶

Reparametrization.¶

Projected gradient.¶

Generalizing the projected gradient method with divergences.¶

Duality between mirror descent and natural gradient.¶

Conclusions.¶

Acknowledgements.¶

Rendering 3-d surface plots for social media

three.js¶

Computing the function values¶

Constructing the 3-d surface¶

But it shows up seen from above and looks ugly!¶

Winning Arguments and Attitude Change on Reddit

What makes /r/ChangeMyView special?¶

Taking a look at the dataset¶

Reproducing some results¶

Training a simple classifier¶

Checking out the features¶

Word Mover’s Distance in Python

Prepare some word embeddings¶

Reproducing the demo above¶

Document classification¶

Comparison with other models¶

What have we learned?¶

Flask-SocketIO on OpenShift: fallback on another port

TL; DR¶

The current state of affairs¶

The problem in a nutshell¶

Making things work¶

Kemeny-Young Optimal Rank Aggregation in Python

Kendall’s Tau distance¶

Kemeny-Young rank aggregation¶

Integer Programming formulation¶

References¶

Site move

BibTeX-powered publications list for Pelican with pelican-bibtex

Hook¶

Backstory¶

Possible extensions¶

Really the most common english idioms?

Corpus Linguistics¶

Cognitive salience vs. social salience¶

The idioms¶

Recalculating the rank¶

Conclusions¶

Scikit-learn-speed: An overview on the final day

The scikit-learn-speed benchmark platform¶

Make your work reusable: the trade-off between good design and get-it-working-now¶

Never work quietly (unless you’re a ninja)¶

The speedups¶

Write tests, tests, tests!¶

Optimization doesn’t have to be ugly¶

Small speed-ups can propagate¶

Inverses and pseudoinverses. Numerical issues, speed, symmetry.

The Moore-Penrose pseudoinverse¶

How to compute the pseudoinverse on paper¶

Numerical issues¶

Numerical rank vs. actual rank¶

Scipy implementations¶

Improving the symmetric case¶

The scikit-learn-speed ship has set sail! Faster than ever, with multi-step benchmarks!

Multi-step benchmarks¶

How much time were we wasting?¶

Behind the scenes¶

Profiler output, benchmark standard deviation and other goodies in scikit-learn-speed

What it looks like now¶

What made this possible¶

Scikit-learn-speed HTML reports teaser

Memory benchmarking with vbench

On why my %memit fails on OSX

Line # Mem usage Increment Line Contents¶

Line # Mem usage Increment Line Contents¶

More on memory benchmarking

%mprun¶

%memit¶

Installation instructions¶

Quick memory usage benchmarking in IPython

What makes `/r/ChangeMyView` special?¶