# Some Linear Algebra with Cython

Carl Vogel    |   @slendrmeans   |   February 2013

## Introduction (Attention Conservation Notice)

This notebook contains information that is new and useful. But the parts that are new are not useful, and the parts that are useful are not new. It exists because I wanted to experiment in Cython and in the IPython notebook. The code here is all but useless except for pedagogy. Virtually no one has any rightful business coding up their own linear algebra routines. And the mathematical and algorithmic content is elementary.

Which is all useful for my purpose: when you try out new tools you want to do it on well-known problems. And the excessive verbiage and formulas give me an excuse to tinker with the notebooks typographical capabilities.

As such this may be of no use to anyone besides the author. Consider yourself warned.

Comments regarding my amateur Cython, or any other topic, are welcome.

## A linear system

Our goal is to understand and, if possible, solve the system of $n$ linear equations

$$
\begin{align}
a_{00}\,x_0 + a_{01}\,x_1 + \ldots + a_{0,n-1}\,x_{n-1}   &= b_0 \\\
a_{10}\,x_0  + a_{11}\,x_1 + \ldots + a_{1,n-1}\,x_{n-1}   &= b_1 \\\
\vdots &  \\\
a_{n-1,0}\,x_0 + a_{n-1,1}\,x_1 + \ldots + a_{n-1,n-1}\,x_{n-1} &= b_{n-1}\ .
\end{align}
$$

In the system, the $a_{ij}$s and $b_i$s are known, while the $x_i$s are the unkown variables we wish to solve for. In other words, solving the system means finding the values for the $x_i$s using the $a_{ij}$s and $b_i$s.

Using matrix notation, we can write the system as

$$
\begin{pmatrix}
a_{00} & a_{01} & \ldots & a_{0,n-1} \\\
a_{10} & a_{11} & \ldots & a_{1,n-1} \\\
\vdots &  &   \ddots & \vdots \\\
a_{n-1,0} & a_{n-1,1} & \ldots & a_{n-1,n-1}
\end{pmatrix} \,
\begin{pmatrix} x_0 \\\ x_1 \\\ \vdots \\\ x_{n-1}\end{pmatrix}
=
\begin{pmatrix} b_0 \\\ b_1 \\\ \vdots \\\ b_{n-1}\end{pmatrix}\
$$

or $Ax = b$. In this form, a solution to the system is the vector $x$ that satisfies the equation.

----------------

##### Exercise: Matrix multiplication in Cython

To work with linear systems, we’ll want to be able to multiply matrices with vectors, like in the equation above, but also with other matrices.

To use Cython in the notebook, we have to load the `cythonmagic` extension. We’ll also want to load numpy and scipy modules form, among other things, benchmarks for our Cython functions.

```python
%load_ext cythonmagic
import numpy as np
import scipy.linalg as la
```

Using `%%cython` cell magic, we can write and compile a Cython module within a notebook cell.

We want to be able to multiply a matrix $A$ by either a vector $x$, like above, or another matrix $B$. Unlike, say, C++, Cython does not allow use to overload a function by using different arguments. (As far as I know.) For example, we can’t define two versions of a function `matprod` that takes either one 2-D array and one 1-D array, or two 2-D arrays.

But Cython is flexible, and lets us define functions where not all arguments are typed. So we’ll write a wrapper function `matprod` that has the second argument untyped. Then, based on whether the second argument is a vector or a matrix, dispatches to the appropriate C-like (typed) function.

```cython
%%cython
cimport cython
import numpy as np
cimport numpy as np

def matprod(np.ndarray[double, ndim = 2] A, B):
    '''
    Matrix-by-vector or matrix-by-matrix multiplication.

    The arguments are dispatched to one of two functions
    depending on whether B is a vector or a matrix.
    '''
    if B.ndim == 1:
       # B is a vector
       return matvecprod(A, B)
    else:
        # B is a matrix
        return matmatprod(A, B)

@cython.boundscheck(False)
@cython.wraparound(False)
cdef np.ndarray[double, ndim=2] matmatprod(
    np.ndarray[double, ndim=2] A,
    np.ndarray[double, ndim=2] B):
    '''
    Matrix-matrix multiplication.
    '''
    cdef:
        int i, j, k
        int A_n = A.shape[0]
        int A_m = A.shape[1]
        int B_n = B.shape[0]
        int B_m = B.shape[1]
        np.ndarray[double, ndim=2] C

    # Are matrices conformable?
    assert A_m == B_n, \
        'Non-conformable shapes.'

    # Initialize the results matrix.
    C = np.zeros((A_n, B_m))
    for i in xrange(A_n):
        for j in xrange(B_m):
            for k in xrange(A_m):
                C[i, j] += A[i, k] * B[k, j]
    return C

@cython.boundscheck(False)
@cython.wraparound(False)
cdef np.ndarray[double, ndim=1] matvecprod(
    np.ndarray[double, ndim=2] A,
    np.ndarray[double, ndim=1] b):
    '''
    Matrix-vector multiplication.
    '''
    cdef:
        Py_ssize_t i, j, k
        Py_ssize_t A_n = A.shape[0]
        Py_ssize_t A_m = A.shape[1]
        Py_ssize_t b_n = b.shape[0]
        np.ndarray[double, ndim=1] c

    # Are matrices conformable?
    assert A_m == b_n, \
        'Non-conformable shapes.'

    # Initialize the results matrix.
    c = np.zeros(A_n)
    for i in xrange(A_n):
            for k in xrange(b_n):
                c[i] += A[i, k] * b[k]
    return c
```

If the above compiles successfully, nothing should happen. Otherwise an output cell with a compilation error message and traceback will appear.

For speed comparisons, the following is a pure Python version of matrix-matrix multiplication.

```python
def pymatmatprod(A, B):
    '''
    Matrix-matrix multiplication
    '''
    A_n, A_m = A.shape
    B_n, B_m = B.shape
    assert A_m == B_n, "Non-conformable shapes."
    C = np.zeros((A_n, B_m))
    for i in xrange(A_n):
        for j in xrange(B_m):
            for k in xrange(A_m):
                C[i, j] += A[i, k] * B[k, j]
    return C
```

We create some small sample matrices to test the functions.

```python
# A is 2x3
A = np.array([[2.0, 0.25, -1.0],
              [3.0, 0.0 ,  5.0]])
# B is 3x2
B = np.array([[-3.0,  0.5],
              [ 2.0,  1.5],
              [ 4.0, -4.0]])
# C is 2x2
C = np.array([[1.0,  1.5],
              [2.5, -1.0]])
# b is 3x1 (a vector)
b = np.array([1.0, -2.0, 0.5])
```

And check to see they give the same results.

```python
print 'Cython:'
print '-------'
print "A x B =\n", matprod(A, B), "\n"
print "A x b =\n", matprod(A, b), "\n"
print 'Numpy dot:'
print '----------'
print "A x B =\n", np.dot(A, B), "\n"
print "A x b =\n", np.dot(A, b), "\n"
print 'Python loops:'
print '-------------'
print "A x B =\n", pymatmatprod(A, B), "\n"
```

    Cython:
    -------
    A x B =
    [[ -9.5     5.375]
     [ 11.    -18.5  ]]

    A x b =
    [ 1.   5.5]

    Numpy dot:
    ----------
    A x B =
    [[ -9.5     5.375]
     [ 11.    -18.5  ]]

    A x b =
    [ 1.   5.5]

    Python loops:
    -------------
    A x B =
    [[ -9.5     5.375]
     [ 11.    -18.5  ]]

We want to make sure the function doesn't try to multiply non-conformable matrices. Without this check, depending on how we code the function, we might not get an error, but instead nonsense.

```python
# Non-comformable matrices
print matprod(A, C)
```

Now that it works correctly, we can check the speed of our Cython function. It appears to be a bit slower than numpy's `dot` function, but much faster than pure Python.

```python
%timeit np.dot(A, B)
```

    1000000 loops, best of 3: 1.33 µs per loop

```python
%timeit matprod(A, B)
```

    100000 loops, best of 3: 3.53 µs per loop

```python
%timeit pymatmatprod(A, B)
```

    10000 loops, best of 3: 23.3 µs per loop

-------------------------

##### Note: Numpy arrays in Cython using buffers or MemoryViews

In the functions above, we declared a variable to be a numpy array with, for example,

    np.ndarray[double, ndim=2] x

This is an example of creating a Numpy array buffer. There is a newer method [recommended](http://docs.cython.org/src/userguide/memoryviews.html) in the Numpy documentation, using typed MemoryViews. Using this method, we would have instead declared

    double[:, :] x

or

    double[:, ::1] x

The `::1` slice indicates that `x` is a C-contiguous array; i.e. its columns are one memory-location apart. This is a much simpler syntax than the array buffer declaration. MemoryViews are also supposed to be faster and more flexible than buffers. In my experience they are when working on larger arrays, but on smaller problems, there seems to be some overhead involved in creating MemoryViews and coercing them back to arrays.

I’ll use the two notations interchangeably throughout.

-----------

## Solution by matrix inversion

The natural solution to the matrix equation $Ax = b$ is to find the inverse of $A$, denoted $A^{-1}$. $\ A^{-1}$ is the matrix that has the property $A^{-1}A = I$. Pre-multiplying both sides of the equation will then leave us with the solution $x = A^{-1}b$.

It turns out, though, that computing $A^{-1}$ is expensive, and that there are more efficient ways of solving the system. As John Cook says, “There is hardly ever a good reason to invert a matrix.” With that advice, we'll not spend effort on writing algorithms for computing matrix inverses.

What is useful to know about $A^{-1}$ is that it only exists if the linear system has a unique solution. That is, if there is one and only one $x$ that solves $Ax = b.$ If $A$ has an inverse, it's called nonsingular.

Under what circumstances would $A$ *not* have an inverse?
If one of the columns in $A$ can be calculated as a linear combination of the other columns, then $A$ will not have an inverse. To have an inverse, $\,A$’s columns must be linearly independent.

For example, let’s look at the linear system

$$
\begin{pmatrix}
2  & 3  &  1 \\\
0.5  & 2  & -1 \\\
-1 & 5 &  -7
\end{pmatrix}\,
\begin{pmatrix}
x_0 \\\ x_1 \\\ x_2
\end{pmatrix} =
\begin{pmatrix}
10 \\\ -3 \\\ 2
\end{pmatrix}\
$$

Here the third column, $A_{\cdot2}$ is equal to $2\times A_{\cdot0} - 1\times A_{\cdot 1}$, so the columns of this matrix are not linearly independent. This relationship means that $x_2 = 2x_0 - x_1$, so $x_2$ is not an independent variable, and we really only have two variables in three equations. There will be an infinite number of combinations of $x_0$ and $x_1$ that solve the system.

When the columns of $A$ are not linearly independent, and $A$ has no inverse, it’s called singular or degenerate.

## The determinant of a matrix

In the example above, it was easy to see that the columns of the matrix were not linearly independent. For larger matrices, a more reliable method of detecting singular matrices is required.

The determinant of a matrix—a real number denoted $\det(A)$—is an attribute of a square matrix that can be used to tell whether a matrix is singular, and therefore whether the linear system has a solution.

The check is straightforward: when the determinant of a matrix is zero, the matrix is singular, and no solution exists. We can check this with the singular matrix in the example above, using numpy’s `det` function.

```python
A = np.array([[  2, 3,  1],
              [0.5, 2, -1],
              [ -1, 5, -7]])

# A is singular, so it's determinant should be zero.
print "The determinant is", np.linalg.det(A)
```

    The determinant is 0.0

Calculating the determinant, though, is not so easy.

Let’s start with a trivial definition. We’ll say that the determinant of a $1\times1$ matrix $A$ (a scalar), is simply equal to $A$. So, for example, $\det(4) = 4$.

The well-known formula for the determinant of a $2x2$ matrix

$$
A =
\begin{pmatrix}
a & b \\\
c & d
\end{pmatrix}
$$

is $\det(A) = ad - bc$. We can break this formula down and generalize it to larger matrices.

First, let’s define the $(i, j)$  minor of $A$, denoted $A_{ij}$ as the matrix that results from removing row $i$ and column $j$ from $A$. For example, the (0, 0) minor of the $2\times2$ matrix above is:

$$
\begin{pmatrix}
\cdot & \cdot \\\
\cdot & d
\end{pmatrix} = d.
$$

Similarly the (0, 1) minor is:

$$
\begin{pmatrix}
\cdot & \cdot \\\
c & \cdot
\end{pmatrix} = c.
$$

We can now re-write the formula for the determinant as

$$
\det(A) = a_{00}\det(A_{00}) - a_{01}\det(A_{01})\ ,
$$

since the the minors $A_{0i}, i = 0, 1$ are scalars so are equal to their determinants by our definition above. Even better, we can take care of the minus sign by noting that

$$
\det(A) = (-1^0)\;a_{00}\det(A_{00}) + (-1^1)\;a_{01}\det(A_{01})\ .
$$

Everything in this formula can now be generalized to the determinant of an arbitrary $n\times n$ matrix.

$$
\det(A) = \sum_{i=0}^{n-1}(-1^i)\;a_{0i}\det(A_{0i})
$$

Our choice to move across row 0 of the matrix was arbitrary; we could have chosen to go across any row or down any column of the matrix, as long as we obtained the associated minor and computed the correct sign on each term. For example, we could have used column 2, in which case we would have had:

$$
\det(A) = \sum_{i=0}^{n-1}(-1^{i+2})\;a_{i2}\det(A_{i2})
$$

This flexibility can often come in handy. For example, if a row or column has a lot of zeros in it, we can exploit that to cut down on the number of calculations needed since terms in the sum get zeroed out.

Lastly, notice that these definitions are recursive. That is, to find the determinant of an $n\times n$ matrix $A$, we have to find the determinants of the $(n-1)\times(n-1)$ minors $A_{0i}$ (of which there are $n$). To find the determinants of *these* matrices, we have to compute the determinants of their $(n-2)\times(n-2)$ minors (of which there are $n-1$). So we are now computing $n\times(n-1)$ determinants of $(n-2)\times(n-2)$ matrices. This continues all the way down until the minor matrices are scalars, at which point we know the determinants by definition.

The recursive equation gives us a simple and elegant way to express all this computation. Furthermore we can write our code using this recursive equation directly, by writing a `determinant` function that calls itself. But as is often the case this elegance comes at a cost, and we’ll find that this recursive method gets very computationally expensive.

-------------------

##### Exercise: Computing determinants with recursive functions

The following code implements the recursive algorithm for calculating the determinant. Note the compiler flag after `%%cython`; necessary since we'll use a function from C's `math` library.

```cython
%%cython -lm
# Note the lm flag, used to import the C math library.
import cython
import numpy as np
cimport numpy as np
# Using C's power function instead of Python, hence the
# math library link flag above.
from libc.math cimport pow

@cython.boundscheck(False)
@cython.wraparound(False)
cpdef double determinant(double[:, :] M):
    '''
    Compute the determinant of a square nxn matrix using
    the recursive formula:

        det(M) = sum_{i=0}^{n-1} (-1^i)*M[0,i]*det(M_minor(0,i))

    where M_minor(0,i) is the (n-1)x(n-1) matrix formed by
    removing row 0 and column i from M.
    '''

    assert M.shape[0] == M.shape[1], 'Matrix is not square.'

    cdef int i, j
    cdef int n = M.shape[0]
    cdef double det = 0.0
    cdef double coef
    cdef double[:, :] M_minor = np.empty((n-1, n-1))

    if n == 1:
        # If M is a scalar (1x1) just return it
        return M[0, 0]
    else:
        # If M is nxn, then get its (n-1)x(n-1) minors
        # (one for each of M's n columns) and compute
        # their determinants and add them to the summation.
        for j in xrange(n):
            coef = pow(-1, j) * M[0, j]
            _get_minor(M, M_minor, 0, j)
            det += coef * determinant(M_minor)
        return det

@cython.boundscheck(False)
@cython.wraparound(False)
cdef void _get_minor(double[:, :] M, double[:, :] M_minor,
                    int row, int col):
    '''
    Return the minor of a matrix, by removing a specified
    row and column

    If M is nxn, then _get_minor(M, row, col) will fill in
    the (n-1)x(n-1) matrix M_minor by removing row `row`
    and column `col` from M.
    '''
    cdef:
        int n = M.shape[0]
        int i_to, j_to, i_from, j_from

    # _from indicates the index of the original
    # matrix M, _to, indicates the index of the
    # result matrix M_minor.
    i_from = 0
    for i_to in xrange(n-1):
        if (i_to == row):
            # This is the row to exclude from the
            # minor, so skip it.
            i_from += 1
        j_from = 0
        for j_to in xrange(n-1):
            if (j_to == col):
            # This is the column to exclude from the
            # minor, so skip it.
                j_from += 1

            M_minor[i_to, j_to] = M[i_from, j_from]
            j_from += 1

        i_from += 1
```

Testing the function on a sample matrix, suggests it’s correctly coded.

```python

# A is 3x3
A = np.array([[ 2, -1,   4],
              [-1,  3, 0.5],
              [ 5, -9,  11]])

print 'Numpy: ', np.linalg.det(A)
print 'Cython: ', determinant(A)
```

    Numpy:  37.5
    Cython:  37.5

And it is even quite fast on a $3\times3$ matrix.

```python
print 'Numpy time:'
%timeit np.linalg.det(A)
print 'Recursive Cython time'
%timeit determinant(A)
```

    Numpy time:
    10000 loops, best of 3: 62.7 µs per loop
    Recursive Cython time
    10000 loops, best of 3: 26.1 µs per loop

But—as feared—it is deadly slow on even just a somewhat larger, $10\times 10$ matrix.

```python
# B is 10x10
print 'Numpy time:'
B = np.random.randn(100).reshape(10, 10)
%timeit np.linalg.det(B)
print 'Recursive Cython time'
%timeit determinant(B)
```

    Numpy time:
    10000 loops, best of 3: 65.6 µs per loop
    Recursive Cython time
    1 loops, best of 3: 16.2 s per loop

---------------------------

## Linear systems with triangular matrices

At this point, the elegance of mathematics has been thwarted by the dirty business of computing. We would like to know whether a linear system is solvable, which we can do by calculating it’s determinant. But the mathematical formula we have derived for it computes terribly. Even if we could compute the determinant and find the system to be solvable, we have been warned against using the algebraically sensible method of matrix inversion to solve it.

Let’s go down a new road. Our system, once more, is:

$$
\begin{pmatrix}
a_{00} & a_{01} & \ldots & a_{0,n-1} \\\
a_{10} & a_{11} & \ldots & a_{1,n-1} \\\
\vdots &  &   \ddots & \vdots \\\
a_{n-1,0} & a_{n-1,1} & \ldots & a_{n-1,n-1}
\end{pmatrix} \,
\begin{pmatrix} x_0 \\\ x_1 \\\ \vdots \\\ x_{n-1}\end{pmatrix}
=
\begin{pmatrix} b_0 \\\ b_1 \\\ \vdots \\\ b_{n-1}\end{pmatrix}\
$$

But imagine that our system was of the form:

$$
\begin{pmatrix}
l_{00} & 0 & 0 & \ldots & 0 \\\
l_{10} & l_{11} & 0 & \ldots & 0 \\\
\vdots &  &  & \ddots & \vdots\\\
l_{n-1,0} & l_{n-1,1} & l_{n-1,2} & \ldots & l_{n-1,n-1}
\end{pmatrix}\,
\begin{pmatrix} y_0 \\\ y_1 \\\ \vdots \\\ y_{n-1}\end{pmatrix}
=
\begin{pmatrix} c_0 \\\ c_1 \\\ \vdots \\\ c_{n-1}\end{pmatrix}\
$$

or $Ly = c$. The matrix $L$ is lower triangular, which means all the elements above its diagonal are zero.

This is a simple system to solve. The first row gives us the value of $x_0$; once we have that, substituting it into the second row gives us $x_1$. We can then roll this process down to solve for a new variable in each row until we’ve solve the whole system. This process is called forward substitution. The formula for any $y_i$ in the system above is

$$
y_i = \frac{1}{l_{ii}}\left(c_i - \sum_{k=0}^{i-1}l_{ik}\,y_k\right),
$$

which only depends on the previous values of $y\,$: $y_{i-1}, y_{i-2}, \ldots, y_0$.

The process, of course, is just as simple with an upper triangular matrix:

$$
\begin{pmatrix}
u_{01} & u_{11} & \ldots & u_{0, n-2} & u_{0,n-1} \\\
0 & u_{11} & \ldots & u_{1, n-2}& u_{1,n-1} \\\
\vdots &  & \ddots &  &\vdots\\\
0 & 0 & \ldots & 0 & u_{n-1,n-1}
\end{pmatrix}\,
\begin{pmatrix} y_0 \\\ y_1 \\\ \vdots \\\ y_{n-1}\end{pmatrix}
=
\begin{pmatrix} c_0 \\\ c_1 \\\ \vdots \\\ c_{n-1}\end{pmatrix}\
$$

or $Uy = c$. Here we would simply move up the rows, substituting and solving for one new variable each time; a process called, unsurprisingly, backwards substitution.

### Determinants of triangular matrices

Another convenient property of triangular matrices is that their determinants are just the products of their diagonal entries. This is easy to prove using the recursive determinant formula defined above. For a lower triangular matrix, just take the determinant across the top row. Only the first term is non-zero, so we have

$$
\det(L) = l_{00}\det(L_{00}).
$$

The minor $L_{00}$ is just another lower triangular matrix, so its determinant is

$$
\det(L_{00}) = l_{11}\det(L_{00_{00}}),
$$

giving us

$$
\det(L) = l_{00}\cdot l_{11}\det(L_{00_{00}}).
$$

We can imagine proceding down this route, substituting minors, until all that remains is the product $\det(L) = l_{00}\cdot l_{11}\cdots l_{n-1,n-1}\,.$

The process is the same for an upper triangular matrix. If we take the determinant down the first column, we'll find each subsequent minor matrix in the recursion is also an upper triangular matrix, so it’s determinant is also just the product of its diagonals, $u_{00}\cdot u_{11}\cdots u_{n-1, n-1}$.

The implication of this is that a triangular matrix is non-singular or invertible so long as none of its diagonal elements are zero. This makes intuitive sense if you imagine proceeding through each step of forward or backward substitution—a zero on the diagonal would break the chain of sequential subsitutions.

---------------

##### Exercise: Coding forward- and backward-substitution solution methods

The forward and backward substitution algorithms are straightforward to code. Below are two functions, expecting either a lower or upper triangular matrix, that solve the respective system.

```cython
%%cython -lm
import cython
import numpy as np
cimport numpy as np
from libc.math cimport fabs

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cpdef np.ndarray[double, ndim=1] forward_sub_solve(
    np.ndarray[double, ndim=2] L,
    np.ndarray[double, ndim=1] b):
    '''
    Solve, by forward substition, the system Lx = b, where
    L is a lower triangular matrix.

    Note that the code does not check whether L is lower
    triangular. Calling the function with an arbitrary
    '''

    assert L.shape[1] == b.shape[0], \
        'Matrix and vector are incompatible shapes.'
    assert fabs(L[0, 0]) >= 10e-16, \
        'Zero element on diagonal.'

    # No. of variables to solve
    cdef int n = b.shape[0]

    # Loop indices
    cdef int i, k

    # Initialize the solution vector
    cdef np.ndarray[double, ndim=1] y = np.zeros(b.shape[0])
    y[0] = b[0] / L[0,0]

    cdef double sum_term = 0.0
    for i in xrange(1, n):
        if fabs(L[i, i]) <= 10e-16:
            raise ValueError('Zero element on diagonal.')
        sum_term = 0.0
        for k in xrange(0, i):
            sum_term += L[i, k] * y[k]
        y[i] = (b[i] - sum_term) / L[i, i]

    return y

cpdef np.ndarray[double, ndim=2] backward_sub_solve(
    np.ndarray[double, ndim=2] U,
    np.ndarray[double, ndim=1] b):
    '''
    Solve, by backward substition, the system Ux = b, where
    U is an upper triangular matrix.

    Note that the code does not check whether U is upper
    triangular. Calling the function with an arbitrary
    square matrix will result in nonsense.
    '''
    assert U.shape[1] == b.shape[0], \
        'Matrix and vector are incompatible shapes.'
    assert fabs(U[0, 0]) >= 10e-16, \
        'Zero element on diagonal.'

    # No. of variables to solve
    cdef int n = b.shape[0]

    # Loop indices
    cdef int i, k

    # Initialize the solution vector
    cdef np.ndarray[double, ndim=1] y = np.zeros(b.shape[0])
    y[n-1] = b[n-1] / U[n-1, n-1]

    cdef double sum_term = 0.0
    for i in xrange(n-2, -1, -1):
        if fabs(U[i, i]) <= 10e-16:
            raise ValueError('Zero element on diagonal.')
        sum_term = 0.0
        for k in xrange(i+1, n):
            sum_term += U[i, k] * y[k]
        y[i] = (b[i] - sum_term) / U[i,i]

    return y
```

The functions give identical results to numpy's `solve` function on some sample triangular matrices.

```python
# L is 3x3 lower-triangular
L = np.array([[  2,  0, 0],
             [1.5,  -3, 0],
             [ -5, 0.5, 4]])

# U is 3x3 upper-triangular
U = np.array([[ -1,    3,  -5],
              [  0, -0.5,   2],
              [  0,    0, 1.5]])

b = np.array([-1., 5., 3.])

print 'Cython (forward):', forward_sub_solve(L, b)
print 'Numpy (forward): ', np.linalg.solve(L, b)

print 'Cython (backward):', backward_sub_solve(U, b)
print 'Numpy (backward): ', np.linalg.solve(U, b)

```

    Cython (forward): [-0.5        -1.91666667  0.36458333]
    Numpy (forward):  [-0.5        -1.91666667  0.36458333]
    Cython (backward): [-15.  -2.   2.]
    Numpy (backward):  [-15.  -2.   2.]

Forward and backward substitution are quite fast. Here they're compared to numpy's `solve` function, which isn't quite fair, since the latter is more general. But it gives us a frame of reference.

```python
print 'Numpy solve:'
%timeit np.linalg.solve(L, b)
print 'Cython forward substitution'
%timeit forward_sub_solve(L, b)
print 'Numpy solve:'
%timeit np.linalg.solve(U, b)
print 'Cython backwards substitution'
%timeit backward_sub_solve(U, b)
```

    Numpy solve:
    10000 loops, best of 3: 26.2 µs per loop
    Cython forward substitution
    100000 loops, best of 3: 4.32 µs per loop
    Numpy solve:
    10000 loops, best of 3: 26.2 µs per loop
    Cython backwards substitution
    100000 loops, best of 3: 4.69 µs per loop

Let's also time the function on a large, $1000\times 1000$ matrix, to make sure its performance scales.

```python
L = np.random.randn(1e6).reshape((1e3,1e3))
cond = np.subtract.outer(np.arange(1e3), np.arange(1e3))
L[cond < 0] = 0.
b = np.random.randn(1e3)

%timeit forward_sub_solve(L, b)
```

    1000 loops, best of 3: 452 µs per loop

----------------

## The LU(P) decomposition

We’ve established that triangular systems are easy to solve and can be implemented in fast code. This is fortunate, becuase it turns out that any square matrix can be represented as the product of a lower and an upper triangular matrix. That is,

$$
A = LU\,.
$$

This representation is called the LU decomposition of $A$. Using this decomposition, the linear system $Ax = b$ can be represented as

$$
\begin{align}
Ax &= b\\\
LUx &= b\\\
Ly  &= b\,,
\end{align}
$$

where $y \equiv Ux$. We can easily solve for $y$ using forward substitution. Having done that, we have

$$
Ux = y\,,
$$

and we can now solve for $x$ by forward substituion.

The algorithm—actually there is more than one—to find $L$ and $U$ is not complicated. It's easiest to see working through an example. Consider the $3\times 3$ matrix
$$
A =
\begin{pmatrix}
2 & -1 & 4 \\\
-1 & 3 & \frac{1}{2} \\\
5 & -9 & 11
\end{pmatrix}\,.
$$

We initialize $U = A$, and $L = I$. Then we proceed to set the sub-diagonal element of U to zero using gaussian elimination. In the first step, we'll remove the sub-diagonal entries in the first column of $U$. There is a separate elimination for each row. To eliminate $u_{10}$ we multiply each entry in the first row by $\frac{u_{10}}{u_{00}} = -\frac{1}{2}$ and substract it from the second row. That is $u_{1j} \rightarrow u_{1j} + \frac{1}{2} u_{0j}$ for $j = 0,\ldots,2$.

To eliminate $u_{20}$ we similarly substract the first rows times $\frac{u_{20}}{u_{00}} = \frac{5}{2}$ from the third row.

We represent these elimination steps in $L$ by setting $l_{10}$ to $-\frac{1}{2}$ and $l_{20}$ to $\frac{5}{2}$.

This step leaves us with

$$
L = \begin{pmatrix}
1 & 0 & 0 \\\
-\frac{1}{2} & 1 & 0 \\\
\frac{5}{2} & 0 & 1
\end{pmatrix}
$$
and

$$
U =
\begin{pmatrix}
2 & -1 & 4 \\\
0 & \frac{5}{2} & \frac{5}{2} \\\
0 & -\frac{13}{2} & 1\end{pmatrix}\,.
$$

The next step repeats the first, but for the second column of $U$. Here we only need to eliminate one sub-diagonal entry, $u_{21}$. We can do this by subtracting the second row times $\frac{u_{21}}{u_{11}} = -\frac{13}{5}$. Like before, we set $l_{21}$ to $-\frac{13}{5}$ to represent the elimination. We now have

$$
L = \begin{pmatrix}
1 & 0 & 0 \\\
-\frac{1}{2} & 1 & 0 \\\
\frac{5}{2} & -\frac{13}{5} & 1
\end{pmatrix}
$$
and
$$
U =
\begin{pmatrix}
2 & -1 & 4 \\\
0 & \frac{5}{2} & \frac{5}{2} \\\
0 & 0 & \frac{15}{2}
\end{pmatrix}\,.
$$

This completes the process since $U$ is now upper triangular and $L$ is lower triangular.

Let’s test that our calculations are correct. If we did everything correctly, we should recover $A$ from $LU$.

```python
A = np.array([[2.,  -1., 4.],
              [-1.,  3., 0.5],
              [ 5., -9., 11.]])

L = np.array([[1, 0, 0],
              [-1./2, 1, 0],
              [5/2., -13./5, 1]])

U = np.array([[2, -1, 4],
              [0, 5./2, 5./2],
              [0, 0, 15./2]])

if np.all(np.abs(matprod(L, U) - A) < 10e-14):
    print 'A = LU, OK!'
else:
    print 'A != LU, Not OK!'
```

    A = LU, OK!

### The LU decompositon with pivoting

In practice, the LU decomposition algorithm outlined above can sometimes be numerically unstable. At each step, $k$, of the algorithm, we calculate $\frac{u_{ik}}{u_{kk}}$ for $i > k$. If any of the diagonals, $u_{kk}$ (called the pivots are near zero, this calculation can overflow.

To avoid this, an extra step, called pivoting is added to the algorithm. At each step $k$, before performing the gaussian elimination, we swap the rows of $U$ to get the largest possible magnitude for $u_{kk}$. This helps avoid the overflow problem.

In the example above, we start with

$$
U =
\begin{pmatrix}
2 & -1 & 4 \\\
-1 & 3 & \frac{1}{2} \\\
5 & -9 & 11
\end{pmatrix}\,.
$$

Here, we would first switch the first row with the third row, since $5 > 2$. So

$$
U =
\begin{pmatrix}
5 & -9 & 11 \\\
-1 & 3 & \frac{1}{2} \\\
2 & -1 & 4
\end{pmatrix}\,.
$$

The gaussian elimination of the first column would then give us

$$
L =
\begin{pmatrix}
1 & 0 & 0 \\\
-\frac{1}{5} & 1 & 0 \\\
\frac{2}{5} & 0 & 1
\end{pmatrix}
$$

and

$$
U =
\begin{pmatrix}
5 & -9 & 11 \\\
0 & \frac{6}{5} & \frac{27}{10} \\\
0 & \frac{13}{5} & -\frac{2}{5}
\end{pmatrix}\,.
$$

We would also represent the row swap by adding a permutation matrix, $P$, where

$$
P =
\begin{pmatrix}
0 & 0 & 1 \\\
0 & 1 & 0 \\\
1 & 0 & 0
\end{pmatrix}\,.
$$

When this permuation matrix is multiplied by the original $U$, it swaps $U\,$'s rows in the desired way.

In the second step, we’ll have to swap the second and third rows, since $\frac{13}{5} > \frac{6}{5}$. We record this permutation in two ways: first by swapping the second and third rows in $P$; second by swapping the second and third rows of the *first column* of $L$. So we have

$$
P =
\begin{pmatrix}
0 & 0 & 1 \\\
1 & 0 & 0 \\\
0 & 1 & 0
\end{pmatrix}\,.
$$

and

$$
L =
\begin{pmatrix}
1 & 0 & 0 \\\
\frac{2}{5} & 1 & 0 \\\
-\frac{1}{5} & \frac{6}{13} & 1
\end{pmatrix}\,.
$$

Performing the gaussian elimination step on (the swapped-row) $U$ gives us

$$
U =
\begin{pmatrix}
5 & -9 & 11 \\\
0 & \frac{13}{5} & -\frac{2}{5}\\\
0 & 0 & \frac{75}{26}
\end{pmatrix}\,.
$$

With $U$ upper triangular and $L$ lower triangular, the process is complete. But with the pivoting, we no longer have the decomposition  $A = LU$, but instead $PA = LU$. The addition of the permutation adds only a trivial step to solving the system $Ax = b$ using decomposition, and protects us from potential numerical problems.

As before, we’ll check our calculations to make sure the $L$, $U$, and $P$ matrices we calculated satisfy $PA = LU$.

```python
A = np.array([[2.,  -1., 4.],
              [-1.,  3., 0.5],
              [ 5., -9., 11.]])

L = np.array([[1,         0, 0],
              [2./5,      1, 0],
              [-1/5., 6./13, 1]])

U = np.array([[5,   -9,       11],
              [0, 13./5,   -2./5],
              [0,     0, 75./26]])

P = np.array([[0., 0., 1.],
              [1., 0., 0.],
              [0., 1., 0.]])

if np.all(np.abs(matprod(L,U) - matprod(P, A)) < 10e-14):
    print 'LU = PA, OK!'
else:
    print 'LU != PA, Not OK!'

```

    LU = PA, OK!

--------------------

##### Exercise: The LUP algorithm

Generalizing from the example above, the code below performs LUP decomposition for an arbitrary square matrix. There are several points to note.

1. To keep the code clear, the pivoting step is assigned to helper functions called within the main function.
2. The function also keeps track of the number of row swaps performed. This information will come in handy later.
3. Since we’re working with MemoryViews in this function, they are coerced back to arrays in the `return`. Otherwise the function would return MemoryView objects in the result tuple.

```cython
%%cython -lm
import cython
import numpy as np
cimport numpy as np
from libc.math cimport pow

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cpdef lup_decomp(double[:, ::1] A):
    '''
    Perform the LUP decomposition of A:
        PA = LU

    Returns a tuple, (L, U, P, nswaps), where
    nswaps is the number of permutations made
    while performing the decomposition.
    '''

    assert A.shape[0] == A.shape[1], 'Not a square matrix.'

    cdef:
        int n = A.shape[0]
        int i_piv, k, j, h
        int nswaps = 0
        double[:, ::1] U = A.copy()
        double[:, ::1] P = np.eye(n)
        double[:, ::1] L = np.eye(n)

    for k in xrange(n-1):
        # Find the pivot row and permute matrices
        i_piv = get_pivot(U, k)
        if i_piv != k:
            nswaps += 1
            swap_rows(U, k, i_piv, k, n)
            swap_rows(P, k, i_piv, 0, n)
            if k > 0:
                swap_rows(L, k, i_piv, 0, k)

        # Gaussian eliminate sub-diagonal elements of U.
        for j in xrange(k+1, n):
            L[j, k] = U[j, k] / U[k, k]
            for h in xrange(k, n):
                U[j, h] -=  U[k, h]  * L[j, k]

    return (np.asarray(L), np.asarray(U), np.asarray(P), nswaps)

@cython.boundscheck(False)
@cython.wraparound(False)
cdef int get_pivot(double[:, ::1] U, int k):
    '''
    Find the pivot row of column k in a matrix.
    The pivot row is the row i (>= k) of U for which
    abs(U[i, k]) is largest.
    '''
    cdef:
        int j
        int n = U.shape[0]
        int i_piv = k

    for j in xrange(k + 1, n):
        if abs(U[j, k]) > abs(U[i_piv, k]):
            i_piv = j
    return i_piv

@cython.boundscheck(False)
@cython.wraparound(False)
cdef void swap_rows(double[:, ::1] M, int row1, int row2, int col_start, int col_end):
    '''
    A helper function to swap segments of rows in a matrix.

    M[row1, col_start:col_end] <-> M[row2, col_start:col_end]
    '''
    cdef:
        double[::1] first = M[row1, col_start:col_end].copy()

    M[row1, col_start:col_end] = M[row2, col_start:col_end].copy()
    M[row2, col_start:col_end] = first
```

As always, we test its accuracy

```python
A = np.array([[2.,  -1., 4.],
              [-1.,  3., 0.5],
              [ 5., -9., 11.]])

L, U, P, nswaps = lup_decomp(A)
print 'L ='
print L
print 'U ='
print U
print 'P ='
print P
```

    L =
    [[ 1.          0.          0.        ]
     [ 0.4         1.          0.        ]
     [-0.2         0.46153846  1.        ]]
    U =
    [[  5.          -9.          11.        ]
     [  0.           2.6         -0.4       ]
     [  0.           0.           2.88461538]]
    P =
    [[ 0.  0.  1.]
     [ 1.  0.  0.]
     [ 0.  1.  0.]]

Scipy has a function `lu` in its `linalg` module that performs LUP decompositions. (Oddly, numpy does not).

```python
P, L, U = la.lu(A)
print 'L ='
print L
print 'U ='
print U
print 'P ='
print P
```

    L =
    [[ 1.          0.          0.        ]
     [ 0.4         1.          0.        ]
     [-0.2         0.46153846  1.        ]]
    U =
    [[  5.          -9.          11.        ]
     [  0.           2.6         -0.4       ]
     [  0.           0.           2.88461538]]
    P =
    [[ 0.  1.  0.]
     [ 0.  0.  1.]
     [ 1.  0.  0.]]

Note that our results match scipy’s except for the $P$ matrix. This is because scipy perform the decomposition $A = LUP$. Therefore what scipy returns as $P$ is actually the inverse of what we’ve defined as $P$. The inverse of a permutation matrix is its transpose (obvious if you think about it), and that's what we see here. So our function appears accurate.

Now, timings.

```python
A = np.random.randn(1e4).reshape((100, 100))
print 'Scipy lu'
%timeit la.lu(A)
print 'Cython'
%timeit lup_decomp(A)
```

    Scipy lu
    1000 loops, best of 3: 255 µs per loop
    Cython
    100 loops, best of 3: 2.88 ms per loop

On a $100 \times 100$ matrix, our Cython function is ten times slower than scipy. Not great news, but perhaps not surprising given that scipy is calling a highly-optimized Fortran function. There are also probably some low-level optimizations we could make to our code. But this is fast enough to be satisfactory.

-----------

### The determinant redux

Recall that above we convinced ourselves that the determinant of a triangular matrix is just the product of its diagonal entries. Since the determinant of the product of two matrices is the product of their determinants, then for an LU decomposition we have

$$
\begin{align}
\det(A) &= \det(LU)\\\
        &= \det(L)\times\det(U)\\\
        &= \prod_{i=0}^{n-1}l_{ii}\times\prod_{i=0}^{n-1}u_{ii}\\\
        &= \prod_{i=0}^{n-1}u_{ii}\,.
\end{align}
$$

The last line obtains, becuase $L$ has ones along its diagonal. (There are alternative $LU$ decompositions where this isn’t so. For the LUP decomposition we have

$$
\begin{align}
\det(A) &= \det(L)\times \det(U) \times \det\left(P^T\right)\\\
        &= \det\left(P^T\right)\times\prod_{i=0}^{n-1}u_{ii}\,.
\end{align}
$$

It can be shown (but not so easily) that the determinant of $P^T$ is equal to $(-1)^\textrm{# swaps}$. So if we swapped two rows during the decomposition, the determinant would be 1, if we swapped three, it would be $-1$. This is why we recorded the number of row swaps in the `lup_decomp` function above.

So the LUP decomposition not only provides an efficient method for solving a linear system, it also provides an efficient method for computing determinants.

--------------------

##### Exercise: Computing the determinant via LUP decomposition

With the code for the LUP decomposition already written, writing a determinant function is straightforward. Because each `%%cython` cell in the IPython notebook is an independent module, we have to include all the LUP decomposition code in the cell below. But everything below the `lup_determinant` function is identical to what was coded above.

```cython
%%cython -lm
import cython
import numpy as np
cimport numpy as np
from libc.math cimport pow

@cython.boundscheck(False)
@cython.wraparound(False)
cpdef double lup_determinant(double[:, ::1] A):
    assert A.shape[0] == A.shape[1], 'Requires square matrix.'
    cdef:
        int n = A.shape[0]
        double[:, ::1] L
        double[:, ::1] U
        double[:, ::1] P
        int nswaps
        int i

    L, U, P, nswaps = lup_decomp(A)

    cdef double det = pow(-1, nswaps)
    for i in xrange(n):
        det *= U[i, i]
    return det

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cpdef lup_decomp(double[:, ::1] A):
    '''
    Perform the LUP decomposition of A:
        PA = LU

    Returns a tuple, (L, U, P, nswaps), where
    nswaps is the number of permutations made
    while performing the decomposition.
    '''

    assert A.shape[0] == A.shape[1], 'Not a square matrix.'

    cdef:
        int n = A.shape[0]
        int i_piv, k, j, h
        int nswaps = 0
        double[:, ::1] U = A.copy()
        double[:, ::1] P = np.eye(n)
        double[:, ::1] L = np.eye(n)

    for k in xrange(n-1):
        # Find the pivot row and permute matrices
        i_piv = get_pivot(U, k)
        if i_piv != k:
            nswaps += 1
            swap_rows(U, k, i_piv, k, n)
            swap_rows(P, k, i_piv, 0, n)
            if k > 0:
                swap_rows(L, k, i_piv, 0, k)

        # Gaussian eliminate sub-diagonal elements of U.
        for j in xrange(k+1, n):
            L[j, k] = U[j, k] / U[k, k]
            for h in xrange(k, n):
                U[j, h] -=  U[k, h]  * L[j, k]

    return (np.asarray(L), np.asarray(U), np.asarray(P), nswaps)

@cython.boundscheck(False)
@cython.wraparound(False)
cdef int get_pivot(double[:, ::1] U, int k):
    '''
    Find the pivot row of column k in a matrix.
    The pivot row is the row i (>= k) of U for which
    abs(U[i, k]) is largest.
    '''
    cdef:
        int j
        int n = U.shape[0]
        int i_piv = k

    for j in xrange(k + 1, n):
        if abs(U[j, k]) > abs(U[i_piv, k]):
            i_piv = j
    return i_piv

@cython.boundscheck(False)
@cython.wraparound(False)
cdef void swap_rows(double[:, ::1] M, int row1, int row2, int col_start, int col_end):
    '''
    A helper function to swap segments of rows in a matrix.

    M[row1, col_start:col_end] <-> M[row2, col_start:col_end]
    '''
    cdef:
        double[::1] first = M[row1, col_start:col_end].copy()

    M[row1, col_start:col_end] = M[row2, col_start:col_end].copy()
    M[row2, col_start:col_end] = first
```

As always, we test accuracy and speed.

```python
B = np.random.randn(100).reshape((10, 10))
print 'Numpy'
print np.linalg.det(B)
print 'Cython, LUP determinant'
print lup_determinant(B)
```

    Numpy
    19.8309766936
    Cython, LUP determinant
    19.8309766936

```python
print 'Numpy'
%%timeit np.linalg.det(B)
print 'Cython, LUP determinant'
%%timeit lup_determinant(B)
```

    Numpy
    10000 loops, best of 3: 63.3 µs per loop
    Cython, LUP determinant
    1000 loops, best of 3: 236 µs per loop

Our function is still slower than numpy, but in the ballpark. It is dramatically faster than the ill-fated recursive function.

------------------

## Synthesis: Solving a linear system with LUP decomposition

With all the puzzle pieces on the table, the code below arranges them into a linear system solver. The function `lup_solve` simply wraps the determinant, decomposition, and backward and forward substition code already written.

```cython
%%cython -lm
import cython
import numpy as np
cimport numpy as np
from libc.math cimport pow, fabs

def lup_solve(double[:, ::1] A, double[::1] b):
    # Perform the LUP decomposition.
    cdef:
        double[:, ::1] L
        double[:, ::1] U
        double[:, ::1] P
        int nswaps
    L, U, P, nswaps = lup_decomp(A)

    # Is the determinant non-zero?
    cdef double det
    det = determinant_from_lup(U, nswaps)
    assert fabs(det) > 10e-16, 'Zero determinant'

    # Decomposition provides PA = LU therefore
    # Ax = b -> PAx = Pb -> LUx = Pb, which
    # can be solve by backward and forward
    # induction.

    # Permute b
    cdef double [::1] b_permute = matvecprod(P, b)

    # Solve Ly = Pb, y := Ux
    cdef double[::1] y = forward_sub_solve(L, b_permute)

    # Solve Ux = y
    cdef double[::1] x = backward_sub_solve(U, y)

    return np.asarray(x)

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cpdef lup_decomp(double[:, ::1] A):
    '''
    Perform the LUP decomposition of A:
        PA = LU

    Returns a tuple, (L, U, P, nswaps), where
    nswaps is the number of permutations made
    while performing the decomposition.
    '''

    assert A.shape[0] == A.shape[1], 'Not a square matrix.'

    cdef:
        int n = A.shape[0]
        int i_piv, k, j, h
        int nswaps = 0
        double[:, ::1] U = A.copy()
        double[:, ::1] P = np.eye(n)
        double[:, ::1] L = np.eye(n)

    for k in xrange(n-1):
        # Find the pivot row and permute matrices
        i_piv = get_pivot(U, k)
        if i_piv != k:
            nswaps += 1
            swap_rows(U, k, i_piv, k, n)
            swap_rows(P, k, i_piv, 0, n)
            if k > 0:
                swap_rows(L, k, i_piv, 0, k)

        # Gaussian eliminate sub-diagonal elements of U.
        for j in xrange(k+1, n):
            L[j, k] = U[j, k] / U[k, k]
            for h in xrange(k, n):
                U[j, h] -=  U[k, h]  * L[j, k]

    return (np.asarray(L), np.asarray(U), np.asarray(P), nswaps)

@cython.boundscheck(False)
@cython.wraparound(False)
cdef int get_pivot(double[:, ::1] U, int k):
    '''
    Find the pivot row of column k in a matrix.
    The pivot row is the row i (>= k) of U for which
    abs(U[i, k]) is largest.
    '''
    cdef:
        int j
        int n = U.shape[0]
        int i_piv = k

    for j in xrange(k + 1, n):
        if abs(U[j, k]) > abs(U[i_piv, k]):
            i_piv = j
    return i_piv

@cython.boundscheck(False)
@cython.wraparound(False)
cdef void swap_rows(double[:, ::1] M, int row1, int row2, int col_start, int col_end):
    '''
    A helper function to swap segments of rows in a matrix.

    M[row1, col_start:col_end] <-> M[row2, col_start:col_end]
    '''
    cdef:
        double[::1] first = M[row1, col_start:col_end].copy()

    M[row1, col_start:col_end] = M[row2, col_start:col_end].copy()
    M[row2, col_start:col_end] = first

@cython.boundscheck(False)
@cython.wraparound(False)
cdef double determinant_from_lup(double[:, ::1] U, int nswaps):
    '''
    Find the determinant of a matrix from its LUP decomposition
    U is the upper-triangular matrix from the decomposition.
    nswaps is the number of pivot swaps made.
    Both are returned by lup_decomposition; the L and P
    matrices are not needed.
    '''
    cdef double det = pow(-1, nswaps)
    for i in xrange(U.shape[0]):
        det *= U[i, i]
    return det

cimport cython
import numpy as np
cimport numpy as np

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cdef double[::1] forward_sub_solve(double[:, ::1] L, double[::1] b):
    '''
    Solve, by forward substition, the system Lx = b, where
    L is a lower triangular matrix.

    Note that the code does not check whether L is lower
    triangular. Calling the function with an arbitrary
    '''

    assert L.shape[1] == b.shape[0], \
        'Matrix and vector are incompatible shapes.'
    assert fabs(L[0, 0]) >= 10e-16, \
        'Zero element on diagonal.'

    # No. of variables to solve
    cdef int n = b.shape[0]

    # Loop indices
    cdef int i, k

    # Initialize the solution vector
    cdef double[::1] y = np.zeros(b.shape[0])
    y[0] = b[0] / L[0,0]

    cdef double sum_term = 0.0
    for i in xrange(1, n):
        if fabs(L[i, i]) <= 10e-16:
            raise ValueError('Zero element on diagonal.')
        sum_term = 0.0
        for k in xrange(0, i):
            sum_term += L[i, k] * y[k]
        y[i] = (b[i] - sum_term) / L[i, i]

    return y

cdef double[::1] backward_sub_solve(double[:, ::1] U, double[::1] b):
    '''
    Solve, by backward substition, the system Ux = b, where
    U is an upper triangular matrix.

    Note that the code does not check whether U is upper
    triangular. Calling the function with an arbitrary
    square matrix will result in nonsense.
    '''
    assert U.shape[1] == b.shape[0], \
        'Matrix and vector are incompatible shapes.'
    assert fabs(U[0, 0]) >= 10e-16, \
        'Zero element on diagonal.'

    # No. of variables to solve
    cdef int n = b.shape[0]

    # Loop indices
    cdef int i, k

    # Initialize the solution vector
    cdef double[::1] y = np.zeros(b.shape[0])
    y[n-1] = b[n-1] / U[n-1, n-1]

    cdef double sum_term = 0.0
    for i in xrange(n-2, -1, -1):
        if fabs(U[i, i]) <= 10e-16:
            raise ValueError('Zero element on diagonal.')
        sum_term = 0.0
        for k in xrange(i+1, n):
            sum_term += U[i, k] * y[k]
        y[i] = (b[i] - sum_term) / U[i,i]

    return y

@cython.boundscheck(False)
@cython.wraparound(False)
cdef double[::1] matvecprod(double[:, ::1] A, double[::1] b):
    '''
    Matrix-vector multiplication.
    '''
    cdef:
        Py_ssize_t i, j, k
        Py_ssize_t A_n = A.shape[0]
        Py_ssize_t A_m = A.shape[1]
        Py_ssize_t b_n = b.shape[0]
        double[::1] c

    # Are matrices conformable?
    assert A_m == b_n, \
        'Non-conformable shapes.'

    # Initialize the results matrix.
    c = np.zeros(A_n)
    for i in xrange(A_n):
            for k in xrange(b_n):
                c[i] += A[i, k] * b[k]
    return c
```

Having already tested its component functions, the solver ought to be correct. It matches numpy’s solver on a sample $3\times 3$ system.

```python
A = np.array([[1., -2, 2], [4, 1, 3], [-2, 3, 1]])
b = np.array([-10, 4., .25])

print 'Cython:'
print lup_solve(A, b)
print 'Numpy solve:'
print np.linalg.solve(A, b)
```

    Cython:
    [ 2.75     3.03125 -3.34375]
    Numpy solve:
    [ 2.75     3.03125 -3.34375]

While it is substantially slower than numpy—about 6 times—the absolute speed is acceptable on this small system.

```python
print 'Cython:'
%timeit lup_solve(A, b)
print 'Numpy solve:'
%timeit np.linalg.solve(A, b)
```

    Cython:
    10000 loops, best of 3: 153 µs per loop
    Numpy solve
    10000 loops, best of 3: 26.4 µs per loop

It solves a much larger, $1000\times 1000$ matrix in under a second; about 20 times slower than numpy.

```python
# A larger 1000 x 1000 system.
M = np.random.randn(1e6).reshape((1000, 1000))
c = np.random.randn(1000).reshape((1000,))

# Timings
print 'Cython'
%timeit lup_solve(M, c)
print 'Numpy solve'
%timeit np.linalg.solve(M, c)

# Check that they are the same to tolerance
if np.abs(lup_solve(M, c) - np.linalg.solve(M, c)).sum() > 10e-9:
    print 'Cython != Numpy, not OK'
else:
    print 'Cython = Numpy, OK'
```

## Iterative Methods: A whole other route

Iterative methods are an altnerative means of solving linear systems. They are not always successful, but when they are, they can be very efficient.

Let $Q$ be some matrix that is easily invertible; some cadidates are mentioned below. Consider the following re-arrangement of the matrix equation:

$$
\begin{align}
Ax &= b \\\
 0 &= b - Ax \\\
Qx &= b - Ax + Qx \\\
Qx &= b + (Q - A)\,x \\\
x  &= Q^{-1}b + (I-Q^{-1}A)\,x \,.
\end{align}
$$

The last line suggests an iteration on $x$ along the following lines.

$$
\begin{align}
x^{(0)} &\leftarrow \tilde{x} \ \textrm{(guess)} \\\
x^{(k+1)} &\leftarrow Q^{-1}b + (I-Q^{-1}A)\,x^{(k)}\\\
\textrm{until} & \left|x^{(k-1)} - x^{(k)}\right| \le \varepsilon
\end{align}
$$

This is the generic form of an iterative solver. Specific methods differ in their choice of $Q$ or implementation details.

The Gauss-Siedel method is a simple, relatively robust implementation of the iteration where $Q$ is an upper triangular matrix whose entries  are the upper-triangular entries of $A$. There is a somewhat simpler method, called the Jacobi method, where $Q$ is taken to be a diagonal matrix comprised of the diagonal elements of $A$. The Gauss-Siedel often has more robust convergence, though that depends on the properties of $A$.

Both methods converge better when the $A$ is diagonally dominant, when the value of each diagonal element is larger than the sum of entries in the column and row of that element. Or:

$$
\left|a_{ii}\right| > \sum_{i=1\,;\,j\ne i}^n\left|a_{ij}\right| \;\;\; \forall\, i
$$

______________________

##### Exercise: The Gauss-Siedel Algorithm

The function below implements the Gauss-Siedel method. Note that the matrix $Q$ is never formally referred to in the code. It’s inversion is baked into the iterative formula. Similarities with the forward substitution formulas coded above are not coincidental.

```cython
%%cython -lm
import cython
import numpy as np
cimport numpy as np
from libc.math cimport abs

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.cdivision(True)
cpdef gauss_siedel_solve(double[:, ::1] A, double[::1] b,
                         double tol = 10e-12, int max_iter = 100):
    '''
    Solve a linear system Ax = b using the Gauss-Siedel iterative
    method.
    '''

    assert A.shape[0] == A.shape[1], 'Matrix is not square.'
    assert A.shape[1] == b.shape[0], \
        'Non-conformable matrix and vector.'
    cdef:
        int n = A.shape[0]
        double[::1] x = np.ones(n)
        int iter_i = 0
        double tol_i = 10e12 # Large number to start.
        double sum_term = 0.
        double new_x_i
        int i, j

    while (tol_i > tol):
        tol_i = 0.
        for i in xrange(n):
            assert abs(A[i, i]) > 10e-16, 'Zero on diagonal detected.'
            sum_term = 0.
            for j in xrange(n):
                if j != i:
                    sum_term += A[i, j] * x[j]
            new_x_i = (b[i] - sum_term) / A[i, i]
            tol_i += abs(new_x_i - x[i])
            x[i] = new_x_i

        iter_i += 1
        if iter_i > max_iter:
            print 'Max iterations, solution may not have converged.'
            print 'Matrix may not be diagonally dominant.'
            break

    return (np.asarray(x), iter_i, tol_i)
```

On a randomly generated $5\times 5$ system, the Gauss-Seidel converges to the solution in 10 to 12 iterations.
Note how $A$ was agumented to be diagonally dominant. Without that change, the Gauss-Seidel method typically does not converge.

```python
# Randomly generate a 5 x 5 linear system.
A = np.random.randn(25).reshape((5,5)) + np.eye(5) * 10
b = np.random.randn(5)

print 'Numpy Solve'
print np.linalg.solve(A, b)
x, niter, tol = gauss_siedel_solve(A, b)
print 'Cython Gauss Siedel'
print x, '\n  # Iterations: ', niter
```

    Numpy Solve
    [ 0.05977965  0.00185487  0.00081283 -0.00602107  0.11776965]
    Cython Gauss Siedel
    [ 0.05977965  0.00185487  0.00081283 -0.00602107  0.11776965]
      # Iterations:  11

To time the method, we’ll use a large sparse system. The cell below generates a matrix fully populated on the diagonal, but with mostly zeros elsewhere.

```python
# Randomly generate a 1000 x 1000 sparse linear system
n = 1000
n_entries = int(0.1 * n)
A = np.eye(n) * np.random.randn(n)
off_diag_entries = np.random.randn(n_entries)
i = 0
while (i < n_entries):
    fill_row = np.random.random_integers(0, n)
    fill_col = np.random.random_integers(0, n)
    if (fill_row == fill_col):
        continue
    else:
        A[fill_row, fill_col] = off_diag_entries[i]
        i += 1

b = np.random.randn(1000)
```

For this system, the Gauss Siedel method is substantially faster, and it converges in only a few iterations. Checking the result shows the solution it converged to is essentially the same as numpy.

```python
print 'Numpy solve'
%timeit np.linalg.solve(A, b)
print 'Cython Gauss-Siedel'
%timeit gauss_siedel_solve(A, b)
```

    Numpy solve
    10 loops, best of 3: 26.1 ms per loop
    Cython Gauss-Siedel
    100 loops, best of 3: 3.34 ms per loop

```python
y_np = np.linalg.solve(A, b)
y_gs, niter, tol = gauss_siedel_solve(A, b)
print 'Gauss Siedel # iterations:', niter
print 'Sum abs. difference between numpy and Gauss-Seidel:', np.abs(y_np - y_gs).sum()

```

    Gauss Siedel # iterations: 3
    Sum abs. difference: 2.87827053858e-13


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 7

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Mar. 5 at **12:30pm.**

## Exercise: The Labor-Leisure Tradeoff

\begin{align}
\frac{\varphi}{1-L_t} & = \frac{(1-\alpha)A_tK_t^{\alpha}L_t^{-\alpha}}{C_t} \tag{1}
\end{align}

**Questions**

1. Explain words why the left-hand side of equation (1) represents the marginal cost to the household of working. A complete answer will make use of the term *marginal utility* .
2. Explain words why the right-hand side of equation (1) represents the marginal benefit to the household of supplying labor (i.e., working). A complete answer will make use of the terms *marginal utility* and *marginal product*.
3. Holding everything else constant, according to equation (1), what effect will an increase in TFP have on equilibrium labor? Explain the economic intuition behind your answer.
4. Holding everything else constant, according to equation (1), what effect will an increase in household consumption have on equilibrium labor? Explain the economic intuition behind your answer.

**Answers**

1. The left-hand side is the derivative of the household's period $t$ utility flow with respect to $1-L_t$ and is therefore the marginal utility of leisure. A marginal increase in work effort leads to a marginal decrease in leisure of equal magnitude so the left-hand side of equation (1) reflects the utility cost at the margin of working. <!-- answer -->
2. The right-hand side of equation (1) is the marginal product of labor times the marginal utility of consumption. A marginal increase in work effort raises household income, and therefore consumption, by the marginal product of labor: $(1-\alpha)A_t K_t^{\alpha}L_t^{-\alpha}$. Accodring to the chain rule: $\partial u /\partial L= (\partial u /\partial C)(\partial C /\partial L)$, and so equation (1) is the additional utility at the margin of working.<!-- answer -->
3. Multiply both sides of equation (1) by $L^{\alpha}$ and obtain $\varphi L^{\alpha}/(1-L_t) = (1-\alpha)A_tK_t^{\alpha}/C_t$. Increasing TFP increases the right-hand side and therefore increases labor supplied because the left-hand side is an increasing function of $L$. Intuitively, an increase in TFP increases the margianl product of labor which effectively raises the price of leisure relative to consumption so the household cuts back on leisure. <!-- answer -->
4. Increasing consumption decreases labor supplied. Intuitively, an increase in consumption reduces the marginal utility of income, reduces the household's incentive to earn income from working, and so the household takes enjoys more leisure. <!-- answer -->

## Exercise: The Euler Equation

\begin{align}
\frac{1}{C_t} & = \beta \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1}L_{t+1}^{1-\alpha} +1-\delta }{C_{t+1}}\right]\tag{2}
\end{align}

**Questions**

1. Explain words why the left-hand side of equation (2) represents the marginal cost to the household of saving (i.e., building new capital). A complete answer will make use of the term *marginal utility* .
2. Explain words why the right-hand side of equation (2) represents the marginal benefit to the household of saving. A complete answer will make use of the terms *marginal utility* and *marginal product*.
3. Holding everything else constant, according to equation (2), what effect will an increase in TFP in period $t+1$ have on the household's choice for capital in period $t+1$? Explain the economic intuition behind your answer.
3. Holding everything else constant, according to equation (2), what effect will an increase in consumption in period $t$ have on the household's choice for capital in period $t+1$? Explain the economic intuition behind your answer.
3. Holding everything else constant, according to equation (2), what effect will an increase in consumption in period $t+1$ have on the household's choice for capital in period $t+1$? Explain the economic intuition behind your answer.

**Answers**

1. The left-hand side is the derivative of the household's period $t$ utility flow with respect to $C_t$ and is therefore the marginal utility of consumption in period $t$. A marginal increase in saving reduces current consuption by an equal magnitude and so the left-hand side of equation (1) reflects the utility cost at the margin of saving. <!-- answer -->
2. The right-hand side of equation (2) represents the marginal benefit to the household of saving because a marginal increase in period $t+1$ capital increases period $t+1$ consumption by the marginal product of capital plus the share of that capital that doesn't depreciate and so, by the chain rule, the increase in the $t+1$ utility flow is: $(\alpha A_{t+1} K^{\alpha-1}L^{1-\alpha}+1-\delta)/C_{t+1}$. Since the houshold realizes this change in the future, it's discounted by the subjective discount factor $\beta$. <!-- answer -->
3. An increase in TFP in $t+1$ would raise the $t+1$ marginal product of capital and so, since the left-hand side is unchanged, period $t+1$ capital increases to so that the marginal product of capital remains unchanged. Intuitively, an increase in future TFP raises the return to saving (in terms of future consumption) and so the household saves more. <!-- answer -->
3. An increase in consumption in $t$ lowers the marginal utility of consumption in period $t$. Therefore the household will try to save more and so period $t+1$ capital increases. Intuitively, a household that suddenly gains more consumption in the current period will try to also increase future consumption by saving more. <!-- answer -->
4. An increase in consumption in $t+1$ lowers the marginal utility of consumption in period $t+1$. Therefore the household will try to save less and so period $t+1$ capital increases. Intuitively, a household that suddenly gains more consumption in the next period period will try to also increase current consumption by saving less. <!-- answer -->