# Linear Algebra

## Contents

- [Linear Algebra](#Linear-Algebra)
  - [Overview](#Overview)
  - [Vectors](#Vectors)
  - [Matrices](#Matrices)
  - [Solving Systems of Equations](#Solving-Systems-of-Equations)
  - [Eigenvalues and Eigenvectors](#Eigenvalues-and-Eigenvectors)
  - [Further Topics](#Further-Topics)
  - [Exercises](#Exercises)
  - [Solutions](#Solutions)

## Overview

Linear algebra is one of the most useful branches of applied mathematics for economists to invest in

For example, many applied problems in economics and finance require the solution of a linear system of equations, such as

$$
\begin{array}{c}
    y_1 = a x_1 + b x_2 \\
    y_2 = c x_1 + d x_2
\end{array}
$$

or, more generally,

$$
\begin{array}{c}
    y_1 = a_{11} x_1 + a_{12} x_2 + \cdots + a_{1k} x_k \\
    \vdots  \\
    y_n = a_{n1} x_1 + a_{n2} x_2 + \cdots + a_{nk} x_k
\end{array} \tag{1}
$$

The objective here is to solve for the “unknowns” $ x_1, \ldots, x_k $ given $ a_{11}, \ldots, a_{nk} $ and $ y_1, \ldots, y_n $

When considering such problems, it is essential that we first consider at least some of the following questions

- Does a solution actually exist?
- Are there in fact many solutions, and if so how should we interpret them?
- If no solution exists, is there a best “approximate” solution?
- If a solution exists, how should we compute it?

These are the kinds of topics addressed by linear algebra

In this lecture we will cover the basics of linear and matrix algebra, treating both theory and computation

We admit some overlap with [this lecture](getting_started_julia/fundamental_types.html), where operations on Julia arrays were first explained

Note that this lecture is more theoretical than most, and contains background
material that will be used in applications as we go along

## Vectors

A *vector* is an element of a vector space.

Vectors can be added together and scaled (multiplied) by scalars.

Vectors can be written as $ x = [x_1, \ldots, x_n] $

The set of all $ n $-vectors is denoted by $ \mathbb R^n $

For example, $ \mathbb R ^2 $ is the plane, and a vector in $ \mathbb R^2 $ is just a point in the plane

Traditionally, vectors are represented visually as arrows from the origin to
the point

The following figure represents three vectors in this manner

### Setup

```julia
using InstantiateFromURL
activate_github("QuantEcon/QuantEconLecturePackages", tag = "v0.9.8");
```

```julia
using LinearAlgebra, Statistics, Compat, Plots
gr(fmt=:png);
```

```julia
x_vals = [0 0 0 ; 2 -3 -4]
y_vals = [0 0 0 ; 4 3 -3.5]

plot(x_vals, y_vals, arrow = true, color = :blue,
     legend = :none, xlims = (-5, 5), ylims = (-5, 5),
     annotations = [(2.2, 4.4, "[2, 4]"),
                    (-3.3, 3.3, "[-3, 3]"),
                    (-4.4, -3.85, "[-4, -3.5]")],
     xticks = -5:1:5, yticks = -5:1:5,
     framestyle = :origin)
```

### Vector Operations

The two most common operators for vectors are addition and scalar multiplication, which we now describe

As a matter of definition, when we add two vectors, we add them element by element

$$
x + y
=
\left[
\begin{array}{c}
    x_1 \\
    x_2 \\
    \vdots \\
    x_n
\end{array}
\right]
+
\left[
\begin{array}{c}
     y_1 \\
     y_2 \\
    \vdots \\
     y_n
\end{array}
\right]
:=
\left[
\begin{array}{c}
    x_1 + y_1 \\
    x_2 + y_2 \\
    \vdots \\
    x_n + y_n
\end{array}
\right]
$$

Scalar multiplication is an operation that takes a number $ \gamma $ and a
vector $ x $ and produces

$$
\gamma x
:=
\left[
\begin{array}{c}
    \gamma x_1 \\
    \gamma x_2 \\
    \vdots \\
    \gamma x_n
\end{array}
\right]
$$

Scalar multiplication is illustrated in the next figure

```julia
# illustrate scalar multiplication

x = [2]
scalars = [-2 1 2]
vals = [0 0 0; x * scalars]
labels = [(-3.6, -4.2, "-2x"), (2.4, 1.8, "x"), (4.4, 3.8, "2x")]

plot(vals, vals, arrow = true, color = [:red :red :blue],
     legend = :none, xlims = (-5, 5), ylims = (-5, 5),
     annotations = labels, xticks = -5:1:5, yticks = -5:1:5,
     framestyle = :origin)
```

In Julia, a vector can be represented as a one dimensional Array

Julia Arrays allow us to express scalar multiplication and addition with a very natural syntax

```julia
x = ones(3)
```

    3-element Array{Float64,1}:
     1.0
     1.0
     1.0

```julia
y = [2, 4, 6]
```

    3-element Array{Int64,1}:
     2
     4
     6

```julia
x + y
```

    3-element Array{Float64,1}:
     3.0
     5.0
     7.0

```julia
4x  # equivalent to 4 * x and 4 .* x
```

    3-element Array{Float64,1}:
     4.0
     4.0
     4.0

### Inner Product and Norm

The *inner product* of vectors $ x,y \in \mathbb R ^n $ is defined as

$$
x' y := \sum_{i=1}^n x_i y_i
$$

Two vectors are called *orthogonal* if their inner product is zero

The *norm* of a vector $ x $ represents its “length” (i.e., its distance from the zero vector) and is defined as

$$
\| x \| := \sqrt{x' x} := \left( \sum_{i=1}^n x_i^2 \right)^{1/2}
$$

The expression $ \| x - y\| $ is thought of as the distance between $ x $ and $ y $

Continuing on from the previous example, the inner product and norm can be computed as
follows

```julia
using LinearAlgebra
```

```julia
dot(x, y)               # Inner product of x and y
```

    12.0

```julia
sum(prod, zip(x, y))    # Gives the same result
```

    12.0

```julia
norm(x)                 # Norm of x
```

    1.7320508075688772

```julia
sqrt(sum(abs2, x))         # Gives the same result
```

    1.7320508075688772

### Span

Given a set of vectors $ A := \{a_1, \ldots, a_k\} $ in $ \mathbb R ^n $, it’s natural to think about the new vectors we can create by performing linear operations

New vectors created in this manner are called *linear combinations* of $ A $

In particular, $ y \in \mathbb R ^n $ is a linear combination of $ A := \{a_1, \ldots, a_k\} $ if

$$
y = \beta_1 a_1 + \cdots + \beta_k a_k
\text{ for some scalars } \beta_1, \ldots, \beta_k
$$

In this context, the values $ \beta_1, \ldots, \beta_k $ are called the *coefficients* of the linear combination

The set of linear combinations of $ A $ is called the *span* of $ A $

The next figure shows the span of $ A = \{a_1, a_2\} $ in $ \mathbb R ^3 $

The span is a 2 dimensional plane passing through these two points and the origin

```julia
# fixed linear function, to generate a plane
f(x, y) = 0.2x + 0.1y

# lines to vectors
x_vec = [0 0; 3 3]
y_vec = [0 0; 4 -4]
z_vec = [0 0; f(3, 4) f(3, -4)]

# draw the plane
n = 20
grid = range(-5, 5, length = n)
z2 = [ f(grid[row], grid[col]) for row in 1:n, col in 1:n ]
wireframe(grid, grid, z2, fill = :blues, gridalpha =1 )
plot!(x_vec, y_vec, z_vec, color = [:blue :green], linewidth = 3, labels = "",
      colorbar = false)
```

#### Examples

If $ A $ contains only one vector $ a_1 \in \mathbb R ^2 $, then its
span is just the scalar multiples of $ a_1 $, which is the unique line passing through both $ a_1 $ and the origin

If $ A = \{e_1, e_2, e_3\} $ consists  of the *canonical basis vectors* of $ \mathbb R ^3 $, that is

$$
e_1
:=
\left[
\begin{array}{c}
     1 \\
     0 \\
     0
\end{array}
\right]
, \quad
e_2
:=
\left[
\begin{array}{c}
     0 \\
     1 \\
     0
\end{array}
\right]
, \quad
e_3
:=
\left[
\begin{array}{c}
     0 \\
     0 \\
     1
\end{array}
\right]
$$

then the span of $ A $ is all of $ \mathbb R ^3 $, because, for any
$ x = (x_1, x_2, x_3) \in \mathbb R ^3 $, we can write

$$
x = x_1 e_1 + x_2 e_2 + x_3 e_3
$$

Now consider $ A_0 = \{e_1, e_2, e_1 + e_2\} $

If $ y = (y_1, y_2, y_3) $ is any linear combination of these vectors, then $ y_3 = 0 $ (check it)

Hence $ A_0 $ fails to span all of $ \mathbb R ^3 $

### Linear Independence

As we’ll see, it’s often desirable to find families of vectors with relatively large span, so that many vectors can be described by linear operators on a few vectors

The condition we need for a set of vectors to have a large span is what’s called linear independence

In particular, a collection of vectors $ A := \{a_1, \ldots, a_k\} $ in $ \mathbb R ^n $ is said to be

- *linearly dependent* if some strict subset of $ A $ has the same span as $ A $
- *linearly independent* if it is not linearly dependent

Put differently, a set of vectors is linearly independent if no vector is redundant to the span, and linearly dependent otherwise

To illustrate the idea, recall [the figure](#la-3dvec) that showed the span of vectors $ \{a_1, a_2\} $ in $ \mathbb R ^3 $ as a plane through the origin

If we take a third vector $ a_3 $ and form the set $ \{a_1, a_2, a_3\} $, this set will be

- linearly dependent if $ a_3 $ lies in the plane
- linearly independent otherwise

As another illustration of the concept, since $ \mathbb R ^n $ can be spanned by $ n $ vectors
(see the discussion of canonical basis vectors above), any collection of
$ m > n $ vectors in $ \mathbb R ^n $ must be linearly dependent

The following statements are equivalent to linear independence of $ A := \{a_1, \ldots, a_k\} \subset \mathbb R ^n $

1. No vector in $ A $ can be formed as a linear combination of the other elements
1. If $ \beta_1 a_1 + \cdots \beta_k a_k = 0 $ for scalars $ \beta_1, \ldots, \beta_k $, then $ \beta_1 = \cdots = \beta_k = 0 $

(The zero in the first expression is the origin of $ \mathbb R ^n $)

### Unique Representations

Another nice thing about sets of linearly independent vectors is that each element in the span has a unique representation as a linear combination of these vectors

In other words, if $ A := \{a_1, \ldots, a_k\} \subset \mathbb R ^n $ is
linearly independent and

$$
y = \beta_1 a_1 + \cdots \beta_k a_k
$$

then no other coefficient sequence $ \gamma_1, \ldots, \gamma_k $ will produce
the same vector $ y $

Indeed, if we also have $ y = \gamma_1 a_1 + \cdots \gamma_k a_k $,
then

$$
(\beta_1 - \gamma_1) a_1 + \cdots + (\beta_k - \gamma_k) a_k = 0
$$

Linear independence now implies $ \gamma_i = \beta_i $ for all $ i $

## Matrices

Matrices are a neat way of organizing data for use in linear operations

An $ n \times k $ matrix is a rectangular array $ A $ of numbers with $ n $ rows and $ k $ columns:

$$
A =
\left[
\begin{array}{cccc}
    a_{11} & a_{12} & \cdots & a_{1k} \\
    a_{21} & a_{22} & \cdots & a_{2k} \\
    \vdots & \vdots &  & \vdots \\
    a_{n1} & a_{n2} & \cdots & a_{nk}
\end{array}
\right]
$$

Often, the numbers in the matrix represent coefficients in a system of linear equations, as discussed at the start of this lecture

For obvious reasons, the matrix $ A $ is also called a vector if either $ n = 1 $ or $ k = 1 $

In the former case, $ A $ is called a *row vector*, while in the latter it is called a *column vector*

If $ n = k $, then $ A $ is called *square*

The matrix formed by replacing $ a_{ij} $ by $ a_{ji} $ for every $ i $ and $ j $ is called the *transpose* of $ A $, and denoted $ A' $ or $ A^{\top} $

If $ A = A' $, then $ A $ is called *symmetric*

For a square matrix $ A $, the $ i $ elements of the form $ a_{ii} $ for $ i=1,\ldots,n $ are called the *principal diagonal*

$ A $ is called *diagonal* if the only nonzero entries are on the principal diagonal

If, in addition to being diagonal, each element along the principal diagonal is equal to 1, then $ A $ is called the *identity matrix*, and denoted by $ I $

### Matrix Operations

Just as was the case for vectors, a number of algebraic operations are defined for matrices

Scalar multiplication and addition are immediate generalizations of the vector case:

$$
\gamma A
=
\gamma
\left[
\begin{array}{ccc}
    a_{11} &  \cdots & a_{1k} \\
    \vdots & \vdots  & \vdots \\
    a_{n1} &  \cdots & a_{nk} \\
\end{array}
\right]
:=
\left[
\begin{array}{ccc}
    \gamma a_{11} & \cdots & \gamma a_{1k} \\
    \vdots & \vdots & \vdots \\
    \gamma a_{n1} & \cdots & \gamma a_{nk} \\
\end{array}
\right]
$$

and

$$
A + B =
\left[
\begin{array}{ccc}
    a_{11} & \cdots & a_{1k} \\
    \vdots & \vdots & \vdots \\
    a_{n1} & \cdots & a_{nk} \\
\end{array}
\right]
+
\left[
\begin{array}{ccc}
    b_{11} & \cdots & b_{1k} \\
    \vdots & \vdots & \vdots \\
    b_{n1} & \cdots & b_{nk} \\
\end{array}
\right]
:=
\left[
\begin{array}{ccc}
    a_{11} + b_{11} &  \cdots & a_{1k} + b_{1k} \\
    \vdots & \vdots & \vdots \\
    a_{n1} + b_{n1} &  \cdots & a_{nk} + b_{nk} \\
\end{array}
\right]
$$

In the latter case, the matrices must have the same shape in order for the definition to make sense

We also have a convention for *multiplying* two matrices

The rule for matrix multiplication generalizes the idea of inner products discussed above,
and is designed to make multiplication play well with basic linear operations

If $ A $ and $ B $ are two matrices, then their product $ A B $ is formed by taking as its
$ i,j $-th element the inner product of the $ i $-th row of $ A $ and the
$ j $-th column of $ B $

There are many tutorials to help you visualize this operation, such as [this one](http://www.mathsisfun.com/algebra/matrix-multiplying.html), or the discussion on the [Wikipedia page](https://en.wikipedia.org/wiki/Matrix_multiplication)

If $ A $ is $ n \times k $ and $ B $ is $ j \times m $, then
to multiply $ A $ and $ B $ we require $ k = j $, and the
resulting matrix $ A B $ is $ n \times m $

As perhaps the most important special case, consider multiplying $ n \times k $ matrix $ A $ and $ k \times 1 $ column vector $ x $

According to the preceding rule, this gives us an $ n \times 1 $ column vector

$$
A x =
\left[
\begin{array}{ccc}
    a_{11} &  \cdots & a_{1k} \\
    \vdots & \vdots  & \vdots \\
    a_{n1} &  \cdots & a_{nk}
\end{array}
\right]
\left[
\begin{array}{c}
    x_{1}  \\
    \vdots  \\
    x_{k}
\end{array}
\right] :=
\left[
\begin{array}{c}
    a_{11} x_1 + \cdots + a_{1k} x_k \\
    \vdots \\
    a_{n1} x_1 + \cdots + a_{nk} x_k
\end{array}
\right] \tag{2}
$$

>**Note**
>
>$ A B $ and $ B A $ are not generally the same thing

Another important special case is the identity matrix

You should check that if $ A $ is $ n \times k $ and $ I $ is the $ k \times k $ identity matrix, then $ AI = A $

If $ I $ is the $ n \times n $ identity matrix, then $ IA = A $

### Matrices in Julia

Julia arrays are also used as matrices, and have fast, efficient functions and methods for all the standard matrix operations

You can create them as follows

```julia
A = [1 2
     3 4]
```

    2×2 Array{Int64,2}:
     1  2
     3  4

```julia
typeof(A)
```

    Array{Int64,2}

```julia
size(A)
```

    (2, 2)

The `size` function returns a tuple giving the number of rows and columns

To get the transpose of `A`, use `transpose(A)` or, more simply, `A'`

There are many convenient functions for creating common matrices (matrices of zeros, ones, etc.) — see [here](getting_started_julia/fundamental_types.html#creating-arrays)

Since operations are performed elementwise by default, scalar multiplication and addition have very natural syntax

```julia
A = ones(3, 3)
```

    3×3 Array{Float64,2}:
     1.0  1.0  1.0
     1.0  1.0  1.0
     1.0  1.0  1.0

```julia
2I
```

    UniformScaling{Int64}
    2*I

```julia
A + I
```

    3×3 Array{Float64,2}:
     2.0  1.0  1.0
     1.0  2.0  1.0
     1.0  1.0  2.0

To multiply matrices we use the `*` operator

In particular, `A * B` is matrix multiplication, whereas `A .* B` is element by element multiplication

### Matrices as Maps

Each $ n \times k $ matrix $ A $ can be identified with a function $ f(x) = Ax $ that maps $ x \in \mathbb R ^k $ into $ y = Ax \in \mathbb R ^n $

These kinds of functions have a special property: they are *linear*

A function $ f \colon \mathbb R ^k \to \mathbb R ^n $ is called *linear* if, for all $ x, y \in \mathbb R ^k $ and all scalars $ \alpha, \beta $, we have

$$
f(\alpha x + \beta y) = \alpha f(x) + \beta f(y)
$$

You can check that this holds for the function $ f(x) = A x + b $ when $ b $ is the zero vector, and fails when $ b $ is nonzero

In fact, it’s [known](https://en.wikipedia.org/wiki/Linear_map#Matrices) that $ f $ is linear if and *only if* there exists a matrix $ A $ such that $ f(x) = Ax $ for all $ x $

## Solving Systems of Equations

Recall again the system of equations [(1)](#equation-la-se)

If we compare [(1)](#equation-la-se) and [(2)](#equation-la-atx), we see that [(1)](#equation-la-se) can now be
written more conveniently as

$$
y = Ax \tag{3}
$$

The problem we face is to determine a vector $ x \in \mathbb R ^k $ that solves [(3)](#equation-la-se2), taking $ y $ and $ A $ as given

This is a special case of a more general problem: Find an $ x $ such that $ y = f(x) $

Given an arbitrary function $ f $ and a $ y $, is there always an $ x $ such that $ y = f(x) $?

If so, is it always unique?

The answer to both these questions is negative, as the next figure shows

```julia
f(x) = 0.6cos(4x) + 1.3
grid = range(-2, 2, length = 100)
y_min, y_max = extrema( f(x) for x in grid )
plt1 = plot(f, xlim = (-2, 2), label = "f")
hline!(plt1, [f(0.5)], linestyle = :dot, linewidth = 2, label = "")
vline!(plt1, [-1.07, -0.5, 0.5, 1.07], linestyle = :dot, linewidth = 2, label = "")
plot!(plt1, fill(0, 2), [y_min y_min; y_max y_max], lw = 3, color = :blue,
      label = ["range of f" ""])
plt2 = plot(f, xlim = (-2, 2), label = "f")
hline!(plt2, [2], linestyle = :dot, linewidth = 2, label = "")
plot!(plt2, fill(0, 2), [y_min y_min; y_max y_max], lw = 3, color = :blue,
      label = ["range of f" ""])
plot(plt1, plt2, layout = (2, 1), ylim = (0, 3.5))
```

In the first plot there are multiple solutions, as the function is not one-to-one, while
in the second there are no solutions, since $ y $ lies outside the range of $ f $

Can we impose conditions on $ A $ in [(3)](#equation-la-se2) that rule out these problems?

In this context, the most important thing to recognize about the expression
$ Ax $ is that it corresponds to a linear combination of the columns of $ A $

In particular, if $ a_1, \ldots, a_k $ are the columns of $ A $, then

$$
Ax = x_1 a_1 + \cdots + x_k a_k
$$

Hence the range of $ f(x) = Ax $ is exactly the span of the columns of $ A $

We want the range to be large, so that it contains arbitrary $ y $

As you might recall, the condition that we want for the span to be large is [linear independence](#la-li)

A happy fact is that linear independence of the columns of $ A $ also gives us uniqueness

Indeed, it follows from our [earlier discussion](#la-unique-reps) that if $ \{a_1, \ldots, a_k\} $ are linearly independent and $ y = Ax = x_1 a_1 + \cdots + x_k a_k $, then no $ z \not= x $ satisfies $ y = Az $

### The $ n \times n $ Case

Let’s discuss some more details, starting with the case where $ A $ is $ n \times n $

This is the familiar case where the number of unknowns equals the number of equations

For arbitrary $ y \in \mathbb R ^n $, we hope to find a unique $ x \in \mathbb R ^n $ such that $ y = Ax $

In view of the observations immediately above, if the columns of $ A $ are
linearly independent, then their span, and hence the range of $ f(x) =
Ax $, is all of $ \mathbb R ^n $

Hence there always exists an $ x $ such that $ y = Ax $

Moreover, the solution is unique

In particular, the following are equivalent

1. The columns of $ A $ are linearly independent
1. For any $ y \in \mathbb R ^n $, the equation $ y = Ax $ has a unique solution

The property of having linearly independent columns is sometimes expressed as having *full column rank*

#### Inverse Matrices

Can we give some sort of expression for the solution?

If $ y $ and $ A $ are scalar with $ A \not= 0 $, then the
solution is $ x = A^{-1} y $

A similar expression is available in the matrix case

In particular, if square matrix $ A $ has full column rank, then it possesses a multiplicative
*inverse matrix* $ A^{-1} $, with the property that $ A A^{-1} = A^{-1} A = I $

As a consequence, if we pre-multiply both sides of $ y = Ax $ by $ A^{-1} $, we get $ x = A^{-1} y $

This is the solution that we’re looking for

#### Determinants

Another quick comment about square matrices is that to every such matrix we
assign a unique number called the *determinant* of the matrix — you can find
the expression for it [here](https://en.wikipedia.org/wiki/Determinant)

If the determinant of $ A $ is not zero, then we say that $ A $ is
*nonsingular*

Perhaps the most important fact about determinants is that $ A $ is nonsingular if and only if $ A $ is of full column rank

This gives us a useful one-number summary of whether or not a square matrix can be
inverted

### More Rows than Columns

This is the $ n \times k $ case with $ n > k $

This case is very important in many settings, not least in the setting of linear regression (where $ n $ is the number of observations, and $ k $ is the number of explanatory variables)

Given arbitrary $ y \in \mathbb R ^n $, we seek an $ x \in \mathbb R ^k $ such that $ y = Ax $

In this setting, existence of a solution is highly unlikely

Without much loss of generality, let’s go over the intuition focusing on the case where the columns of
$ A $ are linearly independent

It follows that the span of the columns of $ A $ is a $ k $-dimensional subspace of $ \mathbb R ^n $

This span is very “unlikely” to contain arbitrary $ y \in \mathbb R ^n $

To see why, recall the [figure above](#la-3dvec), where $ k=2 $ and $ n=3 $

Imagine an arbitrarily chosen $ y \in \mathbb R ^3 $, located somewhere in that three dimensional space

What’s the likelihood that $ y $ lies in the span of $ \{a_1, a_2\} $ (i.e., the two dimensional plane through these points)?

In a sense it must be very small, since this plane has zero “thickness”

As a result, in the $ n > k $ case we usually give up on existence

However, we can still seek a best approximation, for example an
$ x $ that makes the distance $ \| y - Ax\| $ as small as possible

To solve this problem, one can use either calculus or the theory of orthogonal
projections

The solution is known to be $ \hat x = (A'A)^{-1}A'y $ — see for example
chapter 3 of these notes

### More Columns than Rows

This is the $ n \times k $ case with $ n < k $, so there are fewer
equations than unknowns

In this case there are either no solutions or infinitely many — in other words, uniqueness never holds

For example, consider the case where $ k=3 $ and $ n=2 $

Thus, the columns of $ A $ consists of 3 vectors in $ \mathbb R ^2 $

This set can never be linearly independent, since it is possible to find two vectors that span
$ \mathbb R ^2 $

(For example, use the canonical basis vectors)

It follows that one column is a linear combination of the other two

For example, let’s say that $ a_1 = \alpha a_2 + \beta a_3 $

Then if $ y = Ax = x_1 a_1 + x_2 a_2 + x_3 a_3 $, we can also write

$$
y
= x_1 (\alpha a_2 + \beta a_3) + x_2 a_2 + x_3 a_3
= (x_1 \alpha + x_2) a_2 + (x_1 \beta + x_3) a_3
$$

In other words, uniqueness fails

### Linear Equations with Julia

Here’s an illustration of how to solve linear equations with Julia’s built-in linear algebra facilities

```julia
A = [1.0 2.0; 3.0 4.0];
```

```julia
y = ones(2, 1);  # A column vector
```

```julia
det(A)
```

    -2.0

```julia
A_inv = inv(A)
```

    2×2 Array{Float64,2}:
     -2.0   1.0
      1.5  -0.5

```julia
x = A_inv * y  # solution
```

    2×1 Array{Float64,2}:
     -0.9999999999999998
      0.9999999999999999

```julia
A * x  # should equal y (a vector of ones)
```

    2×1 Array{Float64,2}:
     1.0
     1.0000000000000004

```julia
A \ y  # produces the same solution
```

    2×1 Array{Float64,2}:
     -1.0
      1.0

Observe how we can solve for $ x = A^{-1} y $ by either via `inv(A) * y`, or using `A \ y`

The latter method is preferred because it automatically selects the best algorithm for the problem based on the types of `A` and `y`

If `A` is not square then  `A \ y` returns the least squares solution $ \hat x = (A'A)^{-1}A'y $

## Eigenvalues and Eigenvectors

Let $ A $ be an $ n \times n $ square matrix

If $ \lambda $ is scalar and $ v $ is a non-zero vector in $ \mathbb R ^n $ such that

$$
A v = \lambda v
$$

then we say that $ \lambda $ is an *eigenvalue* of $ A $, and
$ v $ is an *eigenvector*

Thus, an eigenvector of $ A $ is a vector such that when the map $ f(x) = Ax $ is applied, $ v $ is merely scaled

The next figure shows two eigenvectors (blue arrows) and their images under $ A $ (red arrows)

As expected, the image $ Av $ of each $ v $ is just a scaled version of the original

```julia
A = [1 2
     2 1]
evals, evecs = eigen(A)

a1, a2 = evals
eig_1 = [0 0; evecs[:,1]']
eig_2 = [0 0; evecs[:,2]']
x = range(-5, 5, length = 10)
y = -x

plot(eig_1[:, 2], a1 * eig_2[:, 2], arrow = true, color = :red,
     legend = :none, xlims = (-3, 3), ylims = (-3, 3), xticks = -3:3, yticks = -3:3,
     framestyle = :origin)
plot!(a2 * eig_1[:, 2], a2 * eig_2, arrow = true, color = :red)
plot!(eig_1, eig_2, arrow = true, color = :blue)
plot!(x, y, color = :blue, lw = 0.4, alpha = 0.6)
plot!(x, x, color = :blue, lw = 0.4, alpha = 0.6)
```

The eigenvalue equation is equivalent to $ (A - \lambda I) v = 0 $, and
this has a nonzero solution $ v $ only when the columns of $ A -
\lambda I $ are linearly dependent

This in turn is equivalent to stating that the determinant is zero

Hence to find all eigenvalues, we can look for $ \lambda $ such that the
determinant of $ A - \lambda I $ is zero

This problem can be expressed as one of solving for the roots of a polynomial
in $ \lambda $ of degree $ n $

This in turn implies the existence of $ n $ solutions in the complex
plane, although some might be repeated

Some nice facts about the eigenvalues of a square matrix $ A $ are as follows

1. The determinant of $ A $ equals  the product of the eigenvalues
1. The trace of $ A $ (the sum of the elements on the principal diagonal) equals the sum of the eigenvalues
1. If $ A $ is symmetric, then all of its eigenvalues are real
1. If $ A $ is invertible and $ \lambda_1, \ldots, \lambda_n $ are its eigenvalues, then the eigenvalues of $ A^{-1} $ are $ 1/\lambda_1, \ldots, 1/\lambda_n $

A corollary of the first statement is that a matrix is invertible if and only if all its eigenvalues are nonzero

Using Julia, we can solve for the eigenvalues and eigenvectors of a matrix as
follows

```julia
A = [1.0 2.0; 2.0 1.0];
```

```julia
evals, evecs = eigen(A);
```

```julia
evals
```

    2-element Array{Float64,1}:
     -1.0
      3.0

```julia
evecs
```

    2×2 Array{Float64,2}:
     -0.707107  0.707107
      0.707107  0.707107

Note that the *columns* of `evecs` are the eigenvectors

Since any scalar multiple of an eigenvector is an eigenvector with the same
eigenvalue (check it), the eig routine normalizes the length of each eigenvector
to one

### Generalized Eigenvalues

It is sometimes useful to consider the *generalized eigenvalue problem*, which, for given
matrices $ A $ and $ B $, seeks generalized eigenvalues
$ \lambda $ and eigenvectors $ v $ such that

$$
A v = \lambda B v
$$

This can be solved in Julia via `eigen(A, B)`

Of course, if $ B $ is square and invertible, then we can treat the
generalized eigenvalue problem as an ordinary eigenvalue problem $ B^{-1}
A v = \lambda v $, but this is not always the case

## Further Topics

We round out our discussion by briefly mentioning several other important
topics

### Series Expansions

Recall the usual summation formula for a geometric progression, which states
that if $ |a| < 1 $, then $ \sum_{k=0}^{\infty} a^k = (1 - a)^{-1} $

A generalization of this idea exists in the matrix setting

#### Matrix Norms

Let $ A $ be a square matrix, and let

$$
\| A \| := \max_{\| x \| = 1} \| A x \|
$$

The norms on the right-hand side are ordinary vector norms, while the norm on
the left-hand side is a *matrix norm* — in this case, the so-called
*spectral norm*

For example, for a square matrix $ S $, the condition $ \| S \| < 1 $ means that $ S $ is *contractive*, in the sense that it pulls all vectors towards the origin [1]

#### Neumann’s Theorem

Let $ A $ be a square matrix and let $ A^k := A A^{k-1} $ with $ A^1 := A $

In other words, $ A^k $ is the $ k $-th power of $ A $

Neumann’s theorem states the following: If $ \| A^k \| < 1 $ for some
$ k \in \mathbb{N} $, then $ I - A $ is invertible, and

$$
(I - A)^{-1} = \sum_{k=0}^{\infty} A^k \tag{4}
$$

#### Spectral Radius

A result known as Gelfand’s formula tells us that, for any square matrix $ A $,

$$
\rho(A) = \lim_{k \to \infty} \| A^k \|^{1/k}
$$

Here $ \rho(A) $ is the *spectral radius*, defined as $ \max_i |\lambda_i| $, where $ \{\lambda_i\}_i $ is the set of eigenvalues of $ A $

As a consequence of Gelfand’s formula, if all eigenvalues are strictly less than one in modulus,
there exists a $ k $ with $ \| A^k \| < 1 $

In which case [(4)](#equation-la-neumann) is valid

### Positive Definite Matrices

Let $ A $ be a symmetric $ n \times n $ matrix

We say that $ A $ is

1. *positive definite* if $ x' A x > 0 $ for every $ x \in \mathbb R ^n \setminus \{0\} $
1. *positive semi-definite* or *nonnegative definite* if $ x' A x \geq 0 $ for every $ x \in \mathbb R ^n $

Analogous definitions exist for negative definite and negative semi-definite matrices

It is notable that if $ A $ is positive definite, then all of its eigenvalues
are strictly positive, and hence $ A $ is invertible (with positive
definite inverse)

### Differentiating Linear and Quadratic forms

The following formulas are useful in many economic contexts.  Let

- $ z, x $ and $ a $ all be $ n \times 1 $ vectors
- $ A $ be an $ n  \times n $ matrix
- $ B $ be an $ m \times n $ matrix and $ y $ be an $ m  \times 1 $ vector

Then

1. $ \frac{\partial a' x}{\partial x} = a $
1. $ \frac{\partial A x}{\partial x} = A' $
1. $ \frac{\partial x'A x}{\partial x} = (A + A') x $
1. $ \frac{\partial y'B z}{\partial y} = B z $
1. $ \frac{\partial y'B z}{\partial B} = y z' $

Exercise 1 below asks you to apply these formulas

### Further Reading

The documentation of the linear algebra features built into Julia can be found [here](https://docs.julialang.org/en/stable/manual/linear-algebra/)

Chapters 2 and 3 of the [Econometric Theory](http://www.johnstachurski.net/emet.html) contains
a discussion of linear algebra along the same lines as above, with solved exercises

If you don’t mind a slightly abstract approach, a nice intermediate-level text on linear algebra
is [[Janich94]](zreferences.html#janich1994)

## Exercises

### Exercise 1

Let $ x $ be a given $ n \times 1 $ vector and consider the problem

$$
v(x) =  \max_{y,u} \left\{ - y'P y - u' Q u \right\}
$$

subject to the linear constraint

$$
y = A x + B u
$$

Here

- $ P $ is an $ n \times n $ matrix and $ Q $ is an $ m \times m $ matrix
- $ A $ is an $ n \times n $ matrix and $ B $ is an $ n \times m $ matrix
- both $ P $ and $ Q $ are symmetric and positive semidefinite

(What must the dimensions of $ y $ and $ u $ be to make this a well-posed problem?)

One way to solve the problem is to form the Lagrangian

$$
\mathcal L = - y' P y - u' Q u + \lambda' \left[A x + B u - y\right]
$$

where $ \lambda $ is an $ n \times 1 $ vector of Lagrange multipliers

Try applying the formulas given above for differentiating quadratic and linear forms to obtain the first-order conditions for maximizing $ \mathcal L $ with respect to $ y, u $ and minimizing it with respect to $ \lambda $

Show that these conditions imply that

1. $ \lambda = - 2 P y $
1. The optimizing choice of $ u $ satisfies $ u = - (Q + B' P B)^{-1} B' P A x $
1. The function $ v $ satisfies $ v(x) = - x' \tilde P x $ where $ \tilde P = A' P A - A'P B (Q + B'P B)^{-1} B' P A $

As we will see, in economic contexts Lagrange multipliers often are shadow prices

>**Note**
>
>If we don’t care about the Lagrange multipliers, we can substitute the constraint into the objective function, and then just maximize $ -(Ax + Bu)'P (Ax + Bu) - u' Q u $ with respect to $ u $.  You can verify that this leads to the same maximizer.

## Solutions

Thanks to [Willem Hekman](https://qutech.nl/person/willem-hekman/) and Guanlong Ren
for providing this solution.

### Exercise 1

We have an optimization problem:

$$
v(x) = \max_{y,u} \{ -y'Py - u'Qu \}
$$

s.t.

$$
y = Ax + Bu
$$

with primitives

- $ P $ be a symmetric and positive semidefinite $ n \times n $
  matrix.
- $ Q $ be a symmetric and positive semidefinite $ m \times m $
  matrix.
- $ A $ an $ n \times n $ matrix.
- $ B $ an $ n \times m $ matrix.

The associated Lagrangian is :

$$
L = -y'Py - u'Qu + \lambda' \lbrack Ax + Bu - y \rbrack
$$

#### 1.

Differentiating Lagrangian equation w.r.t y and setting its derivative
equal to zero yields

$$
\frac{ \partial L}{\partial y} = - (P + P') y - \lambda = - 2 P y - \lambda = 0 \:,
$$

since P is symmetric.

Accordingly, the first-order condition for maximizing L w.r.t. y implies

$$
\lambda = -2 Py \:.
$$

#### 2.

Differentiating Lagrangian equation w.r.t. u and setting its derivative
equal to zero yields

$$
\frac{ \partial L}{\partial u} = - (Q + Q') u - B'\lambda = - 2Qu + B'\lambda = 0 \:.
$$

Substituting $ \lambda = -2 P y $ gives

$$
Qu + B'Py = 0 \:.
$$

Substituting the linear constraint $ y = Ax + Bu $ into above
equation gives

$$
Qu + B'P(Ax + Bu) = 0
$$

$$
(Q + B'PB)u + B'PAx = 0
$$

which is the first-order condition for maximizing L w.r.t. u.

Thus, the optimal choice of u must satisfy

$$
u = -(Q + B'PB)^{-1}B'PAx \:,
$$

which follows from the definition of the first-oder conditions for
Lagrangian equation.

#### 3.

Rewriting our problem by substituting the constraint into the objective
function, we get

$$
v(x) = \max_{u} \{ -(Ax+ Bu)'P(Ax+Bu) - u'Qu \} \:.
$$

Since we know the optimal choice of u satisfies \$ u = -(Q +
B’PB)^{-1}B’PAx \$, then

$$
v(x) =  -(Ax+ B u)'P(Ax+B u) - u'Q u  \,\,\,\, with \,\,\,\, u = -(Q + B'PB)^{-1}B'PAx
$$

To evaluate the function

$$
\begin{align}
v(x) &=  -(Ax+ B u)'P(Ax+Bu) - u'Q u \\
&= -(x'A' + u'B')P(Ax+Bu) - u'Q u \\
&= - x'A'PAx - u'B'PAx - x'A'PBu - u'B'PBu - u'Qu \\
&= - x'A'PAx - 2u'B'PAx - u'(Q + B'PB) u
\end{align}
$$

For simplicity, denote by $ S := (Q + B'PB)^{-1} B'PA $, then \$ u =
-Sx\$.

Regarding the second term $ - 2u'B'PAx $,

$$
\begin{align}
- 2u'B'PAx &= -2 x'S'B'PAx  \\
& = 2 x'A'PB( Q + B'PB)^{-1} B'PAx
\end{align}
$$

Notice that the term $ (Q + B'PB)^{-1} $ is symmetric as both P and Q
are symmetric.

Regarding the third term $ - u'(Q + B'PB) u $,

$$
\begin{align}
- u'(Q + B'PB) u &= - x'S' (Q + B'PB)Sx \\
&= -x'A'PB(Q + B'PB)^{-1}B'PAx
\end{align}
$$

Hence, the summation of second and third terms is
$ x'A'PB(Q + B'PB)^{-1}B'PAx $.

This implies that

$$
\begin{align}
 v(x) &= - x'A'PAx - 2u'B'PAx - u'(Q + B'PB) u\\
 &= - x'A'PAx + x'A'PB(Q + B'PB)^{-1}B'PAx \\
 &= -x'[A'PA - A'PB(Q + B'PB)^{-1}B'PA] x
\end{align}
$$

Therefore, the solution to the optimization problem
$ v(x) = -x' \tilde{P}x $ follows the above result by denoting
$ \tilde{P} := A'PA - A'PB(Q + B'PB)^{-1}B'PA $.

**Footnotes**

[1] Suppose that $ \|S \| < 1 $. Take any nonzero vector $ x $, and let $ r := \|x\| $. We have $ \| Sx \| = r \| S (x/r) \| \leq r \| S \| < r = \| x\| $. Hence every point is pulled towards the origin.


```python
import pandas as pd
import numpy as np
import scipy
import matplotlib.pyplot as plt
from sklearn import tree
from IPython.display import Image
%matplotlib inline
from sklearn import preprocessing
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, roc_curve
```

### Gradient boosting

You may recall that we last encountered gradients when discussing the gradient descent algorithm in the context of fitting linear regression models.  For a particular regression model with n parameters, an n+1 dimensional space existed defined by all the parameters plus the cost/loss function to minimize.  The combination of parameters and loss function define a surface within the space.  The regression model is fitted by moving down the steepest 'downhill' gradient until we reach the lowest point of the surface, where all possible gradients are 'uphill.'  The final model is made up of the parameter estimates that define that location on the surface.

Throughout all iterations of the gradient descent algorithm for linear regression, one thing remains constant: The underlying data used to estimate the parameters and calculate the loss function never changes.  In gradient boosting, however, the underlying data do change.

Each time we run a decision tree, we extract the residuals.  Then we run a new decision tree, using those residuals as the outcome to be predicted.  After reaching a stopping point, we add together the predicted values from all of the decision trees to create the final gradient boosted prediction.

Gradient boosting can work on any combination of loss function and model type, as long as we can calculate the derivatives of the loss function with respect to the model parameters.  Most often, however, gradient boosting uses decision trees, and minimizes either the  residual (regression trees) or the negative log-likelihood (classification trees).

Let’s go through a simple regression example using Decision Trees as the base predictors (of course Gradient Boosting also works great with regression tasks). This is called Gradient Tree Boosting, or Gradient Boosted Regression Trees. First, let’s fit a `DecisionTreeRegressor` to the training set.

```python
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
```

```python
from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Now train a second `DecisionTreeRegressor` on the residual errors made by the first predictor:

```python
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Then we train a third regressor on the residual errors made by the second predictor:

```python
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
```

    DecisionTreeRegressor(max_depth=2, random_state=42)

Now we have an ensemble containing three trees. It can make predictions on a new instance simply by adding up the predictions of all the trees:

```python
X_new = np.array([[0.8]])
```

```python
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
```

```python
y_pred
```

    array([0.75026781])

The figure below represents the predictions of these three trees in the left column, and the ensemble’s predictions in the right column. In the first row, the ensemble has just one tree, so its predictions are exactly the same as the first tree’s predictions. In the second row, a new tree is trained on the residual errors of the first tree. On the right you can see that the ensemble’s predictions are equal to the sum of the predictions of the first two trees. Similarly, in the third row another tree is trained on the residual errors of the second tree. You can see that the ensemble’s predictions gradually get better as trees are added to the ensemble.

**Run the below cell to develop a visual representation.**

```python
def plot_predictions(regressors, X, y, axes, label=None, style="r-", data_style="b.", data_label=None):
    x1 = np.linspace(axes[0], axes[1], 500)
    y_pred = sum(regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)
    plt.plot(X[:, 0], y, data_style, label=data_label)
    plt.plot(x1, y_pred, style, linewidth=2, label=label)
    if label or data_label:
        plt.legend(loc="upper center", fontsize=16)
    plt.axis(axes)

plt.figure(figsize=(11,11))

plt.subplot(321)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h_1(x_1)$", style="g-", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Residuals and tree predictions", fontsize=16)

plt.subplot(322)
plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1)$", data_label="Training set")
plt.ylabel("$y$", fontsize=16, rotation=0)
plt.title("Ensemble predictions", fontsize=16)

plt.subplot(323)
plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_2(x_1)$", style="g-", data_style="k+", data_label="Residuals")
plt.ylabel("$y - h_1(x_1)$", fontsize=16)

plt.subplot(324)
plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1)$")
plt.ylabel("$y$", fontsize=16, rotation=0)

plt.subplot(325)
plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5], label="$h_3(x_1)$", style="g-", data_style="k+")
plt.ylabel("$y - h_1(x_1) - h_2(x_1)$", fontsize=16)
plt.xlabel("$x_1$", fontsize=16)

plt.subplot(326)
plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8], label="$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$")
plt.xlabel("$x_1$", fontsize=16)
plt.ylabel("$y$", fontsize=16, rotation=0)

#save_fig("gradient_boosting_plot")
plt.show()
```

Now that you have solid understanding of Gradient Boosting in the regression scenario, let's apply the same algorithm to a classification problem. Specifically, the Titanic dataset and predicting survival.

**Use pandas read csv to load in the Titantic data set into a dataframe called df.**

Hint: in this case you can use [dropna()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html) to just throw away any incomplete rows. For the purpose of this exercise we will disregard them but obviously in the real world you need to be much more careful and decide how to handle incomplete observations.

```python
df = pd.read_csv("titanic.csv")
df.dropna(inplace=True)
```

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 183 entries, 1 to 889
    Data columns (total 12 columns):
     #   Column       Non-Null Count  Dtype
    ---  ------       --------------  -----
     0   PassengerId  183 non-null    int64
     1   Survived     183 non-null    int64
     2   Pclass       183 non-null    int64
     3   Name         183 non-null    object
     4   Sex          183 non-null    object
     5   Age          183 non-null    float64
     6   SibSp        183 non-null    int64
     7   Parch        183 non-null    int64
     8   Ticket       183 non-null    object
     9   Fare         183 non-null    float64
     10  Cabin        183 non-null    object
     11  Embarked     183 non-null    object
    dtypes: float64(2), int64(5), object(5)
    memory usage: 18.6+ KB

**Print the levels of the categorical data using 'select_dtypes'.**

## Expected Categories from Research Online

The below columns seem like categories to me.
Question: Discuss categories / this section.
__A__: See answer below in Project's Expected Categories section.

```python
df_categories = df.copy()
df_categories = df_categories[["Survived", "Pclass", "Sex", "Embarked"]].astype("category")
df_categories.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 183 entries, 1 to 889
    Data columns (total 4 columns):
     #   Column    Non-Null Count  Dtype
    ---  ------    --------------  -----
     0   Survived  183 non-null    category
     1   Pclass    183 non-null    category
     2   Sex       183 non-null    category
     3   Embarked  183 non-null    category
    dtypes: category(4)
    memory usage: 2.6 KB

```python
df_categories = pd.DataFrame(df_categories.nunique()).reset_index()
df_categories.columns = ['Features', 'Number of Categories']
df_categories.sort_values(by="Number of Categories", ascending=False)
df_categories
```

      Features
      Number of Categories

      0
      Survived
      2

      1
      Pclass
      3

      2
      Sex
      2

      3
      Embarked
      3

## Project's Expected Categories
Based off the code further below it looks like only 'object' types are considered categories.
Question: Is this is expected?
__A__: Yes. These are the expected categories because the machine learning model needs everything to be transformed into numbers.

```python
dfo = df.select_dtypes(include='object')
dfo.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 183 entries, 1 to 889
    Data columns (total 5 columns):
     #   Column    Non-Null Count  Dtype
    ---  ------    --------------  -----
     0   Name      183 non-null    object
     1   Sex       183 non-null    object
     2   Ticket    183 non-null    object
     3   Cabin     183 non-null    object
     4   Embarked  183 non-null    object
    dtypes: object(5)
    memory usage: 8.6+ KB

```python
dfo_unique = pd.DataFrame(dfo.nunique()).reset_index()
dfo_unique.columns = ['Features', 'Number of Categories']
dfo_unique.sort_values(by="Number of Categories", ascending=False)
dfo_unique
```

      Features
      Number of Categories

      0
      Name
      183

      1
      Sex
      2

      2
      Ticket
      127

      3
      Cabin
      133

      4
      Embarked
      3

**Create dummy features for the categorical features and add those to the 'df' dataframe. Make sure to also remove the original categorical columns from the dataframe.**

```python
df = pd.DataFrame(df.drop(dfo.columns,axis =1)).merge(pd.get_dummies(dfo.drop(['Name','Cabin','Ticket'],axis =1)),left_index=True,right_index=True).drop(['PassengerId'],axis =1)
print(df.shape)
df.head()
```

    (183, 11)

      Survived
      Pclass
      Age
      SibSp
      Parch
      Fare
      Sex_female
      Sex_male
      Embarked_C
      Embarked_Q
      Embarked_S

      1
      1
      1
      38.0
      1
      0
      71.2833
      1
      0
      1
      0
      0

      3
      1
      1
      35.0
      1
      0
      53.1000
      1
      0
      0
      0
      1

      6
      0
      1
      54.0
      0
      0
      51.8625
      0
      1
      0
      0
      1

      10
      1
      3
      4.0
      1
      1
      16.7000
      1
      0
      0
      0
      1

      11
      1
      1
      58.0
      0
      0
      26.5500
      1
      0
      0
      0
      1

**Print the null values for each column in the dataframe.**

```python
# Expect this to be 0 because all the null values were dropped when reading in the data.
df.isnull().sum()
```

    Survived      0
    Pclass        0
    Age           0
    SibSp         0
    Parch         0
    Fare          0
    Sex_female    0
    Sex_male      0
    Embarked_C    0
    Embarked_Q    0
    Embarked_S    0
    dtype: int64

**Create the X and y matrices from the dataframe, where y = df.Survived **

```python
X = df.drop('Survived', axis=1)
y = df.Survived
```

**Apply the standard scaler to the X matrix.**

```python
scaler = preprocessing.StandardScaler().fit(X)
X_scaled = scaler.transform(X)
```

**Split the X_scaled and y into 75/25 training and testing data subsets..**

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=123)
```

**Run the cell below to test multiple learning rates in your gradient boosting classifier.**

```python
learning_rates = [0.05, 0.1, 0.25, 0.5, 0.75, 1]
for learning_rate in learning_rates:
    gb = GradientBoostingClassifier(n_estimators=20, learning_rate = learning_rate, max_features=2, max_depth = 2, random_state = 0)
    gb.fit(X_train, y_train)
    print("Learning rate: ", learning_rate)
    print("Accuracy score (training): {0:.3f}".format(gb.score(X_train, y_train)))
    print("Accuracy score (validation): {0:.3f}".format(gb.score(X_test, y_test)))
    print()
```

    Learning rate:  0.05
    Accuracy score (training): 0.752
    Accuracy score (validation): 0.717

    Learning rate:  0.1
    Accuracy score (training): 0.839
    Accuracy score (validation): 0.783

    Learning rate:  0.25
    Accuracy score (training): 0.847
    Accuracy score (validation): 0.761

    Learning rate:  0.5
    Accuracy score (training): 0.898
    Accuracy score (validation): 0.761

    Learning rate:  0.75
    Accuracy score (training): 0.912
    Accuracy score (validation): 0.739

    Learning rate:  1
    Accuracy score (training): 0.912
    Accuracy score (validation): 0.761

**Apply the best learning rate to the model fit and predict on the testing set. Print out the confusion matrix and the classification report to review the model performance.**

A learning rate of 0.1 had the highest validation accuracy score of 0.783 with a lower training accuracy score of 0.839.
On the other hand, a learning rate of 1 had the highest training accuracy score of 0.912; but a lower validation accuracy score of 0.761.

Therefore, using the learning rate of 0.1 because the validation accuracy score is what was used on the test set.

```python
gradient_boosting = GradientBoostingClassifier(learning_rate=0.1, random_state=123)
gradient_boosting.fit(X_train, y_train)
y_pred = gradient_boosting.predict(X_test)

print(f"The confusion matrix:\n{confusion_matrix(y_test, y_pred)}")
print(f"The classification report:\n{classification_report(y_test, y_pred)}")
```

    The confusion matrix:
    [[ 8  5]
     [ 5 28]]
    The classification report:
                  precision    recall  f1-score   support

               0       0.62      0.62      0.62        13
               1       0.85      0.85      0.85        33

        accuracy                           0.78        46
       macro avg       0.73      0.73      0.73        46
    weighted avg       0.78      0.78      0.78        46

**Calculate the ROC for the model as well.**

### [What is the AUC-ROC curve?](https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/)
The Receiver Operator Characteristic (ROC) curve is an evaluation metric for binary classification problems. It is a probability curve that plots the TPR against FPR at various threshold values and essentially separates the ‘signal’ from the ‘noise’. The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve.

The higher the AUC, the better the performance of the model at distinguishing between the positive and negative classes.

```python
y_pred_prob = gradient_boosting.predict_proba(X_test)[:, 1]
print(f"The Area Under the Curve is {round(roc_auc_score(y_test, y_pred_prob), 4)}.")
```

    The Area Under the Curve is 0.8473.

### [What are Sensitivity and Specificity?](https://www.analyticsvidhya.com/blog/2020/06/auc-roc-curve-machine-learning/)

#### Sensitivity / True Positive Rate / Recall
\begin{equation}
Sensitivity = \frac{TP}{TP + FN}
\end{equation}

Sensitivity tells us what proportion of the positive class got correctly classified.

A simple example would be to determine what proportion of the actual sick people were correctly detected by the model.

#### False Negative Rate
\begin{equation}
FNR = \frac{FN}{TP + FN}
\end{equation}

False Negative Rate (FNR) tells us what proportion of the positive class got incorrectly classified by the classifier.

A higher TPR and a lower FNR is desirable since we want to correctly classify the positive class.

#### Specificity / True Negative Rate
\begin{equation}
Specificity = \frac{TN}{TN + FP}
\end{equation}

Specificity tells us what proportion of the negative class got correctly classified.

Taking the same example as in Sensitivity, Specificity would mean determining the proportion of healthy people who were correctly identified by the model.

#### False Positive Rate
\begin{equation}
FPR = \frac{FP}{TN + FP} = 1 - Specificity
\end{equation}

FPR tells us what proportion of the negative class got incorrectly classified by the classifier.

A higher TNR and a lower FPR is desirable since we want to correctly classify the negative class.

```python
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)
plt.plot([0, 1], [0, 1])
plt.plot(fpr, tpr)
plt.xlabel("False Positive Rate (1 - Specificity)")
plt.ylabel("True Positive Rate (Sensitivity)")
plt.title("ROC")
plt.show()

```


```python
from ipywidgets import interactive, interact
import matplotlib.pyplot as plt
import numpy as np
import ipywidgets as widgets
import sympy as sym
import seaborn as sns
import plotly.graph_objects as go
from plotly.offline import init_notebook_mode, iplot
from numba import jit
import pandas as pd
import base64
import pandas as pd
from IPython.display import HTML

init_notebook_mode(connected=True)
jit(nopython=True, parallel=True)
sns.set()

```

# Interactive partial sums of sine

Zoufiné Lauer-Baré

```python

class plot():

    def __init__(self, preWidgetN):

        self.N = preWidgetN

        x,y,n ,k = sym.symbols('x, y,n,k', real=True)
        X=np.linspace(0, 10, 100)

        f = sym.Sum((-1)**k*(x**(2*k+1))/(sym.factorial(2*k+1)),(k,0, n))
        #f = sym.Sum((-1)**k*(x**(2*k))/(sym.factorial(2*k)),(k,0, n))
        #print(sym.latex(f))
        f = f.subs(n, self.N.value)
        f = sym.lambdify(x, f)
        self.trace1 = go.Scatter(x=X, y=np.sin(X),
                            mode='lines+markers',
                            name='sin'
                               )
        self.trace2 = go.Scatter(x=X, y=f(X),
                            mode='lines',
                            name=r'$\sum_{k=0}^{%s} \frac{\left(-1\right)^{k} x^{2 k + 1}}{\left(2 k + 1\right)!}$' %(self.N.value)
                               )

        layout = go.Layout(template='plotly_dark', title="Partial sums of sine")

        self.fig = go.FigureWidget(data=[self.trace1, self.trace2],
                                   layout = layout,
                                   layout_yaxis_range=[-3 , 3],
                                  )

        #self.fig.update_layout(title="Plot Title",)

    def sineSeries(self, change):

        x,y,n ,k = sym.symbols('x, y,n,k', real=True)
        X=np.linspace(0, 10, 100)
        f = sym.Sum((-1)**k*(x**(2*k+1))/(sym.factorial(2*k+1)),(k,0, n))
        #f = sym.Sum((-1)**k*(x**(2*k))/(sym.factorial(2*k)),(k,0, n))
        f = f.subs(n, self.N.value)
        f = sym.lambdify(x, f)

    def create_download_link(self, df, title = "Download CSV file", filename = "data.csv"):
        csv = data.to_csv()
        b64 = base64.b64encode(csv.encode())
        payload = b64.decode()
        html = '<a download="{filename}" href="data:text/csv;base64,{payload}" target="_blank">{title}</a>'
        html = html.format(payload=payload,title=title,filename=filename)
        return HTML(html)

        with self.fig.batch_update():
            self.fig.data[1].x = X
            self.fig.data[1].y = f(X)
            self.fig.data[1].name = r'$\sum_{k=0}^{%s} \frac{\left(-1\right)^{k} x^{2 k + 1}}{\left(2 k + 1\right)!}$' %(self.N.value)

        return

    def show(self):
        #button = widgets.Button(description="Download image")
        self.N.observe(self.sineSeries, names='value')
        df = pd.DataFrame(data = [[1,2],[3,4]], columns=['Col 1', 'Col 2'])
        display(self.N,  self.fig)

        #self.create_download_link(df)
        #self.fig.write_image("fig1.svg")
        return
```

```python
N = widgets.IntSlider(min=0, max=20, step=1, value=0, description='partial sum order')

p = plot(N)

p.show()
```

    IntSlider(value=0, description='partial sum order', max=20)

    FigureWidget({
        'data': [{'mode': 'lines+markers',
                  'name': 'sin',
                  'type': 'scat…

```python
import base64
import pandas as pd
from IPython.display import HTML

def create_download_link( df, title = "Download CSV file", filename = "data.csv"):
    csv = df.to_csv()
    b64 = base64.b64encode(csv.encode())
    payload = b64.decode()
    html = '<a download="{filename}" href="data:text/csv;base64,{payload}" target="_blank">{title}</a>'
    html = html.format(payload=payload,title=title,filename=filename)
    return HTML(html)

df = pd.DataFrame()
df['sinus'] = np.sin(np.linspace(0, 10))
create_download_link(df)
```

<a download="data.csv" href="data:text/csv;base64,LHNpbnVzCjAsMC4wCjEsMC4yMDI2Njc5MzY1NDgyMDA5NQoyLDAuMzk2OTI0MTQ4OTI0OTIyMzQKMywwLjU3NDcwNjA0MTIxNjE3OTIKNCwwLjcyODYzNDc4MzQ2OTM1MDQKNSwwLjg1MjMyMTU2OTcxOTYxODQKNiwwLjk0MDYzMjc4NTExMjQ4NjcKNywwLjk4OTkwMzA3NjM3MjEyMzkKOCwwLjk5ODA4NzQ4MjEzNDcxODMKOSwwLjk2NDg0NjMwODk4Mzc2MzIKMTAsMC44OTE1NTkyMzA0MTEwMDM3CjExLDAuNzgxMjY4MDIzNTI2MjYzOAoxMiwwLjYzODU1MDMyMDIyNjYwMjEKMTMsMC40NjkzMjk2MTI3NzcyMDEKMTQsMC4yODA2MjkzOTk1MTQzNTY4NAoxNSwwLjA4MDI4MTY3NDg0MjgxMzUKMTYsLTAuMTIzMzk4MTM3MzYyMTc4NzEKMTcsLTAuMzIxOTU2MzE1MDcyNjE4NwoxOCwtMC41MDcxNTE3MDk0ODQ1MTQ0CjE5LC0wLjY3MTI5Nzc5MzU1MTkzMjEKMjAsLTAuODA3NTgxNjkwOTY4MzM2MwoyMSwtMC45MTAzNDY5NDQzMTA3ODI4CjIyLC0wLjk3NTMyODI4NjA2NzA0NTYKMjMsLTAuOTk5ODI4NjY4Mzg0MDg5NgoyNCwtMC45ODI4MzEyMDM5MjU2MzA2CjI1LC0wLjkyNTA0MTM3MTczODIwMjkKMjYsLTAuODI4ODU3NzM2MzczMDQyNwoyNywtMC42OTgyNzIzOTU1NjUzOTk2CjI4LC0wLjUzODcwNTI4ODM4NjE1NjMKMjksLTAuMzU2Nzc5MjQwODk4OTM4MQozMCwtMC4xNjAwNDUwODYwNDMyNTA1NwozMSwwLjA0MzMzMTczMzM2ODY4MzQ2CjMyLDAuMjQ0OTEwMDcxMDExOTc5MwozMywwLjQzNjMyMzQyNjQ3MTgxOTMKMzQsMC42MDk2MjcxOTY0OTA4MzIzCjM1LDAuNzU3NjI4NDE1MzkyNzIwMgozNiwwLjg3NDE4NDI5ODgxOTczMzUKMzcsMC45NTQ0NTcxOTk3Mzg3NTE5CjM4LDAuOTk1MTE1Mzk0Nzc3NjYzNgozOSwwLjk5NDQ3MTM2NzI2MzYxNjgKNDAsMC45NTI1NTE4NDc1MzE0NjA0CjQxLDAuODcxMDk2NzAzNDgyMzIwNwo0MiwwLjc1MzQ4NjcyNzQzOTYzNzYKNDMsMC42MDQ2MDMzMTY1MDYxNTQzCjQ0LDAuNDMwNjI1ODcwMzgyNzM3Mwo0NSwwLjIzODc3NTMxNTY0NDAzMDg3CjQ2LDAuMDM3MDE0NDAxNDg1MDYyMzcKNDcsLTAuMTY2MjgyNzkzODQ4NzU2NAo0OCwtMC4zNjI2Nzg0Mjg4MjY1NDg4CjQ5LC0wLjU0NDAyMTExMDg4OTM2OTgK" target="_blank">Download CSV file

```python

```


# DL Indaba Practical 5
# Variational Auto-Encoders

**Background**

In this practical, our only aim is for you to understand what a variational auto-encoder (VAE) is. Some people prefer to call it "auto-encoding variational Bayes", which is technically more correct.

If you are mathematically inclined, you are welcome to look at https://arxiv.org/pdf/1312.6114.pdf, but it is not neccessary for the scope of this practical.

**What is expected of you?**

1. Run through all the cells below, in order, and consider the output. You should see a loss that becomes smaller and smaller, and some images that will, over time, start resembling MNIST digits.
2. Step through the cells again, top to bottom. As you do this, try to answer all the questions as they arise.

```
import itertools
import matplotlib as mpl
import numpy as np
import os
import tensorflow as tf
import time

from matplotlib import pyplot as plt

import tensorflow.contrib.slim as slim
from tensorflow.examples.tutorials.mnist import input_data

sg = tf.contrib.bayesflow.stochastic_graph
st = tf.contrib.bayesflow.stochastic_tensor
distributions = tf.contrib.distributions

```

```
tf.reset_default_graph()

try:
  flags = tf.app.flags
  flags.DEFINE_string('data_dir', '/tmp/dat/', 'Directory for data')
  flags.DEFINE_string('logdir', '/tmp/log/', 'Directory for logs')

  flags.DEFINE_integer('latent_dim', 100, 'Latent dimensionality of model')
  flags.DEFINE_integer('batch_size', 64, 'Minibatch size')
  flags.DEFINE_integer('n_samples', 1, 'Number of samples to save')
  flags.DEFINE_integer('print_every', 1000, 'Print every n iterations')
  flags.DEFINE_integer('hidden_size', 200, 'Hidden size for neural networks')
  # flags.DEFINE_integer('n_iterations', 1, 'number of iterations')
except:
  pass

FLAGS = flags.FLAGS

# Flags are immutable once defined. If you were to ever run big experiments,
# you'd use something like FLAGS.n_iterations, but as you want to try things in
# the cells below, we'll just make n_iterations something that you can change
# to easily re-run.

N_ITERATIONS = 10000

```

Before continuing, you'll have to sharpen your mental mathematical pencils, and step through a derivation.

# An "evidence lower bound"

"Evidence? Of what?," you might ask. Bayes's theorem states that

\begin{align}
p_{\theta}(z|x) & = \frac{p_{\theta}(z|x) \, p(z)}{p_{\theta}(x)} \\
\textrm{posterior} & = \frac{\textrm{likelihood} \times \textrm{prior}}{\textrm{evidence (or marginal likelihood)}}
\end{align}

and the denominator is an average of the likelihood over all prior assignments of $z$. Of course we can't compute it in most cases, and will find a way to approximate it as best we could.

**What would we like to optimize?**

We would like to optimize $\log p_{\theta}(x)$ over $\theta$. It's the best description of the data, averaged over all prior settings of $z$. But we cannot compute it. Oh dear!

**A lower bound, a.k.a. "sensible approximations rock!"**

For every data point (or MNIST image) $x$, we define the bound on the log marginal likelihood as:

\begin{align}
\log p_{\theta}(x) & = \log \int p_{\theta}(x, z) \, \mathrm{d} z \\
& = \log \int p_{\theta}(x | z) \, p(z) \, \mathrm{d} z \\
& = \log \int q_{\phi}(z | x) \left[ \frac{ p_{\theta}(x | z) \, p(z) }{q_{\phi}(z | x)} \right] \mathrm{d} z \\
& \ge \int q_{\phi}(z | x) \log \left[ \frac{ p_{\theta}(x | z) \, p(z) }{q_{\phi}(z | x)} \right] \mathrm{d} z \\
& = \mathbb{E}_{q_{\phi}(z|x)}\Big[ \log p_{\theta}(x | z) + \log p(z) - \log q_{\phi}(z | x)\Big] \\
& \equiv \mathcal{L}
\end{align}

We have a few pieces:
1. The likelihood $p_{\theta}(x | z)$, also called the "generative network". It is sometimes called the "decoder" as it "decodes" $x$ from $z$.
2. The variational approximation $q_{\phi}(z | x)$, also called the "inference network". Sometimes it is called the "encoder", as it "encodes" $x$ into $z$.

In the code below, we are going to define $p_{\theta}(x | z)$ and $q_{\phi}(z | x)$.

**Take a breather**

Explain to your neighbour what is going on here. Is there anything that you don't understand?

**Deep dive 101**

Your challenge is to find how $\theta$ and $\phi$ are represented in the code.
1. What are they?
2. Where are they?
Try to answer the other questions in the code below.

Note that in the code, there are many images $x$ that are stacked in a batch x. The bound refers to one such image, and we assume that they are generated i.i.d., we sum the bounds...

```
def inference_network(x, latent_dim, hidden_size):
  """Construct an inference network q(z|x).

  Args:
    x: A batch of MNIST digits.
    latent_dim: The latent dimensionality of z.
    hidden_size: The size of the neural net hidden layers.
  Returns:
    mu: The mean of q(z|x)
    sigma: The component-wise standard deviation of q(z|x)
  """

  # Question: What is the shape of x as you run the code with MNIST images?
  # Answer: It has shape (batch_size, 28, 28, 1).

  with slim.arg_scope([slim.fully_connected], activation_fn=tf.nn.relu):
    net = slim.flatten(x)

    # Question: Now that we've flattened x, what is its shape now, as `net`?
    # Hint: You might have to put a `print(net.get_shape())` somewhere around
    #       here, run the `train()` cell to contruct the TF graph, and see what
    #       you get. Debugging is part of life!
    # Answer: ...

    # At this point, `net` contains the input to the network. We will now
    # transform it layer-by-layer:

    net = slim.fully_connected(net, hidden_size)

    # Question: What is the shape of the first layer, as `net`?
    # Answer: ...

    # Question: Write down the mathematical expression for
    #    net = slim.fully_connected(net, hidden_size). It it was not wrapped in
    #    "with slim.arg_scope(..., activation_fn=tf.nn.relu)", would the
    #    mathematical expression look different? Why?
    # Answer: ...

    net = slim.fully_connected(net, hidden_size)

    # Question: What is the shape of the second layer, as `net`?
    # Answer: ...

    gaussian_params = slim.fully_connected(
        net, latent_dim * 2, activation_fn=None)

    # Question: What is the shape of the third (output) layer,
    #    `gaussian_params`?
    # Answer: ...

  # We're taking the output of the MLP (multi-layer perceptron) and split it
  # into two parts: the first latent_dim outputs become the means, while the
  # second latent_dim outputs become the standard deviations.

  # Challenge 1. Can you write the entire network as a mathematical function?
  # Challenge 2. Can you draw it?
  # Challenge 3. Can you say which parameters constitute \phi?
  #
  # Challenge 4. (You can skip this challenge and do it at home).
  #              Can you see it in the actual computation graph?
  #              Hint: You can write the graph to file with
  #
  #              tf.train.write_graph(tf.get_default_graph().as_graph_def(),
  #                                   directory, 'mygraph.pbtxt')

  # The mean parameter is unconstrained
  mu = gaussian_params[:, :latent_dim]

  # The standard deviation must be positive. Parametrize with a softplus and
  # add a small epsilon for numerical stability
  sigma = 1e-6 + tf.nn.softplus(gaussian_params[:, latent_dim:])

  # Question: Can you draw the softplus function, log(exp(inputs) + 1)?
  # Answer: ...

  return mu, sigma

def generative_network(z, hidden_size):
  """Build a generative network to return the logits of p(x|z).

  Args:
    z: Samples of latent variables.
    hidden_size: Size of the hidden state of the neural net.

  Returns:
    bernoulli_logits: logits for the Bernoulli likelihood of the data.
  """
  with slim.arg_scope([slim.fully_connected], activation_fn=tf.nn.relu):

    net = slim.fully_connected(z, hidden_size)
    net = slim.fully_connected(net, hidden_size)
    bernoulli_logits = slim.fully_connected(net, 784, activation_fn=None)
    bernoulli_logits = tf.reshape(bernoulli_logits, [-1, 28, 28, 1])

    # Challenge 5. Can you again write the entire network as a mathematical
    #              function, draw it, and say which parameters constitute
    #              \theta?

    # Hold that thought... Can you (later) say why "logits" are returned, and
    # why the are not transformed here to a probability?
  return bernoulli_logits
```

Now, look for the line `q_z = st.StochasticTensor(distributions.Normal(loc=q_mu, scale=q_sigma))` in `train()`.

**Your challenge will be to rewrite that line, so that `st.StochasticTensor(...)` is not used.**

First of all, you need to know what it does.
Here, you will encounter the **reparameterization trick**. This rewrites:

\begin{align}
\mathcal{L} & = \mathbb{E}_{q_{\phi}(z|x)}\Big[ \log p_{\theta}(x | z) + \log p(z) - \log q_{\phi}(z | x)\Big] \\
& = \mathbb{E}_{z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))}\Big[ \log p_{\theta}(x | z) \Big]
- \mathbb{E}_{z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))}\left[ \log \frac{ q_{\phi}(z | x) }{p(z)} \right] \\
& = \mathbb{E}_{z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))}\Big[ \log p_{\theta}(x | z) \Big]
- \textrm{Kullback Leibler divergence} \ .
\end{align}

You can turn to your favourite souce, and find an expression for the KL divergence:
https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence ... It is function of a mean $\mu_{\theta}(x)$ and variance $\sigma^2_{\theta}(x)$, and we can look in Wikipedia on how to compute it.

Note that the mean $\mu_{\theta}(x)$ and variance $\sigma^2_{\theta}(x)$ are parameterized as multi-layer perceptrons (MLPs) in `def inference_network(x, latent_dim, hidden_size)`.

**Easy there, what's the catch?**

Well, we can write the KL divergence part as a function, that we can differentiate (thank you, TensorFlow! back in the day people had to do this by hand!). But what about the first expectation?

For the first expectation, we'd have to do something called a "Monte Carlo estimate" because we can't browse Wikipedia and look for a nice and analytic form of the function.

**Monte Carlo estimate**

Take $L$ samples $z^{(l)}$ from $z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))$, and call them $z^{(1)}, z^{(2)}, \ldots, z^{(L)}$. The "sampling process" essentially maps $x \to z$ stochastically. We now approximate

\begin{align}
\mathbb{E}_{z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))}\Big[ \log p_{\theta}(x | z) \Big] \approx
\frac{1}{L} \sum_{l=1}^{L} \log p_{\theta}(x | z^{(l)}) \ .
\end{align}

This is really awesome, as the expression above maps back $z \to x$. Except for one small detail. If we were to write $x \to z \to x$ as a function, TensorFlow cannot differentiate it for us any more. We cannot take gradients through the samples :-( :-(

**Reparameterization trick**

There is a neat trick, where we put the samples (and stochasticity) *outside* the $x \to z \to x$ path.
The trick is to sample $L$ uniform Gaussian random variables $\epsilon^{(l)} \sim \mathcal{N}(0, 1)$. Can you see that there are no fancy parameters (that we have to adjust) in $\mathcal{N}(0, 1)$?

We can get the same samples from $z$, as we've had before, by *transforming* the $\mathcal{N}(0, 1)$ random variables:
$z^{(l)} = \mu_{\theta}(x) + \epsilon^{(l)} * \sigma_{\theta}(x)$.

\begin{align}
\mathbb{E}_{z \sim \mathcal{N}(\mu_{\theta}(x), \sigma^2_{\theta}(x))}\Big[ \log p_{\theta}(x | z) \Big]
=
\mathbb{E}_{\epsilon \sim \mathcal{N}(0, 1)}\Big[ \log p_{\theta}\Big(x \, \Big| \, z = \mu_{\theta}(x) + \epsilon * \sigma_{\theta}(x)\Big) \Big]
\approx
\frac{1}{L} \sum_{l=1}^{L} \log p_{\theta}\Big(x \, \Big| \, z = \mu_{\theta}(x) + \epsilon^{(l)} * \sigma_{\theta}(x)\Big) \ .
\end{align}

```
def train():
  # Train a Variational Autoencoder on MNIST

  # We need to reset the graph between multiple runs of the cell to avoid
  # TF complaining about creating duplicate variables.
  tf.reset_default_graph()

  # Input placeholders
  with tf.name_scope('data'):
    x = tf.placeholder(tf.float32, [None, 28, 28, 1])

  with tf.variable_scope('variational'):
    # Given x, get the mean and variance of q(z|x).
    q_mu, q_sigma = inference_network(x=x,
                                      latent_dim=FLAGS.latent_dim,
                                      hidden_size=FLAGS.hidden_size)

    with st.value_type(st.SampleValue()):
      # We now draw a sample from z ~ q(z|x).
      #
      # This is the key to this practical. We effectively transform Normal(0,1)
      # noise that it *outside* the computation graph, to represent a sample
      # from Normal(q_mu, q_sigma).
      #
      #   This is the REPARAMETERIZATION TRICK.

      q_z = st.StochasticTensor(distributions.Normal(loc=q_mu, scale=q_sigma))

      # Challenge: Can you write
      #
      # q_z = ...
      #
      # by using tf.random_normal(...) and then transforming it?

  with tf.variable_scope('model'):
    # The likelihood is Bernoulli-distributed with logits given by the
    # generative network
    p_x_given_z_logits = generative_network(z=q_z,
                                            hidden_size=FLAGS.hidden_size)

    p_x_given_z = distributions.Bernoulli(logits=p_x_given_z_logits)

    posterior_predictive_samples = p_x_given_z.sample()

  # You can ignore the "Take samples" bits for now, until the end of the prac.
  # Take samples from the prior
  with tf.variable_scope('model', reuse=True):
    p_z = distributions.Normal(loc=np.zeros(FLAGS.latent_dim, dtype=np.float32),
                               scale=np.ones(FLAGS.latent_dim, dtype=np.float32))
    p_z_sample = p_z.sample(FLAGS.n_samples)
    p_x_given_z_logits = generative_network(z=p_z_sample,
                                            hidden_size=FLAGS.hidden_size)
    prior_predictive = distributions.Bernoulli(logits=p_x_given_z_logits)
    prior_predictive_samples = prior_predictive.sample()

  # You can ignore the "Take samples" bits for now, until the end of the prac.
  # Take samples from the prior with a placeholder
  with tf.variable_scope('model', reuse=True):
    z_input = tf.placeholder(tf.float32, [None, FLAGS.latent_dim])
    p_x_given_z_logits = generative_network(z=z_input,
                                            hidden_size=FLAGS.hidden_size)
    prior_predictive_inp = distributions.Bernoulli(logits=p_x_given_z_logits)
    prior_predictive_inp_sample = prior_predictive_inp.sample()

  # The juice! Convince yourself that you understand the lines below.
  # Build the evidence lower bound (ELBO) or the negative loss
  kl = tf.reduce_sum(distributions.kl_divergence(q_z.distribution, p_z), 1)
  expected_log_likelihood = tf.reduce_sum(p_x_given_z.log_prob(x),
                                          [1, 2, 3])
  elbo = tf.reduce_sum(expected_log_likelihood - kl, 0)

  # Challenge. Can you see how the four lines above relate to the mathematical
  #            expressions you've encountered?
  # Answer: ...

  optimizer = tf.train.RMSPropOptimizer(learning_rate=0.001)

  train_op = optimizer.minimize(-elbo)

  init_op = tf.global_variables_initializer()

  # Run training
  sess = tf.InteractiveSession()
  sess.run(init_op)

  #mnist = read_data_sets(FLAGS.data_dir, one_hot=True)
  mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)

  # Get fixed MNIST digits for plotting posterior means during training
  np_x_fixed, np_y = mnist.test.next_batch(5000)
  np_x_fixed = np_x_fixed.reshape(5000, 28, 28, 1)
  np_x_fixed = (np_x_fixed > 0.5).astype(np.float32)

  for i in range(N_ITERATIONS):
    # Re-binarize the data at every batch; this improves results
    np_x, _ = mnist.train.next_batch(FLAGS.batch_size)
    np_x = np_x.reshape(FLAGS.batch_size, 28, 28, 1)
    np_x = (np_x > 0.5).astype(np.float32)
    sess.run(train_op, {x: np_x})

    t0 = time.time()
    if i % 100 == 0:
      np_elbo = sess.run(elbo, feed_dict={x: np_x})

      print('Iteration: {0:d} ELBO: {1:.3f} Examples/s: {2:.3e}'.format(
          i,
          np_elbo / FLAGS.batch_size,
          FLAGS.batch_size * FLAGS.print_every / (time.time() - t0)))
      t0 = time.time()

    if i % 1000 == 0 and i > 0:
      # Get samples
      np_posterior_samples, np_prior_samples = sess.run(
          [posterior_predictive_samples, prior_predictive_samples],
          feed_dict={x: np_x})

      for k in range(1):   # range(FLAGS.n_samples):
        # Plot the data using Matplotlib
        fig = plt.figure()
        plt.subplot(1, 2, 1)
        plt.imshow(np_posterior_samples[k, :, :, 0])
        plt.title('Posterior sample')

        plt.subplot(1, 2, 2)
        plt.imshow(np_prior_samples[k, :, :, 0])
        plt.title('Prior sample')
        plt.show()

train()

```

```

```

# Acknowledgements

* Based on Jaan Altosaar's wonderful VAE implementation: https://github.com/altosaar/vae/blob/master/vae.py

```

```


# MAP inference

> In classes 6 and 7 we focused our attention to inference methods the find conditional probability queries. In this class, we turn our attention to MAP queries, which are also very useful in several applications (recall the motivating example of the first class).
>
> As a reminder, a MAP query aims to find the most likely assignment to all of the non-evidence variables. Similarly, a marginal MAP query aims to find the most likely assignment to a subset of the variables.

> **Objetives:**
> - To reformulate the Variable Elimination method for MAP inference.
> - To reformulate the Belief Propagation algorithm for MAP inference.

> **References:**
> - Probabilistic Graphical Models: Principles and Techniques, By Daphne Koller and Nir Friedman. Ch. 13.
> - Mastering Probabilistic Graphical Models Using Python, By Ankur Ankan and Abinash Panda. Ch. 3.
> - Probabilistic Graphical Models Specialization, offered through Coursera. Prof. Daphne Koller.

 Imagen recuperada de: https://upload.wikimedia.org/wikipedia/commons/e/ed/Bayes_icon.svg.

___

# 1. Variable Elimination

We begin studying the most fundamental inference algorithm: variable elimination.

## 1.1. Max-product variable elimination

Let's introduce this problem though a simple example.

**Example.** Consider the following BN

```python
from IPython.display import Image
```

```python
Image("figures/simpleExample.png")
```

Or goal is to compute:

$$\begin{align}
\max_{a,b} P(A=a, B=b)  & = \max_{a,b} P(A=a)P(B=b | A=a) \\
                        & = \max_{a} \max_{b} P(A=a)P(B=b | A=a).
\end{align}$$

From this last form, we would want to choose $a$ so that

$$\max_{b} P(A=a)P(B=b | A=a) = P(A=a) \max_{b}P(B=b | A=a)$$

is maximum. Hence,

- a **necessary condition** of the joint MAP assignment $a,b$ is that $b$ is selected to maximize $P(B=b | A=a)$.

- This is not sufficient, however, since we must choose $a$ appropriately.

For instance, consider the following CPDs

```python
Image("figures/simpleExample2.png")
```

Let's denote $\max_{b}P(B=b | A=a) = \phi_1(a)$. Then

- $\phi_1(a^0)=0.9$
- $\phi_1(a^1)=0.55$

Finally,

$$\max_{a} P(a) \phi_1(a) = \max_{a} \phi_2(a)=0.36$$

| $A$ | $\phi_2$           |
| --- | ------------------ |
|$a^0$|$0.9\times0.4=0.36$ |
|$a^1$|$0.55\times0.6=0.33$|

### Important comments:

- One can note that the operation $\max_{b}P(B=b | A=a) = \phi_1(a)$ corresponds to a *max-marginalization* of the $P(B|A)$ factor.

- Just like in conditional probability queries, this step marginalizes out the variable $B$, not by summing, but by maximizing it.

- One key point is: Assume we are given two factors $\phi_1,\phi_2$ and that $X\notin \mathrm{scope}[\phi_1]$, then

  $$\max_{X} \phi_1 \phi_2 = \phi_1 \max_{X} \phi_2.$$

  This is, we can push in the maximization operations (just as we push in the summation operations) over factors that don't involve the variable being maximized. Similarly,

  $$\max_{X} (\phi_1 + \phi_2) = \phi_1 + \max_{X} \phi_2.$$

The final comment directly leads to the **max-product and max-sum variable elimination algorithms** which is completely dual to the **sum-prod variable elimination algorithm** that we studied in class 6.

### Max-product variable elimination algorithm

We will describe the steps for eliminating some variable $Z$ from $\bar{\Phi}$: *Eliminate Var-$Z$ from $\bar{\Phi}$*.

1. Determine the set of factors that involve $Z$:

   $$\bar{\Phi}' = \left\{\phi_i \in \bar{\Phi} : Z  \in \mathrm{scope}[\phi_i]\right\}$$

2. Compute:

   $$\psi = \prod_{\phi_i \in \bar{\Phi}'} \phi_i$$

3. Compute:

   $$\tau = \max_Z \psi$$

   Store the value $z^\ast$ that maximizes $\psi$.

4. Overwrite:

   $$\bar{\Phi} := \left(\bar{\Phi}\setminus \bar{\Phi}'\right) \cup \{\tau\}$$

Thus, the complete algorithm can be described as:

1. The first step is to reduce all factors in $\bar{\Phi}$ acording to the given evidence, if any.

2. For each non-query variable $Z$:
   - Eliminate Var-$Z$ from $\bar{\Phi}$.

3. Multiply all the remaining factors.

## 1.2. Max-sum variable elimination algorithm

Often, the values of some factors are so small that, when multipled together, they may produce values that are less than the processor/memory resolution: **numerical underflow**.

One effective way of dealing with this situation is to consider the $\log$ of the factors.

We have that:

$$P_{\bar{\Phi}} (\bar{X}) \propto \prod_{k} \phi_{k} (\bar{X}_k)$$

and we want,

$$\arg \max_{X} \prod_{k} \phi_{k} (\bar{X}_k) = \arg \max_{X} \log\left(\prod_{k} \phi_{k} (\bar{X}_k)\right) = \arg \max_{X} \sum_{k} \log\left(\phi_{k} (\bar{X}_k)\right),$$

since the $\log$ is an increasing function.

Thus, converting factors with the logarithm allows us to interchange products by sum, obtaining some **practical robustness against numerical underflow**.

The algorithm is completely analogous:

### Max-sum variable elimination algorithm

We will describe the steps for eliminating some variable $Z$ from $\bar{\Phi}$: *Eliminate Var-$Z$ from $\bar{\Phi}$*.

0. Take the $log$ of all the factors:

   $$\theta_k(\bar{X}_k) = \log{\phi_k}(\bar{X}_k).$$

   $$\bar{\Theta} = \{\theta_{1}(\bar{X}_1), \dots, \theta_{m}(\bar{X}_m)\}.$$

1. Determine the set of factors that involve $Z$:

   $$\bar{\Theta}' = \left\{\theta_i \in \bar{\Theta} : Z  \in \mathrm{scope}[\theta_i]\right\}$$

2. Compute:

   $$\psi = \sum_{\theta_i \in \bar{\Theta}'} \phi_i$$

3. Compute:

   $$\tau = \max_Z \psi$$

   Store the value $z^\ast$ that maximizes $\psi$.

4. Overwrite:

   $$\bar{\Theta} := \left(\bar{\Theta}\setminus \bar{\Theta}'\right) \cup \{\tau\}$$

How is this done in `pgmpy`?

```python
Image("figures/restaurant.png")
```

```python
# Import pgmpy.models.BayesianModel

# Import pgmpy.factors.discrete.TabularCPD

```

```python
# Define model skeleton

```

```python
# Define CPDs

```

```python
# Attach CPDs to the model

```

```python
# Check if the model is correctly defined

```

```python
# Import pgmpy.inference.VariableElimination

```

```python
# Create VariableElimination inference object

```

```python
# Map query

```

```python
# Max marginal

```

# 2. Max-sum Belief Propagation

We start by a simple example

### Max-sum elimination in chains

Consider the following chain

```python
Image("figures/chain.png")
```

We want:

$$\begin{align}
\max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) & = \max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) \\
                                                      & = \max_{D} \max_{C} \max_{B} \max_{A} \left(\theta_1(A,B) + \theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E)\right) \\
                                                      & = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \max_{A}\theta_1(A,B)\right)
\end{align}$$

___
**1. Sum of factors**

$$\phi_1(A, B)$$

|   A   |   B   | $\phi_1$ |
| ----- | ----- | -------- |
| $a^0$ | $b^0$ | 5        |
| $a^0$ | $b^1$ | 8        |
| $a^1$ | $b^0$ | 1        |
| $a^1$ | $b^1$ | 0        |
| $a^2$ | $b^0$ | 3        |
| $a^2$ | $b^1$ | 9        |

$$\phi_2(B, C)$$

|   B   |   C   | $\phi_1$ |
| ----- | ----- | -------- |
| $b^0$ | $c^0$ | 5        |
| $b^0$ | $c^1$ | 7        |
| $b^1$ | $c^0$ | 1        |
| $b^1$ | $c^1$ | 2        |

$$\phi_3(A, B, C) = \phi_1(A, B) + \phi_2(B, C)$$

|   A   |   B   |   C   | $\phi_3$     |
| ----- | ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $c^0$ | $5 + 5 = 10$ |
| $a^0$ | $b^0$ | $c^1$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $c^0$ | $8 + 1 = 9$  |
| $a^0$ | $b^1$ | $c^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $c^0$ | $1 + 5 = 6$  |
| $a^1$ | $b^0$ | $c^1$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $c^0$ | $0 + 1 = 1$  |
| $a^1$ | $b^1$ | $c^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $c^0$ | $3 + 5 = 8$  |
| $a^2$ | $b^0$ | $c^1$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $c^0$ | $9 + 1 = 10$ |
| $a^2$ | $b^1$ | $c^1$ | $9 + 2 = 11$ |

**2. Max reduction**

$$\phi_3(A, B, C)$$

|   A   |   B   |   C   | $\phi_3$     |
| ----- | ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $c^0$ | $5 + 5 = 10$ |
| $a^0$ | $b^0$ | $c^1$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $c^0$ | $8 + 1 = 9$  |
| $a^0$ | $b^1$ | $c^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $c^0$ | $1 + 5 = 6$  |
| $a^1$ | $b^0$ | $c^1$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $c^0$ | $0 + 1 = 1$  |
| $a^1$ | $b^1$ | $c^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $c^0$ | $3 + 5 = 8$  |
| $a^2$ | $b^0$ | $c^1$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $c^0$ | $9 + 1 = 10$ |
| $a^2$ | $b^1$ | $c^1$ | $9 + 2 = 11$ |

$$\phi_4(A, B) = \max_{C}\phi_3(A, B, C)$$

|   A   |   B   | $\phi_3$     |
| ----- | ----- | ------------ |
| $a^0$ | $b^0$ | $5 + 7 = 12$ |
| $a^0$ | $b^1$ | $8 + 2 = 10$ |
| $a^1$ | $b^0$ | $1 + 7 = 8$  |
| $a^1$ | $b^1$ | $0 + 2 = 2$  |
| $a^2$ | $b^0$ | $3 + 7 = 10$ |
| $a^2$ | $b^1$ | $9 + 2 = 11$ |

___

See in the whiteboard then show:

$$\begin{align}
\max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) & = \max_{D} \max_{C} \max_{B} \max_{A} \theta(A,B,C,D,E) \\
                                                      & %= \max_{D} \max_{C} \max_{B} \max_{A} \left(\theta_1(A,B) + \theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E)\right) \\
                                                      %& = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \max_{A}\theta_1(A,B)\right) \\
                                                      %& = \max_{D} \max_{C} \max_{B} \left(\theta_2(B,C) + \theta_3(C,D) + \theta_4(D,E) + \tau_1(B)\right) \\
                                                      %& = \max_{D} \max_{C} \left(\theta_3(C,D) + \theta_4(D,E) + \max_{B} (\theta_2(B,C)+ \tau_1(B))\right) \\
                                                      %& = \max_{D} \left(\theta_4(D,E) + \max_{C} (\theta_3(C,D) + \tau_2(C))\right) \\
                                                      %& = \max_{D} \left(\theta_4(D,E) + \tau_3(D)\right) \\
                                                      %& = \tau_4(E)
\end{align}$$

### Max-sum in clique trees

Consider the clique tree of the above chain:

```python
Image("figures/clusterGraphChain.png")
```

The message passing has the same idea in this context:

```python
Image("figures/clusterGraphChain2.png")
```

Then, the forward-backward algorithm we defined for conditional probability queries in clique trees is exactly the same for MAP queries.

We only have to change products by sums/products, and reduction by max-reductions.

How is this done in `pgmpy`?

```python
# Import pgmpy.inference.BeliefPropagation

```

```python
# Create BeliefPropagation inference object

```

```python
# MAP query

```

# 3. Application - Classification

Let's use the example in the first class

```python
# Library imports (sklearn.datasets.load_iris, pandas y numpy)
from sklearn.datasets import load_iris
import pandas as pd
import numpy as np
```

```python
# Data loading (integer parts, for simplicity)
iris = load_iris()
X = iris['data'][:, :2].astype(int)
y = iris['target']
df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width'])
df['target'] = y
df
```

| Type of flower | label  |
| -------------- | ------ |
| Setosa         | 0      |
| Versicolor     | 1      |
| Virginica      | 2      |

```python
# Import matplotlib.pyplot
import matplotlib.pyplot as plt
```

```python
# Scatter plot (largo vs. ancho), color regarding type of flower. Add a bit of noise
l = df['sepal_length'] + np.random.normal(0, 0.2, (len(df), ))
w = df['sepal_width'] + np.random.normal(0, 0.2, (len(df), ))
plt.figure(figsize=(8, 5))
plt.scatter(l, w, c=['bgr'[c] for c in df['target']])
plt.show()
```

```python
# Import sklearn.model_selection.train_test_split

```

```python
# Train/Test split

```

```python
Image('figures/iris.png')
```

```python
# Flower model

```

```python
# Generate train frame

```

```python
# Train model

```

```python
# Generate test frame

```

```python
# Predict (MAP Assignment)

```

```python
# Accuracy

```

# Announcements

## 1. Quiz.

Created with Jupyter by Esteban Jiménez Rodríguez.