# Linear algebra overview

Linear algebra is the study of **vectors** and **linear transformations**. This notebook introduces concepts form linear algebra in a birds-eye overview. The goal is not to get into the details, but to give the reader a taste of the different types of thinking: computational, geometrical, and theoretical, that are used in linear algebra.

## Chapters overview

  - 1/ Math fundamentals
  - 2/ Intro to linear algebra
      - Vectors
      - Matrices
      - Matrix-vector product representation of linear transformations
      - Linear property: $f(a\mathbf{x} + b\mathbf{y}) = af(\mathbf{x}) + bf(\mathbf{y})$
  - 3/ Computational linear algebra
    - Gauss-Jordan elimination procedure
      - Augemnted matrix representaiton of systems of linear equations
      - Reduced row echelon form
    - Matrix equations
    - Matrix operations
      - Matrix product
      - Determinant
      - Matrix inverse
  - 4/ Geometrical linear algebra
    - Points, lines, and planes
    - Projection operation
    - Coordinates
    - Vector spaces
    - Vector space techniques
  - 5/ Linear transformations
    - Vector functions
    - Input and output spaces
    - Matrix representation of linear transformations
    - Column space and row spaces of matrix representations
    - Invertible matrix theorem
  - 6/ Theoretical linear algebra
    - Eigenvalues and eigenvectors
    - Special types of matrices
    - Abstract vectors paces
    - Abstract inner product spaces
    - Gram–Schmidt orthogonalization
    - Matrix decompositions
    - Linear algebra with complex numbers
  - 7/ Applications
  - 8/ Probability theory
  - 9/ Quantum mechanics
  - Notation appendix

```python
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()

# a vector is a special type of matrix (an n-vector is either a nx1 or a 1xn matrix)
Vector = Matrix  # define alias Vector so I don't have to explain this during video
Point = Vector   # define alias Point for Vector since they're the same thing

# setup plotting
%matplotlib inline
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, plot_line, plot_plane, autoscale_arrows
```

# 1/ Math fundamentals

Linear algebra builds upon high school math concepts like:
  - Numbers (integers, rationals, reals, complex numbers)
  - Functions ($f(x)$ takes an input $x$ and produces an output $y$)
  - Basic rules of algebra
  - Geometry (lines, curves, areas, triangles)
  - The cartesian plane

```python

```

# 2/ Intro to linear algebra

Linear algebra is the study of vectors and matrices.

## Vectors

```python
# define two vectors
u = Vector([2,3])
v = Vector([3,0])
u
```

```python
v
```

```python
plot_vecs(u, v)
autoscale_arrows()
```

## Vector operations

  - Addition (denoted $\vec{u}+\vec{v}$)
  - Subtraction, the inverse of addition (denoted $\vec{u}-\vec{v}$)
  - Scaling (denoted $\alpha \vec{u}$)
  - Dot product (denoted $\vec{u} \cdot \vec{v}$)
  - Cross product (denoted $\vec{u} \times \vec{v}$)

### Vector addition

```python
# algebraic
u+v
```

```python
# graphical
plot_vecs(u, v)
plot_vec(v, at=u, color='b')
plot_vec(u+v, color='r')
autoscale_arrows()
```

### Basis

When we describe the vector as the coordinate pair $(4,6)$, we're implicitly using the *standard basis* $B_s = \{ \hat{\imath}, \hat{\jmath} \}$. The vector $\hat{\imath} \equiv (1,0)$ is a unit-length vector in the $x$-direciton,
and $\hat{\jmath} \equiv (0,1)$ is a unit-length vector in the $y$-direction.

To be more precise when referring to vectors, we can indicate the basis as a subscript of every cooridnate vector $\vec{v}=(4,6)_{B_s}$, which tells $\vec{v}= 4\hat{\imath}+6\hat{\jmath}=4(1,0) +6(0,1)$.

```python
# the standard basis
ihat = Vector([1,0])
jhat = Vector([0,1])

v = 4*ihat + 6*jhat
v
```

```python
# geomtrically...
plot_vecs(ihat, jhat, 4*ihat, 6*jhat, v)
autoscale_arrows()
```

The same vector $\vec{v}$ will correspond to the a different pair of coefficients if a differebt basis is used.
For example, if we use the basis $B^\prime = \{ (1,1), (1,-1) \}$, the same vector $\vec{v}$ must be expressed as $\vec{v} = 5\vec{b}_1 +(-1)\vec{b}_2=(5,-1)_{B^\prime}$.

```python
# another basis B' = { (1,1), (1,-1) }
b1 = Vector([ 1, 1])
b2 = Vector([ 1, -1])

v = 5*b1 + (-1)*b2
v

# How did I know 5 and -1 are the coefficients w.r.t basis {b1,b2}?
# Matrix([[1,1],[1,-1]]).inv()*Vector([4,6])
```

```python
# geomtrically...
plot_vecs(b1, b2, 5*b1, -1*b2, v)
autoscale_arrows()
```

## Matrix operations

  - Addition (denoted $A+B$)
  - Subtraction, the inverse of addition (denoted $A-B$)
  - Scaling by a constant $\alpha$  (denoted $\alpha A$)
  - Matrix-vector product (denoted $A\vec{x}$, related to linear transformations)
  - Matrix product (denoted $AB$)
  - Matrix inverse (denoted $A^{-1}$)
  - Trace (denoted $\textrm{Tr}(A)$)
  - Determinant (denoted $\textrm{det}(A)$ or $|A|$)

In linear algebra we'll extend the notion of funttion $f:\mathbb{R}\to \mathbb{R}$, to functions that act on vectors called *linear transformations*.  We can understand the properties of linear transformations $T$ in analogy with ordinary functions:

\begin{align*}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}
    \end{array} \\
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \\
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
     \textrm{output }      T(\vec{x})\in \mathbb{R}^m \\
    g\circ\! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \\
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{inverse transformation } T^{-1} \\
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T \\
    \textrm{image of } f
         & \ \Leftrightarrow \
             \begin{array}{l}
             \textrm{image of } T
              \end{array}
\end{align*}

## Linear property

$$
  T(a\mathbf{x}_1 + b\mathbf{x}_2) = aT(\mathbf{x}_1) + bT(\mathbf{x}_2)
$$

## Matrix-vector product representation of linear transformations

Equivalence between linear transformstions $T$ and matrices $M_T$:

$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$

$$
  \vec{y} = T(\vec{x})
  \qquad
  \Leftrightarrow
  \qquad
  \vec{y} = M_T\vec{x}
$$

# 3/ Computational linear algebra

## Gauss-Jordan elimination procedure

Suppose you're asked to solve for $x_1$ and $x_2$ in the following system of equations

\begin{align*}
1x_1 + 2x_2 &= 5 \\
3x_1 + 9x_2 &= 21.
\end{align*}

```python
# represent as an augmented matrix
AUG = Matrix([
  [1, 2,   5],
  [3, 9,  21]])
AUG
```

```python
# eliminate x_1 in second equation by subtracting 3x times the first equation
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG
```

```python
# simplify second equation by dividing by 3
AUG[1,:] = AUG[1,:]/3
AUG
```

```python
# eliminate x_2 from first equation by subtracting 2x times the second equation
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG
```

This augmented matrix is in *reduced row echelon form* (RREF), and corresponds to the system of equations:

\begin{align*}
1x_1  \ \  \qquad &= 1 \\
       1x_2 &= 2,
\end{align*}

so the the solution is $x_1=1$ and $x_2=2$.

## Matrix equations

page 150

## Matrix product

```python
a,b,c,d,e,f, g,h,i,j = symbols('a b c d e f  g h i j')
A = Matrix([[a,b],
            [c,d],
            [e,f]])
B = Matrix([[g,h],
            [i,j]])

A, B
```

```python
A*B
```

```python
def mat_prod(A, B):
    """Compute the matrix product of matrices A and B."""
    assert A.cols == B.rows, "Error: matrix dimensions not compatible."
    m, ell = A.shape  # A is a  m x ell  matrix
    ell, n = B.shape  # B is a  ell x n  matrix
    C = zeros(m,n)
    for i in range(0,m):
        for j in range(0,n):
            C[i,j] = A[i,:].dot(B[:,j])
    return C

mat_prod(A,B)
```

```python
# mat_prod(B,A)
```

## Determinant

```python
a,b,c,d  = symbols('a b c d')
A = Matrix([[a,b],
            [c,d]])

A.det()
```

```python
# Consider the parallelogram with sides:
u1 = Vector([3,0])
u2 = Vector([2,2])

plot_vecs(u1,u2)
plot_vec(u1, at=u2, color='k')
plot_vec(u2, at=u1, color='b')
autoscale_arrows()

# What is the area of this parallelogram?
```

```python
# base = 3, height = 2, so area is 6
```

```python
# Compute the area of the parallelogram with sides u1 and u2 using the deteminant
A = Matrix([[3,0],
            [2,2]])
A.det()
```

## Matrix inverse

For an invertible matrix $A$, the matrix inverse $A^{-1}$ acts to undo the effects of $A$:

$$
 A^{-1} A \vec{v} = \vec{v}.
$$

The effect applying $A$ followed by $A^{-1}$ (or the other way around) is the identity transformation:

$$
 A^{-1}A \ = \ \mathbb{1} \ = \ AA^{-1}.
$$

```python
A = Matrix([[1, 2],
            [3, 9]])
A
```

```python
# Compute deteminant to check if inverse matrix exists
A.det()
# if non-zero, then inverse exists
```

```python
A.inv()
```

```python
A.inv()*A
```

### Adjugate-matrix formula

The *adjugate matrix* of the matrix $A$ is obtained by replacing each entry of the matrix with a partial determinant calculation (called *minors*). The minor $M_{ij}$ is the determinant of $A$ with its $i$th row and $j$th columns removed.

```python
A.adjugate() / A.det()
```

### Augmented matrix approach

$$
 \left[ \, A \, | \,  \mathbb{1} \, \right]
 \qquad
 -\textrm{Gauss-Jordan elimination}\rightarrow
 \qquad
 \left[ \, \mathbb{1} \, | \,  A^{-1} \, \right]
$$

```python
AUG = A.row_join(eye(2))
AUG
```

```python
# performd row operations
AUG[1,:] = AUG[1,:] - 3*AUG[0,:]
AUG[1,:] = AUG[1,:]/3
AUG[0,:] = AUG[0,:] - 2*AUG[1,:]
AUG
```

```python
AUG[:,2:5]
```

### Using elementary matrices

Each row operation $\mathcal{R}_i$ can be represented as an elementary matrix $E_i$. The elementary matrix of a given row operation is obtained by performing the row operation on the identity matrix.

```python
E1 = eye(2)
E1[1,:] = E1[1,:] - 3*E1[0,:]

E2 = eye(2)
E2[1,:] = E2[1,:]/3

E3 = eye(2)
E3[0,:] = E3[0,:] - 2*E3[1,:]

E1,E2,E3
```

```python
# the sequence of three row operations transforms the matrix A into RREF
E3*E2*E1*A
```

Recall definition $A^{-1}A=\mathbb{1}$, and we just observed that $E_3E_2E_1 A =\mathbb{1}$, so it must be that $A^{-1}=E_3E_2E_1$.

```python
E3*E2*E1
```

# 4/ Geometrical linear algebra

Points, lines, and planes are geometrical objects that are conveniently expressed using the language of vectors.

## Points

A point $p=(p_x,p_y,p_z)$ refers to a single location in $\mathbb{R}^3$.

```python
p = Point([2,4,5])
p
```

## Lines

A line is a one dimensional infinite subset of $\mathbb{R}^3$ that can be described as

$$
   \ell:  \{ p_o + \alpha \vec{v} \ | \ \forall \alpha \in \mathbb{R} \}.
$$

```python
po = Point([1,1,1])
v = Vector([1,1,0])

plot_line(v, po)
```

## Planes

A plane is a two-dimensional infinite subset of $\mathbb{R}^3$ that can be described in one of three ways:

The *general equation*:

$$
  P: \left\{ \,  Ax+By+Cz=D \, \right\}
$$

The *parametric equation*:

$$
  P: \{  p_{\textrm{o}}+s\,\vec{v} + t\,\vec{w}, \ \forall s,t \in \mathbb{R} \},
$$

which defines a plane that that contains the point $p_{\textrm{o}}$ and the vectors $\vec{v}$ and $\vec{w}$.

Or the *geometric equation*:

$$
  P: \left\{ \vec{n} \cdot [ (x,y,z) - p_{\textrm{o}} ] = 0 \,\right\},
$$

which defines a plane that contains point $p_{\textrm{o}}$ and has normal vector $\hat{n}$.

```python
# plot plane     2x + y + z = 5
normal = Vector([2,   1,  1])
D = 5

plot_plane(normal, D)
```

## Projection operation

A projection of the vector $\vec{v}$ in the direction $\vec{d}$ is denoted $\Pi_{\vec{d}}(\vec{v})$. The formula for computing the projections uses the dot product operation:

$$
   \Pi_{\vec{d}}(\vec{v})
      \ \equiv \
      (\vec{v} \cdot \hat{d}) \hat{d}
      \ =  \
      \left(\vec{v} \cdot \frac{\vec{d}}{\|\vec{d}\|} \right) \frac{\vec{d}}{\|\vec{d}\|}.
$$

```python
def proj(v, d):
    """Computes the projection of vector `v` onto direction `d`."""
    return v.dot( d/d.norm() )*( d/d.norm() )
```

```python
v = Vector([2,2])
d = Vector([3,0])
proj_v_on_d = proj(v,d)

plot_vecs(d, v, proj_v_on_d)
autoscale_arrows()
```

The basic projection operation can be used to compute projection onto planes, and compute distances between geomteirc objects (page 192).

## Bases and coordinate projections

page 194

  - Different types of bases
    - Orthonormal
    - Orthogonal
    - Generic
  - Change of basis operation

## Vector spaces

page 199

## Vector space techniques

page 211

# 5/ Linear transformations

page 223

## Vector functions

Functions that take vectors as inputs and produce vectors as outputs:

$$
  T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}
$$

## Matrix representation of linear transformations

$$
  T : \mathbb{R}^n \to \mathbb{R}^m
  \qquad
  \Leftrightarrow
  \qquad
  M_T \in \mathbb{R}^{m \times n}
$$

## Input and output spaces

We can understand the properties of linear transformations $T$, and their matrix representations $M_T$ in analogy with ordinary functions:

\begin{align*}
    \textrm{function }
     f:\mathbb{R}\to \mathbb{R}
         & \ \Leftrightarrow \,
         \begin{array}{l}
    \textrm{linear transformation }
    T:\mathbb{R}^{n}\! \to \mathbb{R}^{m}		 \\
    \textrm{represented by the matrix } M_T \in \mathbb{R}^{m \times n}
    \end{array} \\
    %
    \textrm{input }  x\in \mathbb{R}
         & \ \Leftrightarrow \
        \textrm{input }  \vec{x} \in \mathbb{R}^n           \\
    %\textrm{compute }
    \textrm{output } f(x) \in \mathbb{R}
         & \ \Leftrightarrow \
    %      \textrm{compute matrix-vector product }
     \textrm{output }      T(\vec{x}) \equiv M_T\vec{x} \in \mathbb{R}^m \\
    %\textrm{function composition  }
    g\circ\! f \: (x) =  g(f(x))
         & \ \Leftrightarrow \
    %      \textrm{matrix product }
    S(T(\vec{x})) \equiv M_SM_T \vec{x} \\
    \textrm{function inverse } f^{-1}
         & \ \Leftrightarrow \
          \textrm{matrix inverse } M_T^{-1} \\
    \textrm{zeros of } f
         & \ \Leftrightarrow \
          \textrm{kernel of } T  \equiv  \textrm{null space of } M_T \equiv \mathcal{N}(A)  \\
    \textrm{image of } f
         & \ \Leftrightarrow \
             \begin{array}{l}
             \textrm{image of } T \equiv  \textrm{column space of } M_T  \equiv \mathcal{C}(A)
              \end{array}
\end{align*}

Observe we refer to the linear transformation $T$ and its matrix representation $M_T$ interchangeably.

## Finding matrix representations

page 234

## Invertible matrix theorem

page 250

# 6/ Theoretical linear algebra

## Eigenvalues and eigenvectors

An eigenvector of the matirx $A$ is a special input vector, for which the matrix $A$ acts as a scaling:

$$
  A\vec{e}_\lambda = \lambda\vec{e}_\lambda,
$$

where $\lambda$ is called the *eigenvalue* and $\vec{e}_\lambda$ is the corresponding eigenvector.

```python
A = Matrix([[1, 5],
            [5, 1]])
A
```

```python
A*Vector([1,0])
```

```python
A*Vector([1,1])
```

The *characterisitic polynomial* of the matrix $A$ is defined as

$$
  p(\lambda) \equiv \det(A-\lambda \mathbb{1}).
$$

```python
l = symbols('lambda')
(A-l*eye(2)).det()
```

```python
# the roots of the characteristic polynomial are the eigenvalues of A
solve( (A-l*eye(2)).det(), l)
```

```python
# or call `eigenvals` method
A.eigenvals()
```

```python
A.eigenvects()
# can also find eigenvects using (A-6*eye(2)).nullspace() and (A+4*eye(2)).nullspace()
```

```python
Q, Lambda = A.diagonalize()
Q, Lambda
```

```python
Q*Lambda*Q.inv()  # == eigendecomposition of A
```

## Special types of matrices

page 271

## Abstract vectors paces

Generalize vector techniques to other vector like quantities. Allow us to talk about basis, dimention, etc.

page 277

## Abstract inner product spaces

Use geometrical notions like length and orthogonaloty for abstract vectors.

page 281

## Gram–Schmidt orthogonalization

page 288

## Matrix decompositions

page 292

## Linear algebra with complex numbers

page 298

# Applications

  - Chapter 7: General applications
  - Chapter 8: Probability theory
  - Chapter 9: Quantum mechanics

# Notation appendix

Check out page 499 for notation.


# Asymptotic solutions in short-times

Projectile motion in a linear potential field with images is described by the equation

$$y_{\tau \tau} + \alpha \frac{1}{(1 + \epsilon y)^2} + 1= 0,$$

with $y(0) = \epsilon$ and $y_{\tau}(0)=1$, and where $\epsilon \ll 1$ is expected.

```python
import sympy as sym
from sympy import init_printing
init_printing(order='rev-lex')
```

```python
y, eps, a, b, t, alpha = sym.symbols('y, epsilon, a, b, t, alpha')
y0 = sym.Function('y0')(t)
y1 = sym.Function('y1')(t)
y2 = sym.Function('y2')(t)
y3 = sym.Function('y3')(t)
y4 = sym.Function('y4')(t)
```

```python
y = sym.Eq(y0 + eps*y1 + eps**2*y2 + eps**3*y3 + eps**4*y4) # naive expansion

class f(sym.Function):

    @classmethod
    def eval(cls, y):
        return y.lhs.diff(t,t) + alpha /(1 + eps * y.lhs)**2
        #return y.lhs.diff(tau, tau) + eps/y.lhs**2
y
```

```python
the_series = sym.series(f(y), eps, x0=0, n=5)
by_order = sym.collect(the_series, eps, evaluate=False)
the_series
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
sym.Eq(by_order[1].removeO())
```

```python
eqn = sym.Eq(by_order[1].removeO())    #1 + y0(tau).diff(tau, tau))
soln0 = sym.dsolve(eqn, y0)
constants = sym.solve([soln0.rhs.subs(t,0) - 0, \
                      soln0.rhs.diff(t).subs(t,0) - 1])
C1, C2 = sym.symbols('C1 C2')
soln0 = soln0.subs(constants)
print(sym.latex(soln0))
soln0
```

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
by_order[eps]
```

```python
try:
    eqn = sym.Eq(by_order[eps].replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps])

soln1 = sym.dsolve(eqn, y1)
constants = sym.solve([soln1.rhs.subs(t,0) - 0, \
                      soln1.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln1 = soln1.subs(constants)
soln1

```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
by_order[eps**2]
```

```python
try:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs))

soln2 = sym.dsolve(eqn, y2)
constants = sym.solve([soln2.rhs.subs(t,0) - 0, \
                      soln2.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln2 = soln2.subs(constants)
sym.factor(soln2)
```

### $\mathcal{O} \left( \epsilon^3 \right) \mbox{Solution}$

```python
by_order[eps**3]
```

```python
try:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs))

soln3 = sym.dsolve(eqn, y3)
constants = sym.solve([soln3.rhs.subs(t,0) - 0, \
                      soln3.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln3 = soln3.subs(constants)
sym.factor(soln3)
```

### $\mathcal{O} \left( \epsilon^4 \right) \mbox{Solution}$

```python
by_order[eps**4]
```

```python
try:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs).replace(
        y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs))

soln4 = sym.dsolve(eqn, y4)
constants = sym.solve([soln4.rhs.subs(t,0) - 0, \
                      soln4.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln4 = soln4.subs(constants)
sym.factor(soln4)
```

### $\mbox{Composite Solution}$

```python
y_comp = sym.symbols('y_{comp}', cls=sym.Function)
try:
    y_comp = sym.Eq(y_comp, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)
except NameError:
    y_comp = sym.Eq(y_comp, eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)

#print(sym.latex(y_comp))
print(str(y_comp.rhs.subs(t, 1)))
y_comp

```

### $\mbox{The Trajectory}$

```python
def savefig(filename, pics):
    if pics == True:
        plt.savefig('../doc/figures/{}.pgf'.format(filename), bbox_inches='tight', dpi=400)
    else:
        pass
pics = False
```

```python
d = 6.47/100
R = np.linspace(0.00193, 0.00492, 4)
phi = d**2/(2*np.pi*R**2)
1/phi*20
```

    array([0.11181908, 0.25712708, 0.4620742 , 0.72666042])

```python
import matplotlib.pyplot as plt
import matplotlib
import numpy as np
import scipy as sp

%config InlineBackend.figure_format = 'retina'
#plt.rc('text', usetex=True)
#plt.rc('font', family='serif')
#plt.rcParams['figure.dpi'] = 300
%matplotlib inline
matplotlib.rcParams.update(
{   'text.color': 'k',
    'xtick.color': 'k',
    'ytick.color': 'k',
    'axes.labelcolor': 'k'
})
plt.rc('font', size=14)

eps_val = [0.1, .2, .5, 1.][::-1]
eps_val = np.linspace(0.87, 23.4, 4)[::-1]
linestyle = ['rs--', 'bo-', 'cv-.', 'k+:', 'm']
tt = sp.arange(0,3,0.001)
al = [2, 1., .5, .01]
al = phi

fig, axs = plt.subplots(2,2, figsize=(10, 8), sharex='col', sharey='row')
fig.subplots_adjust(hspace = .2, wspace=.2)
axs = axs.ravel()
i = 0
for aas in al:
    yc = y_comp.rhs.subs(alpha, aas)
    #plt.figure(figsize=(6, 4), dpi=100)
    for keys, vals in enumerate(eps_val):
        y_compP = sym.lambdify(t, yc.subs(eps, vals), 'numpy')
        if aas == 2:
            label='$\mathbf{E}\mbox{u}=$'+ ' {}'.format(vals).rstrip('0').rstrip('.')
        else:
            label=None
        axs[i].plot(tt, y_compP(tt), linestyle[keys],label=label,
                 markevery=100)
    axs[i].set_ylim(ymin=0., ymax=1)
    axs[i].set_xlim(xmax=1.5)
    axs[i].tick_params(axis='both', which='major', labelsize=16)
    leg = axs[i].legend(title = r'$\phi = $' + ' {:1.2f}'.format(aas).rstrip('0').rstrip('.'), loc=2)
    leg.get_frame().set_linewidth(0.0)
    i += 1

fig.text(0.5, -0.01, r'$t^*$', ha='center', fontsize=20)
fig.text(-0.03, 0.5, r'$y^*$', va='center', rotation='vertical', fontsize=20)
fig.tight_layout()
savefig('short_times', pics)
plt.show()
```

```python
eps_val = [.01, .1, 1.][::-1]
linestyle = ['rs--', 'bo-', 'cv-.', 'k+:', 'm']
tt = sp.arange(0,2.5,0.001)
yc = y_comp.rhs.subs(alpha, eps*0.0121 + 0.2121)

plt.figure(figsize=(6, 4))#, dpi=100)
for keys, vals in enumerate(eps_val):
    y_compP = sym.lambdify(t, yc.subs(eps, vals), 'numpy')
    plt.plot(tt, y_compP(tt), linestyle[keys], label='$\mathbf{E}\mbox{u} =$'+ ' {}'.format(vals).rstrip('0').rstrip('.'),
             markevery=100)
plt.ylim(ymin=0., ymax=0.5)
plt.xlim(xmax=2.05)
plt.legend()

plt.xlabel(r'$t^*$')
plt.ylabel(r'$y^*$')
#savefig('short_times_better', pics)
plt.show()
```

## Time aloft

```python
y2 = sym.symbols('y2', cls=sym.Function)
y2 = sym.Function('y2')(t)
try:
    y2 = sym.Eq(y2, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs) # + eps**2*soln2.rhs)
except NameError:
    y2 = sym.Eq(y2, eps*soln1.rhs + eps**2*soln2.rhs)
y2.rhs
#y2.diff(t)
```

```python
tau0, tau1, tau2 = sym.symbols('tau0 tau1 tau2')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2)
y3 = y2.rhs.subs(t, tau.lhs).series(eps)
col = sym.collect(y3, eps, evaluate=False)
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
#tau0 = 2
sym.Eq(col[1].removeO())
```

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
order_eps = col[eps].subs(tau0, 2)
order_eps
soln_eps = sym.solve(order_eps, tau1)
```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
order_eps2 = col[eps**2].subs(tau0, 2).subs(tau1, soln_eps[0])
order_eps2
soln_eps2 = sym.solve(order_eps2, tau2)
```

### Composite Solution
Using the linear regression for Im.

```python
tau0, tau1, tau2 = sym.symbols('tau0 tau1 tau2')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2)
tau = tau.subs(tau0, 2).subs(tau1, soln_eps[0]).subs(tau2, soln_eps2[0])
print(str(tau.subs(alpha, eps*0.0121 + 0.2121)))
tau.subs(alpha, eps*0.0121 + 0.2121)
```

```python
ttt = np.arange(0.01, 2.,0.001)
#betas = [bet]
linestyle = ['k','rs--', 'bo-', 'cv-.', 'k+:', 'm']
plt.figure(figsize=(6, 4), dpi=100)

#taun = tau.subs(beta, vals)
tau_soln = sym.lambdify(eps, tau.lhs.subs(alpha, eps*0.0121 + 0.2121), 'numpy')
plt.semilogx(ttt, tau_soln(ttt), 'k', markevery=100)
plt.xlabel(r'$\mathbf{E}\mbox{u}$')
plt.ylabel(r'$t_f$')
#plt.legend()
#savefig('drag', pics)
plt.show();
```

```python

```


Refresher Course on Matrix Analysis and Optimization

Part II: Numerical Optimization

Franck Iutzeler Jerome Malick

# 1- the Gradient Method

We consider two $\mathbb{R}^2 \to \mathbb{R}$ convex functions with the same global minimizer $(3,1)$ but quite different *shapes* and see how this impacts the performance of gradient-based algorithms. The considered functions $f$ and $g$ and their 3D are:

\begin{array}{rrcll}
f: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & 4 (x_1-3)^2 + 2(x_2-1)^2
\end{array}
 \begin{array}{rrcll}
g: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & \log( 1 + \exp(4 (x_1-3)^2 ) + \exp( 2(x_2-1)^2 ) ) - \log(3) .
\end{array}

In numerical optimization, it is customary to define *oracles* for the functions. These are functions that take a point of the ambiant space as an input and return the value of the function at this point (or its gradient, Hessian, etc.)

```python
# Oracle for function f
def f(x):
    """
    Function f
    """
    x1 = x[0]
    x2 = x[1]
    return 4*(x1-3)**2+2*(x2-1)**2
```

Plotting the level sets of the functions can be useful in 2D.

```python
import numpy as np
import matplotlib.pyplot as plt

npoints = 200
xmin = -0.5
xmax = 5.5
ymin = -0.5
ymax = 5.5
```

```python
X,Y = np.meshgrid(np.linspace(xmin,xmax,npoints),np.linspace(ymin,ymax,npoints))

Points = [  np.array([x,y]) for (x,y) in zip(X.ravel(), Y.ravel())]

Z = np.array([ f(p) for p in Points ]).reshape(X.shape)
```

```python
plt.contour(X,Y,Z,levels=[0.1,1,2,5,10,25])
```

> **Question a:** Construct a function `g` that will be the oracle for the second function
> \begin{array}{rrcll}
g: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  & \log( 1 + \exp(4 (x_1-3)^2 ) + \exp( 2(x_2-1)^2 ) ) - \log(3) .
\end{array}

```python

```

> **Question b:** Create a function `f_grad` that returns $\nabla f(x)$ (as a Numpy array) from an input vector $x$.
> Do the same for function $g$

```python

```

```python

```

> **Question c:** Implement a constant stepsize gradient method `gradient_algorithm(f,f_grad,x0,step,PREC,ITE_MAX)` that takes:
> * `f` and `f_grad`: respectively functions and gradient oracles
> * `x0`: a starting point
> * `step`: a (real) stepsize
> * `PREC` and `ITE_MAX`: two stopping criteria for i) sought precision (in gradient norm); and ii) maximum number of iterations;
>
> and returns a tuple with `x` the final point and `x_tab` the matrix of all vectors stacked vertically

```python

```

> **Question d:** Test your gradient descent function on $f$ and $g$:
> 1. Verify that the final point is close to the sought minimizer $(3,1)$
> 2. Observe the behavior of the iterates (you can plot them on top of the contour plot)
> 3. Change the stepsize and give the values for which the algorithm i) diverges; and ii) oscillates. Compare with theoretical limits by computing the Lipschitz constant of the gradients

```python

```

# 2- Application to Regression

We now get back to the problem of predicting the final grade of a student from various features treated in the Matrix part of the course.

We remind that mathematically, from the $m_{learn}  \times (n+1)$  *learning matrix* $A_{learn}$
($m_{learn} = 300$, $n=27$)  comprising of the features values of each training student in line, and the vector of the values of the target features $b_{learn}$;  we seek a size-$(n+1)$ *regression vector* that minimizes the squared error between  $A_{learn} x$ and $b_{learn}$. This problem boils down to the following least square problem:
$$ \min_{x\in\mathbb{R}^{n+1}} s(x) =  \frac{1}{2} \|  A_{learn} x - b_{learn} \|_2^2 . $$

```python
import numpy as np

# File reading
dat_file = np.load('data/student.npz')
A_learn = dat_file['A_learn']
b_learn = dat_file['b_learn']
A_test = dat_file['A_test']
b_test = dat_file['b_test']

m = 395 # number of read examples (total:395)
n = 27 # features
m_learn = 300
```

> **Question a:** Construct the oracles for function $s$ and its gradient as in the previous section

```python

```

> **Question b:** Compute the Lipschitz constant of the gradient of $s$. Find a solution to the minimization of $s$ using your gradient algorithm. Compare with Numpy's Least Square routine.

```python

```

> **Question c:** Generate random Gaussian matrix/vector couples $A,b$ with different sizes, change the *shape* of $A$ from *tall* (nb. of rows >> nb. of cols.) to *fat* (nb. of rows << nb. of cols.). Compare the execution time of constant stepsize gradient with fixed precision for the different couples.

```python

```

# 3- Beyond constant stepsizes

The *Rosenbrock* function $r$ writes

\begin{array}{rrcll}
r: & \mathbb{R}^2 & \to &\mathbb{R}\\
& (x_1,x_2) & \mapsto  &  (1-x_1)^2 + 100(x_2-x_1^2)^2 .
\end{array}

This function has a global minimum at $(1,1)$ with value $0$.

> **Question a:** Construct function and gradient oracles for $r$.

```python

```

> **Question b:** Try to minimize $r$ using your constant stepsize gradient function. Can you find a stepsize for which the algorithm converges?

```python

```

The observed problem is typical of poorly conditionned functions. At the beggining, gradients are large and so a small stepsize has to be employed but near the optimal value, the gradients are very small and thus the progress is very slow.

> **Question c:** Try to devise a gradient method where you change the stepsize during the algorithms to mend this problem.

```python

```

A systematic approch to stepsize tuning is the use of a [line-search](https://en.wikipedia.org/wiki/Line_search) method.

> **Question d:** Implement a gradient algorithm with [Armijo-Wolfe](https://en.wikipedia.org/wiki/Wolfe_conditions) linesearch using Scipy's [line_search](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.line_search.html) routine.
>
> *Warning:* Since you are doing gradient descent, your *search direction* is the *opposite* of the gradient.

```python

```

# 4- Application to Classification

Binary classification is another popular problem in machine learning. Instead of predicting a numerical value, the goal is now to classify the student into two classes: $+1$ -- *pass* i.e. final grade $\geq 10$; and $-1$ -- *fail*.

To this purpose, we create a class vector $c_{learn}$  from the observation vector $b_{learn}$ by simply setting $c_{learn}(i) = +1 $ if  $b_{learn}(i)\geq10$ and $-1$ otherwise.

Then, the most common approach is to minimize the logistic loss regularized by a squared norm:
\begin{equation}
\min_{x\in\mathbb{R}^{n+1}}  \ell(x) = \sum_{i=1}^{m_{learn} } \log\left( 1 + \exp\left( -c_{learn}(i) a_i^{\mathrm{T}}  x   \right) \right)  + \frac{1}{m}\|x\|^2
\end{equation}
where $a^\mathrm{T}_i$ is the $i$-th row of $A_{learn}$.

Finally, from a solution $x^\star$ of this problem, one can classify a new example, represented by its feature vector $a$, as such: the quantity $p(a) = \frac{1}{1+\exp(- a^\mathrm{T} x^\star)}$ estimates the probability of belonging to class $1$; thus, one can decide class $+1$ if for instance $ p(a) \geq 0.5$; otherwise, decide class $-1$.

```python
c_learn = np.sign(b_learn-(10-1e-15))
c_learn
```

    array([-1., -1.,  1.,  1.,  1.,  1.,  1., -1.,  1.,  1., -1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1., -1.,  1.,  1.,  1.,  1.,  1., -1., -1.,
            1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1., -1.,  1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1., -1., -1.,  1.,  1.,  1., -1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1.,  1., -1.,  1.,  1.,  1., -1., -1.,  1.,
            1.,  1., -1., -1.,  1.,  1.,  1., -1.,  1.,  1.,  1.,  1.,  1.,
            1., -1.,  1.,  1., -1.,  1.,  1., -1., -1.,  1.,  1., -1., -1.,
            1., -1.,  1.,  1.,  1.,  1.,  1.,  1., -1., -1.,  1.,  1., -1.,
            1.,  1., -1.,  1.,  1.,  1.,  1.,  1.,  1.,  1., -1.,  1.,  1.,
            1., -1.,  1.,  1.,  1.,  1.,  1., -1.,  1.,  1., -1., -1.,  1.,
           -1., -1.,  1.,  1., -1., -1., -1., -1.,  1.,  1., -1., -1.,  1.,
            1., -1.,  1., -1.,  1., -1.,  1., -1.,  1.,  1., -1.,  1., -1.,
            1.,  1.,  1.,  1., -1., -1., -1.,  1., -1.,  1.,  1.,  1., -1.,
            1., -1.,  1.,  1., -1., -1., -1.,  1., -1., -1.,  1., -1.,  1.,
            1., -1.,  1.,  1.,  1.,  1., -1.,  1.,  1., -1., -1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1.,  1.,  1.,  1., -1.,  1., -1., -1.,  1.,
            1., -1., -1.,  1.,  1., -1.,  1.,  1., -1., -1., -1.,  1., -1.,
           -1.,  1.,  1.,  1., -1.,  1.,  1., -1.,  1.,  1.,  1., -1.,  1.,
           -1.,  1.,  1.,  1.,  1., -1.,  1.,  1., -1.,  1., -1.,  1.,  1.,
           -1., -1.,  1., -1.,  1., -1., -1.,  1., -1.,  1.,  1.,  1., -1.,
            1., -1.,  1., -1., -1.,  1.,  1.,  1.,  1., -1., -1.,  1.,  1.,
            1.,  1.,  1., -1., -1., -1.,  1., -1.,  1.,  1.,  1.,  1.,  1.,
            1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1., -1., -1.,  1.,
            1.])

> **Question a:** Compute the gradient of $q(t)  = \log(1+\exp(t))$. Is the function is convex? Deduce that $\ell$ is convex and compute its gradient.

```python

```

As you can see if you plot it, this kind of function has steep slopes but also very flat valleys. Hence, constant stepsize might not perform well.

> **Question b:** Construct the suitable function and gradient simulators in order to use your gradient descent function with Armijo-Wolfe linesearch.

```python

```

> **Question c:** From a final point of the optimization algorithm above, generate a decision vector corresponding to the testing set $A_{test}$. Evaluate the classification error.

```python

```


```python
from __future__ import division

import numpy as np
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
sns.set_style('whitegrid')
sns.set_palette('colorblind')

np.random.seed(40997)
```

```python
import datagenerators as dg

```

```python
observed_data_0 = dg.generate_dataset_0()
observed_data_0.tail()
len(observed_data_0[observed_data_0.x == 0]), len(observed_data_0[observed_data_0.x == 1])
# X = wearing hat, Y = productivity
```

    (252, 248)

### Estimate Conditional

```latex
%%latex
\begin{align}
\mathbb{E}[Y=1|X=1] - \mathbb{E}[Y=1|X=0]
\end{align}
```

\begin{align}
\mathbb{E}[Y=1|X=1] - \mathbb{E}[Y=1|X=0]
\end{align}

```python
def estimate_uplift(ds):
    c, t = ds[ds.x == 0], ds[ds.x==1]
    delta = t.y.mean() - c.y.mean()
    # 95% confidence interval for standard normal is [-1.96, 1.96]
    err = 1.96*np.sqrt( (t.y.var() / t.shape[0] + c.y.var() /c.shape[0]) )
    return {'estimated_effect': delta, 'standard_err': err}
```

```python
estimate_uplift(observed_data_0)
```

    {'estimated_effect': -0.10816692268305167,
     'standard_err': 0.087296361223711094}

### Chi2 test for categorical data

```python
from scipy.stats import chi2_contingency
contingency_table = (
    observed_data_0
    .assign(placeholder=1)
    .pivot_table(index='x', columns='y', values='placeholder', aggfunc='sum')
    .values
)
contingency_table

_, p, _, _ = chi2_contingency(contingency_table, lambda_='log-likelihood')
p
# p > .05, say and so Null hypothesis holds. ????
# There is a significant relationship between wearing hat (X) and productivity (Y) ???
```

    0.019712554881038489

```python
def run_ab_test(datagenerator, nsamples=10000, filter_=None):
    '''
    Generate nsamples using datagenerator with 50% X=1 and the other X=0.
    Run the results through uplift to get the estimated conditional effect
    (avg treatment effect)
    '''
    nsamples_a = int(nsamples/2)
    nsamples_b = nsamples - nsamples_a
    setX = np.concatenate([np.ones(nsamples_a), np.zeros(nsamples_b)]).astype(np.int64)
    ds = datagenerator(nsamples, set_X=setX)
    if (filter_ != None):
        ds = ds[filter_(ds)].copy()
    return estimate_uplift(ds)

run_ab_test(dg.generate_dataset_0)
# low p-value < .05 indicates null hypothesis needs to be rejected - that is there is no relationship between
# wearing a hat and productivity. Why this reversal? Because in our A/B test we randomized the X's, and it teases
# out the relation between X, Y
```

    {'estimated_effect': 0.19500000000000006, 'standard_err': 0.019224274761603016}

## Causality
Average Treatment Effect: $ \Delta = E[Y_1] - E[Y_0] $, where $Y_1$ is the outcome when the entire population is treated, and $Y_0$ is the outcome when no one is treated (control). In RCT, $Y_1, Y_0 \perp X$. However, in Observational studies, we assume $ Y_1, Y_0 \perp X \text{ } | \text{ }Z$, where $Z$ are the control variates. Therefore, $P(X, Y_0, Y_1, Z) = P(Z)P(X|Z)P(Y_0,Y_1|Z)$

```python
observed_with_confounders = dg.generate_dataset_0(show_z=True)
print(estimate_uplift(observed_with_confounders.loc[lambda df: df.z==0]))
print(estimate_uplift(observed_with_confounders.loc[lambda df: df.z==1]))
#
# We see positive effects of hat and productivity in both splits, with Z=1 (skilled) and Z=0 (unskilled)
#
```

    {'estimated_effect': 0.25333333333333335, 'standard_err': 0.052167040467244345}
    {'estimated_effect': 0.12639553429027117, 'standard_err': 0.072451436661551447}

### Approach 1: Modeling Counter factual
Use estimators of $Y_0, Y_1$

$\hat{Y_0}(Z) = E[Y|Z, X=0] $

$\hat{Y_1}(Z) = E[Y|Z, X=1] $

ATE = $\Delta = \frac{1}{N} \sum_i(\hat{Y_1}(z_i) - \hat{Y_0}(z_i))$

```python
observed_1 = dg.generate_dataset_1()
observed_1.plot.scatter(x='z', y='y', c='x', cmap='rainbow_r', colorbar=True)
plt.title('Controls (RED) vs. Treated')
# set title color
# tobj = plt.title('Controls vs. Treated')
# plt.setp(tobj, color='r')
```

```python
observed_1.head() #, observed_1.tail()
```

      x
      y
      z

      0
      1
      0.958249
      0.717579

      1
      1
      0.669238
      0.483711

      2
      1
      1.375049
      0.943936

      3
      0
      1.218325
      0.503694

      4
      1
      1.210632
      0.993055

```python
sns.kdeplot(observed_1.loc[lambda df: df.x == 0].y, color='r', label='control')
sns.kdeplot(observed_1.loc[lambda df: df.x == 1].y, color='g', label='treated')

```

```python
print("Observed ATE: {estimated_effect:.3f}, ({standard_err:.3f})".format(**estimate_uplift(observed_1)))

```

    Observed ATE: 0.085, (0.037)

```python
sns.kdeplot(observed_1[lambda df: df.x == 0].z, color='r', label='control')
sns.kdeplot(observed_1[lambda df: df.x == 1].z, color='g', label='treated')
```

```python
# Real ATE by running A/B test
print('Real ATE: {estimated_effect:.3f}, ({standard_err:.3f})'.format(**run_ab_test(dg.generate_dataset_1)))
```

    Real ATE: -0.520, (0.009)

#### Linear Model for the counterfactual
$Y_0 = \alpha + \beta Z + \epsilon $

$Y_1 = Y_0 + \gamma $

pip install causalinference

```python
from causalinference import CausalModel
cm = CausalModel(
    Y=observed_1.y.values, # y, outcome
    D=observed_1.x.values, # x, treatment
    X=observed_1.z.values  # z confounders
)
cm.est_via_ols(adj=1)
print(cm.estimates)
#
# The package does really well because the input data is linear and designed to
# meet the expectations
#
```

    Treatment Effect Estimates: OLS

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE     -0.510      0.030    -16.917      0.000     -0.569     -0.451

```python
#
# Check against a data that is not designed to such assumptions
#
observed_2 = dg.generate_dataset_2()
observed_2.plot.scatter(x='z',y='y',c='x', cmap='rainbow_r', colorbar=True)
```

```python
sns.kdeplot(observed_2[lambda df: df.x==0].y, color='r', label='control')
sns.kdeplot(observed_2[lambda df: df.x==1].y, color='g', label='treated')

```

```python
sns.kdeplot(observed_2[lambda df: df.x==0].z, color='r', label='control')
sns.kdeplot(observed_2[lambda df: df.x==1].z, color='g', label='treated')

```

```python
#
# Here the effects are not additive
#
print('Observed ATE: {estimated_effect:.3f} ({standard_err:.3f})'.format(**estimate_uplift(observed_2)))
print('Real ATE: {estimated_effect:.3f} ({standard_err:.3f})'.format(**run_ab_test(dg.generate_dataset_2)))
```

    Observed ATE: 0.641 (0.040)
    Real ATE: 0.569 (0.011)

```python
cm = CausalModel(
    Y=observed_2.y.values,
    D=observed_2.x.values,
    X=observed_2.z.values
)
cm.est_via_ols(adj=1)
print(cm.estimates)
```

    Treatment Effect Estimates: OLS

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      0.244      0.074      3.292      0.001      0.099      0.390

```python
#
# Estimates by matching
#
cm = CausalModel(
    Y=observed_2.y.values,
    D=observed_2.x.values,
    X=observed_2.z.values
)

cm.est_via_matching()
print(cm.estimates)
```

    Treatment Effect Estimates: Matching

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      0.535      0.139      3.863      0.000      0.264      0.807
               ATC      0.979      0.170      5.759      0.000      0.646      1.312
               ATT      0.074      0.168      0.439      0.661     -0.256      0.404

### Approach 2: Covariate Imbalance (matching?)

```python
#
# Imbalanced control vs treatment
#
observed_3 = dg.generate_dataset_3()
observed_3.plot.scatter(x='z',y='y',c='x', cmap='rainbow_r', colorbar=False)
#plt.xticks(np.arange(observed_3.z.min(),observed_3.z.max(), .1))
plt.xlabel('skill')
plt.ylabel('productivity')
# plt.show()

#print(observed_3.z.min(),observed_3.z.max())
# actual response curves (data generators)
z = np.linspace(0,1,100)
y0 = np.where(z>=0.4, -4*(z-0.4), 0)
y1 = np.where(z<0.6, -4*(z-0.6), 0) + 1.

plt.plot(z, y0, color='r' )
plt.plot(z, y1, color='b' )

print("Observed ATE: {estimated_effect:.3f}, ({standard_err:.3f})".format(**estimate_uplift(observed_3)))
print("Real ATE: {estimated_effect:.3f}, ({standard_err:.3f})".format(**run_ab_test(dg.generate_dataset_3)))
```

```python
# OLS estimator
cm = CausalModel(
    Y=observed_3.y.values,
    D=observed_3.x.values,
    X=observed_3.z.values
)
cm.est_via_ols()
print(cm.estimates)
```

    Treatment Effect Estimates: OLS

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      1.933      0.052     37.451      0.000      1.832      2.034
               ATC      1.940      0.055     35.303      0.000      1.832      2.047
               ATT      1.929      0.066     29.189      0.000      1.799      2.058

```python
# Matching
cm = CausalModel(
    Y=observed_3.y.values,
    D=observed_3.x.values,
    X=observed_3.z.values
)
cm.est_via_matching()
print(cm.estimates)
```

    Treatment Effect Estimates: Matching

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      1.877      0.163     11.521      0.000      1.557      2.196
               ATC      1.830      0.277      6.597      0.000      1.286      2.374
               ATT      1.906      0.196      9.734      0.000      1.522      2.290

```python
print(cm.summary_stats)
```

    Summary Statistics

                           Controls (N_c=194)         Treated (N_t=306)
           Variable         Mean         S.d.         Mean         S.d.     Raw-diff
    --------------------------------------------------------------------------------
                  Y       -0.109        0.450        1.214        0.453        1.323

                           Controls (N_c=194)         Treated (N_t=306)
           Variable         Mean         S.d.         Mean         S.d.     Nor-diff
    --------------------------------------------------------------------------------
                 X0        0.268        0.203        0.686        0.199        2.078

### Approach 3: Inverse Propensity Score Weighting
How likely is it for a subject to have received treatment?

$\hat{p}(Z) = P[X|Z] $

Probability of potential outcome, $Y_i$:
$P(Y_i) = P(Y_i|X=i) P(X=i)$

So, we can estimate $E[Y_i] $ (inverse propensity score weight estimator, IPS),

$ = \Delta_{IPS} = \frac{1}{N}\big{(}\sum_{x_i = 1}\frac{y_i}{\hat{p}(z_i)} - \sum_{x_i = 0}\frac{y_i}{1. - \hat{p}(z_i)}\big{)}$

```python
cm = CausalModel(
    Y=observed_1.y.values,
    D=observed_1.x.values,
    X=observed_1.z.values
)
cm.est_propensity_s()
propensity = cm.propensity['fitted']
#print(propensity)

df = observed_1

def compute_ipse(df, propensity):
    df['ips'] = np.where(df.x == 1, 1./propensity, 1./(1.-propensity))
    df['ipsw'] = df.y * df.ips

    ipse = (df[df.x==1].ipsw.sum() - df[df.x==0].ipsw.sum())/(1. * df.shape[0])
    return ipse
    # print(ipse)
#
# Compare this to the real ATE for observed_1 from previously (way above)
# Real ATE: -0.520, (0.009)
#
print(compute_ipse(df, propensity))
```

    -0.486394457147

```python
#
# If we used logistic regression to estimate the propensity score, the outcome is slightly poorer
#
def compute_propensity_lg(df):
    from sklearn.linear_model import LogisticRegression
    lg = LogisticRegression()
    X = df.z.values[:, np.newaxis]
    #print(X)
    #X = df.z.values.reshape(-1,1)
    y = df.x.values
    lg.fit(X,y)
    return lg.predict_proba(X)[:,1] # probability of predicting class '1'

df = dg.generate_dataset_1()
print(compute_ipse(df, compute_propensity_lg(df)))
# The propensity computed by logistic regression is lower? (because it is not inversely weighted?)

```

    -0.435559813357

### Approach 4: Doubly Robust Weighted Estimator
This technique combines the inverse propensity score weighted and the linear estimators

```python
observed_1 = dg.generate_dataset_1()
cm = CausalModel(
    Y=observed_1.y.values,
    D=observed_1.x.values,
    X=observed_1.z.values
)
cm.est_propensity_s()
cm.est_via_weighting()
print(cm.estimates)
```

    Treatment Effect Estimates: Weighting

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE     -0.453      0.038    -12.013      0.000     -0.527     -0.379

### Unconfoundedness and Propensity Score
A stronger assumption than $Y_1, Y_0 \perp X \text{ }| \text{ } Z$ is

$ Y_1, Y_0 \perp X \text{ } | \text{ } \hat{p}(Z)$, that is given propensity

### Approach 5: Trimming (Calipers?)
Imbalances in covariates can cause issues. A solution is to make predictions for counterfactuals in regions where there is a good overlap, and "trim" points where there is not any overlap. For high dimensional data it is difficult to define good overlap. Trimming based on propensity scores is one approach

```python
# Look at dataset 3, where there is poor overlap
cm = CausalModel(
    Y=observed_3.y.values,
    D=observed_3.x.values,
    X=observed_3.z.values
)

cm.est_propensity_s()
cm.trim_s()
cm.est_via_matching()
print(cm.estimates)
#
# Real ATE: 2.456, (0.012)
#
```

    Treatment Effect Estimates: Matching

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      1.824      0.113     16.199      0.000      1.604      2.045
               ATC      1.848      0.218      8.473      0.000      1.420      2.275
               ATT      1.807      0.094     19.183      0.000      1.623      1.992

```python
# Visualize the trimmed data
propensity = cm.propensity['fitted']
#print(cm.cutoff)
#print(propensity[:20])
mask = (propensity > cm.cutoff) & ( propensity < 1. - cm.cutoff)
#print(mask[:20])
def plot_actuals(plt):
    z = np.linspace(0,1,100)
    y0 = np.where(z >=0.4, -4*(z-.4), 0 )
    y1 = np.where(z < 0.6, -4*(z-.6), 0) + 1
    plt.plot(z,y0,'r')
    plt.plot(z,y1,'b')

plt = observed_3[mask].plot.scatter(x='z',y='y',c='x', cmap='rainbow_r', colorbar=False)
plot_actuals(plt)

filter_ = lambda df: (df.z > 0.2) & (df.z < 0.7)
print("Observed ATE[mask]: {estimated_effect:.3f}, ({standard_err:.3f})".format(**estimate_uplift(observed_3[mask])))
print("Real ATE[mask]: {estimated_effect:.3f}, ({standard_err:.3f})".format(**run_ab_test(dg.generate_dataset_3,
                                                                                filter_=filter_)))

```

### Approach 6: Stratification (or blocking estimator)
Another use of the propensity score is in the stratified or blocking estimator. It consists of grouping data points into groups of similar propensity and to estimate the ATE within these groups.

```python
# Use stratify to specify the boundaries or stratify_s to have it picked automatically

observed_1 = dg.generate_dataset_1()
cm = CausalModel(
    Y=observed_1.y.values,
    D=observed_1.x.values,
    X=observed_1.z.values
)

cm.est_propensity_s()
cm.stratify_s()
print(cm.strata)
```

    Stratification Summary

                  Propensity Score         Sample Size     Ave. Propensity   Outcome
       Stratum      Min.      Max.  Controls   Treated  Controls   Treated  Raw-diff
    --------------------------------------------------------------------------------
             1     0.108     0.162        58         6     0.133     0.127    -0.428
             2     0.162     0.209        25         6     0.185     0.194    -0.463
             3     0.209     0.269        19        12     0.227     0.234    -0.467
             4     0.269     0.508        82        43     0.374     0.385    -0.507
             5     0.512     0.782        41        84     0.655     0.669    -0.453
             6     0.784     0.909        18       106     0.838     0.852    -0.332

```python
# Compute overall ATE
cm.est_via_blocking()
print(cm.estimates)
```

    Treatment Effect Estimates: Blocking

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE     -0.458      0.029    -15.966      0.000     -0.514     -0.401
               ATC     -0.473      0.033    -14.338      0.000     -0.537     -0.408
               ATT     -0.443      0.033    -13.478      0.000     -0.508     -0.379

```python
# Just an aside
# Compute a confidence interval from sample
# https://stackoverflow.com/questions/15033511/compute-a-confidence-interval-from-sample-data
#
import numpy as np
import scipy.stats

def mean_confidence_interval(data, confidence=0.95):
    a = 1.0 * np.array(data)
    n = len(a)
    m, se = np.mean(a), scipy.stats.sem(a)
    h = se * scipy.stats.t.ppf((1 + confidence) / 2., n-1)
    return m, h, m-h, m+h

a = np.random.randn(1000,)
sns.distplot(a)
# import matplotlib.pyplot as plt
# plt.scatter(a, range(len(a)))
# plt.show()
# print(a[:10])
#print(scipy.stats.norm.stats(a, 'mvsk'))
print(np.mean(a), np.var(a))
print(mean_confidence_interval(a,.95))
```

### Various Methods
Let's try all these

```python
data_gen = dg.generate_exercise_dataset_2
ds = data_gen()

print("Observed ATE: {estimated_effect:.3f} ({standard_err:.3f})".format(**estimate_uplift(ds)))
print("Real ATE: {estimated_effect:.3f} ({standard_err:.3f})".format(**run_ab_test(data_gen)))

zs = [c for c in ds.columns if c.startswith("z")]
cm = CausalModel(
    Y=ds.y.values,
    D=ds.x.values,
    X=ds[zs].values # confounders
)
print(zs)

cm.est_via_ols()
cm.est_via_matching()

#ipsw
cm.est_propensity_s()
cm.est_via_weighting()

#stratified
cm.stratify_s()
cm.est_via_blocking()

print(cm.estimates)
```

    Observed ATE: 0.118 (0.665)
    Real ATE: 4.504 (0.186)
    ['z_0', 'z_1', 'z_2', 'z_3', 'z_4']

    Treatment Effect Estimates: OLS

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      0.474      0.321      1.476      0.140     -0.155      1.104
               ATC      0.750      0.332      2.260      0.024      0.100      1.401
               ATT      0.346      0.335      1.030      0.303     -0.312      1.003

    Treatment Effect Estimates: Matching

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      1.768      0.498      3.552      0.000      0.792      2.743
               ATC      1.598      0.625      2.556      0.011      0.373      2.824
               ATT      1.846      0.548      3.367      0.001      0.771      2.922

    Treatment Effect Estimates: Weighting

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      3.654      0.956      3.821      0.000      1.780      5.528

    Treatment Effect Estimates: Blocking

                         Est.       S.e.          z      P>|z|      [95% Conf. int.]
    --------------------------------------------------------------------------------
               ATE      1.201      0.333      3.605      0.000      0.548      1.854
               ATC     -0.054      0.468     -0.115      0.908     -0.972      0.864
               ATT      1.787      0.351      5.086      0.000      1.098      2.475

```python
cm.estimates
```

    {'ols': {'ate': 0.47419777090420356, 'ate_se': 0.32116657752282168, 'atc': 0.75020996241911453, 'att': 0.34550000418610732, 'atc_se': 0.33195237063060334, 'att_se': 0.33546747808044225}, 'matching': {'atc': 1.5983416141984819, 'att': 1.8464926008999762, 'ate': 1.7675805871289012, 'atc_se': 0.62526736068696021, 'att_se': 0.54847357409903819, 'ate_se': 0.49758450947551408}, 'weighting': {'ate': 3.6538297598650313, 'ate_se': 0.95614235242183243}, 'blocking': {'ate': 1.2013569062060201, 'atc': -0.053889561123241191, 'att': 1.7866477810017753, 'ate_se': 0.33320254112880332, 'atc_se': 0.46840424000538011, 'att_se': 0.35128984048406586}}

```python
y = []
yerr = []
x_label = []

for method, result in dict(cm.estimates).items():
    y.append(result['ate'])
    yerr.append(result['ate_se'])
    x_label.append(method)

x = np.arange(len(y))

plt.errorbar(x=x, y=y, yerr=yerr, linestyle='none', capsize=5, marker='o')
plt.xticks(x, x_label)
plt.title('Estimated Effect Size', fontsize=18)
plt.hlines(4.5, -.5, 3.5, linestyles='dashed')
plt.xlim(-.5,3.5)
```

```python

```


# Vector-space models: dimensionality reduction

```python
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Fall 2020"
```

## Contents

1. [Overview](#Overview)
1. [Set-up](#Set-up)
1. [Latent Semantic Analysis](#Latent-Semantic-Analysis)
  1. [Overview of the LSA method](#Overview-of-the-LSA-method)
  1. [Motivating example for LSA](#Motivating-example-for-LSA)
  1. [Applying LSA to real VSMs](#Applying-LSA-to-real-VSMs)
  1. [Other resources for matrix factorization](#Other-resources-for-matrix-factorization)
1. [GloVe](#GloVe)
  1. [Overview of the GloVe method](#Overview-of-the-GloVe-method)
  1. [GloVe implementation notes](#GloVe-implementation-notes)
  1. [Applying GloVe to our motivating example](#Applying-GloVe-to-our-motivating-example)
  1. [Testing the GloVe implementation](#Testing-the-GloVe-implementation)
  1. [Applying GloVe to real VSMs](#Applying-GloVe-to-real-VSMs)
1. [Autoencoders](#Autoencoders)
  1. [Overview of the autoencoder method](#Overview-of-the-autoencoder-method)
  1. [Testing the autoencoder implementation](#Testing-the-autoencoder-implementation)
  1. [Applying autoencoders to real VSMs](#Applying-autoencoders-to-real-VSMs)
1. [Other methods](#Other-methods)

## Overview

The matrix weighting schemes reviewed in the first notebook for this unit deliver solid results. However, they are not capable of capturing higher-order associations in the data.

With dimensionality reduction, the goal is to eliminate correlations in the input VSM and capture such higher-order notions of co-occurrence, thereby improving the overall space.

As a motivating example, consider the adjectives _gnarly_ and _wicked_ used as slang positive adjectives.  Since both are positive, we expect them to be similar in a good VSM. However, at least stereotypically, _gnarly_ is Californian and _wicked_ is Bostonian. Thus, they are unlikely to occur often in the same texts, and so the methods we've reviewed so far will not be able to model their similarity.

Dimensionality reduction techniques are often capable of capturing such semantic similarities (and have the added advantage of shrinking the size of our data structures).

## Set-up

* Make sure your environment meets all the requirements for [the cs224u repository](https://github.com/cgpotts/cs224u/). For help getting set-up, see [setup.ipynb](setup.ipynb).

* Make sure you've downloaded [the data distribution for this course](http://web.stanford.edu/class/cs224u/data/data.tgz), unpacked it, and placed it in the current directory (or wherever you point `DATA_HOME` to below).

```python
from torch_glove import TorchGloVe
import numpy as np
from np_glove import GloVe
import os
import pandas as pd
import scipy.stats
from torch_autoencoder import TorchAutoencoder
import utils
import vsm
```

```python
# Set all the random seeds for reproducibility:

utils.fix_random_seeds()
```

```python
DATA_HOME = os.path.join('data', 'vsmdata')
```

```python
imdb5 = pd.read_csv(
    os.path.join(DATA_HOME, 'imdb_window5-scaled.csv.gz'), index_col=0)
```

```python
imdb20 = pd.read_csv(
    os.path.join(DATA_HOME, 'imdb_window20-flat.csv.gz'), index_col=0)
```

```python
giga5 = pd.read_csv(
    os.path.join(DATA_HOME, 'giga_window5-scaled.csv.gz'), index_col=0)
```

```python
giga20 = pd.read_csv(
    os.path.join(DATA_HOME, 'giga_window20-flat.csv.gz'), index_col=0)
```

## Latent Semantic Analysis

Latent Semantic Analysis (LSA) is a prominent dimensionality reduction technique. It is an application of __truncated singular value decomposition__ (SVD) and so uses only techniques from linear algebra (no machine learning needed).

### Overview of the LSA method

The central mathematical result is that, for any matrix of real numbers $X$ of dimension $m \times n$, there is a factorization of $X$ into matrices $T$, $S$, and $D$ such that

$$X_{m \times n} = T_{m \times m}S_{m\times m}D_{n \times m}^{\top}$$

The matrices $T$ and $D$ are  __orthonormal__ – their columns are length-normalized and orthogonal to one another (that is, they each have cosine distance of $1$ from each other). The singular-value matrix $S$ is a diagonal matrix arranged by size, so that the first dimension corresponds to the greatest source of variability in the data, followed by the second, and so on.

Of course, we don't want to factorize and rebuild the original matrix, as that wouldn't get us anywhere. The __truncation__ part means that we include only the top $k$ dimensions of $S$. Given our row-oriented perspective on these matrices, this means using

$$T[1{:}m, 1{:}k]S[1{:}k, 1{:}k]$$

which gives us a version of $T$ that includes only the top $k$ dimensions of variation.

To build up intuitions, imagine that everyone on the Stanford campus is associated with a 3d point representing their position: $x$ is east–west, $y$ is north–south, and $z$ is zenith–nadir. Since the campus is spread out and has relatively few deep basements and tall buildings, the top two dimensions of variation will be $x$ and $y$, and the 2d truncated SVD of this space will leave $z$ out. This will, for example, capture the sense in which someone at the top of Hoover Tower is close to someone at its base.

### Motivating example for LSA

We can also return to our original motivating example of _wicked_ and _gnarly_. Here is a matrix reflecting those assumptions:

```python
gnarly_df = pd.DataFrame(
    np.array([
        [1,0,1,0,0,0],
        [0,1,0,1,0,0],
        [1,1,1,1,0,0],
        [0,0,0,0,1,1],
        [0,0,0,0,0,1]], dtype='float64'),
    index=['gnarly', 'wicked', 'awesome', 'lame', 'terrible'])

gnarly_df
```

      0
      1
      2
      3
      4
      5

      gnarly
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0

      wicked
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0

      awesome
      1.0
      1.0
      1.0
      1.0
      0.0
      0.0

      lame
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0

      terrible
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0

No column context includes both _gnarly_ and _wicked_ together so our count matrix places them far apart:

```python
vsm.neighbors('gnarly', gnarly_df)
```

    gnarly      0.000000
    awesome     0.292893
    wicked      1.000000
    lame        1.000000
    terrible    1.000000
    dtype: float64

Reweighting doesn't help. For example, here is the attempt with Positive PMI:

```python
vsm.neighbors('gnarly', vsm.pmi(gnarly_df))
```

    gnarly      0.000000
    awesome     0.292893
    wicked      1.000000
    lame        1.000000
    terrible    1.000000
    dtype: float64

However, both words tend to occur with _awesome_ and not with _lame_ or _terrible_, so there is an important sense in which they are similar. LSA to the rescue:

```python
gnarly_lsa_df = vsm.lsa(gnarly_df, k=2)
```

```python
vsm.neighbors('gnarly', gnarly_lsa_df)
```

    gnarly      0.0
    wicked      0.0
    awesome     0.0
    lame        1.0
    terrible    1.0
    dtype: float64

### Applying LSA to real VSMs

Here's an example that begins to convey the effect that this can have empirically.

First, the original count matrix:

```python
vsm.neighbors('superb', imdb5).head()
```

    superb    0.000000
    .         0.814956
    e         0.889682
    k         0.891759
    g         0.894064
    dtype: float64

And then LSA with $k=100$:

```python
imdb5_svd = vsm.lsa(imdb5, k=100)
```

```python
vsm.neighbors('superb', imdb5_svd).head()
```

    superb     0.000000
    notch      0.015580
    talents    0.028776
    poor       0.029328
    voice      0.031802
    dtype: float64

A common pattern in the literature is to apply PMI first. The PMI values tend to give the count matrix a normal (Gaussian) distribution that better satisfies the assumptions underlying SVD:

```python
imdb5_pmi = vsm.pmi(imdb5, positive=False)
```

```python
imdb5_pmi_svd = vsm.lsa(imdb5_pmi, k=100)
```

```python
vsm.neighbors('superb', imdb5_pmi_svd).head()
```

    superb         0.000000
    outstanding    0.017650
    terrific       0.021071
    fantastic      0.033293
    brilliant      0.050633
    dtype: float64

### Other resources for matrix factorization

The [sklearn.decomposition](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) module contains an implementation of LSA ([TruncatedSVD](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html#sklearn.decomposition.TruncatedSVD)) that you might want to switch to for real experiments:

* The `sklearn` version is more flexible than the above in that it can operate on both dense matrices (Numpy arrays) and sparse matrices (from Scipy).

* The `sklearn` version will make it easy to try out other dimensionality reduction methods in your own code; [Principal Component Analysis (PCA)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA) and [Non-Negative Matrix Factorization (NMF)](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html#sklearn.decomposition.NMF) are closely related methods that are worth a look.

## GloVe

### Overview of the GloVe method

[Pennington et al. (2014)](http://www.aclweb.org/anthology/D/D14/D14-1162.pdf) introduce an objective function for semantic word representations. Roughly speaking, the objective is to learn vectors for words $w_{i}$ and $w_{j}$ such that their dot product is proportional to their probability of co-occurrence:

$$w_{i}^{\top}\widetilde{w}_{k} + b_{i} + \widetilde{b}_{k} = \log(X_{ik})$$

The paper is exceptionally good at motivating this objective from first principles. In their equation (6), they define

$$w_{i}^{\top}\widetilde{w}_{k} = \log(P_{ik}) = \log(X_{ik}) - \log(X_{i})$$

If we allow that the rows and columns can be different, then we would do

$$w_{i}^{\top}\widetilde{w}_{k} = \log(P_{ik}) = \log(X_{ik}) - \log(X_{i} \cdot X_{*k})$$

where, as in the paper, $X_{i}$ is the sum of the values in row $i$, and $X_{*k}$ is the sum of the values in column $k$.

The rightmost expression is PMI by the equivalence $\log(\frac{x}{y}) = \log(x) - \log(y)$, and hence we can see GloVe as aiming to make the dot product of two learned vectors equal to the PMI!

The full model is a weighting of this objective:

$$\sum_{i, j=1}^{|V|} f\left(X_{ij}\right)
  \left(w_i^\top \widetilde{w}_j + b_i + \widetilde{b}_j - \log X_{ij}\right)^2$$

where $V$ is the vocabulary and $f$ is a scaling factor designed to diminish the impact of very large co-occurrence counts:

$$f(x)
\begin{cases}
(x/x_{\max})^{\alpha} & \textrm{if } x < x_{\max} \\
1 & \textrm{otherwise}
\end{cases}$$

Typically, $\alpha$ is set to $0.75$ and $x_{\max}$ to $100$ (though it is worth assessing how many of your non-zero counts are above this; in dense word $\times$ word matrices, you could be flattening more than you want to).

### GloVe implementation notes

* The implementation in `np_glove.GloVe` is the most stripped-down, bare-bones version of the GloVe method I could think of. As such, it is quite slow.

* `torch_glove.py` includes a vectorized implementation that is much, much faster, so we'll mainly use that. It is based on the method and implementation introduced in [Dingwall and Potts 2018](https://www.aclweb.org/anthology/N18-2034/).

* For really large jobs, [the official C implementation released by the GloVe team](http://nlp.stanford.edu/projects/glove/) is probably the best choice.

### Applying GloVe to our motivating example

GloVe should do well on our _gnarly/wicked_ evaluation, though you will see a lot variation due to the small size of this VSM:

```python
gnarly_glove_mod = GloVe(n=2, max_iter=1000)
```

```python
gnarly_glove = gnarly_glove_mod.fit(gnarly_df)
```

    Finished epoch 999 of 1000; error is 3.371482852000653e-055

```python
vsm.neighbors('gnarly', gnarly_glove)
```

    gnarly      0.000000
    terrible    0.115135
    wicked      0.151883
    lame        0.171145
    awesome     1.602795
    dtype: float64

### Testing the GloVe implementation

It is not easy analyze GloVe values derived from real data, but the following little simulation suggests that `np_glove.glove` is working as advertised: it does seem to reliably deliver vectors whose dot products are proportional to the log co-occurrence probability:

```python
glove_test_count_df = pd.DataFrame(
    np.array([
        [10.0,  2.0,  3.0,  4.0],
        [ 2.0, 10.0,  4.0,  1.0],
        [ 3.0,  4.0, 10.0,  2.0],
        [ 4.0,  1.0,  2.0, 10.0]]),
    index=['A', 'B', 'C', 'D'],
    columns=['A', 'B', 'C', 'D'])
```

```python
glove_test_mod = GloVe(n=4, max_iter=1000)
```

```python
glove_test_df = glove_test_mod.fit(glove_test_count_df)
```

    Finished epoch 999 of 1000; error is 7.633385001520849e-054

The `score` method for `GloVe` performs the correlation test that seems appropriate for GloVe. We can roughly summarize the test as follows: calculate the Pearson correlation of `G.dot(G.T)` and `log P(X)`, where `G` is the learned GloVe embedding and `log P(X)` gives the log probabilities. The test exempts `0` count cells when calculating the correlation.

```python
glove_test_mod.score(glove_test_count_df)
```

    0.9566649559987255

### Applying GloVe to real VSMs

The `vsm.glove` implementation is too slow to use on real matrices. The version in the `torch_glove` module is significantly faster, making its use possible even without a GPU (and it will be very fast indeed on a GPU machine):

```python
glove_model = TorchGloVe()
imdb5_glv = glove_model.fit(imdb5)
```

    cuda

    Finished epoch 1000 of 1000; error is 225738.7109375

We can again use the `score` method to see how we did:

```python
glove_model.score(imdb5)
```

    0.4575722721534565

```python
vsm.neighbors('superb', imdb5_glv).head()
```

    superb         0.000000
    excellent      0.073973
    outstanding    0.085165
    terrific       0.116188
    fantastic      0.127722
    dtype: float64

## Autoencoders

An autoencoder is a machine learning model that seeks to learn parameters that predict its own input. This is meaningful when there are intermediate representations that have lower dimensionality than the inputs. These provide a reduced-dimensional view of the data akin to those learned by LSA, but now we have a lot more design choices and a lot more potential to learn higher-order associations in the underyling data.

### Overview of the autoencoder method

The module `torch_autoencoder` uses PyToch to implement a simple one-layer autoencoder:

$$
\begin{align}
h &= \mathbf{f}(xW + b_{h}) \\
\widehat{x} &= hW^{\top} + b_{x}
\end{align}$$

Here, we assume that the hidden representation $h$ has a low dimensionality like 100, and that $\mathbf{f}$ is a non-linear activation function (the default for `TorchAutoencoder` is `tanh`). These are the major design choices internal to the network. It might also be meaningful to assume that there are two matrices of weights $W_{xh}$ and $W_{hx}$, rather than using $W^{\top}$ for the output step.

The objective function for autoencoders will implement some kind of assessment of the distance between the inputs and their predicted outputs. For example, one could use the one-half mean squared error:

$$\frac{1}{m}\sum_{i=1}^{m} \frac{1}{2}(\widehat{X[i]} - X[i])^{2}$$

where $X$ is the input matrix of examples (dimension $m \times n$) and $X[i]$ corresponds to the $i$th example.

When you call the `fit` method of `TorchAutoencoder`, it returns the matrix of hidden representations $h$, which is the new embedding space: same row count as the input, but with the column count set by the `hidden_dim` parameter.

For much more on autoencoders, see the 'Autoencoders' chapter of [Goodfellow et al. 2016](http://www.deeplearningbook.org).

### Testing the autoencoder implementation

Here's an evaluation that is meant to test the autoencoder implementation – we expect it to be able to full encode the input matrix because we know its rank is equal to the dimensionality of the hidden representation.

```python
def randmatrix(m, n, sigma=0.1, mu=0):
    return sigma * np.random.randn(m, n) + mu

def autoencoder_evaluation(nrow=1000, ncol=100, rank=20, max_iter=20000):
    """This an evaluation in which `TfAutoencoder` should be able
    to perfectly reconstruct the input data, because the
    hidden representations have the same dimensionality as
    the rank of the input matrix.
    """
    X = randmatrix(nrow, rank).dot(randmatrix(rank, ncol))
    ae = TorchAutoencoder(hidden_dim=rank, max_iter=max_iter)
    ae.fit(X)
    X_pred = ae.predict(X)
    mse = (0.5 * (X_pred - X)**2).mean()
    return(X, X_pred, mse)
```

```python
ae_max_iter = 100

_, _, ae = autoencoder_evaluation(max_iter=ae_max_iter)

print("Autoencoder evaluation MSE after {0} evaluations: {1:0.04f}".format(ae_max_iter, ae))
```

### Applying autoencoders to real VSMs

You can apply the autoencoder directly to the count matrix, but this could interact very badly with the internal activation function: if the counts are all very high or very low, then everything might get pushed irrevocably towards the extreme values of the activation.

Thus, it's a good idea to first normalize the values somehow. Here, I use `vsm.length_norm`:

```python
imdb5_l2 = imdb5.apply(vsm.length_norm, axis=1)
```

```python
imdb5_l2_ae = TorchAutoencoder(
    max_iter=100, hidden_dim=50, eta=0.001).fit(imdb5_l2)
```

    Stopping after epoch 28. Training loss did not improve more than tol=1e-05. Final error is 0.0009803352440940216.

```python
vsm.neighbors('superb', imdb5_l2_ae).head()
```

    superb     0.000000
    lizard     0.001003
    fill       0.001045
    concert    0.001068
    gritty     0.001128
    dtype: float64

This is very slow and seems not to work all that well. To speed things up, one can first apply LSA or similar:

```python
imdb5_l2_svd100 = vsm.lsa(imdb5_l2, k=100)
```

```python
imdb_l2_svd100_ae = TorchAutoencoder(
    max_iter=1000, hidden_dim=50, eta=0.01).fit(imdb5_l2_svd100)
```

    Stopping after epoch 29. Training loss did not improve more than tol=1e-05. Final error is 0.0007643999852007255.

```python
vsm.neighbors('superb', imdb_l2_svd100_ae).head()
```

    superb         0.000000
    outstanding    0.017357
    terrific       0.035065
    exceptional    0.036933
    magnificent    0.052532
    dtype: float64

## Other methods

Learning word representations is one of the most active areas in NLP right now, so I can't hope to offer a comprehensive summary. I'll settle instead for identifying some overall trends and methods:

* The label __word2vec__ picks out a family of models in which the embedding for a word $w$ is trained to predict the words that co-occur with $w$. This intuition can be cashed out in numerous ways. The skip-gram model, due to [Mikolov et al. 2013](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality), is arguably the closest to the models discussed above. For instance, see [Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization) for a proof that the skip-gram model's objective reduces to PMI shifted by a constant value.

* The LexVec model of [Salle et al. 2016](https://aclanthology.coli.uni-saarland.de/papers/P16-2068/p16-2068) combines the core insight of GloVe (learn vectors that approximate PMI) with the insight from word2vec that we should additionally try to push words that don't appear together farther apart in the VSM. (GloVe simply ignores 0 count cells and so can't do this.)

* Many apparently diverse models can be expressed as matrix factorization methods like SVD/LSA. See especially
[Singh and Gordon 2008](http://www.cs.cmu.edu/~ggordon/singh-gordon-unified-factorization-ecml.pdf),
[Levy and Golberg 2014](http://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization), [Cotterell et al. 2017](https://www.aclweb.org/anthology/E17-2028/).

* Subword modeling ([reviewed briefly in the previous notebook](vsm_01_distributional.ipynb#Subword-information)) is increasingly yielding dividends. (It would already be central if most of NLP focused on languages with complex morphology!) Check out the papers at the Subword and Character-Level Models for NLP Workshops: [SCLeM 2017](https://sites.google.com/view/sclem2017/home), [SCLeM 2018](https://sites.google.com/view/sclem2018/home).

* Contextualized word representations have proven valuable in many contexts. These methods do not provide representations for individual words, but rather represent them in their linguistic context. This creates space for modeling how word senses vary depending on their context of use. We will study these methods later in the quarter, mainly in the context of identifying ways that might achieve better results on your projects.


# Determination of the refractive index

Considering the following experimental setup

which can be modeled as

From the lab class notes we know that

\begin{equation}
x = t \sin(\theta)\left\lbrace 1 - \left[\frac{1 - \sin^2(\theta)}{n_{2}^2 -\sin^2(\theta) }\right]^\frac{1}{2}\right\rbrace.
\end{equation}

The raw data look something like (green is reference beam, black is for oil and pink is for water).

```python
# Notebook magic
%matplotlib notebook

# Import the packages that are needed
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import curve_fit
from scipy.stats.distributions import chi2

# Use latex for labels
from matplotlib import rc
rc('font',**{'family':'serif','serif':['Computer Modern']})
rc('text', usetex=True)
```

## Analysis

### Define the data
Define some variables containing the data aquired (remember to note down the unit of measure).

```python
# Data
delta_theta = 40/60 * np.pi/180    # [rad] Offset angle
theta_oil = np.pi/180 * np.array([-40/60, 5, 10, 15, 20, 25, 30, 35]) + delta_theta    # [rad]
theta_water = np.pi/180 * np.array([-40/60, 15, 20, 25, 30, 35]) + delta_theta   # [rad]

err_theta = np.pi/180 * 10/(60 * np.sqrt(12))    # [rad] Resolution error of protractor (10 minutes of arc)

x_oil = 1e-3 * np.array([0, 2, 4.5, 6.5, 9, 12, 15, 18])    # [m]
x_water = 1e-3 * np.array([0, 5, 7, 9.5, 12, 14])    # [m]
err_x = 1e-3 / np.sqrt(12)    # [m]

t = 0.074    # [m] internal width of the pool
```

Let's have a look at the data ...

```python
plt.figure()
plt.errorbar(theta_oil, x_oil, xerr=err_theta, yerr=err_x, marker='.', capsize=2,
                   elinewidth=1, markeredgewidth=1, linestyle='', label='Data oil')
plt.errorbar(theta_water, x_water, xerr=err_theta, yerr=err_x, marker='.', capsize=2,
                   elinewidth=1, markeredgewidth=1, linestyle='', label='Data water')

plt.grid()
plt.xlabel(r'$\theta\ [\mbox{rad}]$')
plt.ylabel(r'$x\ [\mbox{m}]$ ')
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

### Fit
Let's use the function ``curve_fit`` to use a non-linear least squares algorithm to make the two fits.

The function ``curve_fit`` allows to use the y-uncertainties as the weights of the fit and it returns the *optimal parameters* and the *covariance matrix* (which is useful to determine the uncertainties on the fir paramters).

First let's define the fit function ...

```python
# Fit function
def my_fit_n(alpha, n):
    return t * np.sin(alpha) * (1 - np.sqrt((1 - np.sin(alpha)**2)/(n**2 - np.sin(alpha)**2)))
```

```python
plt.figure()
plt.errorbar(theta_oil, x_oil, xerr=err_theta, yerr=err_x, marker='.', capsize=2,
                   elinewidth=1, markeredgewidth=1, linestyle='', label='Data oil')
plt.errorbar(theta_water, x_water, xerr=err_theta, yerr=err_x, marker='.', capsize=2,
                   elinewidth=1, markeredgewidth=1, linestyle='', label='Data water')

# Define a reasonable x-axis for the fit
angles = np.linspace(0, 0.7)

plt.plot(angles, my_fit_n(angles, 1.5), label='Model oil: $n=1.5$')
plt.plot(angles, my_fit_n(angles, 1.33), label='Model water: $n=1.33$')

# Use non-linear least squares to fit a function, f, to data
# Oil
popt_oil, pcov_oil = curve_fit(my_fit_n, theta_oil, x_oil, sigma=err_x * np.ones(len(x_oil)), absolute_sigma=True)
# Water
popt_water, pcov_water = curve_fit(my_fit_n, theta_water, x_water, sigma=err_x * np.ones(len(x_water)), absolute_sigma=True)

plt.plot(angles, my_fit_n(angles, *popt_oil), linestyle='--', label=r'Fit oil: $n_{fit}=\,$%.2f' % popt_oil)
plt.plot(angles, my_fit_n(angles, *popt_water), linestyle='--', label='Fit water: $n_{fit}=\,$%.2f'% popt_water)

plt.grid()
plt.xlabel(r'$\theta\ [\mbox{rad}]$')
plt.ylabel(r'$x\ [\mbox{m}]$ ')
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

## Results
Here we report the results.

Under reasonable assumptions, the uncertainties on the fit paramters can be estimated as the square root of the diagonal elements of the covariance matrix $\mathbf{C}=\left[c_{ij}\right]$:

\begin{equation}
\sigma_i= \sqrt{c_{ii}}
\end{equation}

```python
print('******************** RESULTS ********************')
print('The optimal parameters (with 68% confidence) are:')
print('Refractive index oil:   %f +/- %f' % (popt_oil, np.sqrt(pcov_oil)))
print('Refractive index water: %f +/- %f' % (popt_water, np.sqrt(pcov_water)))
```

    ******************** RESULTS ********************
    The optimal parameters (with 68% confidence) are:
    Refractive index oil:   1.499474 +/- 0.008395
    Refractive index water: 1.342760 +/- 0.006557

### Chi-squared test
We perform quickly a chi-squared test to check if our fits make sense or not ...

```python
# Function that computes the reduced chi-square of my data
def chi2test(x_data, y_data, exp_err, fit_func, p_opt, false_alarm):
    # DOF is equal to the number of data points minus the number of parameters of the fit
    DOF = len(x_data) - len(p_opt)
    red_chi2 = np.sum((y_data - fit_func(x_data, *p_opt))**2 / exp_err **2) / DOF

    # Compare reduced chi-square values with the acceptable interval
    low_limit = chi2.ppf(false_alarm/2, df=DOF) / DOF
    up_limit = chi2.ppf(1 - false_alarm/2, df=DOF) / DOF

    print('***************** CHI-SQUARED TEST *****************')

    if red_chi2 <= up_limit and red_chi2 >= low_limit:
        print('PASSED!')
    else:
        print('FAILED!')

    print('The reduced chi-square of the data is %f' % red_chi2)
    print('The acceptation interval with %.0f%% - confidence is: [%f, %f]\n' % ((1-false_alarm)*100, low_limit, up_limit))
    return red_chi2

red_chi2_oil = chi2test(theta_oil, x_oil, err_x, my_fit_n, popt_oil, 0.05)
red_chi2_water = chi2test(theta_water, x_water, err_x, my_fit_n, popt_water, 0.05)
```

    ***************** CHI-SQUARED TEST *****************
    PASSED!
    The reduced chi-square of the data is 1.152352
    The acceptation interval with 95% - confidence is: [0.241410, 2.287538]

    ***************** CHI-SQUARED TEST *****************
    PASSED!
    The reduced chi-square of the data is 0.972747
    The acceptation interval with 95% - confidence is: [0.166242, 2.566500]

### Conclusions
The idex of refraction found for the oil $n_{oil} = 1.52 \pm 0.02$ is consistent with the typical values found in literature (ranging from $1.4$ to $1.6$).

The idex of refraction found for water $n_{\tiny \mbox{H}_2\mbox{O}} = 1.36 \pm 0.01$ is barely consistent (within $3\sigma$) with the nominal value $1.33$.