# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 6 Optimizing Functions

Chapter 6 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

By the end of this session, you should be able to:
- Understand general approaches to optimize functions either for minima or maxima (i.e. "extrema")
- Understand how to use derivatives in multiple dimensions to categorize extrema
- Using `scipy.optimize` to do numeric optimization of complicated functions
  - (We'll have more examples for both algebra/calculus and numerical optimization in recitation)

### Why

In chemistry and physics, we often want to determine the maximum or minimum value of a function of one or many variables. Examples include characterizing the minima, maxima, and saddle points on a potential energy surface.

Remember from Calculus, that at a maximum or a minimum, the derivative must be zero.

To decide on the type of extrema, we look at the derivative:

$$
\left.\frac{d^{2} f(x)}{d x^{2}}\right|_{x=x^{*}}=\left\{\begin{array}{ll}
<0, & \text { maximum } \\
=0, & \text { inflection point } \\
>0, & \text { minimum }
\end{array}\right.
$$

In other words, the second derivative is negative at a maxima because it "points down." At a minima, the second derivative is positive and "points up." At an inflection point, the derivative isn't zero, but the second derivative is, because on one side it is positive and on the other side it is negative.

Thus, optimizing functions in one dimension is pretty easy, if sometimes tedious.
1. Take derivatives and find where the first derivative is zero
2. Look at the second derivatives, to categorize as a minima / maxima / inflection point
3. Then compare values of the function at those points to see if it's a local minima / max or the global minima / max.

### Many Variables

Not surprisingly, we can use a similar technique in multiple dimensions.

If we have a function $f(x,y)$ in two dimensions, then to have an extrema:

$$
\frac{\partial f}{\partial x}=0 \quad \frac{\partial f}{\partial y}=0
$$

In other words, we need to see the partial derivative with respect to **both / all** variables be zero.

We can then categorize the type of minima / maxima with the [Hessian]. (Later, we will see that this is the *determinant* of the Hessian matrix, for when we have more than 2 variables.)

$$
D=\left(\frac{\partial^{2} f}{\partial x^{2}}\right)\left(\frac{\partial^{2} f}{\partial y^{2}}\right)-\left(\frac{\partial^{2} f}{\partial x \partial y}\right)\left(\frac{\partial^{2} f}{\partial y \partial x}\right)
$$

$$
\left.\frac{\partial^{2} f}{\partial x^{2}}\right|_{\left(x^{*}, y^{*}\right)}=\left\{\begin{array}{ll}
<0 & D>0 & \text { maximum } \\
>0 & D>0 & \text { minimum } \\
 & D < 0 & \text { saddle-point }
\end{array}\right.
$$

### Example:

Let's try the example of a potential energy surface (i.e., a double-well potential):

$$
V(x, y)=\left(x^{2}-1\right)^{2}+y^{2}
$$

```python
from sympy import init_session
init_session()
```

```python
V = (x**2 - 1)**2 + y**2
```

```python
# okay, let's see where the partial derivative with x is zero...
diff(V, x)
```

```python
# we can use the Sympy solve() function
# although that one isn't very hard
solve( diff(V, x) )
```

```python
# now let's do y
diff(V, y)
```

Okay, so for this potential energy function, we have three zero points:
- (-1, 0)
- (0, 0)
- (+1, 0)

What kind of points are these?

```python
# here's the two-variable Hessian test...
D = diff(V, x, 2)*diff(V, y, 2) - diff(V, x, y)*diff(V, y, x)
D # print it
```

```python
# we can use the .subs() method to substitute values (-1, 0)
D.subs([ (x, -1), (y, 0) ])
```

```python
# check (0,0)
D.subs([ (x, 0), (y, 0) ])
```

```python
# now check (+1, 0)
D.subs([ (x, 1), (y, 0) ])
```

To recap:
- (-1, 0): D is positive, second derivative is positive, this is a minima
- (0, 0): D is negative, this must be a saddle point
- (+1, 0): D is positive, second derivative is positive, this is a minima

So we can use the Hessian D to establish the type of minima / maxima / saddle point.

### More Complicated Functions

Sometimes in chemistry, functions are complicated enough that we can't easily find all the zeros.

When?

Let's imagine I want to optimize the geometry of a molecule. Each atom can move independently of the other atoms, so that's:
- X, Y, and Z displacement for atom 1
- X, Y, and Z displacement for atom 2
- .. etc.

So for N atoms, that's 3 displacements per atom, or 3N. To be precise, I'll need to subtract out cases where I move all the atoms the same amount in the X, Y, or Z directions - that's just moving the whole molecule. And I'll need to subtract our cases where I rotate the whole molecule along the X, Y, or Z axis.. so generally I have 3N-6 variabels to optimize. For water, that's 3 variables, but even for ethane, that's 8 atoms and 18 variables to optimize at once.

How do we optimize ethane or something more complicated?

Since we can presumably calculate the potential energy as a function of the atom positions, we calculate:

$$
V(1, 2, 3 .. 8)
$$

Then we calculate the gradients on each atom:

$$
\boldsymbol{F} = -\boldsymbol{\nabla} V
$$

We push the atoms a little bit, and repeat. How much do we push the atoms? There are a few methods:
- [steepest gradient descent](https://en.wikipedia.org/wiki/Gradient_descent)
- [conjugate gradient](https://en.wikipedia.org/wiki/Conjugate_gradient_method)
- [BFGS](https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm) or limited-memory L-BFGS

In some sense these are all methods that determine how much you should move along the gradient, and then what should you do .. to eventually end up at the extrema. (You pick whether you're going uphill or downhill to Point State Park.)

### Example:

A standard case for optimizing multi-variate functions is the [Rosenbrock function](https://en.wikipedia.org/wiki/Rosenbrock_function):

$$
f(\mathbf{x})=\sum_{i=1}^{N-1} 100\left(x_{i+1}-x_{i}^{2}\right)^{2}+\left(1-x_{i}\right)^{2}
$$

So you pick some number of dimensions N, and the minimum is clearly when all the $x_i = 1$

```python
import numpy as np
# the scipy.optimize module has the Rosenbrock function
from scipy.optimize import rosen
# it also has a bunch of minimize methods
from scipy.optimize import minimize

# here are our initial values - 5 dimensions
# all somewhat close to 1
x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])

rosen(x0) # close to 1, but still pretty big
```

```python
# we can minimize using the Conjugate Gradients method "CG"
# the 'disp' True part will show us some information about how many steps
optima = minimize(rosen, x0, method='CG', options={'disp': True})
```

```python
# the variable result has a bunch of things in it
dir(optima)
```

```python
# the most useful is 'x' - the array of minimized values
print(optima.x)
# also useful is 'fun' - the value of the function at optima.x
# e.g., rosen(optima.x)
print(optima.fun)
```

In short, we can use the Conjugate Gradients method to optimize the function - it took 67 steps and 834 evaluations of the Rosenbrock function. Are there better algorithms?

```python
# Let's try BFGS instead of "CG"
optima = minimize(rosen, x0, method='BFGS', options={'disp': True})
```

```python
# let's see the optimized x values and rosen(optima.x)
print(optima.x)
print(optima.fun)
```

So the BFGS method takes fewer steps then conjugate gradients (25 vs. 67) and fewer function evaluations (180 vs. 834).

If we had a more complicated function, a slower computer, .. maybe we're doing quantum mechanics and calculating the energy of the molecule takes time.. clearly we want to use efficient optimization methods. BFGS and the L-BFGS method are pretty good if you have can easily calculate the gradient.

Now if you can also easily calculate the Hessian, you can use another method - a modified Newton's method "Newton-CG"

```python
# here, we import the known derivative and Hessian of the Rosenbrock function
# - the Gradient (rosen_der)
# - the Hessian (rosen_hess)
from scipy.optimize import rosen_der, rosen_hess

# Newton-CG is a little more complicated
# because we have to give these too
optima = minimize(rosen, x0, method='Newton-CG',
               jac=rosen_der, hess=rosen_hess,
               options={'disp': True})
```

To summarize:
- Conjugate Gradients (only need gradient, 67 steps, 834 function calls)
- BFGS (also only need gradient, 25 steps, 180 function calls)
- Newton-CG (gradient and Hessian, 21 steps, 30 function calls)

In short, when we have really complicated functions, we optimize numerically using one of these methods, ideally BFGS or a Newton-CG method depending on  whether we have the gradient and/or Hessian to compute.

Beyond these techniques, there is a whole area of mathematics on "[optimization theory](https://en.wikipedia.org/wiki/Mathematical_optimization)"

Gradient-free techniques include:
- [Particle swarm optimization](https://en.wikipedia.org/wiki/Particle_swarm_optimization)
- [Simultaneous perturbation stochastic approximation](https://www.jhuapl.edu/SPSA/)
- [Genetic algorithm](https://en.wikipedia.org/wiki/Genetic_algorithm) - also work with discrete or non-continuous functions
- [Simulated annealing](https://en.wikipedia.org/wiki/Simulated_annealing)
- [Simplex / Nelder-Mead](https://en.wikipedia.org/wiki/Nelder–Mead_method)

Many of these gradient-free techniques involve estimating a [*surrogate function*](https://en.wikipedia.org/wiki/Surrogate_model) -- a simpler system to be optimized, e.g. [Bayesian optimization / Gaussian processs regression](https://en.wikipedia.org/wiki/Bayesian_optimization). (Some texts call this ["kriging"](https://en.wikipedia.org/wiki/Kriging).)

**Multiple Objectives**

Some cases, you may want to optimize more than one property that have a trade-off. This is known as [Pareto optimization](https://en.wikipedia.org/wiki/Pareto_efficiency), for example fuel efficiency and engine horsepower in a car.

In quantum chemistry, there is often a trade-off between accuracy and speed. You may make approximations that result in a faster but less accurate method (e.g., you approximate or ignore certain time-consuming integrals). Or you may want a higher accuracy calculation, but that means it will take longer.

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000


# Probabilistic Programming 4: Latent Variable & Dynamic Models

#### Goal
  - Understand how to estimate latent variables in models.
  - Understand how to estimate states in dynamical models.

#### Materials
  - Mandatory
    - This notebook
    - Lecture notes on latent variable models
    - Lecture notes on dynamical models
  - Optional
    - [Review of latent variable models](https://doi.org/10.1146/annurev-statistics-022513-115657)
    - [Bayesian Filtering & Smoothing](https://www.cambridge.org/core/books/bayesian-filtering-and-smoothing/C372FB31C5D9A100F8476C1B23721A67)
    - [Differences between Julia and Matlab / Python](https://docs.julialang.org/en/v1/manual/noteworthy-differences/index.html).

Note that none of the material below is new. The point of the Probabilistic Programming sessions is to solve practical problems so that the concepts from Bert's lectures become less abstract.

```julia
using Pkg
Pkg.activate("./workspace")
Pkg.instantiate()
IJulia.clear_output();
```

```julia
using JLD
using Statistics
using StatsBase
using LinearAlgebra
using ProgressMeter
using LaTeXStrings
using ForneyLab
using Plots
pyplot();

import LinearAlgebra: I
import ForneyLab: unsafeMean
include("../scripts/clusters.jl");
include("../scripts/filters.jl");
```

## Problem: Stone Tools

Archeologists have asked for your help in analyzing data on stone tools. It is believed that primitive humans created tools by striking stones with others. During this process, the stone loses flakes, which have been preserved. The archeologists have recovered these flakes from various locations and time periods and want to know whether this stone tool shaping process has improved over the centuries.

### Data

The data is available from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/StoneFlakes). Each instance represents summary information of the stone flakes for a particular site. We will be using the attributes _flaking angle_ (FLA) and the _proportion of the dorsal surface worked_ (PROZD) for now.

```julia
dataset = load("../datasets/stoneflakes.jld");
```

I've done some pre-processing on the data set, namely [z-scoring](https://nl.wikipedia.org/wiki/Z-score) and removing two outliers. This reduces the scale of the attributes which helps numerical stability during optimization.

Now let's visualize the data with a scatterplot.

```julia
scatter(dataset["data"][:,1],
        dataset["data"][:,2],
        label="",
        xlabel="Proportion of worked dorsal surface (PROZD)",
        ylabel="Flaking angle (FLA)")
```

### Model specification

We will be clustering this data with a Gaussian mixture model, to see if we can identify clear types of stone tools. The generative model for a Gaussian mixture consists of:

$$ p(X, z, \phi, \mu, \Lambda) =\ \underbrace{p(X \mid z, \mu, \Lambda)}_{\text{likelihood}}\ \times \ \underbrace{p(z \mid \phi)}_{\text{prior latent variables}} \ \times \ \underbrace{p(\mu \mid \Lambda)\  p(\Lambda)\ p(\phi)}_{\text{prior parameters}}$$

with the likelihood of observation $X_i$ being a Gaussian raised to the power of the latent assignment variables $z$

$$ p(X_i \mid z, \mu, \Lambda) = \prod_{k=1}^{K} \mathcal{N}(X_i \mid \mu_k, \Lambda_k^{-1})^{z_i = k}$$

the prior for each latent variable $z_i$ being a Categorical distribution

$$ p(z_i \mid \phi) = \text{Categorical}(z_i \mid \phi) $$

and priors for the parameters being

$$ \begin{align*}
p(\mu_k \mid \Lambda_k) =&\ \mathcal{N}(\mu_k \mid m_0, l_0^{-1}\Lambda_k^{-1}) \qquad &\text{for all}\ k \\
p(\Lambda_k) =&\ \text{Wishart}(\Lambda_k \mid V_0, n_0) \qquad &\text{for all}\ k \\
p(\phi) =&\ \text{Dirichlet}(\phi \mid a_0) \, ,
\end{align*}$$

We will be implementing this model directly in ForneyLab. If you're unfamiliar with these distributions or with the Gaussian mixture model, have another look at Bert's lectures.

---

First, we will do a bit of bookkeeping.

```julia
# Data dimensionality
num_features = size(dataset["data"],2)

# Sample size
num_samples = size(dataset["data"],1)

# Number of mixture components
num_components = 3;

# Identity matrix (convenience variable)
Id = Matrix{Float64}(I, num_features, num_features);
```

Mixture models can be sensitive to initialization, so we are going to specify the prior parameters explicitly.

```julia
# Prior means
m0 = [ 1.0 0.0 -1.0;
      -1.0 0.0  1.0];

# Prior scale matrices
V0 = cat(Id, Id, Id, dims=3)

# Prior degrees of freedom
n0 = num_features

# Prior concentration parameters
a0 = ones(num_components);
```

Now to start the factor graph.

```julia
# Start a graph
graph1 = FactorGraph()

# Initialize vector variables
z = Vector{Variable}(undef, num_samples)
X = Vector{Variable}(undef, num_samples)
Λ = Vector{Variable}(undef, num_components)
μ = Vector{Variable}(undef, num_components)

# Mixture weights are drawn from a Dirichlet distribution
@RV ϕ ~ Dirichlet(a0)

θ = []
for k = 1:num_components

    # Parameters of k-th component
    @RV Λ[k] ~ Wishart(V0[:,:,k], n0)
    @RV μ[k] ~ GaussianMeanPrecision(m0[:,k], Λ[k])

    push!(θ, μ[k], Λ[k])
end

for i = 1:num_samples

    # Assignment variable
    @RV z[i] ~ Categorical(ϕ)

    # Gaussian mixture component
    @RV X[i] ~ GaussianMixture(z[i], θ...)

    # Add data
    placeholder(X[i], :X, dims=(num_features,), index=i)
end
```

This is another _iid_ setting, which means the graph will be too large to visualize.

The next step is to compile an inference algorithm.

```julia
# Specify recognition factorization (mean-field)
q = PosteriorFactorization(ϕ, μ[1], Λ[1], μ[2], Λ[2], μ[3], Λ[3], z,
                           ids=[:ϕ, :μ_1, :Λ_1, :μ_2, :Λ_2, :μ_3, :Λ_3, :z])

# Generate the algorithm
algorithm = messagePassingAlgorithm(free_energy=true)
source_code = algorithmSourceCode(algorithm, free_energy=true);
eval(Meta.parse(source_code));
```

After that, we feed in data, initialize recognition factors and run the inference procedure.

```julia
# Convert data to a format suited to ForneyLab
observations = [dataset["data"][i,:] for i in 1:num_samples]

# Add to data dictionary
data = Dict(:X => observations)

# Prepare recognition distributions
marginals = Dict()
marginals[:ϕ] = ProbabilityDistribution(Dirichlet, a=ones(num_components,))
for k = 1:num_components
    marginals[:μ_*k] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=m0[:,k], w=Id)
    marginals[:Λ_*k] = ProbabilityDistribution(Wishart, v=Id, nu=num_features)
end
for i = 1:num_samples
    marginals[:z_*i] = ProbabilityDistribution(Categorical, p=ones(num_components,)./num_components)
end

# Number of iterations
num_iterations = 20

# Preallocate free energy tracking array
F = Float64[]

# Execute algorithm
@showprogress for i = 1:num_iterations

    # Update assignments
    stepz!(data, marginals)

    # Update parameters
    stepϕ!(data, marginals)
    stepμ_1!(data, marginals)
    stepΛ_1!(data, marginals)
    stepμ_2!(data, marginals)
    stepΛ_2!(data, marginals)
    stepμ_3!(data, marginals)
    stepΛ_3!(data, marginals)

    # Store variational free energy for visualization
    push!(F, freeEnergy(data, marginals))
end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:28

Alright, we're done. Let's track the evolution of free energy.

```julia
# Plot free energy to check for convergence
plot(1:num_iterations, F, color="black", label="")
xlabel!("Number of iterations")
ylabel!("Free Energy")
```

That looks like it is nicely decreasing. We might want to increase the number of iterations a bit more.

Let's now visualize the cluster on top of the observations.

```julia
# Estimated means (unsafeMean retrieves parameters directly)
μ1_estimated = unsafeMean(marginals[:μ_1])
μ2_estimated = unsafeMean(marginals[:μ_2])
μ3_estimated = unsafeMean(marginals[:μ_3])

# Estimated precisions
Λ1_estimated = unsafeMean(marginals[:Λ_1])
Λ2_estimated = unsafeMean(marginals[:Λ_2])
Λ3_estimated = unsafeMean(marginals[:Λ_3])

# Invert to covariances
Σ1_estimated = inv(Λ1_estimated)
Σ2_estimated = inv(Λ2_estimated)
Σ3_estimated = inv(Λ3_estimated)

# Select dimensions to plot
dims_plot = [1, 2]
dim_limsx = [minimum(dataset["data"][:,dims_plot[1]])-1, maximum(dataset["data"][:,dims_plot[1]])+1]
dim_limsy = [minimum(dataset["data"][:,dims_plot[2]])-1, maximum(dataset["data"][:,dims_plot[2]])+1]

# Plot data and overlay estimated posterior probabilities
plot_clusters(dataset["data"][:, dims_plot],
              μ=[μ1_estimated[dims_plot], μ2_estimated[dims_plot], μ3_estimated[dims_plot]],
              Σ=[Σ1_estimated[dims_plot,dims_plot], Σ2_estimated[dims_plot,dims_plot], Σ3_estimated[dims_plot,dims_plot]],
              x1=range(dim_limsx[1], step=0.01, stop=dim_limsx[2]),
              x2=range(dim_limsy[1], step=0.01, stop=dim_limsy[2]),
              size=(600,400))
```

That doesn't look bad. The three Gaussians nicely cover all samples.

---

### $\ast$ **Try for yourself**

Play around with the number of components. Can you get an equally good coverage with just 2 components? What if you had 4?

---

## Problem: Alpine Railways

The Swiss Federal Railways company operates a series of [mountain railways](https://en.wikipedia.org/wiki/List_of_mountain_railways_in_Switzerland) bringing hikers (in the summer) and skiers (in the winter) up the Alps. They are setting up a new fallback security system where they intend to track trains through cameras and remote sensors. They want you to design a system to keep track of the trains' positions.

### Data

A train going uphill updates its position according to: new position = old position + velocity x length of time-step + noise. The noise represents the train randomly slipping and sliding back down. We observe the train through a remote sensor, producing noisy observations of its position.

You receive a data set with past recordings. Your job is to set up an online filtering system, which can be deployed later on to process the incoming signal in real-time.

```julia
signal = load("../datasets/alpinerails_filtering.jld");

# Unpack data
states = signal["X"]
observations = signal["Y"]
transition = signal["A"]
emission = signal["C"]
process_noise = signal["Q"]
measurement_noise = signal["R"]
T = signal["T"]
Δt = signal["Δt"];

# Size
M = size(states,1)
N = size(observations,1)

# Visualize
plot(1:T, states[1,:], color="red", label="states", grid=false, xlabel="time (t)", ylabel="train position")
scatter!(1:T, observations[1,:], color="black", label="observations")
```

### Model specification

We define noisy observations $y_k \in \mathbb{R}^{1}$ with latent states $x_k \in \mathbb{R}^{2}$. Observations $y_k$ are generated through a Gaussian likelihood centered on an emission matrix $C$ times the current state $x_k$ perturbed by measurement noise with covariance matrix $R$. State transitions follow a Gaussian distribution centered on a transition matrix $A$ times the previous state perturbed by process noise with covariance matrix $Q$. In equation form, these are:

$$\begin{align}
p(x_k \mid x_{k-1}) =&\ \mathcal{N}(x_k \mid A x_{k-1}, Q)\\
p(y_k \mid x_k) =&\ \mathcal{N}(y_k \mid C x_k, R) \, .
\end{align}$$

We have a prior for the previous state $x_{k-1} \sim \mathcal{N}(m_{k-1}, V_{k-1})$. In filtering problems, we feed the estimates of the current states as the parameters for the previous state in the next time-step.

```julia
# Initialize a graph
graph2 = FactorGraph()

# Define initial state prior
@RV x_kmin1 ~ GaussianMeanVariance(placeholder(:m_kmin1, dims=(M,)),
                                   placeholder(:V_kmin1, dims=(M,M)))

# State transition
@RV x_k ~ GaussianMeanVariance(transition * x_kmin1, process_noise)

# Observation likelihood
@RV y_k ~ GaussianMeanVariance(dot(emission, x_k), measurement_noise)

# Tell FL that y is observed
placeholder(y_k, :y_k);

# Visualize subgraph
ForneyLab.draw(graph2)
```

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Generated by graphviz version 2.40.1 (20161225.0304)
 -->
<!-- Title: G Pages: 1 -->
<svg width="464pt" height="746pt"
 viewBox="0.00 0.00 463.50 746.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 742)">
<title>G</title>
<polygon fill="#ffffff" stroke="transparent" points="-4,4 -4,-742 459.5,-742 459.5,4 -4,4"/>
<!-- 11961784858993670058 -->
<g id="node1" class="node">
<title>11961784858993670058</title>
<polygon fill="#d3d3d3" stroke="#000000" points="95,-54 0,-54 0,0 95,0 95,-54"/>
<text text-anchor="middle" x="47.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_V_kmin1</text>
</g>
<!-- 13525923077041460389 -->
<g id="node2" class="node">
<title>13525923077041460389</title>
<polygon fill="none" stroke="#000000" points="204.5,-171 100.5,-171 100.5,-99 204.5,-99 204.5,-171"/>
<text text-anchor="middle" x="152.5" y="-137.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="152.5" y="-127.8" font-family="Times,serif" font-size="9.00" fill="#000000">gaussianmeanvariance_1</text>
</g>
<!-- 13525923077041460389--11961784858993670058 -->
<g id="edge9" class="edge">
<title>13525923077041460389--11961784858993670058</title>
<path fill="none" stroke="#000000" d="M117.2444,-98.7371C103.065,-84.1526 86.9977,-67.6262 73.8615,-54.1146"/>
<text text-anchor="start" x="100.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">V_kmin1</text>
<text text-anchor="start" x="54.8615" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="104.2444" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 17337918697909013489 -->
<g id="node11" class="node">
<title>17337918697909013489</title>
<polygon fill="#d3d3d3" stroke="#000000" points="305,-54 210,-54 210,0 305,0 305,-54"/>
<text text-anchor="middle" x="257.5" y="-24.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_m_kmin1</text>
</g>
<!-- 13525923077041460389--17337918697909013489 -->
<g id="edge2" class="edge">
<title>13525923077041460389--17337918697909013489</title>
<path fill="none" stroke="#000000" d="M187.7556,-98.7371C201.935,-84.1526 218.0023,-67.6262 231.1385,-54.1146"/>
<text text-anchor="start" x="210.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">m_kmin1</text>
<text text-anchor="start" x="212.1385" y="-56.7146" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="172.7556" y="-92.3371" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
</g>
<!-- 12542845186555412082 -->
<g id="node3" class="node">
<title>12542845186555412082</title>
<polygon fill="none" stroke="#000000" points="286.5,-288 212.5,-288 212.5,-216 286.5,-216 286.5,-288"/>
<text text-anchor="middle" x="249.5" y="-254.8" font-family="Times,serif" font-size="9.00" fill="#000000">×</text>
<text text-anchor="middle" x="249.5" y="-244.8" font-family="Times,serif" font-size="9.00" fill="#000000">multiplication_1</text>
</g>
<!-- 12542845186555412082--13525923077041460389 -->
<g id="edge3" class="edge">
<title>12542845186555412082--13525923077041460389</title>
<path fill="none" stroke="#000000" d="M219.6053,-215.9415C207.8042,-201.7071 194.2708,-185.3833 182.463,-171.1409"/>
<text text-anchor="start" x="203.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">x_kmin1</text>
<text text-anchor="start" x="163.463" y="-173.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="200.6053" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 in1 </text>
</g>
<!-- 11282890469948126819 -->
<g id="node5" class="node">
<title>11282890469948126819</title>
<polygon fill="#d3d3d3" stroke="#000000" points="373.5,-162 319.5,-162 319.5,-108 373.5,-108 373.5,-162"/>
<text text-anchor="middle" x="346.5" y="-132.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_1</text>
</g>
<!-- 12542845186555412082--11282890469948126819 -->
<g id="edge1" class="edge">
<title>12542845186555412082--11282890469948126819</title>
<path fill="none" stroke="#000000" d="M279.3947,-215.9415C293.7565,-198.6185 310.6838,-178.201 323.9442,-162.2064"/>
<text text-anchor="start" x="300.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_1</text>
<text text-anchor="start" x="304.9442" y="-164.8064" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="266.3947" y="-209.5415" font-family="Times,serif" font-size="8.00" fill="#000000">3 a </text>
</g>
<!-- 8256772125782865436 -->
<g id="node4" class="node">
<title>8256772125782865436</title>
<polygon fill="none" stroke="#000000" points="185.5,-639 81.5,-639 81.5,-567 185.5,-567 185.5,-639"/>
<text text-anchor="middle" x="133.5" y="-605.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="133.5" y="-595.8" font-family="Times,serif" font-size="9.00" fill="#000000">gaussianmeanvariance_3</text>
</g>
<!-- 14488786852968061199 -->
<g id="node8" class="node">
<title>14488786852968061199</title>
<polygon fill="#d3d3d3" stroke="#000000" points="86.5,-513 32.5,-513 32.5,-459 86.5,-459 86.5,-513"/>
<text text-anchor="middle" x="59.5" y="-483.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_4</text>
</g>
<!-- 8256772125782865436--14488786852968061199 -->
<g id="edge5" class="edge">
<title>8256772125782865436--14488786852968061199</title>
<path fill="none" stroke="#000000" d="M81.39,-579.6969C69.4552,-571.752 58.2919,-561.6051 51.5,-549 45.6426,-538.1293 46.4417,-524.7118 49.1939,-513.0568"/>
<text text-anchor="start" x="51.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_4</text>
<text text-anchor="start" x="30.1939" y="-515.6568" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="68.39" y="-582.2969" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 10313299326062430023 -->
<g id="node12" class="node">
<title>10313299326062430023</title>
<polygon fill="none" stroke="#000000" points="273.5,-522 201.5,-522 201.5,-450 273.5,-450 273.5,-522"/>
<text text-anchor="middle" x="237.5" y="-488.8" font-family="Times,serif" font-size="9.00" fill="#000000">dot</text>
<text text-anchor="middle" x="237.5" y="-478.8" font-family="Times,serif" font-size="9.00" fill="#000000">dotproduct_1</text>
</g>
<!-- 8256772125782865436--10313299326062430023 -->
<g id="edge10" class="edge">
<title>8256772125782865436--10313299326062430023</title>
<path fill="none" stroke="#000000" d="M165.552,-566.9415C178.2048,-552.7071 192.7148,-536.3833 205.3748,-522.1409"/>
<text text-anchor="start" x="188.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">variable_2</text>
<text text-anchor="start" x="186.3748" y="-524.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="150.552" y="-560.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
</g>
<!-- 4493200663958554577 -->
<g id="node6" class="node">
<title>4493200663958554577</title>
<polygon fill="#d3d3d3" stroke="#000000" points="171,-738 96,-738 96,-684 171,-684 171,-738"/>
<text text-anchor="middle" x="133.5" y="-708.8" font-family="Times,serif" font-size="9.00" fill="#000000">placeholder_y_k</text>
</g>
<!-- 4493200663958554577--8256772125782865436 -->
<g id="edge11" class="edge">
<title>4493200663958554577--8256772125782865436</title>
<path fill="none" stroke="#000000" d="M133.5,-683.747C133.5,-670.2495 133.5,-653.7693 133.5,-639.2253"/>
<text text-anchor="start" x="133.5" y="-659.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">y_k</text>
<text text-anchor="start" x="114.5" y="-641.8253" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="114.5" y="-677.347" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
</g>
<!-- 4290618869832858310 -->
<g id="node7" class="node">
<title>4290618869832858310</title>
<polygon fill="#d3d3d3" stroke="#000000" points="178.5,-396 124.5,-396 124.5,-342 178.5,-342 178.5,-396"/>
<text text-anchor="middle" x="151.5" y="-366.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_3</text>
</g>
<!-- 11365605713984537538 -->
<g id="node9" class="node">
<title>11365605713984537538</title>
<polygon fill="#d3d3d3" stroke="#000000" points="455.5,-279 401.5,-279 401.5,-225 455.5,-225 455.5,-279"/>
<text text-anchor="middle" x="428.5" y="-249.8" font-family="Times,serif" font-size="9.00" fill="#000000">clamp_2</text>
</g>
<!-- 17953359042533603836 -->
<g id="node10" class="node">
<title>17953359042533603836</title>
<polygon fill="none" stroke="#000000" points="397.5,-405 293.5,-405 293.5,-333 397.5,-333 397.5,-405"/>
<text text-anchor="middle" x="345.5" y="-371.8" font-family="Times,serif" font-size="9.00" fill="#000000">𝒩</text>
<text text-anchor="middle" x="345.5" y="-361.8" font-family="Times,serif" font-size="9.00" fill="#000000">gaussianmeanvariance_2</text>
</g>
<!-- 17953359042533603836--12542845186555412082 -->
<g id="edge7" class="edge">
<title>17953359042533603836--12542845186555412082</title>
<path fill="none" stroke="#000000" d="M315.9135,-332.9415C304.234,-318.7071 290.8401,-302.3833 279.1541,-288.1409"/>
<text text-anchor="start" x="299.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">variable_1</text>
<text text-anchor="start" x="260.1541" y="-290.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="300.9135" y="-326.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 m </text>
</g>
<!-- 17953359042533603836--11365605713984537538 -->
<g id="edge4" class="edge">
<title>17953359042533603836--11365605713984537538</title>
<path fill="none" stroke="#000000" d="M371.08,-332.9415C383.3689,-315.6185 397.8531,-295.201 409.1997,-279.2064"/>
<text text-anchor="start" x="388.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_2</text>
<text text-anchor="start" x="390.1997" y="-281.8064" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="358.08" y="-326.5415" font-family="Times,serif" font-size="8.00" fill="#000000">3 v </text>
</g>
<!-- 10313299326062430023--4290618869832858310 -->
<g id="edge8" class="edge">
<title>10313299326062430023--4290618869832858310</title>
<path fill="none" stroke="#000000" d="M201.365,-473.2014C182.9168,-464.6125 162.1629,-451.2859 151.5,-432 145.5728,-421.2794 144.8319,-407.8808 145.8738,-396.1993"/>
<text text-anchor="start" x="151.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">clamp_3</text>
<text text-anchor="start" x="126.8738" y="-398.7993" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="182.365" y="-475.8014" font-family="Times,serif" font-size="8.00" fill="#000000">3 in2 </text>
</g>
<!-- 10313299326062430023--17953359042533603836 -->
<g id="edge6" class="edge">
<title>10313299326062430023--17953359042533603836</title>
<path fill="none" stroke="#000000" d="M270.7848,-449.9415C283.9242,-435.7071 298.9923,-419.3833 312.1392,-405.1409"/>
<text text-anchor="start" x="294.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="#ff0000">x_k</text>
<text text-anchor="start" x="293.1392" y="-407.7409" font-family="Times,serif" font-size="8.00" fill="#000000">1 out </text>
<text text-anchor="start" x="251.7848" y="-443.5415" font-family="Times,serif" font-size="8.00" fill="#000000">2 in1 </text>
</g>
</g>
</svg>

We'll now construct the algorithm and infer results.

```julia
# Generate inference algorithm
algorithm = messagePassingAlgorithm(x_k)
source_code = algorithmSourceCode(algorithm)
eval(Meta.parse(source_code));
```

For filtering, we use the same graph in each time-step.

```julia
# Initialize recognition distribution marginals
marginals = Dict(:x_k => vague(GaussianMeanVariance, M))

# Initialize message array
messages = Array{Message}(undef, 5)

# Keep track of estimates
m_x = 10*ones(M,T+1)
V_x = repeat(10*Matrix{Float64}(I,M,M), outer=(1,1,T+1))

@showprogress for k = 1:T

    # Initialize data
    data = Dict(:y_k => observations[k],
                :m_kmin1 => m_x[:,k],
                :V_kmin1 => V_x[:,:,k])

    # Update states
    step!(data, marginals, messages)

    # Store estimates
    m_x[:,k+1] = mean(marginals[:x_k])
    V_x[:,:,k+1] = cov(marginals[:x_k])

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:01

Let's check whether everything went ok. We'll visualize the state estimations.

```julia
# Visualization
plot(1:T, states[1,:], color="red", label="states", grid=false)
plot!(1:T, m_x[1,2:end], ribbon=[sqrt.(V_x[1,1,2:end]), sqrt.(V_x[1,1,2:end])], fillalpha=0.2, color="blue", label="inferred")
scatter!(1:T, observations[1,:], color="black", label="observations")
```

We're going to inspect some messages. Let's open up the algorithm and look up the marginal computation for the final $x_k$. It will be the multiplication of two messages, one consisting of the state transition prediction and the other consisting of the measurement likelihood.

```julia
println(source_code)
```

    begin

    function step!(data::Dict, marginals::Dict=Dict(), messages::Vector{Message}=Array{Message}(undef, 5))

    messages[1] = ruleSPGaussianMeanVarianceOutNPP(nothing, Message(Multivariate, PointMass, m=data[:m_kmin1]), Message(MatrixVariate, PointMass, m=data[:V_kmin1]))
    messages[2] = ruleSPMultiplicationOutNGP(nothing, messages[1], Message(MatrixVariate, PointMass, m=[1.0 0.1; 0.0 1.0]))
    messages[3] = ruleSPGaussianMeanVarianceOutNGP(nothing, messages[2], Message(MatrixVariate, PointMass, m=[0.01 0.0; 0.0 0.1]))
    messages[4] = ruleSPGaussianMeanVarianceMPNP(Message(Univariate, PointMass, m=data[:y_k]), nothing, Message(Univariate, PointMass, m=0.1))
    messages[5] = ruleSPDotProductIn1GNP(messages[4], nothing, Message(Multivariate, PointMass, m=[1.0, 0.0]))

    marginals[:x_k] = messages[3].dist * messages[5].dist

    return marginals

    end

    end # block

Alright, we need messages 3 and 5. Let's visualize them along with the state marginal.

```julia
# Collect state transition prediction
m_statepred = convert(ProbabilityDistribution{Multivariate, GaussianMeanVariance}, messages[3].dist)

# Collect observation likelihood
m_likelihood = convert(ProbabilityDistribution{Multivariate, GaussianMeanVariance}, messages[5].dist)

# Collect corrected prediction
state_marginal = convert(ProbabilityDistribution{Multivariate, GaussianMeanVariance}, marginals[:x_k])

# # Extract x-coordinates
m_statepred_x = ProbabilityDistribution(Univariate, GaussianMeanVariance, m=m_statepred.params[:m][1], v=m_statepred.params[:v][1,1])
m_likelihood_x = ProbabilityDistribution(Univariate, GaussianMeanVariance, m=m_likelihood.params[:m][1], v=m_likelihood.params[:v][1,1])
state_marginal_x = ProbabilityDistribution(Univariate, GaussianMeanVariance, m=state_marginal.params[:m][1], v=state_marginal.params[:v][1,1])

# Plot of the prediction, noisy measurement, and corrected prediction for x-coordinate
plot_messages(m_statepred_x, m_likelihood_x, state_marginal_x)
```

As you can see, the state estimate is a combination of the state prediction, produced by the message from the state transition node, and the observation likelihood, produced by the message from the likelihood.

---

### $\ast$ **Try for yourself**

Re-run the inference procedure and stop at an earlier time-step, for example $k$=2. How does the balance between the state prediction and the observation likelihood differ?

---


# Neuromatch Academy: Week 1, Day 5, Tutorial 3
# Dimensionality Reduction and reconstruction

---

In this notebook we'll learn to apply PCA for dimensionality reduction, using a classic dataset that is often used to benchmark machine learning algorithms: the MNIST dataset of handwritten digits. We'll also learn how to use PCA for reconstruction and denoising.

Steps:
 1. Perform PCA on MNIST dataset.
 2. Calculate the variance explained.
 3. Reconstruct data with different numbers of PCs.
 4. Examine denoising using PCA.

To learn more about MNIST:
* https://en.wikipedia.org/wiki/MNIST_database

---

```python
#@title Video: Logistic regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="ew0-P7-6Nho", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=ew0-P7-6Nho

# Setup
Run these cells to get the tutorial started.

```python
#import libraries
import time                        # import time
import numpy as np                 # import numpy
import scipy as sp                 # import scipy
import math                        # import basic math
import random                      # import basic random number generator functions

import matplotlib.pyplot as plt    # import matplotlib
from IPython import display
```

```python
# @title Figure Settings
fig_w, fig_h = (10, 4)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
plt.style.use('ggplot')

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
```

```python
# @title Helper Functions
# New helper functions

def plot_variance_explained(variance_explained):
  """
  Plots eigenvalues.

  Args:
     variance_explained (numpy array of floats) : Vector of variance explained for each PC

  Returns:
    Nothing.

  """
  plt.figure()
  plt.plot(np.arange(1,len(variance_explained)+1),variance_explained,'o-k')
  plt.xlabel('Number of components')
  plt.ylabel('Variance explained')

def plot_MNIST_reconstruction(X,X_reconstructed):
  """
  Plots 9 images in the MNIST dataset side-by-side with the reconstructed images.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable
     X_reconstructed (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.

  """
  plt.figure()
  ax = plt.subplot(1,2,1)
  k=0
  for k1 in range(3):
    for k2 in range(3):
      k = k+1
      plt.imshow(np.reshape(X[k,:],(28,28)),extent=[(k1+1)*28,k1*28,(k2+1)*28,k2*28],vmin=0,vmax=255)
  plt.xlim((3*28,0))
  plt.ylim((3*28,0))
  plt.tick_params(axis='both',which='both',bottom=False,top=False,labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.title('Data')
  plt.clim([0,250])
  ax = plt.subplot(1,2,2)
  k=0
  for k1 in range(3):
    for k2 in range(3):
      k = k+1
      plt.imshow(np.reshape(np.real(X_reconstructed[k,:]),(28,28)),extent=[(k1+1)*28,k1*28,(k2+1)*28,k2*28],vmin=0,vmax=255)
  plt.xlim((3*28,0))
  plt.ylim((3*28,0))
  plt.tick_params(axis='both',which='both',bottom=False,top=False,labelbottom=False)
  ax.set_xticks([])
  ax.set_yticks([])
  plt.clim([0,250])
  plt.title('Reconstructed')

def plot_MNIST_sample(X):
  """
  Plots 9 images in the MNIST dataset.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    Nothing.

  """
  plt.figure()
  fig, ax = plt.subplots()
  k=0
  for k1 in range(3):
    for k2 in range(3):
      k = k+1
      plt.imshow(np.reshape(X[k,:],(28,28)),extent=[(k1+1)*28,k1*28,(k2+1)*28,k2*28],vmin=0,vmax=255)
  plt.xlim((3*28,0))
  plt.ylim((3*28,0))
  plt.tick_params(axis='both',which='both',bottom=False,top=False,labelbottom=False)
  plt.clim([0,250])
  ax.set_xticks([])
  ax.set_yticks([])

def plot_MNIST_weights(weights):
  """
  Visualize PCA basis vector weights for MNIST. Red = positive weights, blue =
  negative weights, white = zero weight.

  Args:
     weights (numpy array of floats) : PCA basis vector

  Returns:
     Nothing.

  """
  plt.figure()
  fig, ax = plt.subplots()
  cmap = plt.cm.get_cmap('seismic')
  plt.imshow(np.real(np.reshape(weights,(28,28))),cmap=cmap)
  plt.tick_params(axis='both',which='both',bottom=False,top=False,labelbottom=False)
  plt.clim(-.15,.15)
  plt.colorbar(ticks=[-.15,-.1,-.05,0,.05,.1,.15])
  ax.set_xticks([])
  ax.set_yticks([])

def add_noise(X,frac_noisy_pixels):
  """
  Randomly corrupts a fraction of the pixels by setting them to random values.

  Args:
     X (numpy array of floats) : Data matrix
     frac_noisy_pixels (scalar) : Fraction of noisy pixels

  Returns:
     (numpy array of floats) : Data matrix + noise

  """
  X_noisy = np.reshape(X,(X.shape[0]*X.shape[1]))
  N_noise_ixs = int(X_noisy.shape[0] * frac_noisy_pixels)
  noise_ixs = np.random.choice(X_noisy.shape[0],size= N_noise_ixs,replace=False)
  X_noisy[noise_ixs] = np.random.uniform(0,255,noise_ixs.shape)
  X_noisy = np.reshape(X_noisy,(X.shape[0],X.shape[1]))
  return X_noisy

  # Old helper functions from Tutorial 1-2

def change_of_basis(X,W):
  """
  Projects data onto a new basis.

  Args:
    X (numpy array of floats) : Data matrix
                                each column corresponding to a different random variable
    W (numpy array of floats):  new orthonormal basis
                                columns correspond to basis vectors

  Returns:
    (numpy array of floats) : Data matrix expressed in new basis
  """
  Y = np.matmul(X,W)
  return Y

def get_sample_cov_matrix(X):
  """
    Returns the sample covariance matrix of data X

    Args:
      X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

    Returns:
      (numpy array of floats) : Covariance matrix
  """
  X = X - np.mean(X,0)
  cov_matrix = 1./X.shape[0]*np.matmul(X.T,X)
  return cov_matrix

def sort_evals_descending(evals,evectors):
  """
  Sorts eigenvalues and eigenvectors in decreasing order. Also aligns first two
  eigenvectors to be in first two quadrants (if 2D).

  Args:
    evals (numpy array of floats):   Vector of eigenvalues
    evectors (numpy array of floats):   Corresponding matrix of eigenvectors
                                each column corresponds to a different eigenvalue

  Returns:
    (numpy array of floats) : Vector of eigenvalues after sorting
    (numpy array of floats) : Matrix of eigenvectors after sorting
  """
  index = np.flip(np.argsort(evals))
  evals = evals[index]
  evectors = evectors[:,index]
  if evals.shape[0] == 2:
    if np.arccos(np.matmul(evectors[:,0], 1./np.sqrt(2)*np.array([1,1]))) > np.pi/2.:
      evectors[:,0] = -evectors[:,0]
    if np.arccos(np.matmul(evectors[:,1], 1./np.sqrt(2)*np.array([-1,1]))) > np.pi/2.:
      evectors[:,1] = -evectors[:,1]
  return evals, evectors

def pca(X):
  """
  Performs PCA on multivariate data. Eigenvalues are sorted in decreasing order.

  Args:
     X (numpy array of floats):   Data matrix
                                  each column corresponds to a different random variable

  Returns:
    (numpy array of floats) : Data projected onto the new basis
    (numpy array of floats) : Vector of eigenvalues
    (numpy array of floats) : Corresponding matrix of eigenvectors

  """
  X = X - np.mean(X,0)
  cov_matrix = get_sample_cov_matrix(X)
  evals, evectors = np.linalg.eig(cov_matrix)
  evals, evectors = sort_evals_descending(evals,evectors)
  score = change_of_basis(X,evectors)
  return score, evectors, evals

def plot_eigenvalues(evals):
  """
  Plots eigenvalues.

  Args:
     (numpy array of floats) : Vector of eigenvalues

  Returns:
    Nothing.

  """
  plt.figure()
  plt.plot(np.arange(1,len(evals)+1),evals,'o-k')
  plt.xlabel('Component')
  plt.ylabel('Eigenvalue')
  plt.title('Scree plot')
```

# Perform PCA on MNIST dataset.
The MNIST dataset consists of a 70,000 images of individual handwritten digits. Each image is a 28x28 pixel grayscale image. For convenience, each 28x28 pixel image is often unravelled into a single 784 (=28*28) element vector, so that the whole dataset is represented as a 70,000 x 784 matrix. Each row represents a different image, and each column represents a different pixel.

Enter the following cell to load the MNIST dataset and plot the first nine images.

```python
from sklearn.datasets import fetch_openml
mnist = fetch_openml(name = 'mnist_784')
X = mnist.data
plot_MNIST_sample(X)
```

The MNIST dataset has an extrinsic dimensionality of 784, much higher than the 2-dimensional examples used in the previous tutorials! To make sense of this data, we'll use dimensionality reduction. But first, we need to determine the intrinsic dimensionality $K$ of the data. One way to do this is to look for an "elbow" in the scree plot, to determine which eigenvalues are signficant.

#### Exercise
In this exercise you will examine the scree plot in the MNIST dataset.

**Suggestions**
* Perform PCA on the dataset and examine the scree plot.
* When do the eigenvalues appear (by eye) to reach zero? (Hint: use `plt.xlim` to zoom into a section of the plot).

```python
help(pca)
help(plot_eigenvalues)
```

    Help on function pca in module __main__:

    pca(X)
        Performs PCA on multivariate data. Eigenvalues are sorted in decreasing order.

        Args:
           X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable

        Returns:
          (numpy array of floats) : Data projected onto the new basis
          (numpy array of floats) : Vector of eigenvalues
          (numpy array of floats) : Corresponding matrix of eigenvectors

    Help on function plot_eigenvalues in module __main__:

    plot_eigenvalues(evals)
        Plots eigenvalues.

        Args:
           (numpy array of floats) : Vector of eigenvalues

        Returns:
          Nothing.

```python
###################################################################
## Insert your code here to:
##                perform PCA
##                plot the eigenvalues
###################################################################

# score, evectors, evals = ...YOUR CODE HERE to perform PCA
# plot_eigenvalues(evals)
# YOUR CODE HERE to limit the x-axis for zooming
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5-DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_f0861370.py)

*Example output:*

# Calculate the variance explained.
The scree plot suggests that most of the eigenvalues are near zero, with fewer than 100 having large values. Another common way to determine the intrinsic dimensionality is by considering the   variance explained. This can be examined with a cumulative plot of the fraction of the total variance explained by the top $K$ components, i.e.:
\begin{equation}
\text{var explained} = \frac{\sum_{i=1}^K \lambda_i}{\sum_{i=1}^N \lambda_i}
\end{equation}
The intrinsic dimensionality is often quantified by the $K$ necessary to explain a large proportion of the total variance of the data (often a defined threshold, e.g., 90%).

#### Exercise
In this exercise you will plot the explained variance.

**Suggestions**
* Fill in the function below to calculate the fraction variance explained as a function of the number of principal componenets. **Hint:** use `np.cumsum`.
* Plot the variance explained using `plot_variance_explained`.
* How many principal components are required to explain 90% of the variance?
* How does the intrinsic dimensionality of this dataset compare to its extrinsic dimensionality?

```python
help(plot_variance_explained)
```

    Help on function plot_variance_explained in module __main__:

    plot_variance_explained(variance_explained)
        Plots eigenvalues.

        Args:
           variance_explained (numpy array of floats) : Vector of variance explained for each PC

        Returns:
          Nothing.

```python
def get_variance_explained(evals):
  """
  Calculates variance explained from the eigenvalues.

  Args:
     evals (numpy array of floats) : Vector of eigenvalues

  Returns:
    (numpy array of floats) : Vector of variance explained

  """
  ###################################################################
  ## Insert your code here to:
  ##                cumulatively sum the eigenvalues
  ##                normalize by the sum of eigenvalues

  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: calculate explaine variance!")
  ###################################################################
  return variance_explained

###################################################################
## Insert your code here to:
##                calculate and plot the variance explained
###################################################################
# variance_explained = ...
# plot_variance_explained(variance_explained)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5-DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_7af6bcb7.py)

*Example output:*

# Reconstruct data with different numbers of PCs.

```python
#@title Video: Geometric view of data
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="A_a7_hMhjfc", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video

```

    Video available at https://youtube.com/watch?v=A_a7_hMhjfc

Now we have seen that the top 100 or so principal components of the data can explain most of the variance. We can use this fact to perform *dimensionality reduction*, i.e., by storing the data using only 100 components rather than the samples of all 784 pixels. Remarkably, we will be able to reconstruct much of the structure of the data using only the top 100 components. To see this, recall that to perform PCA we projected the data $\bf X$ onto the eigenvectors of the covariance matrix:
\begin{equation}
\bf S = X W
\end{equation}
Since $\bf W$ is an orthogonal matrix, ${\bf W}^{-1} = {\bf W}^T$. So by multiplying by ${\bf W}^T$ on each side we can rewrite this equation as
\begin{equation}
{\bf X = S W}^T.
\end{equation}
This now gives us a way to reconstruct the data matrix from the scores and loadings. To reconstruct the data from a low-dimensional approximation, we just have to truncate these matrices.  Let's call ${\bf S}_{1:K}$ and ${\bf W}_{1:K}$ as keeping only the first $K$ columns of this matrix. Then our reconstruction is:
\begin{equation}
{\bf \hat X = S}_{1:K} ({\bf W}_{1:K})^T.
\end{equation}

#### Exercise
Fill in the function below to reconstruct the data using different numbers of principal components.

**Suggestions**
* Fill in the following function to reconstruct the data based on the weights and scores. Don't forget to add the mean!
* Make sure your function works by reconstructing the data with all $K=784$ components. They two images should look identical.

```python
help(plot_MNIST_reconstruction)
```

    Help on function plot_MNIST_reconstruction in module __main__:

    plot_MNIST_reconstruction(X, X_reconstructed)
        Plots 9 images in the MNIST dataset side-by-side with the reconstructed images.

        Args:
           X (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable
           X_reconstructed (numpy array of floats):   Data matrix
                                        each column corresponds to a different random variable

        Returns:
          Nothing.

```python
def reconstruct_data(score,evectors,X_mean,K):
  """
  Reconstruct the data based on the top K components.

  Args:
     score (numpy array of floats) : Score matrix
     evectors (numpy array of floats) : Matrix of eigenvectors
     X_mean (numpy array of floats) : Vector corresponding to data mean
     K (scalar) : Number of components to include

  Returns:
     (numpy array of floats) : Matrix of reconstructed data

  """
  ###################################################################
  ## Insert your code here to:
  ##                Reconstruct the data from the score and eigenvectors
  ##                Don't forget to add the mean!!

  #X_reconstructed =  Your code here
  #uncomment once you've filled in the function
  raise NotImplementedError("Student excercise: finish reconstructing data function!")
  ###################################################################
  return X_reconstructed

K = 784

## Uncomment below to to:
##                Reconstruct the data based on all components
##                Plot the data and reconstruction
# X_mean = ...
# X_reconstructed = ...
# plot_MNIST_reconstruction(X ,X_reconstructed)

```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5-DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_0edb6db6.py)

*Example output:*

#### Exercise:
Now run the code below and experiment with the slider to reconstruct the data matrix using different numbers of principal components.

**Questions:**
* How many principal components are necessary to reconstruct the numbers (by eye)? How does this relate to the intrinsic dimensionality of the data?
* Do you see any information in the data with only a single principal component?

```python
###### MAKE SURE TO RUN THIS CELL VIA THE PLAY BUTTON TO ENABLE SLIDERS ########

import ipywidgets as widgets

def refresh(K = 100):
  X_reconstructed = reconstruct_data(score,evectors,X_mean,K)
  plot_MNIST_reconstruction(X ,X_reconstructed)
  plt.title('Reconstructed, K={}'.format(K))

_ = widgets.interact(refresh,
  K = (1, 784, 10))
```

#### Exercise:
Next, let's take a closer look at the first principal component by visualizing its corresponding weights.

**Questions**
* Enter `plot_MNIST_weights` to visualize the weights of the first basis vector.
* What structure do you see? Which pixels have a strong positive weighting? Which have a strong negative weighting? What kinds of images would this basis vector differentiate?
* Try visualizing the second and third basis vectors. Do you see any structure? What about the 100th basis vector? 500th? 700th?

```python
help(plot_MNIST_weights)
```

    Help on function plot_MNIST_weights in module __main__:

    plot_MNIST_weights(weights)
        Visualize PCA basis vector weights for MNIST. Red = positive weights, blue =
        negative weights, white = zero weight.

        Args:
           weights (numpy array of floats) : PCA basis vector

        Returns:
           Nothing.

```python
###################################################################
## Insert your code here to:
##                Plot the weights of the first principal component

#plot_MNIST_weights(Your code here)
###################################################################
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D5-DimensionalityReduction/solutions/W1D5_Tutorial3_Solution_d07d8802.py)

*Example output:*

# (Optional Exploration): Examine denoising using PCA.

Finally, we will test how PCA can be used to denoise data. We will add salt-and-pepper noise to the original data and see how that affects the eigenvalues. To do this, we'll use the function `add_noise`, starting with adding noise to 20% of the pixels.
The we'll Perform PCA and plot the variance explained. How many principal components are required to explain 90% of the variance? How does this compare to the original data?

```python
###################################################################
## Here we:
##                Add noise to the data
##                Plot noise-corrupted data
##                Perform PCA on the noisy data
##                Calculate and plot the variance explained
###################################################################
X_noisy = add_noise(X,.2)
score_noisy, evectors_noisy, evals_noisy = pca(X_noisy)
variance_explained_noisy = get_variance_explained(evals_noisy)

with plt.xkcd():
  plot_MNIST_sample(X_noisy)
  plot_variance_explained(variance_explained_noisy)
```

To denoise the data, we can simply project it onto the basis found with the original dataset (`evectors`, not `evectors_noisy`). Then, by taking the top K components of this projection, we have a guess for where the sample should lie in the K-dimensional latent space. We can then reconstruct the data as normal, using the top 50 components. You should play around with the amount of noise and K to build intuition.

```python
###################################################################
## Here we:
##                Subtract the mean of the noise-corrupted data
##                Project onto the original basis vectors evectors
##                Reconstruct the data using the top 50 components
##                Plot the result
###################################################################

X_noisy_mean = np.mean(X_noisy,0)
projX_noisy = np.matmul(X_noisy-X_noisy_mean,evectors)
X_reconstructed = reconstruct_data(projX_noisy,evectors,X_noisy_mean,50)

with plt.xkcd():
  plot_MNIST_reconstruction(X_noisy,X_reconstructed)
```


# Assignment 4: Word Embeddings

Welcome to the fourth (and last) programming assignment of Course 2!

In this assignment, you will practice how to compute word embeddings and use them for sentiment analysis.
- To implement sentiment analysis, you can go beyond counting the number of positive words and negative words.
- You can find a way to represent each word numerically, by a vector.
- The vector could then represent syntactic (i.e. parts of speech) and semantic (i.e. meaning) structures.

In this assignment, you will explore a classic way of generating word embeddings or representations.
- You will implement a famous model called the continuous bag of words (CBOW) model.

By completing this assignment you will:

- Train word vectors from scratch.
- Learn how to create batches of data.
- Understand how backpropagation works.
- Plot and visualize your learned word vectors.

Knowing how to train these models will give you a better understanding of word vectors, which are building blocks to many applications in natural language processing.

## Outline

- [1 The Continuous bag of words model](#1)
- [2 Training the Model](#2)
    - [2.0 Initialize the model](#2)
        - [Exercise 01](#ex-01)
    - [2.1 Softmax Function](#2.1)
        - [Exercise 02](#ex-02)
    - [2.2 Forward Propagation](#2.2)
        - [Exercise 03](#ex-03)
    - [2.3 Cost Function](#2.3)
    - [2.4 Backproagation](#2.4)
        - [Exercise 04](#ex-04)
    - [2.5 Gradient Descent](#2.5)
        - [Exercise 05](#ex-05)
- [3 Visualizing the word vectors](#3)

# 1. The Continuous bag of words model

Let's take a look at the following sentence:
>**'I am happy because I am learning'**.

- In continuous bag of words (CBOW) modeling, we try to predict the center word given a few context words (the words around the center word).
- For example, if you were to choose a context half-size of say $C = 2$, then you would try to predict the word **happy** given the context that includes 2 words before and 2 words after the center word:

> $C$ words before: [I, am]

> $C$ words after: [because, I]

- In other words:

$$context = [I,am, because, I]$$
$$target = happy$$

The structure of your model will look like this:

 Figure 1

Where $\bar x$ is the average of all the one hot vectors of the context words.

 Figure 2

Once you have encoded all the context words, you can use $\bar x$ as the input to your model.

The architecture you will be implementing is as follows:

\begin{align}
 h &= W_1 \  X + b_1  \tag{1} \\
 a &= ReLU(h)  \tag{2} \\
 z &= W_2 \  a + b_2   \tag{3} \\
 \hat y &= softmax(z)   \tag{4} \\
\end{align}

```python
# Import Python libraries and helper functions (in utils2)
import nltk
from nltk.tokenize import word_tokenize
import numpy as np
from collections import Counter
from utils2 import sigmoid, get_batches, compute_pca, get_dict
```

```python
# Download sentence tokenizer
nltk.data.path.append('.')
```

```python
# Load, tokenize and process the data
import re                                                           #  Load the Regex-modul
with open('shakespeare.txt') as f:
    data = f.read()                                                 #  Read in the data
data = re.sub(r'[,!?;-]', '.',data)                                 #  Punktuations are replaced by .
data = nltk.word_tokenize(data)                                     #  Tokenize string to words
data = [ ch.lower() for ch in data if ch.isalpha() or ch == '.']    #  Lower case and drop non-alphabetical tokens
print("Number of tokens:", len(data),'\n', data[:15])               #  print data sample
```

```python
# Compute the frequency distribution of the words in the dataset (vocabulary)
fdist = nltk.FreqDist(word for word in data)
print("Size of vocabulary: ",len(fdist) )
print("Most frequent tokens: ",fdist.most_common(20) ) # print the 20 most frequent words and their freq.
```

#### Mapping words to indices and indices to words
We provide a helper function to create a dictionary that maps words to indices and indices to words.

```python
# get_dict creates two dictionaries, converting words to indices and viceversa.
word2Ind, Ind2word = get_dict(data)
V = len(word2Ind)
print("Size of vocabulary: ", V)
```

```python
# example of word to index mapping
print("Index of the word 'king' :  ",word2Ind['king'] )
print("Word which has index 2743:  ",Ind2word[2743] )
```

# 2 Training the Model

###  Initializing the model

You will now initialize two matrices and two vectors.
- The first matrix ($W_1$) is of dimension $N \times V$, where $V$ is the number of words in your vocabulary and $N$ is the dimension of your word vector.
- The second matrix ($W_2$) is of dimension $V \times N$.
- Vector $b_1$ has dimensions $N\times 1$
- Vector $b_2$ has dimensions  $V\times 1$.
- $b_1$ and $b_2$ are the bias vectors of the linear layers from matrices $W_1$ and $W_2$.

The overall structure of the model will look as in Figure 1, but at this stage we are just initializing the parameters.

### Exercise 01
Please use [numpy.random.rand](https://numpy.org/doc/stable/reference/random/generated/numpy.random.rand.html) to generate matrices that are initialized with random values from a uniform distribution, ranging between 0 and 1.

**Note:** In the next cell you will encounter a random seed. Please **DO NOT** modify this seed so your solution can be tested correctly.

```python
# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: initialize_model
def initialize_model(N,V, random_seed=1):
    '''
    Inputs:
        N:  dimension of hidden vector
        V:  dimension of vocabulary
        random_seed: random seed for consistent results in the unit tests
     Outputs:
        W1, W2, b1, b2: initialized weights and biases
    '''

    np.random.seed(random_seed)

    ### START CODE HERE (Replace instances of 'None' with your code) ###
    # W1 has shape (N,V)
    W1 = None
    # W2 has shape (V,N)
    W2 = None
    # b1 has shape (N,1)
    b1 = None
    # b2 has shape (V,1)
    b2 = None
    ### END CODE HERE ###

    return W1, W2, b1, b2
```

```python
# Test your function example.
tmp_N = 4
tmp_V = 10
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)
assert tmp_W1.shape == ((tmp_N,tmp_V))
assert tmp_W2.shape == ((tmp_V,tmp_N))
print(f"tmp_W1.shape: {tmp_W1.shape}")
print(f"tmp_W2.shape: {tmp_W2.shape}")
print(f"tmp_b1.shape: {tmp_b1.shape}")
print(f"tmp_b2.shape: {tmp_b2.shape}")
```

##### Expected Output

```CPP
tmp_W1.shape: (4, 10)
tmp_W2.shape: (10, 4)
tmp_b1.shape: (4, 1)
tmp_b2.shape: (10, 1)
```

### 2.1 Softmax
Before we can start training the model, we need to implement the softmax function as defined in equation 5:

$$ \text{softmax}(z_i) = \frac{e^{z_i} }{\sum_{i=0}^{V-1} e^{z_i} }  \tag{5} $$

- Array indexing in code starts at 0.
- $V$ is the number of words in the vocabulary (which is also the number of rows of $z$).
- $i$ goes from 0 to |V| - 1.

### Exercise 02
**Instructions**: Implement the softmax function below.

- Assume that the input $z$ to `softmax` is a 2D array
- Each training example is represented by a column of shape (V, 1) in this 2D array.
- There may be more than one column, in the 2D array, because you can put in a batch of examples to increase efficiency.  Let's call the batch size lowercase $m$, so the $z$ array has shape (V, m)
- When taking the sum from $i=1 \cdots V-1$, take the sum for each column (each example) separately.

Please use
- numpy.exp
- numpy.sum (set the axis so that you take the sum of each column in z)

```python
# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: softmax
def softmax(z):
    '''
    Inputs:
        z: output scores from the hidden layer
    Outputs:
        yhat: prediction (estimate of y)
    '''

    ### START CODE HERE (Replace instances of 'None' with your own code) ###

    # Calculate yhat (softmax)
    yhat = None

    ### END CODE HERE ###

    return yhat
```

```python
# Test the function
tmp = np.array([[1,2,3],
                [1,1,1]
               ])
tmp_sm = softmax(tmp)
display(tmp_sm)
```

##### Expected Ouput

```CPP
array([[0.5       , 0.73105858, 0.88079708],
       [0.5       , 0.26894142, 0.11920292]])
```

### 2.2 Forward propagation

### Exercise 03
Implement the forward propagation $z$ according to equations (1) to (3).

\begin{align}
 h &= W_1 \  X + b_1  \tag{1} \\
 a &= ReLU(h)  \tag{2} \\
 z &= W_2 \  a + b_2   \tag{3} \\
\end{align}

For that, you will use as activation the Rectified Linear Unit (ReLU) given by:

$$f(h)=\max (0,h) \tag{6}$$

    Hints

You can use numpy.maximum(x1,x2) to get the maximum of two values

Use numpy.dot(A,B) to matrix multiply A and B

```python
# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: forward_prop
def forward_prop(x, W1, W2, b1, b2):
    '''
    Inputs:
        x:  average one hot vector for the context
        W1, W2, b1, b2:  matrices and biases to be learned
     Outputs:
        z:  output score vector
    '''

    ### START CODE HERE (Replace instances of 'None' with your own code) ###

    # Calculate h
    h = None

    # Apply the relu on h (store result in h)
    h = None

    # Calculate z
    z = None

    ### END CODE HERE ###

    return z, h
```

```python
# Test the function

# Create some inputs
tmp_N = 2
tmp_V = 3
tmp_x = np.array([[0,1,0]]).T
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(N=tmp_N,V=tmp_V, random_seed=1)

print(f"x has shape {tmp_x.shape}")
print(f"N is {tmp_N} and vocabulary size V is {tmp_V}")

# call function
tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print("call forward_prop")
print()
# Look at output
print(f"z has shape {tmp_z.shape}")
print("z has values:")
print(tmp_z)

print()

print(f"h has shape {tmp_h.shape}")
print("h has values:")
print(tmp_h)
```

##### Expected output
```CPP
x has shape (3, 1)
N is 2 and vocabulary size V is 3
call forward_prop

z has shape (3, 1)
z has values:
[[0.55379268]
 [1.58960774]
 [1.50722933]]

h has shape (2, 1)
h has values:
[[0.92477674]
 [1.02487333]]
```

## 2.3 Cost function

- We have implemented the *cross-entropy* cost function for you.

```python
# compute_cost: cross-entropy cost functioN
def compute_cost(y, yhat, batch_size):
    # cost function
    logprobs = np.multiply(np.log(yhat),y) + np.multiply(np.log(1 - yhat), 1 - y)
    cost = - 1/batch_size * np.sum(logprobs)
    cost = np.squeeze(cost)
    return cost
```

```python
# Test the function
tmp_C = 2
tmp_N = 50
tmp_batch_size = 4
tmp_word2Ind, tmp_Ind2word = get_dict(data)
tmp_V = len(word2Ind)

tmp_x, tmp_y = next(get_batches(data, tmp_word2Ind, tmp_V,tmp_C, tmp_batch_size))

print(f"tmp_x.shape {tmp_x.shape}")
print(f"tmp_y.shape {tmp_y.shape}")

tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)

print(f"tmp_W1.shape {tmp_W1.shape}")
print(f"tmp_W2.shape {tmp_W2.shape}")
print(f"tmp_b1.shape {tmp_b1.shape}")
print(f"tmp_b2.shape {tmp_b2.shape}")

tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print(f"tmp_z.shape: {tmp_z.shape}")
print(f"tmp_h.shape: {tmp_h.shape}")

tmp_yhat = softmax(tmp_z)
print(f"tmp_yhat.shape: {tmp_yhat.shape}")

tmp_cost = compute_cost(tmp_y, tmp_yhat, tmp_batch_size)
print("call compute_cost")
print(f"tmp_cost {tmp_cost:.4f}")
```

##### Expected output

```CPP
tmp_x.shape (5778, 4)
tmp_y.shape (5778, 4)
tmp_W1.shape (50, 5778)
tmp_W2.shape (5778, 50)
tmp_b1.shape (50, 1)
tmp_b2.shape (5778, 1)
tmp_z.shape: (5778, 4)
tmp_h.shape: (50, 4)
tmp_yhat.shape: (5778, 4)
call compute_cost
tmp_cost 9.9560
```

## 2.4 Training the Model - Backpropagation

### Exercise 04
Now that you have understood how the CBOW model works, you will train it.

You created a function for the forward propagation. Now you will implement a function that computes the gradients to backpropagate the errors.

```python
# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: back_prop
def back_prop(x, yhat, y, h, W1, W2, b1, b2, batch_size):
    '''
    Inputs:
        x:  average one hot vector for the context
        yhat: prediction (estimate of y)
        y:  target vector
        h:  hidden vector (see eq. 1)
        W1, W2, b1, b2:  matrices and biases
        batch_size: batch size
     Outputs:
        grad_W1, grad_W2, grad_b1, grad_b2:  gradients of matrices and biases
    '''
    ### START CODE HERE (Replace instanes of 'None' with your code) ###

    # Compute l1 as W2^T (Yhat - Y)
    # Re-use it whenever you see W2^T (Yhat - Y) used to compute a gradient
    l1 = None
    # Apply relu to l1
    l1 = None
    # Compute the gradient of W1
    grad_W1 = None
    # Compute the gradient of W2
    grad_W2 = None
    # Compute the gradient of b1
    grad_b1 = None
    # Compute the gradient of b2
    grad_b2 = None
    ### END CODE HERE ###

    return grad_W1, grad_W2, grad_b1, grad_b2
```

```python
# Test the function
tmp_C = 2
tmp_N = 50
tmp_batch_size = 4
tmp_word2Ind, tmp_Ind2word = get_dict(data)
tmp_V = len(word2Ind)

# get a batch of data
tmp_x, tmp_y = next(get_batches(data, tmp_word2Ind, tmp_V,tmp_C, tmp_batch_size))

print("get a batch of data")
print(f"tmp_x.shape {tmp_x.shape}")
print(f"tmp_y.shape {tmp_y.shape}")

print()
print("Initialize weights and biases")
tmp_W1, tmp_W2, tmp_b1, tmp_b2 = initialize_model(tmp_N,tmp_V)

print(f"tmp_W1.shape {tmp_W1.shape}")
print(f"tmp_W2.shape {tmp_W2.shape}")
print(f"tmp_b1.shape {tmp_b1.shape}")
print(f"tmp_b2.shape {tmp_b2.shape}")

print()
print("Forwad prop to get z and h")
tmp_z, tmp_h = forward_prop(tmp_x, tmp_W1, tmp_W2, tmp_b1, tmp_b2)
print(f"tmp_z.shape: {tmp_z.shape}")
print(f"tmp_h.shape: {tmp_h.shape}")

print()
print("Get yhat by calling softmax")
tmp_yhat = softmax(tmp_z)
print(f"tmp_yhat.shape: {tmp_yhat.shape}")

tmp_m = (2*tmp_C)
tmp_grad_W1, tmp_grad_W2, tmp_grad_b1, tmp_grad_b2 = back_prop(tmp_x, tmp_yhat, tmp_y, tmp_h, tmp_W1, tmp_W2, tmp_b1, tmp_b2, tmp_batch_size)

print()
print("call back_prop")
print(f"tmp_grad_W1.shape {tmp_grad_W1.shape}")
print(f"tmp_grad_W2.shape {tmp_grad_W2.shape}")
print(f"tmp_grad_b1.shape {tmp_grad_b1.shape}")
print(f"tmp_grad_b2.shape {tmp_grad_b2.shape}")
```

##### Expected output

```CPP
get a batch of data
tmp_x.shape (5778, 4)
tmp_y.shape (5778, 4)

Initialize weights and biases
tmp_W1.shape (50, 5778)
tmp_W2.shape (5778, 50)
tmp_b1.shape (50, 1)
tmp_b2.shape (5778, 1)

Forwad prop to get z and h
tmp_z.shape: (5778, 4)
tmp_h.shape: (50, 4)

Get yhat by calling softmax
tmp_yhat.shape: (5778, 4)

call back_prop
tmp_grad_W1.shape (50, 5778)
tmp_grad_W2.shape (5778, 50)
tmp_grad_b1.shape (50, 1)
tmp_grad_b2.shape (5778, 1)
```

## Gradient Descent

### Exercise 05
Now that you have implemented a function to compute the gradients, you will implement batch gradient descent over your training set.

**Hint:** For that, you will use `initialize_model` and the `back_prop` functions which you just created (and the `compute_cost` function). You can also use the provided `get_batches` helper function:

```for x, y in get_batches(data, word2Ind, V, C, batch_size):```

```...```

Also: print the cost after each batch is processed (use batch size = 128)

```python
# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: gradient_descent
def gradient_descent(data, word2Ind, N, V, num_iters, alpha=0.03):

    '''
    This is the gradient_descent function

      Inputs:
        data:      text
        word2Ind:  words to Indices
        N:         dimension of hidden vector
        V:         dimension of vocabulary
        num_iters: number of iterations
     Outputs:
        W1, W2, b1, b2:  updated matrices and biases

    '''
    W1, W2, b1, b2 = initialize_model(N,V, random_seed=282)
    batch_size = 128
    iters = 0
    C = 2
    for x, y in get_batches(data, word2Ind, V, C, batch_size):
        ### START CODE HERE (Replace instances of 'None' with your own code) ###
        # Get z and h
        z, h = None
        # Get yhat
        yhat = None
        # Get cost
        cost = None
        if ( (iters+1) % 10 == 0):
            print(f"iters: {iters + 1} cost: {cost:.6f}")
        # Get gradients
        grad_W1, grad_W2, grad_b1, grad_b2 = None

        # Update weights and biases
        W1 = None
        W2 = None
        b1 = None
        b2 = None

        ### END CODE HERE ###

        iters += 1
        if iters == num_iters:
            break
        if iters % 100 == 0:
            alpha *= 0.66

    return W1, W2, b1, b2
```

```python
# test your function
C = 2
N = 50
word2Ind, Ind2word = get_dict(data)
V = len(word2Ind)
num_iters = 150
print("Call gradient_descent")
W1, W2, b1, b2 = gradient_descent(data, word2Ind, N, V, num_iters)
```

##### Expected Output

```CPP
iters: 10 cost: 0.789141
iters: 20 cost: 0.105543
iters: 30 cost: 0.056008
iters: 40 cost: 0.038101
iters: 50 cost: 0.028868
iters: 60 cost: 0.023237
iters: 70 cost: 0.019444
iters: 80 cost: 0.016716
iters: 90 cost: 0.014660
iters: 100 cost: 0.013054
iters: 110 cost: 0.012133
iters: 120 cost: 0.011370
iters: 130 cost: 0.010698
iters: 140 cost: 0.010100
iters: 150 cost: 0.009566
```

Your numbers may differ a bit depending on which version of Python you're using.

## 3.0 Visualizing the word vectors

In this part you will visualize the word vectors trained using the function you just coded above.

```python
# visualizing the word vectors here
from matplotlib import pyplot
%config InlineBackend.figure_format = 'svg'
words = ['king', 'queen','lord','man', 'woman','dog','wolf',
         'rich','happy','sad']

embs = (W1.T + W2)/2.0

# given a list of words and the embeddings, it returns a matrix with all the embeddings
idx = [word2Ind[word] for word in words]
X = embs[idx, :]
print(X.shape, idx)  # X.shape:  Number of words of dimension N each
```

```python
result= compute_pca(X, 2)
pyplot.scatter(result[:, 0], result[:, 1])
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()
```

You can see that man and king are next to each other. However, we have to be careful with the interpretation of this projected word vectors, since the PCA depends on the projection -- as shown in the following illustration.

```python
result= compute_pca(X, 4)
pyplot.scatter(result[:, 3], result[:, 1])
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 3], result[i, 1]))
pyplot.show()
```

NLP/Learn_by_deeplearning.ai/Course 2 - Probabilistic Models/Original/C2_W4_Assignment.ipynb


Contenido bajo licencia Creative Commons BY 4.0 y código bajo licencia MIT. © Juan Gómez y Nicolás Guarín-Zapata 2020. Este material es parte del curso Modelación Computacional en el programa de Ingeniería Civil de la Universidad EAFIT.

# Interpolación en 2D

## Introducción

Acá extenderemos el esquema de interpolación unidimensional estudiado previamente al caso mas general de un dominio bidimensional. Desde el punto de vista geométrico también veremos como un **elemento finito** es solo un dominio espacial canónico descrito por puntos nodales y el correspondiente grupo de funciones de interpolación (o de **forma**).

**Al completar este notebook usted debería estar en la capacidad de:**

* Reconocer el problema de interpolación en dominios bidimensionales como uno de aplicación de los esquemas unidimensionales.

* Formalizar el concepto de un elemento finito como un espacio de interpolación canónico con funciones de interpolación predefinidas.

* Proponer esquemas de interpolación para dominios bidimensionales arbitrarios.

## Dominio bidimensional

Consideremos el dominio cuadrado mostrado en la figura y en el cual queremos aproximar, por medio de interpolación, una función escalar (o vectorial) $f=f(x,y)$. Para ese propósito los puntos negros en la figura representan puntos nodales donde asumimos que la función es conocida. En este caso el polinomio de interpolación, denotado por $p(x,y)$ se construye como:

$$p(x,y) = \sum_{Q=1}^N H_Q(x,y)f_Q$$

donde $Q = 1,...,N$ para un dominio de *N* puntos nodales y donde $H_Q(x,y)$ son las funciones de interpolación o funciones de forma.

Como se detallará a continuación para construir las funciones de interpolación bidimensionales $ H_Q(x,y)$  en realidad aplicamos un proceso de interpolaciones unidimensionales iteradas.

Denotemos como $x_A$ y $x_B$ a las coordenadas de los puntos A y B en el dominio cuadrilátero mostrado en la figura y supongamos que queremos encontrar el valor de la función en el punto A.

El punto A tiene una coordenada en $y$ que es arbitraria pero una coordenada en $x$ constante correspondiente a $x = x_A$ de manera que para un punto A arbitrario a lo largo de la dirección 1-4 (ver figura) el esquema de interpolación es aún unidimensional solamente con dependencia en $y$ y expresado como $f(y , x= A)$ en la figura. Usando polinomios de interpolación de Lagrange unidimensionales la dependencia en $y$ puede ser capturada por:

$$f(x_A , y) = L_1(y) f_1 + L_4(y) f_4$$

Procediendo de manera similar para un punto arbitrario $B$ a lo largo de la dirección 2-3 se tiene que:

$$f(x_B, y) = L_2(y) f_2 + L_3(y) f_3\, .$$

con $f_A$ y $f_B$ conocidos la dependencia en $x$ puede capturarse como:

$$f(x, y) = L_A(x) f(x_A, y) + L_B(x)f(x_B, y)\, .$$

Para llegar a la forma final de las funciones de forma bidimensionales calculamos los polinomios $L_2(y)$, $L_3(y)$, $ L_A(x)$ y $L_B(x)$ y los reemplazamos en las expresiones anteriores. En el caso de un elemento de lado $2.0$ las funciones son:

\begin{align*}
H_1(x,y) & = L_1(x)L_1(y) \equiv \frac{1}{4}(1-x)(1-y)\, ,\\
H_2(x,y) & = L_2(x)L_1(y) \equiv \frac{1}{4}(1+x)(1-y)\, ,\\
H_3(x,y) & = L_2(x)L_2(y) \equiv \frac{1}{4}(1+x)(1+y)\, ,\\
H_4(x,y) & = L_1(x)L_2(y) \equiv \frac{1}{4}(1-x)(1+y)\, .
\end{align*}

### Elemento finito canónico

En la siguiente subrutina codificamos la forma final $H_Q(x,y)$ de las funciones de forma en vez de calcular directamente los polinomios fundamentales en una dimensión de la forma $L_I(y)$ para posteriormente realizar la interpolación iterada. La subrutina llamada ``sha4`` almacena las funciones en una estructura matricial que depende de $x$ e $y$. Asumimos que el elemento es un cuadradado perfecto de lado $\mathcal{l}=2.0$ con puntos nodales en las esquinas correspondiente a una interpolación lineal a lo largo de cada cara.

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
```

```python
%matplotlib notebook
sym.init_printing()
```

```python
def sha4(x, y):
    """
    Compute the shape functions for bi-linear
    square element of size 2.0.
    """
    sh = sym.Matrix([[
            (1 - x)*(1 - y),
            (1 + x)*(1 - y),
            (1 + x)*(1 + y),
            (1 - x)*(1 + y)]])/4
    return sh
```

Este elemento cuadrado es un elemento **canónico** o de referencia en el cual es fácil la realización de las operaciones de interpolación. En una malla real de elementos finitos es de esperar que los elementos estén distorsionados en relación con este elemento canónico. En esos casos la interpolación también se realiza en el espacio del elemento canónico pero ahora tanto la geometría como las funciones son transformadas usando operaciones matemáticas. Estos detalles sin embargo no se discutirán acá.

Las funciones de forma almacenadas en la subrutina corresponden a:

$$H = \frac{1}{4}\begin{bmatrix}(1-x)(1-y)&(1+x)(1-y)&(1+x)(1+y)&(1-x)(1+y)\end{bmatrix}$$

**Preguntas**

- Escriba las funciones de forma asumiendo que el sub-dominio el mismo cuadrado discutido hasta el momento, pero además de los nodos de la esquina también incluye nodos en la mitad de las caras para completar un total de 8 puntos nodales.

- Haga una copia de la subrutina `sha4` y modifíquela para que calcule las funciones de forma para el elemento de 8 nodos.

```python
x, y= sym.symbols('x y')
H = sha4(x, y)
display(H)
```

## Interpolación en un dominio cuadrado

En este paso consideramos un elemento cuadrado conformado por 4 puntos nodales localizados en las esquinas y donde se asumen conocidos los valores de la función. Usaremos estos valores, conjuntamente con las funciones de forma para encontrar un polinomio de interpolación. El polinomio resultante se usa posteriormente para generar valores aproximados de la función en una serie de puntos que conforman una grilla usada para visualizar la solución. La grilla de puntos de observación se genera usando la función `mgrid` de `numpy`.

Note que el sistema de referencia se localiza en el centro del elemento por lo tanto $x \in [-1, 1]$ and $y \in [-1, 1]$. El arreglo unidimensional `u_interp` almacenará los valores interpolados en cada punto de la grilla.

Para realizar la interpolación asumiremos valores nodales de la función en un punto dado $(x , y)$  de manera que podemos obtener el valor interpolado como:

$$u(x,y)\;=\;\left[H(x,y)\right]\left\{u\right\}$$

```python
# Agregue comentarios para aclarar los pasos mas relevantes del siguiente código
li = -1.0
ls = 1.1
dl = 0.05
npts = int((ls - li)/dl)
u_interp = np.zeros((npts, npts))
xx, yy = np.mgrid[li:ls:npts*1j, li:ls:npts*1j]

# Intente diferentes valores nodales
u = sym.Matrix(4, 1, [-0.2, 0.2, -0.2, 0.2])
for i in range(npts):
    for j in range(npts):
        NS = H.subs([(x, xx[i,j]), (y, yy[i,j])])
        up = NS*u
        u_interp[i, j] = up[0]
plt.figure()
plt.contourf(xx, yy, u_interp, cmap="RdYlBu")
plt.axis("image")
```

### Glosario de términos

**Elemento finito canonico:** Sub-dominio no distorsionado de tamaño constante y con funciones de forma únicas. En un caso práctico los elementos difieren en tamaño y nivel de distorión, sin embargo todos ellos son transformados al elemento canónico.

**Funciones de forma:** Funciones de interpolación formuladas sobre un elemento canónico.

**Malla:** Conjunto de elementos finitos que cubren un dominio computacional dado. Se dice que una malla ha sido refinada cuando el tamaño característico de sus elementos se reduce produciendo un mayor número de elementos para cubrir el mismo dominio computacional.

## Actividades para la clase

### Problema 1

Extender el esquema de interpolación en 2D discutido previamente al caso de una función vectorial en el contexto de teoría de la elasticidad con las siguientes consideraciones:

* Asuma que el vector de desplazamientos con componentes horizontal y vertical $u$ y $v$ respectivamente es conocido en cada nodo del dominio cuadrado.

* Usando los valores nodales del vector de desplazamientos calcule las componentes horizontal y vertical a lo largo del elemento (al interior).

* Usando los valores nodales calculo el campo de deformaciones unitarias definido por:

$$\varepsilon_{xx}=\frac12\left(\frac{\partial u}{\partial x}\right)$$

$$\varepsilon_{yy}=\frac12\left(\frac{\partial v}{\partial y}\right)$$

$$\gamma_{yy}=\left(\frac{\partial u}{\partial y}+\frac{\partial v}{\partial x}\right)$$

* Almacene las derivadas de las funciones de forma en una matriz independiente $B$.

### Problema 2

En la actividad que se describe a continuación aplicaremos algunas de las capacidades intrínsecas de interpolación disponibles en Python para resolver un problema de interés en Ingeniería Civil. Utilizaremos algunas funciones del módulo `geopandas` para crear un `GeoDataFrame` a partir de información almacenada en un archivo en [formato shp](https://en.wikipedia.org/wiki/Shapefile) que contiene la geometría de los municipios del Valle de Aburrá.
También utilizaremos la información de las coordenadas, altitudes y aceleraciones de las estaciones de la red acelerográfica que se encuentran en formato csv (archivo separado por comas, del inglés _comma separated value_). Posteriormente, usaremos la información de las estaciones acelerográficas para visualizar las altitudes y aceleraciones máximas. Para esto, Python usa un objeto denominado `Triangulation` que permite realizar operaciones de interpolación y graficación de manera simple.

**Antes de iniciar la actividad**

Consultar el significado de los siguientes términos:

* Módulo `geopandas`:

* GeoDataFrame:

* Análisis geo-espacial:

* Archivo de formato shape:

* `Triangulation`:

Uno de los insumos fundamentales para el diseño de estructuras sismo-resistentes es la aceleración máxima del terreno que se puede esperar ante la ocurrencia de un sismo. Existe amplia evidencia teórica y experimental de que el tren de ondas incidentes desde la fuente, y por ende los movimientos resultantes en el terreno, se ven fuertemente afectados por la topografía superficial. A este fenómeno se le conoce como efectos topográficos.

En el caso del Valle de Aburrá, sobre el cual descansa la ciudad de Medellín, se conoce que este está conformado por una zona relativamente plana y uniforme en el centro del valle, pero que esta rodeado por un perfil topográfico con laderas de pendiente considerable y que pueden generar efectos topográficos. Para tratar de identificar si efectivamente tales efectos son importantes en la ciudad de Medellín se propone utilizar los registros acelerográficos del sismo de Armenia, 1999, capturados por los equipos de la Red Acelerográfica de Medellín (RAM) y tratar de identificar si existe alguna correlación entre las aceleraciones máximas de dichos registros y la topografía del valle. Para estudiar el problema se dispone de los siguientes archivos.

* Archivo separado por comas (csv) denominado `estaciones_siata.csv` el cual contiene información de latitud (columna 2), longitud (columna 3) y altitud (columna 6) para las diferentes estaciones de la RAM. Adicionalmente las columnas 8, 9 y 10 contienen las componentes Norte-Sur, Este-Oeste y vertical de la aceleración máxima del terreno (cm/s²) registrados durante el sismo de Armenia, Colombia, 1999.

* Archivo de extensión `shp` denominado "medellin_colombia_osm_admin.shp" el cual contiene un mapa de Medellín.

* Archivos separados por comas conteniendo los registros acelerográficos para las estaciones de la RAM.

Usando estos archivos se requiere:

* Leer el archivo `shp` que contiene el mapa de Medellín y visualizarlo (usando los módulos `geopandas` y `matplotlib`).

* Leer el archivo `estaciones_siata.csv` para extraer de allí la localización de las estaciones y posteriormente graficarlas sobre el mapa de Medellín.

* Utilizar la función `tricontourf()` para interpolar y visualizar la distribución de la altitud sobre el mapa de Medellín.

* Utilizar la función `tricontourf()` para interpolar y visualizar la distribución de cada una de las componentes de la aceleración sobre el mapa de Medellín.

* Usando las visualizaciones de altitud y de aceleraciones máximas correlacionar estas 2 variables para identificar si hay efectos topográficos.

**Nota: Reportar sus resultados completando este Notebook**

## Formato del notebook

La siguiente celda cambia el formato del Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```