# Module 1: Setting up the problem

## Introduction

Geophysical surveys consist of a similar basic framework. An energy source is delivered into the earth, which can be natural (for example, the Earth's magnetic field) or human-made (current in the ground, acoustic wave energy, etc.), and this stimulates a response according to the variation in physical properties in the subsurface. At the surface, receivers pick up the signal and record this as data.

The goal of inversion is to find a model of the physical property distribution in the earth that produced the data. This is a difficult process because (1) information about a physical property for each datum is encoded in a complex way, and (2) we have a finite amount of data and cannot represent the physical property distribution everywhere.

Inversion is a multistep process, often represented as a workflow.

The goal of this module is to cover the first section of the workflow, which discretizes the data and places the values of the functions onto a mesh. This will be done using the following steps:

(1) Start with an expression that relates a kernel function with the continuous distribution of a physical property.

(2) Discretize this expression, and introduce a simple example problem to illustrate the mathematics in detail.

(3) Define a mesh that organizes our information.

(4) Build up the matrix equation $d = Gm$.

(5) Generalize the form of the problem from the example.

(6) Implement the example problem in Python as a forward problem.

But first, here are some fundamental definitions:

The general mathematical description of the inverse problem can be written as follows:

\begin{equation} F_i[m]= d_i +n_i \quad \text{for} \quad i=1,..,N \; \text{where}\end{equation}

$F[m]$ is a forward modeling operator. $F[m]$ incorporates the survey design and the physics of the problem. In the linear case we represent this as the matrix $G$.

$\bf{m}$ is a generic symbol for a physical property distribution, i.e. the model.

$\bf{d}$ represents the observed data, (sometimes also represented as $\bf{d^{obs}}$).

$\bf{n}$ is a term that represents additive noise.

This is, of course, the most general formulation of the problem. In this module we will consider the simplest case, which is (a) one dimensional (this can be likened to a survey that varies as a function of depth only) and (b) linear, in which our forward modeling operator becomes a matrix **G** in the matrix equation:

\begin{equation}d = Gm\end{equation}

## Step 1: Physical property distribution and the kernel function.

Each datum ($d_i$) collected is a volumetric response, that is to say, every datum measures the response of the whole volume (within the range of the system); it records the superposition of effects caused by all the material in the ground, and is therefore naturally represented as an integral. A "kernel function" or "sensitivity function", $g(x,y,z)$, shows how a datum is affected by all the subsurface. It describes the physics of the problem. The model, $m(x,y,z)$, represents the distribution of a physical property in the volume. Since each datum measures the response of the kernel function with the physical property distribution in the volume, for a continuous medium we can express this relationship as the inner product of the kernel function and the model. In the one dimensional case the expression for the *ith* datum is written as:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \end{equation}

where again:
\begin{equation}
    d :=\text{measured data} \\
    g :=\text{kernel function} \\
    m :=\text{physical property}\\
    a,b := \text{the domain of consideration}\\
\end{equation}

## Step 2: Discretize the expression

While the integral expression describes the inner product in a continuum, it is not possible to maintain this representation on a computer, so the above equation must be discretized. Let us consider the case where we have $N$ data, and as we build up the matrix equation, let $i$ be the rows and $j$ be the columns of our matrix:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_i (x_j) m_j \Delta x\end{equation}

**A "Toy Problem"**

Consider a simple case where we have a one dimensional, linear problem that will generate two data points, $d_1$ and $d_2$, from five physical property values
 ($m_1, m_2, m_3, m_4, m_5$), and these two data points are generated using two kernel functions $g_1$ and $g_2$. Further, let us assume that our domain of interest lies on the interval [0,1]. The equation above is then expressed as the following two equations, one for each datum:

\begin{equation}d_1 = \int_0^1 g_1(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_1 (x_j) m_j \Delta x\end{equation}

\begin{equation}d_2 = \int_0^1 g_2(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_2 (x_j) m_j \Delta x\end{equation}

Given that our problem is small by design, it is instructive to visualize the summation notation as matrix-vector products. Doing so yields the following expressions:

(a) Data. We can collect our two data points into a column vector. Our data in vector notation is given by:

\begin{equation}
d = \left[
\begin{array}{c}
d_1\\
d_2
\end{array}
\right]
\end{equation}

(b) The x-spacing. The x-spacings, $\Delta x$, are represented by a diagonal matrix. Here we are in a one dimensional case, but in general our information will reside in a volume, so let this matrix be represented as $V$. In the most general situation, the x-spacings need not be equal, and there can be significant variation in the distances within a grid, depending on the amount of resolution one desires at a particular location. For the moment, let's ignore this complexity and assume equal spacing in our grid. Then let $V=diag(\Delta x)$, and given the dimensions of our problem, $V$ appears as follows:

\begin{equation}
V=\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\end{equation}

(c) The model. The
model, $m$, is a column vector with five rows:
\begin{equation}
m=
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

(d) The kernel functions. Recall that a kernel function shows how a datum is affected by all the subsurface. The full expression for the kernel function will be developed further in the next sections, but for the moment, let us put each kernel function on a row of a matrix, such that $g_1$ will be on row 1, and $g_2$ on row 2, and define this matrix as $\widetilde{G}$. This will yield the matrix:
\begin{equation}
\widetilde{G}=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\end{equation}

Using the above, the assembled matrix vector forms for these equations becomes:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

Or to put it succinctly, $d = \widetilde{G} \, diag(\Delta x)\, m$.

## Step 3: Set up the mesh

Now that the data has been discretized, let's look at where we put our information. In the three dimensional cases, our information is stored in a cube, where we can put values in the cell centers (the red circle below), on the cube faces (red triangles), the edges (blue triangles), and on the nodes (the apices). This representation is useful because it allows us to represent physical quantities that are conserved across boundaries from on cell to the other. For example, fluxes can be put across the faces of the cube.

Taken from http://docs.simpeg.xyz/en/latest/api_Mesh.html

In a one dimensional case, however, we are only dealing with two elements, cell centers and cell nodes. First, subdivide the one dimensional domain into cell centers and nodes, with spacings of $\Delta x$:

In this instance the model values reside in the cell centers, while the values for the kernel functions reside on the nodes. Schematically, we can think of our 1D mesh as the following picture, where $g$ represents a kernel function and $m$ represents the model parameters.

Note that the kernel function in our example (represented above) has six values and the model values are five; moreover, the x-coordinates where $g(x) $ and $m(x)$ are evaluated are not coincident, and this leads to a complication when we want to perform the inner product of **m** and **g**. What is needed is a way to evaluate the kernel functions at the cell centers. To do this, we employ the trapezoidal rule for approximating integration. Simply put, to obtain the values of the kernel function on the cell centers, take the average of the kernel function values on the adjacent nodes. Let us then define two kernel function matrices, one with values evaluated on the cell centers, $G_c$ (represented by the black circles below), and another with the values evaluated on the nodes, $G_n$ (in white).

The relationship between $G_c $ and $G_n$ is an "averaging matix", $A_v$, such that $G_c = A_v G_n$. Putting each kernel function in the columns for both $G_c$ and $G_n$, and using again the dimensions of our toy problem, this relation appears as follows:

\begin{equation}
\begin{bmatrix}
    g_{c1}(x_1)    & g_{c2}(x_1)  \\
    g_{c1}(x_2)    & g_{c2}(x_2)  \\
    g_{c1}(x_3)    & g_{c2}(x_3)  \\
    g_{c1}(x_4)    & g_{c2}(x_4)  \\
    g_{c1}(x_5)    & g_{c2}(x_5)  \\
\end{bmatrix}
=
\frac{1}{2}
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\end{equation}

Meanwhile, the relationship between $G_c$ and $\widetilde{G}$ in step 2 is such that $G_c = \widetilde{G}^T$.

## Step 4: Build up the matrix equation $d=Gm$

We have all the required building blocks to assemble the matrix equation. At the end of step 2 we arrived at $d = \widetilde{G} \, diag(\Delta x)\, m$, and from the previous step we have $G_c = \widetilde{G}^T = A_v G_n$. Put more orderly, $\widetilde{G} = (A_v G_n)^T$, so substituting  this into our expression gives:

\begin{equation}
d = (A_v G_n)^T \, diag(\Delta x)\, m
\end{equation}

If we group matrices together and let $G = (A_v G_n)^T diag(\Delta x)$ then we arrive at $d=Gm$, our desired form. Referring again to our example, this would appear as follows:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\frac{1}{2}
\left(
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\right)^T
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

## Step 5: Generalize the form of the problem

In the above case we had two data points and five model values. We can generalize this to larger data sets easily. In the case where we have $N$ measured data and $M$ model values, we obtain a matrix equation of the following dimensions:

\begin{equation}
(N \times 1) = [(M \times (M+1)) \; ((M+1) \times N)]^T (M  \times M) (M \times 1)\\
d \qquad = \qquad \qquad  \qquad \quad   [A_v G_n]^T  \qquad \qquad diag(\Delta x) \quad m
\end{equation}

## Step 6: Implement the example problem in Python.

Next, we will build up our matrices one by one in Python. First we will do the small problem exactly as above, and then we will use a larger, but still one-dimensional, problem. To begin, import the SimPEG and numpy packages:

```
# Import Packages
from SimPEG import *
from IPython.html.widgets import interactive
import numpy as np
%pylab inline
```

    Efficiency Warning: Interpolation will be slow, use setup.py!

                python setup.py build_ext --inplace

    Populating the interactive namespace from numpy and matplotlib

    WARNING: pylab import has clobbered these variables: ['interactive']
    `%matplotlib` prevents importing * from pylab and numpy

Let start by formulating the forward problem, that is, let us assume that we have our model values $m$ and seek to generate synthetic data $d$. Recall that we are building up the matrix equation: $d = (A_v G_n)^T \, diag(\Delta x)\, m$, so let's start with the right hand side and go over each matrix, one by one.

(1) The $A_v$ matrix. We can build the "averaging matrix" as follows:

```
M=5              # RecalL is the number of values we have in m
n=M+1            # Define n as the N+1 dimension of the matrix
w=n-1            # Define w and the n-1  dimentsion of the matix
s = (M,n)        # Store matrix values
Av = np.zeros(s) # Create a matrix of zeros of the correct dimensions
                 # and fill in with elements usin the loop below (note the 1/2 is included in here).
for i in range(M):
    j=i
    k=i+1
    Av[i,j] = 0.5
    Av[i,k] = 0.5
print Av
```

    [[ 0.5  0.5  0.   0.   0.   0. ]
     [ 0.   0.5  0.5  0.   0.   0. ]
     [ 0.   0.   0.5  0.5  0.   0. ]
     [ 0.   0.   0.   0.5  0.5  0. ]
     [ 0.   0.   0.   0.   0.5  0.5]]

(2) The kernel matrix on the nodes, $G_n$. To build this matrix, we must first define a sensitivity function that describes some physical phenomenon. For the sake of our example, let us define our kernel function for the toy problem using the following equations. From a physical perspective, one can think of these equations as representing the decay of electromagnetic waves with depth. Note also that $i$ in this case is simply index number (i.e. $j=1,2,3,...$) and is *not* the imaginary number. The $j$ for the kernel functions affects the frequency and decay for each kernel function. Our kernel functions are as follows:

\begin{equation}
g_j(x) = e^{jpx} \cos(2 \pi j q x)
\end{equation}

```
g = lambda x, k, p, q: np.exp(-p*k*x)*np.cos(2*np.pi*q*k*x) # create an anonymous function as above
x=np.linspace(0,1,6)                                          # define the nodes of our x-array
#x = np.array([0., 0.2, 0.4, 0.6, 0.8, 1.])
p = 0.01
q = 0.1
j = np.array([1, 2])
Gn = np.zeros((len(x), len(j)))

for i in range(len(j)):
    f = g(x,i,p,q)
    Gn[:,i] = f
#print f
print Gn
```

    [[ 1.          1.        ]
     [ 1.          0.99013245]
     [ 1.          0.96471657]
     [ 1.          0.92421453]
     [ 1.          0.86932419]
     [ 1.          0.80096714]]

(3) The volume matrix $V$. This simply consists of making an $M \times M$ array of x-spacings, $\Delta x.$

```
# Make the delta x array

Deltax = 0.2*np.ones(M) # set x-spacings
V = np.diag(Deltax)     # create diagonal matrix
print V

```

    [[ 0.2  0.   0.   0.   0. ]
     [ 0.   0.2  0.   0.   0. ]
     [ 0.   0.   0.2  0.   0. ]
     [ 0.   0.   0.   0.2  0. ]
     [ 0.   0.   0.   0.   0.2]]

(4) Input the model values, $m$. Given that we are making a forward problem, we assume these values as given, so we input fictitious values:

```
m = np.array([0.02, 0.05, 0.09, 0.07, 0.04])
print m
```

    [ 0.02  0.05  0.09  0.07  0.04]

(5) Generate our two data values. The remaining step is simply to perform the matrix-vector multiplication:

```
Gc=np.transpose(np.dot(Av, Gn)) # cell center matrix Gc
G = np.dot(Gc,V)                # make G
d = np.dot(G,m)

#Note: in one line our data d, would be:  d = np.dot(np.dot(np.transpose(np.dot(Av, Gn)), V),m)
print d

```

    [ 0.054       0.04999083]

## A Bigger Toy Problem

We have successfully created two synthetic data points. Hopefully this has provided some intuition about how this is put together. But two data points are not interesting to view. Let's rebuild the problem using a larger system and using a more model values and several kernel functions. Let's clear everything and begin again:

```
clear
```

This time, let's use a similar setup as the one provided in the the *Inversion for Applied Geophysics* tutorial, found here: http://www.eos.ubc.ca/ubcgif/iag/tutorials/tutorial-v9.pdf. For this exercise, let's assume that we have 1000 model parameter values and 20 kernel functions, using the same form for the kernel functions as above. Also, let us create a model that is the combination of a Gaussian and a boxcar on a 1D domain between zero and one, similar to the model in the tutorial. Again, we will be performing the forward problem, which means that we will create synthetic data. The general setup of the domain and a plot of the model the model function could go something like this:

```
# Begin by creating a ficticious set of model data

M=1000                      # Set number of model parameters
x=np.linspace(0,1,M+1)      # Define 1D domain on nodes
xc = 0.5*(x[1:] + x[0:-1])  # Define 1D domain on cell centers
m = np.zeros(M)             # preallocate m array

# Define Gaussian function:
gauss = lambda x, a, mean, SD: a*numpy.exp(-(x-mean)**2./(2.*SD**2.)) # create a Gaussian function

# Choose parameters for Gaussian, evaluate, and store in an array, f.
SD=6
mean=0
a=1/(SD*sqrt(2*pi))
x2=np.linspace(-20,20,0.2*M)
f = gauss(x2,a, mean,SD)

# Define a boxcar function:
box = 0.4*np.ones(0.2*M)

# Insert the Gaussian and Boxcar into m:
m[0.2*M:0.4*M]=box
m[0.6*M:0.8*M]=10*f

# Plot
pylab.plot(xc,m)
#pylab.plot(f)
print size(f)
pylab.xlabel('x')
pylab.ylabel('m(x)')
pylab.title('Model, $m(x)$')

print shape(xc)
```

Next, let's build the kernel functions. In this case, let's define the kernel functions as before, with p, q, and the number of output data, N, as follows. We can then create a set of kernel functions on the nodes. The plot for these functions is below:

```
# Make the set of kernel functions

g = lambda x, i, p, q: np.exp(-p*i*x)*np.cos(2*np.pi*q*i*x) # create an anonymous function as immediately above
p = 0.01      # Set values for p, q
q = 0.15
N = 20        # specify number of output data
Gn = np.zeros((M+1,N))

for i in range(N):
    f = g(x,i,p,q)
    Gn[:,i] = f

# Plot
pylab.plot(x,Gn)
pylab.xlabel('x')
pylab.ylabel('g(x)')
pylab.title('Kernel functions, $g(x)$')
```

The averaging matrix can be made much the same as before, although it will be larger in this case. I will print a subsample of the matrix and show it as a plot below:

```
# Make Averaging Matrix
n=M+1            # Define n as the N+1 dimension of the matrix
w=n-1            # Define w and the n-1  dimentsion of the matix
s = (M,n)        # Store matrix values
Av = np.zeros(s) # Create a matrix of zeros of the correct dimensions
                 # and fill in with elements usin the loop below (note the 1/2 is included in here).
for i in range(M):
    j=i
    k=i+1
    Av[i,j] = 0.5
    Av[i,k] = 0.5
print shape(Av)

# Plot
pylab.title('Subset of averaging matrix')
pyplot.imshow(Av[:5,:6], cmap='Greys',  interpolation='nearest')
print(Av[:5,:6])
```

The next step is to make our volume matrix (note its shape, $M \times M$):

```
# make the Volume, "delta x" array

Deltax = 0.2*np.ones(M) # set x-spacings
V = np.diag(Deltax)     # create diagonal matrix
print shape(V)

```

    (1000L, 1000L)

The last step is to create and plot our synthetic data:

```
d = np.dot(np.dot(np.transpose(np.dot(Av, Gn)), V),m)   # Create synthetic data
xd=np.linspace(0,1,len(d))                              # make x array for data

# Plot

pylab.plot(xd,d)
pylab.xlabel('')
pylab.ylabel('d')
pylab.title('Synthetic Data $d$')
```

## Harder, Bettter, Stronger, Faster

Now we have created our first 1D set of synthetic data using numpy exclusively. In real world situations, however, storing matrices explicitly like this is inefficient if not infeasible. For example, for a large data set, the volume matrix consists mainly of zeros, save the diagonal elements (this is a similar case for the averaging matrix). It is not practical to hold on to superfluous information when the operation can be specified without storing so many values. In Python there is a package available capable of performing these operations much more effectively: the Simulation and Parameter Estimation in Geophysics, or "SimPEG" for short (see http://www.simpeg.xyz). As a last exercise, let's see how the same process can be done using SimPEG.

```
g = lambda x, k, p, q: np.exp(-p*k*x)*np.cos(2*np.pi*q*k*x)  # I redfine our kernel functions, as above.

M = Mesh.TensorMesh([1000])                                  # create a tensor mesh
Glist = [g(M.vectorNx,i+1,p,q) for i in range(20)]           # make kernel functions on nodes with the following lines
Glist = []
for i in range(20):
    gi = g(M.vectorNx,i+1,p,q)
    Glist.append(gi)
Gnodes = np.vstack(Glist).T
Gcells = M.aveF2CC * Gnodes                                  # evaluate kernel functions on cell centers directly
V = Utils.sdiag(M.vol)                                       # build volume array
d = np.dot(Gcells.T, V * m)                                  # create data
xd=np.linspace(0,1,len(d))

# Plot
pylab.plot(xd,d)
pylab.xlabel('')
pylab.ylabel('d')
pylab.title('Synthetic Data $d$')
```

Thus far we have created a set of synthetic data. Essentially what we have done is make a forward problem. In the next module, we are going to look at a method for performing and inversion, which will involve casting our inverse problem in terms of an optimization.

.ipynb_checkpoints/Module 1,  Setting up the problem, April-checkpoint.ipynb


# Computational Exercise 1: A Vary Great, Very Harmonic, Wavefunction
We will apply the Variational Method to the Harmonic Oscillator.  In particular, we will optimize
the trial wavefunction given by
\begin{equation}
\Phi(x) = {\rm exp}\left( -\alpha x^2 \right),
\end{equation}
where $\alpha$ is an adjustable parameter that will be variationally optimized.  In particular, the goal
will be to minimize the energy functional given by
\begin{equation}
E[\Phi(x)] = \frac{\int_{-\infty}^{\infty} \Phi^* (x) \: \hat{H} \: \Phi(x) dx }{\int_{-\infty}^{\infty} \Phi^* (x) \: \Phi(x) dx }
\end{equation}
by modulating the trial wavefunction through the adjustable parameter $\alpha$.

The Hamiltonian operator for Harmonic Oscillator is given by
\begin{equation}
\hat{H} = -\frac{\hbar^2}{2m} \frac{d^2}{dx^2} + \frac{1}{2} k \: x^2.
\end{equation}
We will use a system of natural units where $\hbar = 1$, the mass of the particle $m = 1$, and the force constant of
the potential $k = 1$.

### Questions
1. What is the optimal value of $E[\Phi(x)]$ in natural units?  How does this value compare to the true ground state energy of the Harmonic Oscillator?
2. What is the optimal value of $\alpha$ in natural units?  How does $\Phi(x)$ with this value of $\alpha$ compare to the true ground state wavefunction?
3. What is the kinetic energy expectation value of the optimized trial wavefunction in natural units?
4. What is the potential energy expectation value of the optimized trial wavefunction in natural units?

### (Hopefully) Helpful Hints
You can represent your trial wavefunction numerically on the computer by creating an array of x-values, evaluating
the formula for your trial wavefunction at each of those x-values, and then storing each of those function values in another array.  The following lines of code will do this using $numpy$ arrays and the $numpy.exp$ function.

```python
import numpy as np
from matplotlib import pyplot as plt
### create an array of 1000 x-values between x=-20 and x=20 atomic units
x_array = np.linspace(-20,20,1000)
### pick a trial value of alpha... might as well be 1 atomic unit!
alpha = 1.0
### evaluate your trial wavefunction at each of the x-values stored in the array called 'x_array'
Phi = np.exp(-alpha * x_array**2)
Vx = 1/2. * x_array **2

plt.plot(x_array, Phi, 'red', label='Trial')
plt.plot(x_array, Vx, 'green', label='Potential')
plt.xlim(-5,5)
plt.ylim(0,10)
plt.legend()
plt.show()

```

To evaluate the kinetic energy of a trial wavefunction, we will need to take the second derivative of the trial wavefunction.
One way to take derivatives on a computer is to use the method of finite differences.  In particular, the forward finite difference
approximation to the first derivative of your trial wavefunction $\Phi(x)$ at the point $x_0$ can be written symbolically as
\begin{equation}
\frac{d \Phi(x_0)}{dx} = \frac{\Phi(x_0 + dx) - \Phi(x_0)}{dx}
\end{equation}
where in our code, $dx$ will be determined by the range of x-values and the number of x-values used in defining
$x_{array}$.

Create a function called $dfdx$ that takes an array of x-values and an array of function values evaluated at each x-value and
returns the first derivative of the function values with respect to x.  Then, show that calling this function in sequence can be used to compute the second derivative of an array of function values.

A skeleton of the function looks like this:

```python
def dfdx(fx, x):
    ### loop through values of fx and x and use forward finite-differences
    ### to compute the d/dx of fx at each point and store in an array called 'fxp'
    ### make sure to poperly handle
    ### the last point where f(x_0 + dx) is not stored
    ### create a numpy array called fxp that has the same length
    ### and data type as the array of function values you are differentiating
    fxp = np.zeros_like(fx)

    for i in range(0,len(x)-1):
        rise = fx[i+1]-fx[i]
        run = x[i+1]-x[i]
        slope = rise/run
        fxp[i] = slope

    ### once you have computed all fxp values, return the array!
    return fxp
```

And the calling sequence for $dfdx$ to compute the second derivative of a trial wavefunction we defined previously will look like this:

```python
### Call dfdx on Phi to get the first derivative of Phi
Phi_p = dfdx(Phi, x_array)

Phi_wa = -2 * alpha * x_array * Phi

plt.plot(x_array, Phi_p, 'red', label='dfdx result')
plt.plot(x_array, Phi_wa, 'b--', label='Wolfram Alpha result')
plt.legend()
plt.show()

### Call dfdx on Phi_p to get the second derivative of Phi
Phi_pp = dfdx(Phi_p, x_array)
```

```python

```


```python
import numpy as np
import pandas as pd
import math as mt
import matplotlib.pyplot as plt
import sympy
from sklearn.preprocessing import MinMaxScaler
```

# Caso 1: Senoide

```python
m1 = 1.2
m2 = 2.7
k1 = 10
k2 = 20
k3 = 15

omega_n_1 = (k1/m1)**0.5
mr = m2/m1
k32 = k3/k2
omega_r = (k2/k1)**0.5

a1 = 1 + omega_r**2 * (1+mr+k32)
a2 = omega_r**2 * (1+k32*(1+omega_r**2 * mr))

Omega1 = (0.5*(a1-(a1**2 - 4*a2)**0.5))**0.5

Omega2 = (0.5*(a1+(a1**2 - 4*a2)**0.5))**0.5

print("Omega1 = {:.2f} rad/s".format(Omega1))

print("Omega2 = {:.2f} rad/s".format(Omega2))

rasao_1 = mr*omega_r**2/(1+omega_r**2 *mr-Omega1**2)

rasao_2 = omega_r**2 *(1+k32) - Omega2**2/(omega_r**2)

print("rasao_1 = {:.2f}".format(rasao_1))

print("rasao_2 = {:.2f}".format(rasao_2))
```

    Omega1 = 1.16 rad/s
    Omega2 = 2.77 rad/s
    rasao_1 = 1.08
    rasao_2 = -0.33


# Matrix as transformation and interpreting low rank matrix

## Goals

### G1: To understand matrix vector multiplication as transformation of the vector

Multiplying a matrix A with a vector x transforms x

### G2: Understanding low rank matrices as applying transformation on a vector resulting in a subspace of the original vector space

Transforming a vector via a low rank matrix in the shown examples leads to a line

We first study Goal 1. The interpretation of matrix vector product is borrowed from the excellent videos from the [3Blue1Brown channel](https://www.youtube.com/watch?v=kYB8IZa5AuE). I'll first set up the environment by importing a few relevant libraries.

#### Basic imports

```python
import numpy as np
import seaborn as sns
import pandas as pd
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
from sympy import Matrix, MatrixSymbol, Eq, MatMul

sns.reset_defaults()
sns.set_context(context="talk", font_scale=0.75)
%matplotlib inline
%config InlineBackend.figure_format='retina'
```

```python
sympy_A = MatrixSymbol("A", 2, 2)
sympy_x = MatrixSymbol("x", 2, 1)
y = MatrixSymbol("y", 2, 1)

Eq(y, sympy_A*sympy_x, evaluate=False)
```

$\displaystyle y = A x$

Given a matrix `A` and a vector `x`, we are trying to get `y=Ax`. Let us first see the values for a specific instance in the 2d space.

```python
A = np.array([[2, 1], [1, 4]])

x = np.array([1, 1])
Ax = A @ x

Eq(Matrix(Ax), MatMul(Matrix(A), Matrix(x)),evaluate=False)
```

$\displaystyle \left[\begin{matrix}3\\5\end{matrix}\right] = \left[\begin{matrix}2 & 1\\1 & 4\end{matrix}\right] \left[\begin{matrix}1\\1\end{matrix}\right]$

Here, we have A = $\left[\begin{matrix}2 & 1\\1 & 4\end{matrix}\right]$ and x = ${\text{[1 1]}}$

Now some code to create arrows to represent arrows.

```python
def plot_arrow(ax, x, color, label):
    x_head, y_head = x[0], x[1]
    x_tail = 0.0
    y_tail = 0.0
    dx = x_head - x_tail
    dy = y_head - y_tail

    arrow = mpatches.FancyArrowPatch(
        (x_tail, y_tail), (x_head, y_head), mutation_scale=10, color=color, label=label
    )

    ax.add_patch(arrow)
    ax.legend(bbox_to_anchor=(1.6, 1), borderaxespad=0)
```

Now some code to plot the vector corresponding to Ax

```python
def plot_transform(A, x):
    Ax = A @ x
    fig, ax = plt.subplots()
    plot_arrow(ax, x, "k", f"Original (x) {x}")
    plot_arrow(ax, Ax, "g", f"Transformed (Ax) {Ax}")
    plt.xlim((-5, 5))
    plt.ylim((-5, 5))
    plt.grid(alpha=0.1)
    ax.set_aspect("equal")
    plt.title(f"A = {A}")
    sns.despine(left=True, bottom=True)
    plt.tight_layout()
```

```python
plot_transform(np.array([[1.0, 1.0], [1.0, -1.0]]), [1.0, 2.0])
plt.savefig("Ax1.png", dpi=100)
```

In the plot above, we can see that the vector [1, 2] is transformed to [3, -1] via the matrix A.

Let us now write some code to create the rotation matrix and apply it on our input x

```python
def rot(angle):
    theta = np.radians(angle)
    c, s = np.cos(theta), np.sin(theta)
    R = np.array(((c, -s), (s, c)))
    return np.round(R, 2)
```

```python
x = np.array([1.0, 2.0])
plot_transform(rot(90), x)
plt.savefig("Ax2", dpi=100)
```

As we can see above, creating the 90 degree rotation matrix indeed transforms our vector anticlockwise 90 degrees.

Now let us talk about matrices A that are low rank. I am creating a simple low rank matrix where the second row is some constant times the first row.

```python
def plot_lr(x, slope):
    low_rank = np.array([1.0, 2.0])
    low_rank = np.vstack((low_rank, slope * low_rank))
    plot_transform(low_rank, x)
    x_lin = np.linspace(-5, 5, 100)
    y = x_lin * slope
    plt.plot(x_lin, y, alpha=0.4, lw=5, label=f"y = {slope}x")
    plt.legend(bbox_to_anchor=(1.2, 1), borderaxespad=0)
```

```python
plot_lr(x, 1.01)
plt.tight_layout()
plt.savefig("lr-1.png", bbox_inches="tight", dpi=100)
```

```python
plot_lr([1.0, -1.0], 1.01)
plt.tight_layout()
plt.savefig("lr-2.png", bbox_inches="tight", dpi=100)
```

```python
plot_lr([0.5, -0.7], 1.01)
plt.tight_layout()
plt.savefig("lr-3.png", bbox_inches="tight", dpi=100)
```

```python
plot_lr([-1.0, 0.0], 1.01)
plt.tight_layout()
plt.savefig("lr-4.png", bbox_inches="tight", dpi=100)
```

To summarize

In the above plots we can see that changing our x to any vector in the 2d space leads to us to transformed vector not covering the whole 2d space, but on line in the 2d space. One can easily take this learning to higher dimensional matrices A.

```python

```


# Iterative Solvers 2 - From FOM to GMRES

FOM is rarely used in practical applications. A problem is that it does not guarantee that the global residuals reduce from step to step. Remember that it only guarantees that the residual is orthogonal to the current search subspace.

A solution for this problem is provided by GMRES (Generalized Minimal Residuals)

## The basic idea of GMRES

GMRES, like FOM, is built oin the Arnoldi iteration. Let $V_m$ be the current Krylov basis. Remember that we want to find a correction $V_my_y$ to our initial approximation $x_0$ such that

$$
A\left(x_0 + V_my_m\right) \approx b.
$$

In the FOM we imposed the condition that $r_m\bot V_m$, that is $V_m^Tr_m = 0$. In GMRES we impose a different condition, namely we choose $y_m$ such that it minimises the residual $r_m$. Hence,

$$
y_m = \text{arg min}_{y\in\mathbb{R}^m}\|AV_my_m - r_0\|_2.
$$

We can use the Arnoldi recurrence relation $AV_m = V_mH_m + h_{m+1, m}v_{m+1}e_m^T$ to simply the above expression, namely

$$
\begin{align}
\|AV_my_m - r_0\|_2 &= \|\left(V_mH_m + h_{m+1, m}v_{m+1}e_m^T\right)y_m - r_0\|_2\nonumber\\
&= \|V_{m + 1}\left(\begin{bmatrix}H_m \\ h_{m+1, m}e_m^T\end{bmatrix}y_m - \|r_0\|_2e_1\right)\|_2\nonumber\\
&= \|\begin{bmatrix}H_m \\ h_{m+1, m}e_m^T\end{bmatrix}y_m - \|r_0\|_2e_1\|_2
\end{align}
$$

We see that we are left with a small dimensional least-squares problem that can easily be solved. A beautiful feature about GMRES is that due to the structure of $H_m$ we can just update our solution $y$ from step $m$ to step $m+1$ without solving the whole least-squares problem again.

As in the FOM we need to put in some care into proper orthogonalisation of the vectors. For the following experiments we will use the Scipy GMRES implementation, which is well tested and provides a number of useful options.

## Some experiments with GMRES

We start with a very simple setup of a matrix with random normally distributed entries.

```python
import numpy as np
rand = np.random.RandomState(0)

n = 500

A = rand.randn(n, n) / np.sqrt(n)

```

Let us plot the eigenvalues of this matrix.

```python
%matplotlib inline
from matplotlib import pyplot as plt
from scipy.linalg import eigvals

eigenvalues = eigvals(A)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)
ax.plot(np.real(eigenvalues), np.imag(eigenvalues), 'rx', markersize=1)
```

This matrix has nicely clustered eigenvalues which lie roughly in the unit disk.

We will now consider the matrix `Ashift` which is defined as `A + shift * np.eye(n)`, and we will plot the convergence curve of GMRES for different values of the `shift` parameter.

```python
from scipy.sparse.linalg import gmres

shifts = [0, 1, 2, 5, 10]
colors = ['r', 'k', 'y', 'g', 'm']

b = rand.randn(n)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)

for index, shift in enumerate(shifts):
    Ashift = A + shift * np.eye(n)
    residuals = []
    callback = lambda res: residuals.append(res)
    x, _ = gmres(Ashift, b, restart=n, callback=callback, callback_type='pr_norm')
    if len(residuals) > 50:
        residuals = residuals[:50] # Only plot the first 50 residuals
    ax.semilogy(residuals, colors[index] + '-x', markersize=2)

fig.legend(loc='lower center', fancybox=True, shadow=True, ncol=len(shifts), labels=[str(shift) for shift in shifts])

```

We see that the further to the origin the cluster of eigenvalues, the slower the convergence. Convergence theorems for GMRES are only known in certain cases. A heuristical guide is that GMRES converges well if the eigenvalues are clustered together and the cluster is well separated from the origin.

## A remark on restarts

In the above code there is a `restart` parameter. One issue with GMRES is that for high number of iterations it needs a lot of memory and each step is increasingly becoming more expensive since we always have to orthogonalise against all previous vectors, so need to keep them in memory. Restarting is a strategy, where we restart the GMRES iteration with the current iterate. This is frequently used in practice and can keep the required memory low at the cost of increasing the number of required iterations.


##Ejercicio 4 Practica 1
Para cada uno de los siguientes sistemas encontrar todos los puntos de equilibrio y determinar el tipo de cada punto de equilibio aislado.
 * c)

 $$\left\{ \begin{array}{lcc}
            \dot{x}_{1}=(1-x_{1})x_{1}-\frac{2x_{1}x_{2}}{1+x_{1}}\\
             \\ \dot{x}_{2}=(2-\frac{x_{2}}{1+x_{1}})x_{2}
             \end{array}
\right.$$

```python
import sympy as sym
```

```python
#Con esto las salidas van a ser en LaTeX
sym.init_printing(use_latex=True)
```

```python
x_1, x_2 = sym.symbols('x_1 x_2')
```

```python
X = sym.Matrix([x_1, x_2])
X
```

```python
f_1 = (1 - x_1) * x_1 - (2 * x_1 * x_2) / (1 + x_1)
f_1
```

```python
f_2 = (2 - (x_2)/(1 + x_1)) * x_2
f_2
```

```python
F = sym.Matrix([f_1,f_2])
F
```

```python
# puntos de equilibrio del sistema
pes = sym.solve([f_1,f_2])
pes
```

```python
A = F.jacobian(X)
A
```

```python
A_1 = A.subs(pes[0])
A_1
```

```python
A_1.eigenvals()
```

```python

```


Erstellen Sie eine Funktion, welche die Funktionswerte von

$$
f(x) = sin(x)cos(x)^2
$$

zurückgibt für beliebige $x \in R$. Geben Sie zunachst die geplottete Funktion mittels `sympy` im Intervall $[−2,2]$ aus. Dann plotten Sie die Funktion mittels `matplotlib` und `numpy` für jeweils 5,10,15 und 100 aquidistante x-Werte (Achtung auf die Grenzen im Definitionsbereich!). Erstellen Sie fur jede Anzahl der x-Werte einen Subplot, wobei die Funktionswerte jeweils durch Punkte, die linear verbunden sind, dargestellt werden sollen. Jeder Subplot soll sich durch unterschiedliche Symbole der Punkte und Farben unterscheiden. Geben Sie den gesamten Plot (mit den 4 Subplots) aus.

```python
from sympy.plotting import plot
from sympy.functions import cos, sin
from sympy.abc import x

p = plot(
    sin(x)*cos(x)**2,
    xlim = (-2, 2),
    legend = True,
    loc = 'best',
    show = False
)
p.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt
cmap = plt.get_cmap('jet')

fig, axs = plt.subplots(2, 2, sharex = True, sharey = True)
for n, marker, color, ax in zip(
    [5, 10, 15, 100],
    '+x^s',
    [cmap(i) for i in np.linspace(0, 1, 4)],
    axs.reshape(4)
):
    x = np.linspace(-2, 2, n)
    ax.plot(
        x,
        np.sin(x)*np.cos(x)**2,
        marker = marker,
        markersize = 10,
        markeredgecolor = color,
        markerfacecolor = color
    )

fig.set_figwidth(20)
fig.set_figheight(10)
fig.tight_layout()
```


Polynomial $A(x) = \sum_{j=0}^{n-1} a_j x^j$

let coeffs $A = (a_0,...,a_{n-1})$.

that $k=0,...,n-1$
$$
\begin{align}
\text{DFT}(A) &= (\sum_{j=0}^{n-1} a_j e^{-\frac{2\pi i}{n} jk},...)_k \\
    &= (A(\omega_n^k),...)_k,\ \ \  \omega_n^k = \exp(-\frac{2\pi i}{n} k)
\end{align}
$$

then consider convolution, or multiplication of $A(x)$ and $B(x)$. we wanna know coeffs of  $(A*B)(x) = A(x)B(x)$.

for simplicity write $F=DFT$, notice that, $F(A) = (A(\omega_n^k))_k$, that

$$
\begin{align}
    F(A*B)  &= ((A*B)(\omega_n^k))_k = (A(\omega_n^k)B(\omega_n^k))_k = F(A)\cdot  F(B) \\
    A*B &= F^{-1}[F(A)\cdot  F(B)]
\end{align}
$$

so next we need is to compute $F, F^{-1}$ in $O(n\log n)$

Before continue, note the lemmas

$$
\begin{align}
    \omega_{dn}^{dk} = \omega_n^k, n,k,d \geq 0 \\
    \omega_n^{n/2} = \omega_2^1 = -1,  n>0 \\
    (\omega_n^{k+n/2})^2=(\omega_n^k)^2=\omega_{n/2}^k, n>0, \text{even} \\
    \omega_n^{k+n/2} = -\omega_n^k, n\geq 0
\end{align}
$$

then by separate odd,even, i.e. $A(x) = A_0(x^2) + xA_1(x^2)$. combine above lemmas, get

$$
\begin{align}
    A(\omega_n^k) = A_0(\omega_{n/2}^k) + \omega_n^k A_1(\omega_{n/2}^k), 0\leq k < n/2 \\
    A(\omega_n^{k+n/2}) = A_0(\omega_{n/2}^k) - \omega_n^k A_1(\omega_{n/2}^k), k+n/2 \geq n/2
\end{align}
$$

Thus, by divide and conquer solved. what about $F^{-1}$. notice that matrix form

$$
\begin{align}
    y=F(A) = W a \\
    w_{k,j} = \omega_n^{kj}, k,j=0,...n-1\\
    a = F^{-1}(y) = W^{-1}y
\end{align}
$$

by the special form of $W$
$$
\begin{align}
    W^{-1} = \frac{1}{n} \bar{W}\\
    v_{kj} = \frac{1}{n} \bar{w_n} ^{kj}
\end{align}
$$

So, just conjugate and divided by $n$. then same method sovlable.

### NTT

let $\alpha$ replace $\omega_n$ in integer field.

DFT require
$$
\begin{align}
    \sum_{j=0}^{n-1} \alpha^{kj} = 0, k=1,...,n-1
\end{align}
$$
and $\alpha^n=1, \alpha^k \neq 1, k=1,...,n-1$ is sufficient

if $n$ is power of $2$, $\alpha^{n/2} = -1$ is sufficient

if we get the $\alpha$ on $Z_p$, then $p=c2^k+1$, that $\alpha^c$ can be the $\alpha'$ to express conv length $\leq 2^{k-1}$

For in place NTT, leaves' addr. are bit-reversed. (NEED rigorous proof)

and if for $F^{-1}$, one way is calc $\alpha^{-1}$, for each iter. Here is another way. Notice the matrix form. suppose $a$ divided by $n$, then we reverse $[a_1,...,a_{n-1}]$. by the fact $\alpha^{k(n-j)} = \alpha^{-kj}$, thus result is exactly $F^{-1}$.

#### inverse

we wanna know $BA \equiv 1 (\mod x^n)$. by the step that double $B$'s size, with init $B_0=1/A_0$. wlog write $B_1$ repre. $B_n$, $B_2$ repre. $B_{2n}$. that
$$
\begin{align}
    & (B_1 A - 1)^2 \equiv 0 (\mod x^2) \\
    \Rightarrow  & B_2 = 2B_1 - B_1^2 A (\mod x^2)
\end{align}
$$
note, in impl. $B_1^2A$ has $B_1(\mod x)$ part in there since $B_1A \equiv 1(\mod x)$. aka, $(\mod x)$ part remain $B_1$, actually, we only need modify the higher order part, $-B_1^2 A ([x\leq..<x^2])$

#### sqrt

also, the double size technique
$$
\begin{align}
    & (B_1^2 - A)^2 \equiv 0 (\mod x^2) \\
    \Rightarrow  & B_2 = \frac{1}{2} [B_1 + A B_1^{-1}](\mod x^2)
\end{align}
$$
again, notice the old part $B_1$ shall remain, we can only modify new part in impl. and careful $B_0^2=A_0$ when $\neq 1$

#### log

$\log P = \int \frac{P'}{P}$

#### exponent

double step, $B_2 = B_1(1 + A - \log B_1)$

### FWHT(FHT)

Hadamard transform, def, with $H_0 = 1$
$$
\begin{align}
    H_m = \frac{1}{\sqrt{2}} \begin{bmatrix} H_{m-1} & H_{m-1} \\ H_{m-1} & -H_{m-1} \end{bmatrix} \\
    (H_m)_{i,j} = \frac{1}{2^{m/2}} (-1)^{\langle i, j \rangle_b}
\end{align}
$$
where $i,j= 0,1,..,2^m-1$ and $\langle .,.\rangle_b$ is bitwise dot product, aka
```
__builtin_popcount(i&j)
```

when do fwht, don't need bit reversal and generator. besides, there is good property $H_m^2= I$.
One can easy show $H_1^2 = I$ by direct compute or quantum tech. $H_1 = |+\rangle \langle 0| + |-\rangle \langle 1| $. then by induction. $H_m = H_1 \otimes H_{m-1}$. that $H_m^2 = H_1^2 \otimes H_{m-1}^2 = I$. which means we can do $H^{-1}$ by direct do $H$ !

note in impl. we often omit $2^{n/2}$, that use $H' = 2^{n/2}H$. so when do inverse, we need divide by $2^n$.

still, we can do multiply by $A*B = H^{-1} [H(A) \cdot H(B)]$. note $\cdot$ is element-wise mult.

also, suprisingly, for solving equation $A*B = C$, if we put $H$ on each side, that

$H(A) \cdot H(B) = H(C)$, since $\cdot$ is element-wised, thus we can get $A = H^{-1} [H(C) / H(B)]$, element-wise.

Note. for $\sum_i B_i = 0$, aka, not linear indep. the solution may has many, notice that $R = (1,1,...,1)$, that $R*B = 0$. e.g. agc034f.

so we can minus arbitral still solution, but according specific condition, we can get the right one.

### useful links

https://cp-algorithms.com/algebra/fft.html

https://cp-algorithms.com/algebra/polynomial.html#toc-tgt-4

https://en.wikipedia.org/wiki/Discrete_Fourier_transform#Polynomial_multiplication

https://en.wikipedia.org/wiki/Discrete_Fourier_transform_(general)

https://codeforces.com/blog/entry/43499

https://codeforces.com/blog/entry/48798

https://crypto.stanford.edu/pbc/notes/numbertheory/gen.html

https://csacademy.com/blog/fast-fourier-transform-and-variations-of-it

```python

```


# Mínimos Cuadrados

## Descomposición $QR$

```python
import numpy as np
import scipy.linalg as spla
import matplotlib.pyplot as plt
```

```python
def plotVector(A, Q):
    plt.figure(figsize=(6, 6))
    origin = [0], [0] # origin point
    plt.quiver(*origin, A[:,0], A[:,1], angles='xy', scale_units='xy', scale=1, color=['green', 'black'])
    plt.quiver(*origin, Q.T[:,0], Q.T[:,1], angles='xy', scale_units='xy', scale=1, color=['r', 'b'])
    plt.axis('square')
    plt.xlim([-2, 4])
    plt.ylim([-2, 4])
    plt.grid(True)
    plt.xlabel(r"$x$")
    plt.ylabel(r"$y$")
    plt.show()
```

### Decomposición $QR$ reducida

```python
def QR(A, modified=True):
    m, n = A.shape
    Q = np.zeros((m, n))
    R = np.zeros((n, n))
    # Gram-Schmidt orthogonalization
    for j in range(n):
        y = A[:, j]
        if modified:
            for i in range(j):
                R[i, j] = np.dot(Q[:, i], y)
                y = y - R[i, j] * Q[:, i]
        else:
            for i in range(j):
                R[i, j] = np.dot(Q[:, i], A[:, j])
                y = y - R[i, j] * Q[:, i]
        R[j, j] = np.linalg.norm(y)
        Q[:, j] = y / R[j, j]
    return Q, R
```

### Ejemplo en $\mathbb{R}^2$

Para considerar el manejo de arreglos de ```numpy```, recordar que el procedimiento lo realizamos por vectores columnas, es decir,
\begin{equation}
    A=
    \left[
        \begin{array}{c|c}
            & \\
            \mathbf{a}_1 & \mathbf{a}_2 \\
            & \\
        \end{array}
    \right].
\end{equation}

```python
A = np.array([[2., 1.], [1., 3.]])
```

```python
A
```

    array([[2., 1.],
           [1., 3.]])

#### Versión implementada

```python
Q, R = QR(A)
```

```python
np.dot(Q, R)
```

    array([[2., 1.],
           [1., 3.]])

#### Versión ```numpy```

```python
Q2, R2 = np.linalg.qr(A)
```

```python
np.dot(Q2, R2)
```

    array([[2., 1.],
           [1., 3.]])

```python
print("Implementado")
print("Q:\n", Q)
print("R:\n", R)
print()
print("Numpy")
print("Q:\n", Q2)
print("R:\n", R2)
```

    Implementado
    Q:
     [[ 0.89442719 -0.4472136 ]
     [ 0.4472136   0.89442719]]
    R:
     [[2.23606798 2.23606798]
     [0.         2.23606798]]

    Numpy
    Q:
     [[-0.89442719 -0.4472136 ]
     [-0.4472136   0.89442719]]
    R:
     [[-2.23606798 -2.23606798]
     [ 0.          2.23606798]]

La implementación de ```numpy``` no realiza la consideración de $r_{j,j} > 0$.

### ¿Son ortogonales?

```python
np.dot(Q[:,0], Q[:, 1]), np.dot(Q2[:,0], Q2[:, 1])
```

    (0.0, -5.551115123125783e-17)

Al parecer sí :)

### ¿Se cumple $Q^{-1}=Q^*$?

```python
np.linalg.inv(Q), Q.T
```

    (array([[ 0.89442719,  0.4472136 ],
            [-0.4472136 ,  0.89442719]]),
     array([[ 0.89442719,  0.4472136 ],
            [-0.4472136 ,  0.89442719]]))

```python
np.linalg.inv(Q2), Q2.T
```

    (array([[-0.89442719, -0.4472136 ],
            [-0.4472136 ,  0.89442719]]),
     array([[-0.89442719, -0.4472136 ],
            [-0.4472136 ,  0.89442719]]))

Al parecer también.

### Visualmente

```python
plotVector(A, Q)
```

Los vectores verde y negro corresponde a los vectores columnas de $A$. Los vectores rojo y azul corresponde a los vectores ortonormales encontrados por $QR$.

# Ajuste de curvas

## Resolución de mínimos cuadrados utilizando $QR$

```python
def QRLS(A, b):
    Q, R = QR(A) # QR reduced
    x = spla.solve_triangular(R, np.dot(Q.T, b)) # Solve Rx=Q^*b
    return x
```

```python
def plot(t, y, tt, yy, yyy):
    plt.figure(figsize=(12, 6))
    plt.plot(t, y, 'r.')
    plt.plot(tt, yy, label="Normal Equations")
    plt.plot(tt, yyy, label="QR")
    plt.grid(True)
    plt.legend()
    plt.xlabel(r'$t$')
    plt.ylabel(r'$y$')
    plt.show()
```

## Ecuaciones normales

```python
lm = lambda c_1, c_2, t: c_1 + c_2 * t
qm = lambda c_1, c_2, c_3, t: c_1 + c_2 * t + c_3 * t ** 2
em = lambda c_1, c_2, t: c_1 * np.exp(c_2 * t)
sm = lambda c_1, c_2, c_3, c_4, t: c_1 + c_2 * np.cos(2 * np.pi * t) + c_3 * np.sin(2 * np.pi * t) + c_4 * np.cos(4 * np.pi * t)
pm = lambda c_1, c_2, t: c_1 * t ** c_2
```

```python
# A^*A x = A^* b
def normalEquations(A, b):
    return np.linalg.solve(np.dot(A.T, A), np.dot(A.T, b))
```

#### Modelo lineal

```python
lm = lambda c_1, c_2, t: c_1 + c_2 * t
```

```python
n = 200
t_a, t_b = 0, 3
t = np.linspace(t_a, t_b, n)
tt = np.linspace(t_a, t_b, 3 * n)
```

```python
y_l = lm(1, 2, t) + np.random.normal(1.5, .5, n) # Data
A_l = np.ones((n, 2))
A_l[:,1] = t
b_l = y_l
```

```python
p_l = normalEquations(A_l, b_l)
```

```python
p_l_qr = QRLS(A_l, b_l)
```

```python
p_l, p_l_qr
```

    (array([2.61872557, 1.92680618]), array([2.61872557, 1.92680618]))

```python
plot(t, y_l, tt, lm(*p_l, tt), lm(*p_l_qr, tt))
```

Podemos ver para ambos métodos obtenemos los mismos coeficientes del modelo lineal.

## Cuadrático

```python
y_q = qm(1, 2, 3, t) + np.random.normal(5, 3, n)
A_q = np.ones((n, 3))
A_q[:, 1] = t
A_q[:, 2] = t ** 2
b_q = y_q
```

```python
p_l = normalEquations(A_q, b_q)
```

```python
p_l_qr = QRLS(A_q, b_q)
```

```python
plot(t, y_q, tt, qm(*p_l, tt), qm(*p_l_qr, tt))
```

## Periódico

```python
y_s = sm(1, 2, 2, 1, t) + np.random.normal(10, 2, n)
A_s = np.ones((n, 4))
A_s[:, 1] = np.cos(2 * np.pi * t)
A_s[:, 2] = np.sin(2 * np.pi * t)
A_s[:, 3] = np.cos(4 * np.pi * t)
b_s = y_s
```

```python
p_l = normalEquations(A_s, b_s)
```

```python
p_l_qr = QRLS(A_s, b_s)
```

```python
plot(t, y_s, tt, sm(*p_l, tt), sm(*p_l_qr, tt))
```

## Exponencial

```python
y_e = em(1, 2, t) * np.random.normal(10, 2, n)
A_e = np.ones((n, 2))
A_e[:, 1] = t
b_e = np.log(y_e)
```

```python
p_l = normalEquations(A_e, b_e)
```

```python
p_l_qr = QRLS(A_e, b_e)
```

```python
plot(t, y_e, tt,  em(np.exp(p_l[0]), p_l[1], tt),  em(np.exp(p_l_qr[0]), p_l_qr[1], tt))
```

## Comentarios

Podemos obtener los mismos resultados al utilizar **Ecuaciones Normales** que al usar la **Descomposición $QR$**. En teoría hay un método que requiere menos computación, ¿puede mostrarlo experimentalmente?

## Ejemplo 4.16 del libro guía p. 219
### ¿Por qué Gram-Schmidt modificado sobre el clásico?

```python
d = 1e-10
Ab = np.array([
    [1, 1, 1],
    [d, 0, 0],
    [0, d, 0],
    [0, 0, d]
])
```

### Clásico

```python
Qbc, Rbc = QR(Ab, False)
```

```python
print("Q:\n", Qbc)
print("R:\n", Rbc)
```

    Q:
     [[ 1.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 1.00000000e-10 -7.07106781e-01 -7.07106781e-01]
     [ 0.00000000e+00  7.07106781e-01  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  7.07106781e-01]]
    R:
     [[1.00000000e+00 1.00000000e+00 1.00000000e+00]
     [0.00000000e+00 1.41421356e-10 0.00000000e+00]
     [0.00000000e+00 0.00000000e+00 1.41421356e-10]]

¿$Q^*Q = I$?

```python
QTQ_c = np.dot(Qbc.T, Qbc)
QTQ_c
```

    array([[ 1.00000000e+00, -7.07106781e-11, -7.07106781e-11],
           [-7.07106781e-11,  1.00000000e+00,  5.00000000e-01],
           [-7.07106781e-11,  5.00000000e-01,  1.00000000e+00]])

¿$\|Q^*Q - I_n\|_F$?

```python
print(np.linalg.norm(QTQ_c - np.eye(3)))
```

    0.7071067811865477

### Modificado

```python
Qbm, Rbm = QR(Ab)
```

```python
print("Q:\n", Qbm)
print("R:\n", Rbm)
```

    Q:
     [[ 1.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 1.00000000e-10 -7.07106781e-01 -4.08248290e-01]
     [ 0.00000000e+00  7.07106781e-01 -4.08248290e-01]
     [ 0.00000000e+00  0.00000000e+00  8.16496581e-01]]
    R:
     [[1.00000000e+00 1.00000000e+00 1.00000000e+00]
     [0.00000000e+00 1.41421356e-10 7.07106781e-11]
     [0.00000000e+00 0.00000000e+00 1.22474487e-10]]

¿$Q^*Q = I$?

```python
QTQ_m = np.dot(Qbm.T, Qbm)
QTQ_m
```

    array([[ 1.00000000e+00, -7.07106781e-11, -4.08248290e-11],
           [-7.07106781e-11,  1.00000000e+00, -1.66533454e-16],
           [-4.08248290e-11, -1.66533454e-16,  1.00000000e+00]])

¿$\|Q^*Q - I_n\|_F$?

```python
print(np.linalg.norm(QTQ_m - np.eye(3)))
```

    1.1547005383859233e-10

Experimentalmente podemos ver que en este ejemplo los vectores de $Q$ obtenidos con el algoritmo clásico no son ortonormales. Notamos que con el algoritmo modificado no obtenemos $I_n$ exactamente, pero el resultado es mejor.


# Rate of Change
Functions are often visualized as a line on a graph, and this line shows how the value returned by the function changes based on changes in the input value.

## Linear Rate of Change

For example, imagine a function that returns the number of meters travelled by a cyclist based on the number of seconds that the cyclist has been cycling.

Here is such a function:

\begin{equation}q(x) = 2x + 1\end{equation}

We can plot the output for this function for a period of 10 seconds like this:

```python
%matplotlib inline

def q(x):
    return 2*x + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.xticks(range(0,11, 1))
plt.yticks(range(0, 22, 1))
plt.grid()

# Plot x against q(x)
plt.plot(x,q(x), color='green')

plt.show()
```

It's clear from the graph that ***q*** is a *linear* function that describes a slope in which distance increases at a constant rate over time. In other words, the cyclist is travelling at a constant speed.

But what speed?

Speed, or more technically, velocity is a measure of change - it measures how the distance travelled changes over time (which is why we typically express it as a unit of distance per a unit of time, like *miles-per-hour* or *meters-per-second*). So we're looking for a way to measure the change in the line created by the function.

The change in values along the line define its *slope*, which we know from a previous lesson is represented like this:

\begin{equation}m = \frac{\Delta{y}}{\Delta{x}} \end{equation}

We can calculate the slope of our function like this:

\begin{equation}m = \frac{q(x)_{2} - q(x)_{1}}{x_{2} - x_{1}} \end{equation}

So we just need two ordered pairs of ***x*** and ***q(x)*** values from our line to apply this equation.

- After 1 second, ***x*** is 1 and ***q***(1) = **3**.
- After 10 seconds, ***x*** is 10 and ***q***(10) = 21.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{21 - 3}{10 - 1} \end{equation}

This is the same as:

\begin{equation}m = \frac{18}{9} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{2}{1} \end{equation}

So our rate of change is 2/1 or put another way, the cyclist is travelling at 2 meters-per-second.

## Average Rate of Change
OK, let's look at another function that calculates distance travelled for a given number of seconds:

\begin{equation}r(x) = x^{2} + x\end{equation}

Let's take a look at that using Python:

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

plt.show()
```

This time, the function is not linear. It's actually a quadratic function, and the line from 0 seconds to 10 seconds shows an exponential increase; in other words, the cyclist is *accelerating*.

Technically, acceleration itself is a measure of change in velocity over time; and velocity, as we've already discussed, is a measure of change in distance over time. So measuring accelleration is pretty complex, and requires *differential calculus*, which we're going to cover shortly. In fact, even just measuring the velocity at a single point in time requires differential calculus; but we can use algebraic methods to calculate an *average* rate of velocity for a given period shown in the graph.

First, we need to define a *secant* line that joins two points in our exponential arc to create a straight slope. For example, a secant line for the entire 10 second time span would join the following two points:

- 0, ***r***(0)
- 10, ***r***(10)

Run the following Python code to visualize this line:

```python
%matplotlib inline

def r(x):
    return (x)**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([0,10])

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.show()
```

Now, because the secant line is straight, we can apply the slope formula we used for a linear function to calculate the average velocity for the 10 second period:

- At 0 seconds, ***x*** is 0 and ***r***(0) = **0**.
- At 10 seconds, ***x*** is 10 and ***r***(10) = 110.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{110 - 0}{10 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{110}{10} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{11}{1} \end{equation}

So our rate of change is 11/1 or put another way, the cyclist is travelling at an average velocity of 11 meters-per-second over the 10-second period.

Of course, we can measure the average velocity between any two points on the exponential line. Use the following Python code to show the secant line for the period between 2 and 7 seconds, and calculate the average velocity for that period

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([2,7])

# Calculate rate of change
x1 = s[0]
x2 = s[-1]
y1 = r(x1)
y2 = r(x2)
a = (y2 - y1)/(x2 - x1)

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.annotate('Average Velocity = ' + str(a) + ' m/s',((x2+x1)/2, (y2+y1)/2))

plt.show()

```

```python

```


**Imports:**

```python
import numpy as np
from scipy import optimize

import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
prop_cycle = plt.rcParams['axes.prop_cycle']
colors = prop_cycle.by_key()['color']

import ipywidgets as widgets
```

$$
\begin{align}
v(r,m,\Delta) &= \max_{c_{1},c_{2}\in\mathbb{R}_{++}^{2}}\ln(c_{1})+\beta\ln(c_{2}) \\
& \text{u.b.b.} \\
c_{1}+\frac{c_{2}}{1+r}&\leq\Delta m+\frac{(1-\Delta)m}{1+r} \\
\end{align}
$$

# Functions

```python
# solve for consumption
def solve(beta,m,r,Delta_vec):

    w = Delta_vec*m + (1-Delta_vec)*m/(1+r)
    fac = beta*(1+r)
    c1 = w/(1+fac/(1+r))
    c2 = fac*c1

    return c1,c2

# evaluate utility
u_func = lambda c1,c2,beta: np.log(c1) + np.log(c2)
```

```python
beta = 1.0
m = 2.0
ro = 0.0
rl = 1.0
```

```python
Ndelta = 1000
Delta_vec = np.linspace(1e-8,1.0-1e-8,Ndelta)
```

# Plot

```python
fig = plt.figure(figsize=(6,6/1.5),dpi=100)
ax = fig.add_subplot(1,1,1)

# saver
c1,c2 = solve(beta,m,ro,Delta_vec)
I = c1 <= Delta_vec*m
uo = u_func(c1[I],c2[I],beta)
ax.plot(Delta_vec[I],uo,label='utility for saver')

# hand-to-moth
c1 = Delta_vec*m
c2 = (1-Delta_vec)*m
umid = u_func(c1,c2,beta)
ax.plot(Delta_vec,umid,label='utility for hand-to-mouth')

# borrower
c1,c2 = solve(beta,m,rl,Delta_vec)
I = c1 > Delta_vec*m
ul = u_func(c1[I],c2[I],beta)
ax.plot(Delta_vec[I],ul,label='utility for borrow')

# lines
ax.axvline(1/3,color='black',ls='--',label='$\Delta = 1/3$')
ax.axvline(1/2,color='black',ls=':',label='$\Delta = 1/2$')
ax.axhline(-np.log(2),color='black',ls='-.',label='utility = $-\ln2$')

# legend and details
lgd = ax.legend(frameon=True,ncol=1,bbox_to_anchor=(1.05, 1), loc='upper left',)

ax.set_xlim([0.0,1.0])
ax.set_ylim([-5.0,1.0])
ax.set_xlabel('$\Delta$')
ax.set_ylabel('utility');
```

```python
#fig.savefig(f'Lec4_Consumption.pdf', bbox_extra_artists=(lgd,), bbox_inches='tight')
```


```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# set up the grid!
N=200
x_start, x_end= -4.0, 4.0
y_start, y_end= -2.0, 2.0
x=numpy.linspace(x_start, x_end, N)
y=numpy.linspace(y_start, y_end, N)
X, Y=numpy.meshgrid(x, y)

# create the freestream!
u_inf=1.0
u_freestream=u_inf*numpy.ones((N, N), dtype=float)
v_freestream=numpy.zeros((N,N),dtype=float)

# compute the freestream function
psi_freestream=u_inf*Y
```

```python
# get the velocity!

def vel(strength, xs, ys, X, Y):
    u=strength/(2*numpy.pi)*(X-xs)/((X-xs)**2+(Y-ys)**2)
    v=strength/(2*numpy.pi)*(Y-ys)/((X-xs)**2+(Y-ys)**2)
    return u,v
```

```python
# get the stream function!

def stream_func(strength, xs, ys, X, Y):
    psi=strength/(2*numpy.pi)*numpy.arctan2((Y-ys), (X-xs))
    return psi
```

```python
# call everything!

strength_source=5.0
x_source, y_source=-1.0, 0.0

u_source, v_source=vel(strength_source, x_source, y_source, X, Y)
psi_source=stream_func(strength_source, x_source, y_source, X, Y)
```

```python
# superimpose the source of the freestream
u=u_freestream + u_source
v=v_freestream + v_source
psi=psi_freestream + psi_source

width =10
height=5
pyplot.figure(figsize=(width, height))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.streamplot(X, Y, u, v, density=2, linewidth=1, arrowsize=1, arrowstyle='->')
pyplot.scatter(x_source, y_source, color='#CD2305', s=80, marker='o');

# want ot find the stagnation point where u=v=0
# so need to set u and v eqns to 0
x_stag=x_source-(strength_source/(2*numpy.pi*u_inf))
y_stag=y_source
#print(x_stag)

# plot it out!
pyplot.scatter(x_stag, y_stag, color='k', s=80, marker='o')
pyplot.contour(X, Y, psi,levels=[-strength_source / 2, strength_source / 2],
               colors='#CD2305', linewidths=2, linestyles='solid');
```

Challenge: what is the max width of the Rankine Half Body?

Hint: use mass conservation, and think about when x is very large.

=> if we use the law of superposition to get an eqn for the stream function that is based on both the source and the uniform flow, we can then plug in the coordinatess of the stagnation point and the source / sink points to get an equation for the stream function.

=> if we then assume that x approaches infinity then we can say that tan^-1() will approach 0 so the stream function will just be equal to U_inf x y, and we can solve for y which is the width of half the Rankine Half Body.

```python
# solve for the half width of the rankine half body:

stream_s=strength_source/2
h_rank_half=stream_s/u_inf

print('This is half the height of the Rankine Half Body: '+str(h_rank_half))
```

    This is half the height of the Rankine Half Body: 2.5

```python
strength_sink=-5.0
x_sink, y_sink=1.0, 0.0

# get the velocity feild
u_sink, v_sink=vel(strength_sink, x_sink, y_sink, X, Y)
psi_sink=stream_func(strength_sink, x_sink, y_sink, X, Y)

#use superposition:
u=u_freestream + u_source + u_sink
v=v_freestream + v_source + v_sink
psi=psi_freestream + psi_source + psi_sink

# want ot find the stagnation point where u=v=0
# so need to set u and v eqns to 0
m_source=strength_source/(2*numpy.pi)
#print('this is m: '+str(m_source))
m_sink=strength_sink/(2*numpy.pi)
#print('this is m: '+str(m_sink))
x_stag_source=x_source*((1+(2*numpy.abs(m_source)/(numpy.abs(x_source) *u_inf)))**0.5)
#print(x_stag_source)
#print(x_source)
x_stag_sink=x_sink*((1+(2*numpy.abs(m_sink)/(numpy.abs(x_sink) *u_inf)))**0.5)
y_stag=y_source
x_width=x_stag_sink-x_stag_source
#print(x_width)

#plot the streamlines:
width=10
height=5
pyplot.figure(figsize=(width, height))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.streamplot(X, Y, u, v, density=2, linewidth=1, arrowsize=1, arrowstyle='->')
pyplot.scatter([x_source, x_sink], [y_source, y_sink], color='#CD2305', s=80, marker='o');
pyplot.contour(X, Y, psi, levels=[0.0], colors='#CD2305', linewidths=2, linestyles='solid');

# plot it out!
pyplot.scatter(x_stag_source, y_stag, color='k', s=80, marker='o');
pyplot.scatter(x_stag_sink, y_stag, color='k', s=80, marker='o');
```

Challenge Question: what is the length and width of the Rankine oval?

```python
#from sympy import *
# see notes for derivation

# find the width of the rankine oval
rank_w=x_stag_sink-x_stag_source
print('This is the width of the rakine oval: ' +str(rank_w))

'''
# find the height of half of the rankine oval
h=symbols('h')
a=strength_source/(2*numpy.pi*u_inf)
m=strength_source/(2*numpy.pi)
solve((h/numpy.abs(x_source))-(1/(tan((h/a)/(2*m/(u_inf*numpy.abs(x_source)))))), h)
print(h)'''

#plot the streamlines:
width=10
height=5
pyplot.figure(figsize=(width, height))
pyplot.grid(True)
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
pyplot.streamplot(X, Y, u, v, density=2, linewidth=1, arrowsize=1, arrowstyle='->')
pyplot.scatter([x_source, x_sink], [y_source, y_sink], color='#CD2305', s=80, marker='o');
pyplot.contour(X, Y, psi, levels=[0.0], colors='#CD2305', linewidths=2, linestyles='solid');

# plot it out!
pyplot.scatter(x_stag_source, y_stag, color='k', s=80, marker='o');
pyplot.scatter(x_stag_sink, y_stag, color='k', s=80, marker='o');
#pyplot.scatter(psi[100], [100], color='m', s=80, marker='o');
```

```python
# compute the pressure coefficient field
cp=1.0-(u**2 + v**2)/u_inf**2

# plot it out
width = 10
height = (y_end - y_start) / (x_end - x_start) * width
pyplot.figure(figsize=(1.1 * width, height))
pyplot.xlabel('x', fontsize=16)
pyplot.ylabel('y', fontsize=16)
pyplot.xlim(x_start, x_end)
pyplot.ylim(y_start, y_end)
contf = pyplot.contourf(X, Y, cp,
                        levels=numpy.linspace(-2.0, 1.0, 100), extend='both')
cbar = pyplot.colorbar(contf)
cbar.set_label('$C_p$', fontsize=16)
cbar.set_ticks([-2.0, -1.0, 0.0, 1.0])
pyplot.scatter([x_source, x_sink], [y_source, y_sink],
               color='#CD2305', s=80, marker='o')
pyplot.contour(X, Y, psi,
               levels=[0.], colors='#CD2305', linewidths=2, linestyles='solid');
```

```python

```


# From linear to multi-linear regression

In the last notebook we learned about linear regression and using python libraries to make our life simpler. If you want to read about that, have a look at

https://github.com/jegali/DataScience/blob/main/lesson-3-9-ticket-sample.ipynb

The last code snippet dealt with the usage of statsmodel which we will use again in this example.

```python
# a reference to the pandas library
import pandas as pd

# To visualize the data
import matplotlib.pyplot as plt

# This is new. Let's try a library which does
# the linear regression for us
import statsmodels.api as sm

from sklearn import linear_model
from sklearn.linear_model import LinearRegression

# To visualize the data
import matplotlib.pyplot as plt

# the excel file must be in the same directory as this notebook
# be sure to use the right excel data file.
# Udacity has some files named linear-example-data with different content
# I renamed this one to linear-example-data-3-13.xlsx
excel_file= 'linear-example-data-3-13.xlsx'

# via pandas, the contents ae read into a variable or data frame named data
data = pd.read_excel(excel_file)

# let's have a look at the data
print (" Contents of the file ", excel_file)
print(data)

# instead of only using one column for linear regression,
# we use two columns for multilinear regression
# column industry is no number column, so we can not integrate it into the calculation
X = data[['Number of Employees','Value of Contract']]
Y = data['Average Number of Tickets']

# let's do the evaluation with sklearn
# Many roads lead to Rome...
regr = linear_model.LinearRegression()
regr.fit(X, Y)

print ("\n\n")
print ("Linear Regression values evaluated by sklearn: ")
print(" ")
print('Y-intercept: \t', regr.intercept_)
print('Coefficients:\t', regr.coef_)

# let's to the evaluation with statsmodels
# we have to add a constant to the calculation or
# we do not have a Y-intercept
X = sm.add_constant(X)

# build the model
model = sm.OLS(Y,X).fit()
model_prediction = model.predict(X)
model_details = model.summary()

# and print the model summary
print("\n\n")
print("Linear Regression values evaluated by statsmodel: ")
print(" ")
print(model_details)
```

     Contents of the file  linear-example-data-3-13.xlsx
        Average Number of Tickets  Number of Employees  Value of Contract  \
    0                           1                   51              25750
    1                           9                   68              25000
    2                          20                   67              40000
    3                           1                  124              35000
    4                           8                  124              25000
    5                          30                  134              50000
    6                          20                  157              48000
    7                           8                  190              32000
    8                          20                  205              70000
    9                          50                  230              75000
    10                         35                  265              50000
    11                         65                  296              75000
    12                         35                  336              50000
    13                         60                  359              75000
    14                         85                  403              81000
    15                         40                  418              60000
    16                         75                  437              53000
    17                         85                  451              90000
    18                         65                  465              70000
    19                         95                  491             100000

             Industry
    0          Retail
    1        Services
    2        Services
    3          Retail
    4   Manufacturing
    5        Services
    6          Retail
    7          Retail
    8          Retail
    9   Manufacturing
    10  Manufacturing
    11       Services
    12  Manufacturing
    13  Manufacturing
    14       Services
    15         Retail
    16       Services
    17       Services
    18         Retail
    19       Services

    Linear Regression values evaluated by sklearn:

    Y-intercept: 	 -24.26671246720077
    Coefficients:	 [0.10190728 0.00066845]

    Linear Regression values evaluated by statsmodel:

                                    OLS Regression Results
    =====================================================================================
    Dep. Variable:     Average Number of Tickets   R-squared:                       0.878
    Model:                                   OLS   Adj. R-squared:                  0.863
    Method:                        Least Squares   F-statistic:                     60.97
    Date:                       Sun, 27 Dec 2020   Prob (F-statistic):           1.76e-08
    Time:                               14:27:30   Log-Likelihood:                -75.083
    No. Observations:                         20   AIC:                             156.2
    Df Residuals:                             17   BIC:                             159.2
    Df Model:                                  2
    Covariance Type:                   nonrobust
    =======================================================================================
                              coef    std err          t      P>|t|      [0.025      0.975]
    ---------------------------------------------------------------------------------------
    const                 -24.2667      7.020     -3.457      0.003     -39.079      -9.455
    Number of Employees     0.1019      0.028      3.594      0.002       0.042       0.162
    Value of Contract       0.0007      0.000      3.564      0.002       0.000       0.001
    ==============================================================================
    Omnibus:                        0.901   Durbin-Watson:                   2.452
    Prob(Omnibus):                  0.637   Jarque-Bera (JB):                0.511
    Skew:                          -0.384   Prob(JB):                        0.775
    Kurtosis:                       2.851   Cond. No.                     1.70e+05
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
    [2] The condition number is large, 1.7e+05. This might indicate that there are
    strong multicollinearity or other numerical problems.

The model above gave the following equation:

\begin{equation}
Tickets = -24.267 + 0.1019 \cdot [Number \; of \; Employees] + 0.0007 \cdot [Value \; of \; Contract]
\end{equation}

What would the model predict for the number of tickets for a customer with 750 employees and a contract of $13,000?
Inserting the value in the formula we get 61. Below we can see a visualization of the data as a scatter plot

```python
# create an object for linear regression from sklearn
linear_regressor = LinearRegression()

# convert the values in an array that fits the function's need
X = data['Number of Employees'].values.reshape(-1, 1)
Y = data['Average Number of Tickets'].values.reshape(-1, 1)

# and do the regression calculation with sklearn
linear_regressor.fit(X,Y)
# make predictions on Y based on X
Y_pred = linear_regressor.predict(X)

print(" ")
# scatter plot
figure = plt.figure()
ax = figure.add_subplot(1,1,1)
ax.set_ylabel('Avg number of tickets')
ax.set_xlabel('Number of employees')
plt.scatter(X, Y)
# linear regression function
plt.title("Number of employees vs. Avg number of tickets")
plt.plot(X, Y_pred, color='red')

# show the graphs
plt.show()

###############################
###############################
###############################

# convert the values in an array that fits the function's need
X = data['Value of Contract'].values.reshape(-1, 1)
Y = data['Average Number of Tickets'].values.reshape(-1, 1)

# and do the regression calculation with sklearn
linear_regressor.fit(X,Y)
# make predictions on Y based on X
Y_pred = linear_regressor.predict(X)

print(" ")
# scatter plot
figure = plt.figure()
ax = figure.add_subplot(1,1,1)
ax.set_ylabel('Avg number of tickets')
ax.set_xlabel('Value of Contract')
plt.scatter(X, Y)
# linear regression function
plt.title("Value of Contract vs. Avg number of tickets")
plt.plot(X, Y_pred, color='red')

# show the graphs
plt.show()

```

```python

```


Table of Contents

1  Objectives

2  Solving the Line of Best Fit by Guessing

3  The Loss Function

4  The Cost Function

5  Better Way of Guessing: Gradient Descent

5.1  Gradient Descent in Words

5.2  Stepping Down a Hill: Step Size

5.3  Putting It All Together

6  Level Up: Gradient Descent Walk Through

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

# Objectives

- Explain and use the concept of a gradient
- Explain the algorithm of gradient descent
- Describe the effect of the "learning rate" in the algorithm

# Solving the Line of Best Fit by Guessing

Let's say we have some data below:

```python
# Randomly created data in x & y
np.random.seed(27)

x = np.random.rand(30,1).reshape(30)
y_randterm = np.random.normal(0,3,30)
y = 3 + 50 * x + y_randterm
```

Here's the data plotted out:

```python
f, ax = plt.subplots(figsize=(8,6))
ax.scatter(x, y)

ax.set_title('Data Points to Model')
ax.set_xlabel('x', fontsize=14)
ax.set_ylabel('y', fontsize=14)
ax.set_xlim(0,1)
ax.set_ylim(0,60)
plt.tight_layout()
```

If we wanted to make a best-fit line, what would you say it's about? Let's create a couple functions to make this easier to make a guess

```python
# Plotting a guess of a regression line
def regression_formula(x, a, b):
    return a*x + b

def plot_data_and_guess(slope, intercept, ax, x1=x, x2=y, **kwargs):
    '''
    Plot our data and regression line on the given axis.

    Arguments:
        slope : float
            Value for the slope the regression line.

        intercept : float
            Value for the intercept the regression line.

        ax : Axes
            Axis to plot data and regression line

        x1 : array-like
            Values along the x-axis

        x2 : array-like
            Values along the y-axis

    Returns:
        fig : Figure

        ax : Axes
    '''
    # Plot data and regression line
    ax.scatter(x1, x2)
    yhat = regression_formula(x1, slope ,intercept)
    ax.plot(x1, yhat, 'r-', **kwargs)

    # Embelishments
    ax.set_title('Data Points to Model')
    ax.set_xlabel('x', fontsize=14)
    ax.set_ylabel('y', fontsize=14)
    ax.set_xlim(0,1)
    ax.set_ylim(0,60)

    return ax
```

So what do you think the regression parameters are?

```python
# Our guess
guess = {
    'slope': 30,
    'intercept': 0,
    'color':'orange'
}

f, ax = plt.subplots(figsize=(8,6))
plot_data_and_guess(**guess, ax=ax)
```

What would be your next guess be?

- How can we tell when our guess is "better"?
- Could we formalize this?

# The Loss Function

One way we can know how well our guess or _model_ did is to compare the predicted values with the actual values. These are the _residuals_.

So this would give us the error for each data point:

$$ r_i = \hat{y}_i - y_i $$

```python
def calculate_residuals(x_values, y_values, slope, intercept):
    '''Find the residulas for each data point'''
    yhat = intercept + slope*x_values
    errors = y_values - yhat
    return errors
```

This is great but we can go further by having just one number to represent how "bad" or "good" our model was to the real points.

This leads us to the **mean squared error** or **MSE**. This is all the residuals squared and then averaged:

$$ MSE = \frac{1}{n} \sum_{i}^{n} (\hat{y}_i - y_i)^2 $$

```python
def mse(x_values, y_values, slope, intercept):

    resid_sq = calculate_residuals(x_values, y_values, slope, intercept)**2

    return sum(resid_sq)/len(x_values)
```

```python
# Use our guess from earlier
slope = guess.get('slope', 30)
intercept = guess.get('intercept', 0)

mse(x,y,slope,intercept)
```

    262.114879650247

> The function we use to find how bad our model did in prediction is typically called the **loss function**

What we found here is great! We can now compare different models with one another.

If we made a few different guesses, we could make our predictions and then calculate from the _loss function_ how good or bad our model did! We will want to find the _smallest loss_.

# The Cost Function

Now our model changes based on the different model _parameters_ (the coefficients $\beta_i$ for linear regression).

If we imagine all the different ways we can adjust these parameters $\vec{\theta}$ and measure how well the model with the loss or **cost function** $J(\vec{\theta})$, we can plot this as a surface in this multidimensional plane. See the image below:

> Note that the terms **loss function** and **cost function** are frequently used interchangeably. Sometimes they are the same function, but sometimes they differ by making changes in the cost to improve _training_ or _learning_.

Let's try creating the cost function's curve/surface for just one parameter (slope) using our earlier data example:

```python
table = np.zeros((20,2))
# Find the MSE for different slope values
for idx, val in enumerate(range(40, 60)):
    table[idx,0] = val
    table[idx,1] = mse(x, y, slope=val, intercept=0)

plt.figure(figsize=(10,7))
plt.plot(table[:,0], table[:,1], '-')
plt.xlabel("Slope Values", fontsize=14)
plt.ylabel("MSE", fontsize=14)
plt.title("MSE with changes to slope", fontsize=16);
```

Based on this graph, what is the optimal slope value?

How could we extend this to find the best slope _and_ intercept combination?

# Better Way of Guessing: Gradient Descent

So this probably all sounds great! We just need to find the minimum of the cost function!

But there's some bad news; we don't usually know what the cost function (which can be complicated!) "looks" like without trying a whole lot of different parameters $\vec{\theta}$. We'd need an _infinite_ number of parameter combinations to know $J(\vec{\theta})$ completely.

So what can we do?

Well, we can take one "guess" (set of  parameters) and then measure $J(\vec{\theta})$. Then we can adjust our guess/parameters in a "good" direction, "down the hill". This is the basic idea of gradient descent.

> **Gradient descent** is an optimization procedure that uses the _gradient_ (a generalized notion of a derivative) of the cost function.

So how do find this "better" guess? Well, we need to find the best direction to move "downhill" the fastest. We can do this with a generalization of the derivative called the **gradient**:

$$\begin{align}\\
    \large -\nabla J &= \sum_i \dfrac{\partial J}{\partial \theta_i} \\
            &= \frac{\partial J}{\partial \theta_1} +  \frac{\partial J}{\partial \theta_2} + \dots +  \frac{\partial J}{\partial \theta_n}
\end{align}$$

In the multivariate case, the gradient tells us how the function is changing **in each dimension**. A large value of the derivative with respect to a particular variable means that the gradient will have a large component in the corresponding direction. Therefore, **the gradient will point in the direction of steepest increase**.

## Gradient Descent in Words

- Make a guess at where the function attains its minimum value
- Calculate the gradient/derivative at that point
- Use that value to decide how to make your next guess!

Repeat until we get the derivative as close as we like to 0.

If we want to improve our guess at the minimum of our loss function, we'll move in the **opposite direction** of the gradient away from our last guess. Hence we are using the *gradient* of our loss function to *descend* to the minimum value of the relevant loss function.

## Stepping Down a Hill: Step Size

So we now have the basic idea of gradient descent of "going down a hill" and hopefully it's obvious that the steeper the hill, the more we can adjust our parameters to get to "bottom" (optimal parameters) faster.

But a big question is how big of a step do we take?

> The amount we adjust our parameter is determined by out **step size** or **learning rate** $\alpha$

If our steps are _too big_, we risk skipping over the minimum value (optimal parameters).

If our steps are _too small_, it might take us too long to reach the minimum value.

Here's an elegant solution: Make the size of your step **proportional to the value of the derivative at the point where you currently are in parameter space**! If we're very far from the minimum, then our values will be large, and so we therefore can safely take a large step; if we're close to the minimum, then our values will be small, and so we should therefore take a smaller step.

I said the size of the step is proportional to the value of the derivative. The constant of proportionality is often called the **"learning rate"**.

This page helps to explain the dangers of learning rates that are too large and too small: https://www.jeremyjordan.me/nn-learning-rate/.

> Note there are other optimizations we can do for gradient descent and rely on adjusting our cost function or how we take steps or both.

## Putting It All Together

The general algorithm looks like this:

We'll make a guess, $\vec{s}$, at where our loss function attains a minimum. If we're not happy with how close the value of the gradient there is to 0, then we'll make a new guess, and the new guess will be constructed as follows:

$\large\vec{s}_{new} = \vec{s}_{old} - \alpha\nabla f(\vec{s}_{old})$,

where $\alpha$ is the learning rate.

In the one-dimensional case, we'll have:

$\large x_{new} = x_{old} - \alpha\frac{df}{dx}|_{x_{old}}$.

# Level Up: Gradient Descent Walk Through

Let's go back to our original example and implement gradient descent to find the optimal parameters (slope and intercept)

```python
f, ax = plt.subplots(figsize=(8,6))
ax.scatter(x, y)

ax.set_title('Data Points to Model')
ax.set_xlabel('x', fontsize=14)
ax.set_ylabel('y', fontsize=14)

plt.tight_layout()
```

First we need to find the gradient for each cost function (2-dimensions: a & b; slope & intercept):

$$ \frac{\partial}{\partial b} (y_i - (b + ax_i))^2 = -2 \cdot (y_i-ax_i - b)$$

$$ \frac{\partial}{\partial a} (y_i - (b + ax_i))^2 = -2 \cdot x_i \cdot (y_i-ax_i - b)$$

Let's formalize this into a function:

```python
def partial_deriv(a, b, x_i, y_i, respect_to):
    '''
    Get the partial derivative for cost function with respect to slope (a)
    or intercept (b).
    '''
    if respect_to == 'b': # intercept
        return (y_i - (a * x_i + b))
    elif respect_to == 'a': # slope
        return (x_i * (y_i - (a * x_i + b)))
    else:
        print('Choose either respect_to: a or b ')
        return 0
```

Next let's define the step we take (amount we adjust the parameters by) using the gradient and learning rate:

```python
def step_gradient(a, b, x, y, learning_rate):
    db = 0
    da = 0
    # For each data point, update the derivative for the slope & intercept
    N = len(x)
    for i in range(len(x)):
        # Derivatives already pre-done

        # Partial derivatives of loss/cost function with respect to b & a
        db +=  -(2/N) * partial_deriv(a,b,x[i],y[i],respect_to='b')
        da +=  -(2/N) * partial_deriv(a,b,x[i],y[i],respect_to='a')

    # Adjust the slope & intercept by the gradient
    new_b = b - (learning_rate * db)
    new_a = a - (learning_rate * da)

    return (new_a, new_b)
```

Let's try it out and keep track of our guesses

```python
guesses = []
alpha = 0.05
```

```python
# Our guess
guess = {
    'slope': 60,
    'intercept': 10
}

guesses.append(guess)
```

```python
f, ax = plt.subplots(figsize=(8,6))
plot_data_and_guess(**guess, ax=ax);
```

```python
step = step_gradient(guess['slope'], guess['intercept'], x, y, learning_rate=alpha)
step
```

    (59.32566774732662, 8.907387521525408)

```python
mse(x, y, guess['slope'], guess['intercept'])
```

    139.0871456161772

Let's update our guess and try again:

```python
# Our guess using the new step
new_slope, new_intercept = step
guess = {
    'slope': new_slope,
    'intercept': new_intercept
}
guesses.append(guess)

# Getting adjusted parameters
step = step_gradient(guess['slope'], guess['intercept'], x, y, learning_rate=alpha)
display(step)
display(mse(x, y, guess['slope'], guess['intercept']))

# Plotting out our new parameters
f, ax = plt.subplots(figsize=(8,6))
plot_data_and_guess(**guess, ax=ax);
```

Let's repeat this another 200 times:

```python
for i in range(200):
    # Our guess using the new step
    new_slope, new_intercept = step
    guess = {
        'slope': new_slope,
        'intercept': new_intercept
    }
    guesses.append(guess)

    # Getting adjusted parameters
    step = step_gradient(guess['slope'], guess['intercept'], x, y, learning_rate=alpha)
    #  Only display every 10
    if (i % 10) == 0:
        print(f'Step # {i}:')
        display(step)
        display(mse(x, y, guess['slope'], guess['intercept']))
        print('-'*30)
```

    Step # 0:

    (58.213746527513386, 7.138241345889149)

    84.91095906900486

    ------------------------------
    Step # 10:

    (55.406396713236255, 3.1147773046958607)

    16.41718043153871

    ------------------------------
    Step # 20:

    (54.433474791049576, 2.271359654686939)

    11.841733247687406

    ------------------------------
    Step # 30:

    (53.933583029310945, 2.2080909509192184)

    11.253138292463875

    ------------------------------
    Step # 40:

    (53.567441379520695, 2.329421747559343)

    10.947887734613678

    ------------------------------
    Step # 50:

    (53.24974627275583, 2.4879882440187626)

    10.692893172824675

    ------------------------------
    Step # 60:

    (52.958349619022925, 2.6477256203265704)

    10.469777489768061

    ------------------------------
    Step # 70:

    (52.68685224086262, 2.800183114155367)

    10.27389987543192

    ------------------------------
    Step # 80:

    (52.4328236504871, 2.9437376980266974)

    10.101894504303145

    ------------------------------
    Step # 90:

    (52.194872131866866, 3.0784324862477908)

    9.950849482942244

    ------------------------------
    Step # 100:

    (51.97191358189265, 3.2046964099530997)

    9.818210455908313

    ------------------------------
    Step # 110:

    (51.76298676241705, 3.323027933183189)

    9.701734502082958

    ------------------------------
    Step # 120:

    (51.56720448432197, 3.4339181387188096)

    9.599452027603196

    ------------------------------
    Step # 130:

    (51.383738742761345, 3.5378331724915375)

    9.509633449207007

    ------------------------------
    Step # 140:

    (51.21181445859029, 3.6352113359041236)

    9.430759946523628

    ------------------------------
    Step # 150:

    (51.05070551910597, 3.726463725462493)

    9.361497778978336

    ------------------------------
    Step # 160:

    (50.899731541232455, 3.811975667931929)

    9.30067573241181

    ------------------------------
    Step # 170:

    (50.75825495591875, 3.892108272825734)

    9.247265314002345

    ------------------------------
    Step # 180:

    (50.62567830516174, 3.9671999388050314)

    9.200363360587245

    ------------------------------
    Step # 190:

    (50.50144171632311, 4.037567778126529)

    9.159176766292335

    ------------------------------

What does our final result look like?

```python
# Plotting out our new parameters
f, ax = plt.subplots(figsize=(8,6))
plot_data_and_guess(**guesses[-1], ax=ax);
```

Let's take a look at the MSE over the guesses:

```python
mses = [
    mse(x,y,d['slope'],d['intercept']) for d in guesses
]
plt.plot(range(len(mses)),mses)
```

This was what we had for 200 iterations. What could we do to improve or speed up this method?