###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth, C.D. Cooper.

# Spreading out

We're back! This is the fourth notebook of _Spreading out: parabolic PDEs,_  Module 4 of the course [**"Practical Numerical Methods with Python"**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about).

In the [previous notebook](http://nbviewer.ipython.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/04_spreadout/03_Heat_Equation_2D_Explicit.ipynb), we solved a 2D problem for the first time, using an explicit scheme. We know explicit schemes have stability constraints that might make them impractical in some cases, due to requiring a very small time step. Implicit schemes are unconditionally stable, offering the advantage of larger time steps; in [notebook 2](https://github.com/numerical-mooc/numerical-mooc/blob/master/lessons/04_spreadout/02_Heat_Equation_1D_Implicit.ipynb), we look at the 1D implicit solution of diffusion. Already, that was quite a lot of work: setting up a matrix of coefficients and a right-hand-side vector, while taking care of the boundary conditions, and then solving the linear system. And now, we want to do implicit schemes in 2D—are you ready for this challenge?

## 2D Heat conduction

We already studied 2D heat conduction in the previous lesson, but now we want to work out how to build an implicit solution scheme. To refresh your memory, here is the heat equation again:

\begin{equation}
\frac{\partial T}{\partial t} = \alpha \left(\frac{\partial^2 T}{\partial x^2} + \frac{\partial^2 T}{\partial y^2} \right)
\end{equation}

Our previous solution used a Dirichlet boundary condition on the left and bottom boundaries, with $T(x=0)=T(y=0)=100$, and a Neumann boundary condition with zero flux on the top and right edges, with $q_x=q_y=0$.

$$
\left( \left.\frac{\partial T}{\partial y}\right|_{y=0.1} = q_y \right) \quad \text{and} \quad \left( \left.\frac{\partial T}{\partial x}\right|_{x=0.1} = q_x \right)$$

Figure 1 shows a sketch of the problem set up for our hypothetical computer chip with two hot edges and two insulated edges.

####   Figure 1: Simplified microchip problem setup.

### Implicit schemes in 2D

An implicit discretization will evaluate the spatial derivatives at the next time level, $t^{n+1}$, using the unknown values of the solution variable. For the 2D heat equation with central difference in space, that is written as:

\begin{eqnarray}
\frac{T^{n+1}_{i,j} - T^n_{i,j}}{\Delta t} & \nonumber \\
= \alpha &\left( \frac{T^{n+1}_{i+1, j}
- 2T^{n+1}_{i,j} + T^{n+1}_{i-1,j}}{\Delta x^2} + \frac{T^{n+1}_{i, j+1} - 2T^{n+1}_{i,j} + T^{n+1}_{i,j-1}}{\Delta y^2}\right)
\end{eqnarray}

This equation looks better when we put what we *don't know* on the left and what we *do know* on the right. Make sure to work this out yourself on a piece of paper.

\begin{eqnarray}
-\frac{\alpha\Delta t}{\Delta x^2} \left(T^{n+1}_{i-1,j}+T^{n+1}_{i+1,j}\right) &+ \left(1 + 2\frac{\alpha\Delta t}{\Delta x^2} + 2\frac{\alpha\Delta t}{\Delta y^2}\right) T^{n+1}_{i,j} \nonumber \\
&-\frac{\alpha\Delta t}{\Delta y^2} \left(T^{n+1}_{i,j-1}+T^{n+1}_{i,j+1}\right) = T^n_{i,j}
\end{eqnarray}

To make this discussion easier, let's assume that the mesh spacing is the same in both directions and $\Delta x=\Delta y = \delta$:

\begin{equation}
-T^{n+1}_{i-1,j} - T^{n+1}_{i+1,j} + \left(\frac{\delta^2}{\alpha \Delta t} + 4 \right) T^{n+1}_{i,j} - T^{n+1}_{i,j-1}-T^{n+1}_{i,j+1} = \frac{\delta^2}{\alpha \Delta t}T^n_{i,j}
\end{equation}

Just like in the one-dimensional case, $T_{i,j}$ appears in the equation for $T_{i-1,j}$, $T_{i+1,j}$, $T_{i,j+1}$ and $T_{i,j-1}$, and we can form a linear system to advance in time. But, how do we construct the matrix in this case? What are the $(i+1,j)$, $(i-1,j)$, $(i,j+1)$, and $(i,j-1)$ positions in the matrix?

With explicit schemes we don't need to worry about these things. We can lay out the data just as it is in the physical problem.  We had an array `T` that was a 2-dimensional matrix. To fetch the temperature in the next node in the $x$ direction $(T_{i+1,j})$ we just did `T[j,i+1]`, and likewise in the $y$ direction $(T_{i,j+1})$ was in `T[j+1,i]`. In implicit schemes, we need to think a bit harder about how the data is mapped to the physical problem.

Also, remember from the [notebook on 1D-implicit schemes](https://github.com/numerical-mooc/numerical-mooc/blob/master/lessons/04_spreadout/02_Heat_Equation_1D_Implicit.ipynb) that the linear system had $N-2$ elements? We applied boundary conditions on nodes $i=0$ and $i=N-1$, and they were not modified by the linear system. In 2D, this becomes a bit more complicated.

Let's use Figure 1, representing a set of grid nodes in two dimensions, to guide the discussion.

####   Figure 2: Layout of matrix elements in 2D problem

Say we have the 2D domain of size $L_x\times L_y$ discretized in $n_x$ and $n_y$ points. We can divide the nodes into boundary nodes (empty circles) and interior nodes (filled circles).

The boundary nodes, as the name says, are on the boundary. They are the nodes with indices $(i=0,j)$, $(i=n_x-1,j)$, $(i,j=0)$, and $(i,j=n_y-1)$, and boundary conditions are enforced there.

The interior nodes are not on the boundary, and the finite-difference equation acts on them. If we leave the boundary nodes aside for the moment, then the grid will have $(n_x-2)\cdot(n_y-2)$ nodes that need to be updated on each time step. This is the number of unknowns in the linear system. The matrix of coefficients will have $\left( (n_x-2)\cdot(n_y-2) \right)^2$ elements (most of them zero!).

To construct the matrix, we will iterate over the nodes in an x-major order: index $i$ will run faster. The order will be

*  $(i=1,j=1)$
*  $(i=2,j=1)$ ...
*  $(i=nx-2,j=1)$
*  $(i=1,j=2)$
*  $(i=2,j=2)$ ...
*  $(i=n_x-2,j=n_y-2)$.

That is the ordering represented by dotted line on Figure 1. Of course, if you prefer to organize the nodes differently, feel free to do so!

Because we chose this ordering, the equation for nodes $(i-1,j)$ and $(i+1,j)$ will be just before and after $(i,j)$, respectively. But what about $(i,j-1)$ and $(i,j+1)$? Even though in the physical problem they are very close, the equations are $n_x-2$ places apart! This can tie your head in knots pretty quickly.

_The only way to truly understand it is to make your own diagrams and annotations on a piece of paper and reconstruct this argument!_

### Boundary conditions

Before we attempt to build the matrix, we need to think about boundary conditions. There is some bookkeeping to be done here, so bear with us for a moment.

Say, for example, that the left and bottom boundaries have Dirichlet boundary conditions, and the top and right boundaries have Neumann boundary conditions.

Let's look at each case:

**Bottom boundary:**

The equation for $j=1$ (interior points adjacent to the bottom boundary) uses values from $j=0$, which are known. Let's put that on the right-hand side of the equation. We get this equation for all points across the $x$-axis that are adjacent to the bottom boundary:

\begin{equation}
-T^{n+1}_{i-1,1} - T^{n+1}_{i+1,1} + \left(\frac{\delta^2}{\alpha \Delta t} + 4 \right) T^{n+1}_{i,1}-T^{n+1}_{i,j+1} = \frac{\delta^2}{\alpha \Delta t}T^n_{i,1} + T^{n+1}_{i,0}
\end{equation}

**Left boundary:**

Like for the bottom boundary, the equation for $i=1$ (interior points adjacent to the left boundary) uses known values from $i=0$, and we will put that on the right-hand side:

\begin{equation}
- T^{n+1}_{2,j} + \left(\frac{\delta^2}{\alpha \Delta t} + 4 \right) T^{n+1}_{1,j} - T^{n+1}_{1,j-1}-T^{n+1}_{1,j+1} = \frac{\delta^2}{\alpha \Delta t}T^n_{1,j} + T^{n+1}_{0,j}
\end{equation}

**Right boundary:**

Say the boundary condition is $\left. \frac{\partial T}{\partial x} \right|_{x=L_x} = q_x$. Its finite-difference approximation  is

\begin{equation}
\frac{T^{n+1}_{n_x-1,j} - T^{n+1}_{n_x-2,j}}{\delta} = q_x
\end{equation}

We can write $T^{n+1}_{n_x-1,j} = \delta q_x + T^{n+1}_{n_x-2,j}$ to get the finite difference equation for $i=n_x-2$:

\begin{eqnarray}
-T^{n+1}_{n_x-3,j} + \left(\frac{\delta^2}{\alpha \Delta t} + 3 \right) T^{n+1}_{n_x-2,j} - T^{n+1}_{n_x-2,j-1}-T^{n+1}_{n_x-2,j+1} \nonumber\\
= \frac{\delta^2}{\alpha \Delta t}T^n_{n_x-2,j} + \delta q_x
\end{eqnarray}

Not sure about this? Grab pen and paper! _Please_, check this yourself. It will help you understand!

**Top boundary:**

Neumann boundary conditions specify the derivative normal to the boundary: $\left. \frac{\partial T}{\partial y} \right|_{y=L_y} = q_y$. No need to repeat what we did for the right boundary, right? The equation for $j=n_y-2$ is

\begin{eqnarray}
-T^{n+1}_{i-1,n_y-2} - T^{n+1}_{i+1,n_y-2} + \left(\frac{\delta^2}{\alpha \Delta t} + 3 \right) T^{n+1}_{i,n_y-2} - T^{n+1}_{i,n_y-3} \nonumber\\
=\frac{\delta^2}{\alpha \Delta t}T^n_{i,n_y-2} + \delta q_y
\end{eqnarray}

So far, we have then 5 possible cases: bottom, left, right, top, and interior points. Does this cover everything? What about corners?

**Bottom-left corner**

At $T_{1,1}$ there is a Dirichlet boundary condition at $i=0$ and $j=0$. This equation is:

\begin{eqnarray}
- T^{n+1}_{2,1} + \left(\frac{\delta^2}{\alpha \Delta t} + 4 \right) T^{n+1}_{1,1} - T^{n+1}_{1,2} & \nonumber \\
= \frac{\delta^2}{\alpha \Delta t}T^n_{1,1} + T^{n+1}_{0,1} + T^{n+1}_{1,0} &
\end{eqnarray}

**Top-left corner:**

At $T_{1,n_y-2}$ there is a Dirichlet boundary condition at $i=0$ and a Neumann boundary condition at $i=n_y-1$. This equation is:

\begin{eqnarray}
- T^{n+1}_{2,n_y-2} + \left(\frac{\delta^2}{\alpha \Delta t} + 3 \right) T^{n+1}_{1,n_y-2} - T^{n+1}_{1,n_y-3} &\nonumber \\
= \frac{\delta^2}{\alpha \Delta t}T^n_{1,n_y-2} + T^{n+1}_{0,n_y-2} + \delta q_y &
\end{eqnarray}

**Top-right corner**

At $T_{n_x-2,n_y-2}$, there are Neumann boundary conditions at both $i=n_x-1$ and $j=n_y-1$. The finite difference equation is then

\begin{eqnarray}
-T^{n+1}_{n_x-3,n_y-2} + \left(\frac{\delta^2}{\alpha \Delta t} + 2 \right) T^{n+1}_{n_x-2,n_y-2} - T^{n+1}_{n_x-2,n_y-3} &\nonumber \\
= \frac{\delta^2}{\alpha \Delta t}T^n_{n_x-2,n_y-2} + \delta(q_x + q_y)&
\end{eqnarray}

**Bottom-right corner**

To calculate $T_{n_x-2,1}$ we need to consider a Dirichlet boundary condition to the bottom and a Neumann boundary condition to the right. We will get a similar equation to the top-left corner!

\begin{eqnarray}
-T^{n+1}_{n_x-3,1} + \left(\frac{\delta^2}{\alpha \Delta t} + 3 \right) T^{n+1}_{n_x-2,1}-T^{n+1}_{n_x-2,2} & \nonumber \\
= \frac{\delta^2}{\alpha \Delta t}T^n_{n_x-2,1} + T^{n+1}_{n_x-2,0} + \delta q_x &
\end{eqnarray}

Okay, now we are actually ready. We have checked every possible case!

### The linear system

Like in the previous lesson introducing implicit schemes, we will solve a linear system at every time step:

$$[A][T^{n+1}_\text{int}] = [b]+[b]_{b.c.}$$

The coefficient matrix now takes some more work to figure out and to build in code. There is no substitute for you working this out patiently on paper!

The structure of the matrix can be described as a series of diagonal blocks, and lots of zeroes elsewhere. Look at Figure 3, representing the block structure of the coefficient matrix, and refer back to Figure 2, showing the discretization grid in physical space. The first row of interior points, adjacent to the bottom boundary, generates the matrix block labeled $A_1$. The top row of interior points, adjacent to the top boundary generates the matrix block labeled $A_3$. All other interior points in the grid generate similar blocks, labeled $A_2$ on Figure 3.

####   Figure 3: Sketch of coefficient-matrix blocks.

####   Figure 4: Grid points corresponding to each matrix-block type.

The matrix block $A_1$ is

The block matrix $A_2$ is

The block matrix $A_3$ is

Vector $T^{n+1}_\text{int}$ contains the temperature of the interior nodes in the next time step. It is:

\begin{equation}
T^{n+1}_\text{int} = \left[
\begin{array}{c}
T^{n+1}_{1,1}\\
T^{n+1}_{2,1} \\
\vdots \\
T^{n+1}_{n_x-2,1} \\
T^{n+1}_{2,1} \\
\vdots \\
T^{n+1}_{n_x-2,n_y-2}
\end{array}\right]
\end{equation}

Remember the x-major ordering we chose!

Finally, the right-hand side is
\begin{equation}
[b]+[b]_{b.c.} =
\left[\begin{array}{c}
\sigma^\prime T^n_{1,1} + T^{n+1}_{0,1} + T^{n+1}_{1,0} \\
\sigma^\prime T^n_{2,0} + T^{n+1}_{2,0} \\
\vdots \\
\sigma^\prime T^n_{n_x-2,1} + T^{n+1}_{n_x-2,0} + \delta q_x \\
\sigma^\prime T^n_{1,2} + T^{n+1}_{0,2} \\
\vdots \\
\sigma^\prime T^n_{n_x-2,n_y-2} + \delta(q_x + q_y)
\end{array}\right]
\end{equation}

where $\sigma^\prime = 1/\sigma = \delta^2/\alpha \Delta t$. The matrix looks very ugly, but it is important you understand it! Think about it. Can you answer:
   * Why a -1 factor appears $n_x-2$ columns after the diagonal? What about $n_x-2$ columns before the diagonal?
   * Why in row $n_x-2$ the position after the diagonal contains a 0?
   * Why in row $n_x-2$ the diagonal is $\sigma^\prime + 3$ rather than $\sigma^\prime + 4$?
   * Why in the last row the diagonal is $\sigma^\prime + 2$ rather than $\sigma^\prime + 4$?

If you can answer those questions, you are in good shape to continue!

Let's write a function that will generate the matrix and right-hand side for the heat conduction problem in the previous notebook. Remember, we had Dirichlet boundary conditions in the left and bottom, and zero-flux Neumann boundary condition on the top and right $(q_x=q_y=0)$.

Also, we'll import `scipy.linalg.solve` because we need to solve a linear system.

```
import numpy
from scipy.linalg import solve
```

```
def constructMatrix(nx, ny, sigma):
    """ Generate implicit matrix for 2D heat equation with Dirichlet in bottom and right and Neumann in top and left
        Assumes dx = dy

    Parameters:
    ----------
    nx   : int
        number of discretization points in x
    ny   : int
        number of discretization points in y
    sigma: float
        alpha*dt/dx

    Returns:
    -------
    A: 2D array of floats
        Matrix of implicit 2D heat equation
    """

    A = numpy.zeros(((nx-2)*(ny-2),(nx-2)*(ny-2)))

    row_number = 0 # row counter
    for j in range(1,ny-1):
        for i in range(1,nx-1):

            # Corners
            if i==1 and j==1: # Bottom left corner (Dirichlet down and left)
                A[row_number,row_number] = 1/sigma+4 # Set diagonal
                A[row_number,row_number+1] = -1      # fetch i+1
                A[row_number,row_number+nx-2] = -1   # fetch j+1

            elif i==nx-2 and j==1: # Bottom right corner (Dirichlet down, Neumann right)
                A[row_number,row_number] = 1/sigma+3 # Set diagonal
                A[row_number,row_number-1] = -1      # Fetch i-1
                A[row_number,row_number+nx-2] = -1   # fetch j+1

            elif i==1 and j==ny-2: # Top left corner (Neumann up, Dirichlet left)
                A[row_number,row_number] = 1/sigma+3   # Set diagonal
                A[row_number,row_number+1] = -1        # fetch i+1
                A[row_number,row_number-(nx-2)] = -1   # fetch j-1

            elif i==nx-2 and j==ny-2: # Top right corner (Neumann up and right)
                A[row_number,row_number] = 1/sigma+2   # Set diagonal
                A[row_number,row_number-1] = -1        # Fetch i-1
                A[row_number,row_number-(nx-2)] = -1   # fetch j-1

            # Sides
            elif i==1: # Left boundary (Dirichlet)
                A[row_number,row_number] = 1/sigma+4 # Set diagonal
                A[row_number,row_number+1] = -1      # fetch i+1
                A[row_number,row_number+nx-2] = -1   # fetch j+1
                A[row_number,row_number-(nx-2)] = -1 # fetch j-1

            elif i==nx-2: # Right boundary (Neumann)
                A[row_number,row_number] = 1/sigma+3 # Set diagonal
                A[row_number,row_number-1] = -1      # Fetch i-1
                A[row_number,row_number+nx-2] = -1   # fetch j+1
                A[row_number,row_number-(nx-2)] = -1 # fetch j-1

            elif j==1: # Bottom boundary (Dirichlet)
                A[row_number,row_number] = 1/sigma+4 # Set diagonal
                A[row_number,row_number+1] = -1      # fetch i+1
                A[row_number,row_number-1] = -1      # fetch i-1
                A[row_number,row_number+nx-2] = -1   # fetch j+1

            elif j==ny-2: # Top boundary (Neumann)
                A[row_number,row_number] = 1/sigma+3 # Set diagonal
                A[row_number,row_number+1] = -1      # fetch i+1
                A[row_number,row_number-1] = -1      # fetch i-1
                A[row_number,row_number-(nx-2)] = -1 # fetch j-1

            # Interior points
            else:
                A[row_number,row_number] = 1/sigma+4 # Set diagonal
                A[row_number,row_number+1] = -1      # fetch i+1
                A[row_number,row_number-1] = -1      # fetch i-1
                A[row_number,row_number+nx-2] = -1   # fetch j+1
                A[row_number,row_number-(nx-2)] = -1 # fetch j-1

            row_number += 1 # Jump to next row of the matrix!

    return A
```

```
def generateRHS(nx, ny, sigma, T, T_bc):
    """ Generates right-hand side for 2D implicit heat equation with Dirichlet in bottom and left and Neumann in top and right
        Assumes dx=dy, Neumann BCs = 0, and constant Dirichlet BCs

        Paramenters:
        -----------
        nx   : int
            number of discretization points in x
        ny   : int
            number of discretization points in y
        sigma: float
            alpha*dt/dx
        T    : array of float
            Temperature in current time step
        T_bc : float
            Temperature in Dirichlet BC

        Returns:
        -------
        RHS  : array of float
            Right hand side of 2D implicit heat equation
    """
    RHS = numpy.zeros((nx-2)*(ny-2))

    row_number = 0 # row counter
    for j in range(1,ny-1):
        for i in range(1,nx-1):

            # Corners
            if i==1 and j==1: # Bottom left corner (Dirichlet down and left)
                RHS[row_number] = T[j,i]*1/sigma + 2*T_bc

            elif i==nx-2 and j==1: # Bottom right corner (Dirichlet down, Neumann right)
                RHS[row_number] = T[j,i]*1/sigma + T_bc

            elif i==1 and j==ny-2: # Top left corner (Neumann up, Dirichlet left)
                RHS[row_number] = T[j,i]*1/sigma + T_bc

            elif i==nx-2 and j==ny-2: # Top right corner (Neumann up and right)
                RHS[row_number] = T[j,i]*1/sigma

            # Sides
            elif i==1: # Left boundary (Dirichlet)
                RHS[row_number] = T[j,i]*1/sigma + T_bc

            elif i==nx-2: # Right boundary (Neumann)
                RHS[row_number] = T[j,i]*1/sigma

            elif j==1: # Bottom boundary (Dirichlet)
                RHS[row_number] = T[j,i]*1/sigma + T_bc

            elif j==ny-2: # Top boundary (Neumann)
                RHS[row_number] = T[j,i]*1/sigma

            # Interior points
            else:
                RHS[row_number] = T[j,i]*1/sigma

            row_number += 1 # Jump to next row!

    return RHS
```

The solution of the linear system $(T^{n+1}_\text{int})$ contains the temperatures of the interior points at the next time step in a 1D array. We will also create a function that will take the values of $T^{n+1}_\text{int}$ and put them in a 2D array that resembles the physical domain.

```
def map_1Dto2D(nx, ny, T_1D, T_bc):
    """ Takes temperatures of solution of linear system, stored in 1D,
    and puts them in a 2D array with the BCs
    Valid for constant Dirichlet bottom and left, and Neumann with zero
    flux top and right

    Parameters:
    ----------
        nx  : int
            number of nodes in x direction
        ny  : int
            number of nodes in y direction
        T_1D: array of floats
            solution of linear system
        T_bc: float
            Dirichlet BC

    Returns:
    -------
        T: 2D array of float
            Temperature stored in 2D array with BCs
    """
    T = numpy.zeros((ny,nx))

    row_number = 0
    for j in range(1,ny-1):
        for i in range(1,nx-1):
            T[j,i] = T_1D[row_number]
            row_number += 1
    # Dirichlet BC
    T[0,:] = T_bc
    T[:,0] = T_bc
    #Neumann BC
    T[-1,:] = T[-2,:]
    T[:,-1] = T[:,-2]

    return T
```

And to advance in time, we will use

```
def btcs_2D(T, A, nt, sigma, T_bc, nx, ny, dt):
    """ Advances diffusion equation in time with backward Euler

    Parameters:
    ----------
    T: 2D array of float
        initial temperature profile
    A: 2D array of float
        Matrix with discretized diffusion equation
    nt: int
        number of time steps
    sigma: float
        alpha*dt/dx^2
    T_bc : float
        Dirichlet BC temperature
    nx   : int
        Discretization points in x
    ny   : int
        Discretization points in y
    dt   : float
        Time step size

    Returns:
    -------
    T: 2D array of floats
        temperature profile after nt time steps
    """

    j_mid = (numpy.shape(T)[0])/2
    i_mid = (numpy.shape(T)[1])/2

    for t in range(nt):
        Tn = T.copy()
        b = generateRHS(nx, ny, sigma, Tn, T_bc)
        # Use numpy.linalg.solve
        T_interior = solve(A,b)
        T = map_1Dto2D(nx, ny, T_interior, T_bc)

        # Check if we reached T=70C
        if T[j_mid, i_mid] >= 70:
            print ("Center of plate reached 70C at time {0:.2f}s, in time step {1:d}.".format(dt*t, t))
            break

    if T[j_mid, i_mid]<70:
        print ("Center has not reached 70C yet, it is only {0:.2f}C.".format(T[j_mid, i_mid]))

    return T
```

Remember, we want the function to tell us when the center of the plate reaches $70^\circ C$.

##### Dig deeper

For demonstration purposes, these functions are very explicit. But you can see a trend here, right?

Say we start with a matrix with `1/sigma+4` in the main diagonal, and `-1` on the 4 other corresponding diagonals. Now, we have to modify the matrix only where the boundary conditions are affecting. We saw the impact of the Dirichlet and Neumann boundary condition on each position of the matrix, we just need to know in which position to perform those changes.

A function that maps `i` and `j` into `row_number` would be handy, right? How about `row_number = (j-1)*(nx-2)+(i-1)`? By feeding `i` and `j` to that equation, you know exactly where to operate on the matrix. For example, `i=nx-2, j=2`, which is in row `row_number = 2*nx-5`, is next to a Neumann boundary condition: we have to substract one out of the main diagonal (`A[2*nx-5,2*nx-5]-=1`), and put a zero in the next column (`A[2*nx-5,2*nx-4]=0`). This way, the function can become much simpler!

Can you use this information to construct a more general function `generateMatrix`? Can you make it such that the type of boundary condition is an input to the function?

## Heat diffusion in 2D

Let's recast the 2D heat conduction from the previous notebook, and solve it with an implicit scheme.

```
alpha = 1e-4

L = 1.0e-2
H = 1.0e-2

nx = 21
ny = 21
nt = 300

dx = L/(nx-1)
dy = H/(ny-1)

x = numpy.linspace(0,L,nx)
y = numpy.linspace(0,H,ny)

T_bc = 100

Ti = numpy.ones((ny, nx))*20
Ti[0,:]= T_bc
Ti[:,0] = T_bc
```

Luckily, the matrix needs to be generated only once. We'll do that now

```
sigma = 0.25
A = constructMatrix(nx, ny, sigma)
```

We are ready to go!

```
T = Ti.copy()
dt = sigma * min(dx, dy)**2 / alpha
T = btcs_2D(T, A, nt, sigma, T_bc, nx, ny, dt)
```

    Center of plate reached 70C at time 0.16s, in time step 256.

```
361/21
```

    17

And plot,

```
from matplotlib import pyplot
%matplotlib inline
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16

my, mx = numpy.meshgrid(y,x)
pyplot.figure(figsize=(7,7))
pyplot.contourf(my,mx,T,20);
```

Try this out with different values of `sigma`! You'll see that it will always give a stable solution!

Does this result match the explicit scheme from the previous notebook? Do they take the same amount of time to reach $70^\circ C$ in the center of the plate? Now that we can use higher values of `sigma`, we need fewer time steps for the center of the plate to reach $70^\circ C$! Of course, we need to be careful that `dt` is small enough to resolve the physics correctly.

---
###### The cell below loads the style of the notebook

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Properties of Catmull--Rom Splines

CR splines are very popular because they are very easy to use.
Only a sequence of control points has to be specified, the tangents are calculated automatically from the given points.
Therefore, CR splines can be regarded as a subset of cubic Hermite splines.

CR splines are $C^1$ continuous, which means that the incoming and outgoing tangent vectors are equal for each control point.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
import splines
```

```python
def plot_spline(spline, dots_per_second=15, ax=None, **kwargs):
    if ax is None:
        ax = plt.gca()
    total_duration = spline.grid[-1] - spline.grid[0]
    times = spline.grid[0] + np.arange(int(total_duration * dots_per_second) + 1) / dots_per_second
    ax.plot(*spline.evaluate(spline.grid).T,
            linestyle=':', color='lightgrey',
            marker='x', markeredgecolor='black')
    ax.plot(*spline.evaluate(times).T, '.', **kwargs)
    ax.axis('equal');
```

Let's choose a few points for an example:

```python
points = [
    (0.2, -0.5),
    (0, 1.3),
    (1.3, 0.6),
    (2.5, 1.9),
    (3.8, -0.2),
    (2.1, 0.1),
]
```

```python
s = splines.CatmullRom(points, endconditions='closed')
```

```python
plot_spline(s)
```

## Tangent Vectors

The tangent vectors at a given point are calculated based on the line connecting the preceding point and the following point.
In fact, the tangent vector has the same orientation as that line but only half of its length.

In other (more mathematical) words:

\begin{equation}
\boldsymbol{\dot{x}}_i = \frac{\boldsymbol{x}_{i+1} - \boldsymbol{x}_{i-1}}{2}
\end{equation}

This is illustrated for two points in the following plot:

```python
plot_spline(s)
for idx, color in zip([2, 5], ['purple', 'orange']):
    plt.quiver(*s.evaluate(s.grid[idx]), *s.evaluate(s.grid[idx], 1),
               angles='xy', scale_units='xy', scale=1, color=color)
    points = np.row_stack([s.evaluate(s.grid[idx - 1]),
                           s.evaluate(s.grid[idx + 1])])
    plt.plot(*points.T, '--', color=color, linewidth=2)
```

Note that the above equation is only valid for the uniform case, i.e. if the time intervals of all segments are equal to 1 unit of time.

Uniform parametrization typically works very well if the (Euclidean) distances between consecutive points are all quite similar.
However, if the distances are very different, the shape of the spline often turns out to be unexpected.
Most notably, in extreme cases there might be even cusps or self-intersections within a spline segment.

```python
points2 = [
    (0, 0),
    (1.45, 1.5),
    (1.55, 1.5),
    (3, 0),
]
```

```python
s2 = splines.CatmullRom(points2, endconditions='closed')
```

```python
plot_spline(s2)
```

In order to manipulate the shape of the spline (without moving control points), non-uniform times can be chosen.

For example, let's choose a few custom times:

```python
times3 = 0, 0.8, 1, 3, 5
```

```python
s3 = splines.CatmullRom(points2, times3, endconditions='closed')
```

```python
plot_spline(s3)
```

In the non-uniform case, the equation for the tangent vector gets a bit more complicated:

\begin{equation}
\boldsymbol{\dot{x}}_i =
\frac{
(t_{i+1} - t_i)^2 (\boldsymbol{x}_i - \boldsymbol{x}_{i-1}) +
(t_i - t_{i-1})^2 (\boldsymbol{x}_{i+1} - \boldsymbol{x}_i)
}{
(t_{i+1} - t_i)(t_i - t_{i-1})(t_{i+1} - t_{i-1})
}
\end{equation}

The derivation of this equation is shown in [a separate notebook](catmull-rom-non-uniform.ipynb).

Some sources use a simpler equation which is (arguably) not correct (except in the uniform case):

\begin{equation}
\boldsymbol{\dot{x}}_i = \frac{1}{2} \left(
\frac{\boldsymbol{x}_i - \boldsymbol{x}_{i-1}}{t_i - t_{i-1}} +
\frac{\boldsymbol{x}_{i + 1} - \boldsymbol{x}_i}{t_{i + 1} - t_i}
\right)
\end{equation}

See the [notebook about "finite difference" splines](finite-difference.ipynb).

There are even sources which show yet a simpler (but even less correct, except in the uniform case) equation:

\begin{equation}
\boldsymbol{\dot{x}}_i = \frac{\boldsymbol{x}_{i + 1} - \boldsymbol{x}_{i - 1}}{t_{i + 1} - t_{i - 1}}
\end{equation}

TODO: separate notebook with plots that illustrate the differences between those equations.

## Chordal Parametrization

TODO: explain "chordal"

```python
distances = np.linalg.norm(np.diff(points2 + points2[:1], axis=0), axis=1)
distances
```

```python
times4 = np.concatenate([[0], np.cumsum(distances)])
times4
```

```python
s4 = splines.CatmullRom(points2, times4, endconditions='closed')
```

```python
plot_spline(s4)
```

## Centripetal Parametrization

TODO: explain "centripetal"

```python
times5 = np.concatenate([[0], np.cumsum(np.sqrt(distances))])
times5
```

```python
s5 = splines.CatmullRom(points2, times5, endconditions='closed')
```

```python
plot_spline(s5)
```

TODO: `alpha` can be specified, see [API docs](python-module.rst#splines.CatmullRom).

```python
plot_spline(splines.CatmullRom(points2, endconditions='closed', alpha=0), label='uniform')
plot_spline(splines.CatmullRom(points2, endconditions='closed', alpha=0.5), label='centripetal')
plot_spline(splines.CatmullRom(points2, endconditions='closed', alpha=1), label='chordal')
plt.legend(loc='center');
```


# Eniyileme

Kısıtsız eniyileme problemlerinin genel yapısı
\begin{align}
\text{minimize} & & E(w)
\end{align}

* Algoritma :
\begin{align}
w_0 & = \text{başlangıç değeri} \\
\text{for}\;\; & \tau = 1, 2,\dots \\
& w_\tau = w_{\tau-1} + g_\tau
\end{align}

Burada, $g_\tau$ bir arama yönüdür. Amaç, döngüler sonucunda başlangıç noktasından daha düşük (idealde en düşük) bir fonksyon değeri bulmaktır.

## Bayır inişi
* negatif gradyan yönünde arama
\begin{align}
g_\tau & =  - \eta \nabla E(w_{\tau-1})
\end{align}

*  $\nabla E(w) = 0$ olduğunda algoritma çözüm noktasını değiştiremez. Bu tarz noktalara sabit nokta adı verilir.

* Bazen algoritma gerçek en küçük noktaya da yakınsar  $w^* = \arg\min_{w} E(w)$. Burada bu yakınsamanın hangi şartlarda ve hangi hızda gerçekleştiği önemli kavramsal sorulardır.

* Burada, $\eta$'ya *adım büyüklüğü* veya *öğrenme hızı* gibi isimler verilir. Bu parametrenin düzgün seçilmesi pratikte epey deneme yanılma grektirmektedir. Düzgün seçilmezse algoritma yakınsamaz ve patlar, veya yakınsaması çok uzun sürer. Epey kavramsal çalışma var.

* Sadece gradyan hesaplamak gerektiği için kullanılması çok kolay bir algoritmadır. Yapay zeka alanında çok büyük modellerde de uygulanabildiği için çok yaygındır.

```python
%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt

from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
from IPython.display import clear_output, display, HTML
from matplotlib import rc

mpl.rc('font',**{'size': 20, 'family':'sans-serif','sans-serif':['Helvetica']})
mpl.rc('text', usetex=True)

import time
import numpy as np
```

```python
y = np.array([7.04, 7.95, 7.58, 7.81, 8.33, 7.96, 8.24, 8.26, 7.84, 6.82, 5.68])
x = np.array(np.linspace(-1,1,11))
N = len(x)

# Design matrix
#A = np.vstack((np.ones(N), x, x**2, x**3)).T
degree = 9
A = np.hstack([np.power(x.reshape(N,1),i) for i in range(degree+1)])

# Learning rate
eta = 0.001

# initial parameters
w = np.array(np.random.randn(degree+1))
W = []
Err = []
for epoch in range(50000):
    # Error
    err = y-A.dot(w)

    # Total error
    E = np.sum(err**2)/N

    # Gradient
    dE = -2.*A.T.dot(err)/N

    if epoch%100 == 0:
        #print(epoch,':',E)
        # print(w)
        W.append(w)
        Err.append(E)

    # Perfom one descent step
    w = w - eta*dE
```

```python

fig = plt.figure(figsize=(5,5))

left = -1.5
right = 1.5
xx = np.linspace(left,right,50)
AA = np.hstack((np.power(xx.reshape(len(xx),1),i) for i in range(degree+1)))

# Find best
A_orth, R = np.linalg.qr(A)
w_orth, res, rank, s = np.linalg.lstsq(A_orth, y)
w_star = np.linalg.solve(R, w_orth)
yy = AA.dot(w_star)

#ax.set_xlim((2,15))

#dots = plt.Line2D(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
#ax.add_line(dots)
plt.plot(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
plt.plot(xx, yy, linestyle=':', color='k', alpha=0.3)
ln = plt.Line2D(xdata=[], ydata=[], linestyle='-',linewidth=2)

ax = fig.gca()
ax.add_line(ln)
plt.close(fig)

ax.set_xlim((left,right))
ax.set_ylim((5,9))

def plot_gd(iteration=0):
    w = W[iteration]
    f = AA.dot(w)
    #print(w)
    ln.set_ydata(f)
    ln.set_xdata(xx)

    ax.set_title('$E = '+str(Err[iteration])+'$')

    display(fig)

res = interact(plot_gd, iteration=(0,len(W)-1))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

```python
%matplotlib inline

import scipy as sc
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pylab as plt

df_arac = pd.read_csv(u'data/arac.csv',sep=';')
#df_arac[['Year','Car']]

BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

plt.plot(x+BaseYear, y, 'o-')
plt.xlabel('Yil')
plt.ylabel('Araba (Millions)')

plt.show()
```

```python
from itertools import product

def Error_Surface(y, A, left=0, right=1, bottom=0, top=1, step=0.1):
    W0 = np.arange(left,right, step)
    W1 = np.arange(bottom,top, step)

    ErrSurf = np.zeros((len(W1),len(W0)))

    for i,j in product(range(len(W1)), range(len(W0))):
        e = y - A*np.matrix([W0[j], W1[i]]).T
        ErrSurf[i,j] = e.T*e/2

    return ErrSurf

BaseYear = 1995
x = np.matrix(df_arac.Year[0:]).T-BaseYear
y = np.matrix(df_arac.Car[0:]).T/1000000.

# Setup the vandermonde matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

left = -5
right = 15
bottom = -4
top = 6
step = 0.05
ErrSurf = Error_Surface(y, A, left=left, right=right, top=top, bottom=bottom)

plt.figure(figsize=(10,10))
#plt.imshow(ErrSurf, interpolation='nearest',
#           vmin=0, vmax=10000,origin='lower',
#           extent=(left,right,bottom,top), cmap='jet')

plt.contour(ErrSurf,
            vmin=0, vmax=10000,origin='lower', levels=np.linspace(100,5000,10),
            extent=(left,right,bottom,top), cmap='jet')

plt.xlabel('$w_0$')
plt.ylabel('$w_1$')
plt.title('Error Surface')
#plt.colorbar(orientation='horizontal')
plt.show()
```

```python
# An implementation of Gradient Descent for solving linear a system

# Setup the Design matrix
N = len(x)
A = np.hstack((np.ones((N,1)), x))

# Starting point
w = np.matrix('[15; -6]')

# Number of iterations
EPOCH = 5000

# Learning rate: The following is the largest possible fixed rate for this problem
#eta = 0.00016
eta = 0.00015

Error = np.zeros((EPOCH))
W = np.zeros((2,EPOCH))

for tau in range(EPOCH):
    # Calculate the error
    e = y - A*w

    # Store the intermediate results
    W[0,tau] = w[0]
    W[1,tau] = w[1]
    Error[tau] = (e.T*e)/2

    # Compute the gradient descent step
    g = -A.T*e
    w = w - eta*g
    #print(w.T)

w_star = w
plt.figure(figsize=(8,8))
plt.imshow(ErrSurf, interpolation='nearest',
           vmin=0, vmax=1000,origin='lower',
           extent=(left,right,bottom,top),cmap='Blues_r')
plt.xlabel('w0')
plt.ylabel('w1')

ln = plt.Line2D(W[0,:300:1], W[1,:300:1], marker='o',markerfacecolor='w')
plt.gca().add_line(ln)

ln = plt.Line2D(w_star[0], w_star[1], marker='x',markerfacecolor='w')
plt.gca().add_line(ln)
plt.show()

plt.figure(figsize=(8,3))
plt.semilogy(Error)
plt.xlabel('Iteration tau')
plt.ylabel('Error')
plt.show()

```

# Bayır inişini hızlandırmak

## Momentum yöntemleri, a.k.a., ağır top

\begin{align}
p(\tau) & =  \nabla E(w(\tau-1)) + \beta p(\tau-1) \\
w(\tau) & =  w(\tau-1) - \alpha p(\tau)
\end{align}

$\beta=0$, bayır inişi

```python
%matplotlib inline
import matplotlib.pylab as plt

from notes_utilities import pnorm_ball_line

import time
from IPython import display
import numpy as np

#y = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
y = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
#y = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x = np.array([10., 8., 13., 9., 11., 14., 6., 4., 12., 7., 5.])
N = len(x)

# Design matrix
A = np.vstack((np.ones(N), x)).T

w_best, E, rank, s = np.linalg.lstsq(A, y)
err = y-A.dot(w_best)
E_min = np.sum(err**2)/N

def inspect_momentum(alpha = 0.005, beta = 0.97):
    ln = pnorm_ball_line(mu=w_best, A=np.linalg.cholesky(np.linalg.inv(A.T.dot(A))),linewidth=1)
    ln2 = pnorm_ball_line(mu=w_best, A=4*np.linalg.cholesky(np.linalg.inv(A.T.dot(A))),linewidth=1)

    # initial parameters
    w0 = np.array([2., 1.])
    w = w0.copy()
    p = np.zeros(2)

    EPOCHS = 100
    W = np.zeros((2,EPOCHS))
    for epoch in range(EPOCHS):
        # Error
        err = y-A.dot(w)
        W[:,epoch] = w
        # Mean square error
        E = np.sum(err**2)/N

        # Gradient
        dE = -2.*A.T.dot(err)/N
        p = dE + beta*p

#        if epoch%10 == 1:
#            print(epoch,':',E)
            # print(w)

        # Perfom one descent step
        w = w - alpha*p

#    print(E_min)

    plt.plot(W[0,:],W[1,:],'.-b')
    plt.plot(w_best[0],w_best[1],'ro')
    plt.plot(w0[0],w0[1],'ko')
    plt.xlim((1.8,4.3))
    plt.ylim((0,1.2))
    plt.title('$\\alpha = $'+str(alpha)+' $\\beta = $'+str(beta))
    plt.gca().add_line(ln)
    plt.gca().add_line(ln2)
    plt.show()

inspect_momentum(alpha=0.0014088, beta=0)

```

```python
%matplotlib inline
from __future__ import print_function
from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
import matplotlib.pylab as plt
from IPython.display import clear_output, display, HTML

interact(inspect_momentum, alpha=(0, 0.02, 0.001), beta=(0, 0.99, 0.001))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in Jupyter Notebook or JupyterLab, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another notebook frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    <function __main__.inspect_momentum>


# Using UDEs to solve a damped SHO

## Generating the data

First, we'll want to generate data from a damped simple harmonic oscillator modeled by

\begin{equation}
\frac{d^2x}{dt^2}+2\gamma\frac{dx}{dt}+\omega_0^2x=0
\label{DampedSHO}
\end{equation}

where $\gamma$ is the dampening factor and $\omega_0$ is the resonant frequency of the oscillator.

The solution to this ODE is obtained by simple using the trial solution of $x(t)=Ae^{kt}$. Plugging into the ODE we get:

\begin{equation}
\begin{split}
Ak^2e^{kt}+2\gamma Ake^{kt}+\omega_0^2Ake^{kt}&=0\\
Ae^{kt}\left(k^2+2\gamma k+\omega_0^2\right)&=0 \text{ Assume } Ae^{kt}\neq 0\\
k^2+2\gamma k+\omega_0^2&=0\\
k=\frac{-2\gamma}{2}\pm\frac{\sqrt{(2\gamma)^2-4(1)\omega_0^2}}{2}&\\
k=-\gamma\pm\sqrt{\gamma^2-\omega_0^2}
\end{split}
\label{DampedAuxEqn}
\end{equation}

So, the solution to the damped SHO is just

\begin{equation}
x(t)=A(e^{(-\gamma+\omega)t}+e^{(-\gamma-\omega)t})\text{ where } \omega=\sqrt{\gamma^2-\omega_0^2}
\label{DampedSHOSoln}
\end{equation}

and $A$ is the amplitude of oscillation.

Let's go ahead and use $\gamma=.1$, $\omega_0=.5$, $A=10$ for our values to get our data. First we'll need to load in all of our packages.

```julia
##using all of the packages we'll need

using Random
using Statistics
using Flux
using DiffEqFlux
using DataDrivenDiffEq
using LinearAlgebra
using DiffEqSensitivity
using Optim
using ModelingToolkit
using OrdinaryDiffEq
using Plots
using DifferentialEquations
using Measurements
using LaTeXStrings
```

At this point let's plot the analytic solution obtained from standard methods of solving ODEs.

```julia
##Create a plot of the analytic solution of the damped SHO ODE.

#parameters of the damped SHO
x0 = 10;                       #Initial position
v0=0;                          #Initial velocity
ω0 = .5;                       #Resonant frequency of the undamped sysyem (γ=0)
γ = .1;                        #Damping coefficient
ω = sqrt(Complex(γ^2 - ω0^2)); #Frequency of the damped system
Ω1 = -γ + ω;                   #
Ω2 = -γ - ω;                   #

#x(t)=10*exp(-γ*t)*cos(ω*t);
x(t)=(x0/2)*(exp(Ω1*t) + exp(Ω2*t));
v(t)=(x0/2)*(Ω1*exp(Ω1*t) + Ω2*exp(Ω2*t));

#Now make the plot to display the solution (the position)

p1=plot(

#funtion to plot
x,

#label the line representing x(t).
label = "position x in (cm) versus time (s)",

#control the line properties by making it red, with an opacity of 50%, thickness of 3, and solid.
line = (:red, 0.5, 3, :solid),

#start the plot at t=0 and terminate it at 6π seconds.
0,
2π*3,

#label axes and title the plot.
ylabel = "Amplitude (cm)",
xlabel = "time (s)",
title = "Damped SHO (Position): γ=$(γ), ω0=$(ω0)",

#implement numbered ticks along x,y axes in increments of 1 from 0 to 20,10.
xticks = 0:1:20,
yticks = -x0:1:x0,

#fill the areas between the line x(t)=0 and the plot with a cyan coloring that has 25% opacity
#and make the background of the plot ivory
fillrange = 0,
fillalpha = 0.25,
fillcolor = :cyan,
background_color = :ivory
)

#save the figure as a .png and .pdf
savefig("DampedSHOx.png")
savefig(p1,"DampedSHOx.pdf")

##Make the plot of the velocity

p2=plot(
v,

#label the line representing v(t).
label = "velocity x in (cm/s) versus time (s)",

#control the line properties by making it red, with an opacity of 50%, thickness of 3, and solid.
line = (:red, 0.5, 3, :solid),

#start the plot at t=0 and terminate it at 6π seconds.
0,
2π*3,

#label axes and title the plot.
ylabel = "Amplitude (cm/s)",
xlabel = "time (s)",
title = "Damped SHO (Velocity): γ=$(γ), ω0=$(ω0)",

#implement numbered ticks along x,y axes in increments of 1 from 0 to 20,10.
xticks = 0:1:20,
yticks = -x0:1:x0,

#fill the areas between the line x(t)=0 and the plot with a cyan coloring that has 25% opacity
#and make the background of the plot ivory
fillrange = 0,
fillalpha = 0.25,
fillcolor = :cyan,
background_color = :ivory
)

##save the figure as a .png and .pdf
savefig("DampedSHOv.png")
savefig(p2,"DampedSHOv.pdf")

display(p1)
display(p2)
```

## Creating the UDE Model to discover damping

Now that we have the model data generated we want to learn the damping behavior without getting the solution completely from prior knowledge. When we found the solution to the damped SHO previously, we used a number of things:

Newton's 2nd law, Hooke's Law, and assuming a damping force linearly proportional to the velocity (to form the differential equation.

The well-known ansatz for a second order linear ODE with constant coefficients, $x(t)=Ae^{kt}$ and a tad bit of algebra.

The idea here is that often times in the real world you won't be able to simply look at the data, slap together an ODE with relevant boundary conditions, do some math and get an answer. However, you will have access to general properties of a system such as conservation of energy, newton's laws, and etc. The point is that while we don't know everything there is to know about the system under study, we (hopefully) DO know something about what's going on. So, let's use that knowledge when making our model. The rest we'll leave up to a neural network (or some other universal function approximator) to learn.

As a start, let's say we know that we have a data from a system involving an oscillator, we know that the data doesn't quite match up with the simple SHO, but it is SHO-like.

\begin{equation}
\frac{d^2x}{dt^2}+\omega_0^2x+NN=0
\end{equation}

Rather than one 2nd order ODE, we can recast this in terms of a system of two 1st order ODEs. Renaming $x$ to $u_1$ and $v$ to $u_2$

\begin{equation}
\begin{split}
\frac{du_1}{dt}&=u_2\\
\frac{du_2}{dt}&=-\omega_0^2u_1-NN=0
\end{split}
\end{equation}

where $NN$ is the neural network which we want to use to approximate $-2\gamma u_2$.

First, we'll need to generate the position data that we wish to work with. So,

```julia
#Initial values of the oscillator
u0=[x0,v0]
#intial and final times for the solution
ti=0.0
tf=20.0
tspan=(ti,tf)

#This defines our DE system
function SHO(du,u,p,t)
    du[1]=u[2]
    du[2]=-ω0^2*u[1]-2γ*u[2]
end

prob = ODEProblem(SHO,u0,tspan)
sol = solve(prob)

#plot the data
p3=plot(
sol,
label=["Position (cm)" "Velocity (cm/s)"],

#label axes and title the plot.
xlabel = "time (s)",
title = "Damped SHO: γ=$(γ), ω0=$(ω0)",

#implement numbered ticks along x,y axes in increments of 1 from 0 to 20,10.
xticks = 0:1:20,
yticks = -x0:1:x0,

#fill the areas between the line x(t)=0 and the plot with a cyan coloring that has 25% opacity
#and make the background of the plot ivory
fillrange = 0,
fillalpha = 0.25,
fillcolor = [:cyan :pink],
background_color = :ivory
)

##save the figure as a .png and .pdf
savefig("Data.png")
savefig(p3,"Data.pdf")

display(p3)

#output the solution to understand its structure
sol;
```

Now, we'll assign the 1st column of data to x_true, aka our true position and the 2nd column of sol we'll assign to v_true aka our true velocity. These will be the ground truth comparision against which we will test how accurate our model is.

```julia
x_true=sol[1,:];
v_true=sol[2,:];
sol.t;
```

Let's go ahead and generate our data that we'll use in the neural network to build our model.

```julia
du = [0.0, 0.0]

SHO([1.0, 1.0], [0.1, 0,0], 1.0, π)
```

    -0.025

```julia
##Setup the data sample
t0=0;             #Start time of sampling
tf=6;             #End time of sampling
Δt=.3;            #Temporal spacing of sampling data
data_t=t0:Δt:tf;
data=sol(data_t); #The sampled dataset

##Sample the position data and acceleration data
pos_data = [state[1] for state in sol(data_t)];
vel_data = [state[2] for state in sol(data_t)];
accel_data = [SHO(du, data[i], ω0, data_t[i]) for i ∈ 1:length(data_t)]

```

    21-element Vector{Float64}:
     -2.5
     -2.327437206738794
     -2.114191752191627
     -1.8672801599772477
     -1.5940469926759633
     -1.3019764672115435
     -0.9985397752247114
     -0.6910038494857581
     -0.3863137357546734
     -0.0909554744383243
      0.18919300096865765
      0.4489071854798998
      0.6836980818078091
      0.8899162471637888
      1.064739313725047
      1.2061632196380447
      1.313051242865706
      1.3851091263960396
      1.4227862324991927
      1.4272374520004694
      1.400305559669581

Now we need to build the neural network, let's call it layers, that we'll use to as a stand-in for the damping term in the ODE.

```julia
layers=Chain(x->[x],Dense(1,32,tanh),Dense(32,1),first)
```

    Chain(#33, Dense(1, 32, tanh), Dense(32, 1), first)

Let's go ahead and make the first loss function, this will be a simple mean-squared error between what our neural network predicts and what the actual true solution is.

```julia
loss()=sum(abs2,layers(pos_data[i]).-accel_data[i] for i in 1:length(pos_data));
loss()
```

    81.21250037896785

Now we will add in the physics to the machine learning by way of Hooke's Law $F=-kx$ or since we're using acceleration in this case $a=-\omega_0^2x$

```julia
rand_pos = [x0*(2rand()-1) for i in 1:100] # random values in [-x0,x0]
phy_loss() = sum(abs2,layers(x) - (-ω0^2*x) for x in rand_pos);
phy_loss()

```

    427.6554111743035

Here we will combine the simple loss function with the physics informed one.

```julia
λ=.01;
total_loss()=loss()+λ*phy_loss();
total_loss()
```

    85.48905449071088

Now we will train the network and see how well we did.

```julia
##Setup the data sample
t0plot=0;             #Start time of sampling
tfplot=20;             #End time of sampling
Δtplot=.01;            #Temporal spacing of sampling data
data_plot=t0:Δt:tf;
plotit=sol(data_plot); #The sampled dataset

##Sample the position data and acceleration data
pos_plot = [state[1] for state in sol(data_plot)];
vel_plot = [state[2] for state in sol(data_plot)];
accel_plot =[SHO(du, plotit[i], ω0, data_plot[i]) for i ∈ 1:length(data_plot)]

##Train the model
opt = Flux.Descent(0.01)
data = Iterators.repeated((), 5000)
iter = 0
cb = function () #callback function to observe training
  global iter += 1
  if iter % 500 == 0
    display(total_loss())
  end
end
display(total_loss())
Flux.train!(total_loss, Flux.params(layers), data, opt; cb=cb)

learned_accel_plot = layers.(pos_plot)

plot(data_plot,accel_plot,xlabel="t",label="True Force")
plot!(data_plot,learned_accel_plot,label="Predicted Force")
scatter!(data_t,accel_data,label="Force Measurements")
```

    85.48905449071088

    NaN

    NaN

    NaN

    NaN

    NaN

    NaN

    NaN

    NaN

    NaN

    NaN

```julia

```


# Lecture 30
## Chi-Square, Student's t, Multivariate Normal
----

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import chi2, gamma, t
%matplotlib inline

plt.xkcd()
None
```

## $\chi^2$ Distribution

The Chi-square Distribution is denoted as $\chi^2(n)$ or sometimes $\chi_{n}^2$, where $n$ indicates the _degrees of freedom_. It used everywhere (I think you used it before in feature analysis). It is related to Normal distribution.

Let $V = Z_1^2 + Z_2^3 + \dots + Z_n^2$, where the $Z_j$ are i.i.d. $\mathcal{N}(0,1)$. Then by definition, $V \sim \chi^2(n)$.

You will find that in a lot of things involving statistics, the sum of squares of $\mathcal{N}(0,1)$ often pops up.

```python
dof_values = [1,2,3,4,5,6,7,8]
x = np.linspace(0, 10, 1000)

# plot the distributions
fig, ax = plt.subplots(figsize=(12, 6))

for d in dof_values:
    ax.plot(x, chi2.pdf(x, d), lw=4, alpha=0.6, label='df={}'.format(d))

plt.xlim((0,10.0))
plt.ylim((0,0.5))
plt.legend()
ax.title.set_text(r'Examples of $\chi^2_n$ with varying degrees of freedom')
plt.show()
```

### Fact: $\chi^2(1)$ is $Gamma(\frac{1}{2}, \frac{1}{2})$

#### Proof

Let $Y = Z^2$ where $Z \sim \mathcal{N}(0,1)$ and $y \gt 0$.

\begin{align}
    P(Y \le y) &= P(Z^2 \le y) \\
    &= P( -y^{\frac{1}{2}} \le y \le y^{\frac{1}{2}}) \\
    &= \Phi(y^{\frac{1}{2}}) - \Phi(y^{-\frac{1}{2}}) \\
    &= \Phi(y^{\frac{1}{2}}) - \left( 1 - \Phi(y^{\frac{1}{2}}) \right) \\
    &= 2 \, \Phi(y^{\frac{1}{2}}) - 1 \\
    \\
    \Rightarrow f_{Y}(y) &= y^{-\frac{1}{2}} \, \phi(y^{\frac{1}{2}}) \\
    &= \frac{1}{\sqrt{2\pi}} \, y^{-\frac{1}{2}} \, e^{-\frac{y}{2}} \\
    \\
    Gamma\left(\frac{1}{2}, \frac{1}{2}\right) &= \frac{1}{\Gamma(\frac{1}{2})} \, \left(\frac{y}{2}\right)^{\frac{1}{2}} \, e^{-\frac{y}{2}} \, \frac{1}{y} \\
    &= \frac{1}{\sqrt{\pi}} \, \sqrt{\frac{y}{2}} \, e^{-\frac{y}{2}} \, \frac{1}{y} \\
    &= \frac{1}{\sqrt{2\pi}} \, y^{-\frac{1}{2}} \, e^{-\frac{y}{2}} &\blacksquare \\
\end{align}

Here's a quick graph to illustrate.

```python
_, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(14,6))

x = np.linspace(0, 20, 1000)

ax1.plot(x, chi2.pdf(x, 1), lw=4, alpha=0.6, color='#33AAFF', label='df=1')
ax1.set_title('$\chi^2_1$', y=1.02)
ax1.set_xlim((0,20.0))
ax1.set_ylim((0,0.5))
ax1.legend()

# gamma.pdf API: scale = 1 / beta
l = 0.5
ax2.plot(x, gamma.pdf(x, 0.5, scale=1/l), lw=4, alpha=0.6, color='#FF9933', label=r'$\alpha$=1/2, $\lambda$=1/2')
ax2.set_title(r'$Gamma(\frac{1}{2}, \frac{1}{2})$', y=1.02)
ax2.set_xlim((0,20.0))
ax2.set_ylim((0,0.5))
ax2.legend()
None
```

### Fact:  $\chi^2(n)$ is $Gamma\left( \frac{n}{2}, \frac{1}{2} \right)$

It follows then that $\chi^2(n) = Gamma\left( \frac{n}{2}, \frac{1}{2} \right)$

```python
_, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(14,6))

x = np.linspace(0, 20, 1000)

dof_values = [1, 2, 5, 10, 20]
col_alph_values = [0.8, 0.6, 0.5, 0.4, 0.3]

for df,c_alph in zip(dof_values, col_alph_values):
    ax1.plot(x, chi2.pdf(x, df), color='#33AAFF', lw=4, alpha=c_alph, label='df={}'.format(df))
ax1.set_title('$\chi^2_n$ for varying degrees of freedom', y=1.02)
ax1.set_xlim((0,20.0))
ax1.set_ylim((0,0.5))
ax1.legend()

# gamma.pdf API: scale = 1 / lambda
l = 0.5
for alph,c_alph in zip(dof_values, col_alph_values):
    ax2.plot(x, gamma.pdf(x, alph/2, scale=1/l), lw=4, alpha=c_alph, color='#FF9933', label=r'$\alpha$={}/2, $\lambda$=1/2'.format(alph))
ax2.set_title(r'$Gamma(\frac{n}{2}, \frac{1}{2})$ for varying n', y=1.02)
ax2.set_xlim((0,20.0))
ax2.set_ylim((0,0.5))
ax2.legend()
None
```

----

## Student's $t$-Distribution

The Student's $t$-distribution can be described in terms of the standard normal $Z \sim N(0,1)$ and $X^2$  $V(n)$ distributions, so that means it can be entirely described in terms of the standard normal distribution.

Let $T = \frac{Z}{\sqrt{V/n}}$, with $Z \sim \mathcal{N}(0,1)$ and $V \sim \chi^2(n)$, where $Z, V$ are independent.

Then we can write $T \sim t_n$, where $n$ is the degrees of freedom.

$t_1$ does not have a $1^{st}$ moment; $t_2$ does not have a $2^{nd}$ moment; $t_3$ does not have a $3^{rd}$ moment; and so on. Odd moments, if they exist, are 0

```python
dof_values   = [1,2,5,10,30,1E10]
col_alph_values = [0.2, 0.3, 0.4, 0.5, 0.6, 0.8]
x = np.linspace(-5, 5, 1000)

# plot the distributions
fig, ax = plt.subplots(figsize=(8, 6))

for df,c_alph in zip(dof_values, col_alph_values):
    if df > 30:
        dl = r'$+\infty$'
    else:
        dl = df
    ax.plot(x, t.pdf(x, df), lw=4, color='#A93226', alpha=c_alph, label=r'df={}'.format(dl))

plt.xlim((-3.0,3.0))
plt.ylim((0,.43))
plt.ylabel('P(x)')
plt.legend()
plt.title(r'Examples of $t_n$ with varying degrees of freedom', y=1.02)
plt.text(x=3.5, y=0.22, s=r'Fatter tails with fewer degrees of freedom')
plt.text(x=3.5, y=0.19, s=r'Approaches $\mathbb{N}(0,1)$ as $df \rightarrow +\infty$')
plt.show()
```

### Properties

1. symmetric, i.e., $-T \sim t_n$

1. $n=1 \, \Rightarrow $ Cauchy, so $t_1$ does not have a mean

1. $n \ge 2 \Rightarrow \mathbb{E}(T) = \mathbb{E}(Z)\,\mathbb{E}\left(\frac{1}{\sqrt{V/n}}\right) = 0$

1. heavier-tailed than Normal

1. for $n$ large, $t_n$ looks very much like Normal

### Brief interlude: even moments of $Z$ and the Gamma distribution

It was proved earlier that for $Z \sim \mathcal{N}(0,1)$, the *even* moments are such that

\begin{align}
  \mathbb{E}(Z^2) &= 1 \\
  \mathbb{E}(Z^4) &= 1 \times 3 = 3 \\
  \mathbb{E}(Z^6) &= 1 \times 3 \times 5 = 15 &\quad \text{ skip factorial} \\
\end{align}

Now, this was proven using moment-generating functions, but we can also relate this to the Gamma distribution.

\begin{align}
  \mathbb{E}(Z^{2n}) &= \mathbb{E}\left( (Z^2)^n \right) \\
  &= \mathbb{E}\left( \chi^2_1 )^n \right) &\text{ but by definition } Z^2 \text{ is } \chi^2_1 \\
  &= \mathbb{E}\left( Gamma(\frac{1}{2}, \frac{1}{2})^n \right)
\end{align}

... and after this point, we can use our knowledge of the Gamma distribution and LOTUS.

#### Finding $\mathbb{E}(Z^8)$ with $Gamma(\frac{1}{2}, \frac{1}{2})$

\begin{align}
  \mathbb{E}(Z^8) &= \mathbb{E}\left( (Z^2)^4 \right) \\
  &= \mathbb{E}\left( Gamma(\frac{1}{2}, \frac{1}{2})^4 \right) \\
  &= \frac{\Gamma(\alpha + 4)}{\Gamma(\alpha) \, \lambda^4} \\
  &= \frac{(\alpha + 3) \, \Gamma(\alpha + 3)}{\Gamma(\alpha) \, \lambda^4} \\
  &= \frac{(\alpha + 3) \, (\alpha + 2) \, \Gamma(\alpha + 2)}{\Gamma(\alpha) \, \lambda^4} \\
  &= \frac{(\alpha + 3) \, (\alpha + 2) \, (\alpha + 1) \, \Gamma(\alpha + 1)}{\Gamma(\alpha) \, \lambda^4} \\
  &= \frac{(\alpha + 3) \, (\alpha + 2) \, (\alpha + 1) \, \alpha \, \Gamma(\alpha)}{\Gamma(\alpha) \, \lambda^4} \\
  &= \frac{\frac{7}{2} \, \frac{5}{2} \, \frac{3}{2} \, \frac{1}{2}}{\frac{1}{2^4}} \\
  &= 1 \times 3 \times 5 \times 7 \\
  &= 105
\end{align}

You can [double-check this answer on WolframAlpha](http://www.wolframalpha.com/input/?i=expected+value+of+Z%5E8).

### Proof of property 5: Student's $t$ when $n$ get large

Let's prove property 5 above, but use the Law of Large Numbers (c.f. Lesson 29).

* Let $T_n = \frac{Z}{\sqrt{V/n}}$ with $Z_1, Z_2, \dots , Z_n$ i.i.d. $\mathcal{N}(0,1)$
* $V = Z_1^2 + Z_2^2 + \dots + Z_n^2$
* $Z$ is independent of the $Z_j^2$

Now we can choose any distribution for this case as long as it is $\mathcal{N}(0,1)$, and so there is nothing wrong in choosing the same distribution $Z$ for the numerator and all elements of the denominator as well.

Then $\frac{V_n}{n} \rightarrow 1$ with probability 1 by the Law of Large Numbers, since the average $\frac{V_n}{n}$ will approach the true average $Z_1^2$ as $n$ gets large. We know that $Z_1^2 = 1$.

Now, the Law of Large Numbers is with regards to point-wise estimates, so we can further state that $\sqrt{\frac{V_n}{n}} \rightarrow 1$ with probability 1.

So $T_n \rightarrow Z$ with probability 1, since the denominator goes to 1 when you have a large number of degrees of freedom; only the $Z$ in the numerator will be of importance.

\begin{align}
\end{align}

----

## Multivariate Normal

### Definition

Random vector $(X_1, X_2, \cdots , X_k) = \vec{X}$ is Multivariate Normal if every linear combination $t_1 X_1 + t_2 X_2 + \cdots + t_k X_k$ is Normal.

### An example that is multivariate Normal

Let $Z, W$ be i.i.d. $\mathcal{N}(0,1)$. Then $( Z + 2W, 3 \, Z + 5W)$ is multivariate Normal (MVN).

Given constants $s,t$

\begin{align}
  s (Z + 2 \, W) + t (3 \, Z + 5 \, W) &= (s + 3t) Z + (2s + 5t) W \\
\end{align}

But since this is $(s + 3t)$ and $(2s + 5t)$ are just scaling independent Normal random variables $Z$ and $W$ respectively; and since we know that the sum of Normal random variables is also Normal, we know that $(s + 3t) Z + (2s + 5t) W$ is also necessarily a Normal r.v.

### A non-example (NOT multivariate Normal)

Let $Z \sim \mathcal{N}(0,1)$, and let $S$ be a random sign that is independent of $Z$.

Then $Z,SZ$ are marginally $\mathcal{N}(0,1)$ (consider both individually on their own).

But $(Z, SZ)$ is _not_ multivariate normal! Just test this by considering $(Z + SZ)$.

$Z + SZ$ cannot be Normal, since:
* half the time, the sum of $Z$ and $SZ$ will be zero when $S$ is negative
* the other times, the sum will be $2Z$
* this is some mixture of discrete and continuous

## MGF of $\vec{X}$

Now with $X \sim \mathcal{N}(\mu, \sigma^2)$, the moment generating function $M(X)$ is

\begin{align}
  M(X) &= \mathbb{E}(e^{tX}) \\
  &= e^{t\mu + \frac{1}{2} \, t^2 \sigma^2} \\\\
\end{align}

Extending this one-dimensional case to the multidimensional $\vec{X}$:

\begin{align}
  M(\vec{X}) &= \mathbb{E}(e^{\vec{t} \, \vec{X}}) \\
  &= \mathbb{E}(e^{t_1 \, X_1 + t_2 \, X_2 + \cdots + t_n \, X_n}) \\
  &= \mathbb{E}(e^{t_1 \, \mu_1 + t_2 \, \mu_2 + \cdots + t_n \, \mu_n + \frac{1}{2} Var(t_1 \, X_1 + t_2 \, X_2 + \cdots + t_n \, X_n)}
\end{align}

## Theorem: Within an MVN, uncorrelated implies independent

Recall that in general, independence implies uncorrelation, but the vice versa is not always true. In the case of an MVN, however, it **is** true.

In other words, consider vector

\begin{align}
\vec{X} &= \begin{bmatrix}
              \vec{X_1} \\
              \vec{X_2} \\
            \end{bmatrix}
\end{align}

If every component of $\vec{X_1}$ is uncorrelated with every component of $\vec{X_2}$, then $\vec{X_1}$ is independent of $\vec{X_2}$.

### Example

Let $X,Y$ be i.i.d. $\mathcal{N}(0,1)$. Then $(X+Y, X-Y)$ is MVN (_bivariate Normal_ to be precise).

It is easy enough to show that $X+Y$ and $X-Y$ are uncorrelated:

\begin{align}
  Cov(X+Y, X-Y) &= Var(X) + Cov(X,Y) - Cov(X,Y) - Var(Y) \\
  &= Var(X) - Var(Y) \\
  &= 1 - 1 \\
  &= 0
\end{align}

But can we show that $X+Y$ and $X-Y$ are _independent_?

### Proof

Let's try for something a bit more abstract.

We suppose that $X,Y$ are _independent_, zero-mean normal random variables with variances $\sigma_U, \sigma_V$.

Let $U = aX + bY$, and $V = cX + dY$ so that $U,V$ are jointly normal; this is a more general represention of the above example, where $a = 1, b=1, c=1, d=-1$.

Say we have some scalars $t_1, t_2$, and let $Z = t_{1}U + t_{2}V$. Then

\begin{align}
  M_{U,V}(t_1, t_2) &= \mathbb{E}(e^{t_1 \, U + t_2 \, V}) \\
  &= \mathbb{E}(e^{Z}) \\
  &= \mathbb{E}(e^{t_1 \, \mu_U + t_2 \, \mu_V + \frac{1}{2} Var(t_1 \, U + t_2 \, V)} \\
  &= \mathbb{E}(e^{\frac{1}{2} Var(t_1 \, U + t_2 \, V}) \\
  &= \mathbb{E}(e^{\frac{t_U^2 \, \sigma_U^2 + t_V^2 \, \sigma_V^2}{2}})
\end{align}

Now let $U\prime, V\prime$ be independent zero-mean normal random variables with the same variances $\sigma_U, \sigma_V$. Since $U\prime, V\prime$ are independent, they are also _uncorrelated_, and so the moment generating function of their bivariate normal distribution is given by $M_{U\prime,V\prime}(t_1, t_2) = \mathbb{E}(e^{\frac{t_U^2 \, \sigma_U^2 + t_V^2 \, \sigma_V^2}{2}}) $.

Since both $U,V$ and $U\prime,V\prime$ have the same moment generating function, they are boht associated with the same bivariate Normal distribution (they share the same joint PDF).

Thereforce, since $U\prime,V\prime$ are _independent_, we conclude that $U,V$ are also independent. QED.


# Sympy - Symbolic algebra in Python

J.R. Johansson (jrjohansson at gmail.com)

The latest version of this [IPython notebook](http://ipython.org/notebook.html) lecture is available at [http://github.com/jrjohansson/scientific-python-lectures](http://github.com/jrjohansson/scientific-python-lectures).

The other notebooks in this lecture series are indexed at [http://jrjohansson.github.io](http://jrjohansson.github.io).

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

## Introduction

There are two notable Computer Algebra Systems (CAS) for Python:

* [SymPy](http://sympy.org/en/index.html) - A python module that can be used in any Python program, or in an IPython session, that provides powerful CAS features.
* [Sage](http://www.sagemath.org/) - Sage is a full-featured and very powerful CAS enviroment that aims to provide an open source system that competes with Mathematica and Maple. Sage is not a regular Python module, but rather a CAS environment that uses Python as its programming language.

Sage is in some aspects more powerful than SymPy, but both offer very comprehensive CAS functionality. The advantage of SymPy is that it is a regular Python module and integrates well with the IPython notebook.

In this lecture we will therefore look at how to use SymPy with IPython notebooks. If you are interested in an open source CAS environment I also recommend to read more about Sage.

To get started using SymPy in a Python program or notebook, import the module `sympy`:

```python
from sympy import *
```

To get nice-looking $\LaTeX$ formatted output run:

```python
init_printing()

# or with older versions of sympy/ipython, load the IPython extension
#%load_ext sympy.interactive.ipythonprinting
# or
#%load_ext sympyprinting
```

## Symbolic variables

In SymPy we need to create symbols for the variables we want to work with. We can create a new symbol using the `Symbol` class:

```python
x = Symbol('x')
```

```python
(pi + x)**2
```

```python
# alternative way of defining symbols
a, b, c = symbols("a, b, c")
```

```python
type(a)
```

    sympy.core.symbol.Symbol

We can add assumptions to symbols when we create them:

```python
x = Symbol('x', real=True)
```

```python
x.is_imaginary
```

    False

```python
x = Symbol('x', positive=True)
```

```python
x > 0
```

### Complex numbers

The imaginary unit is denoted `I` in Sympy.

```python
1+1*I
```

```python
I**2
```

```python
(x * I + 1)**2
```

### Rational numbers

There are three different numerical types in SymPy: `Real`, `Rational`, `Integer`:

```python
r1 = Rational(4,5)
r2 = Rational(5,4)
```

```python
r1
```

```python
r1+r2
```

```python
r1/r2
```

## Numerical evaluation

SymPy uses a library for artitrary precision as numerical backend, and has predefined SymPy expressions for a number of mathematical constants, such as: `pi`, `e`, `oo` for infinity.

To evaluate an expression numerically we can use the `evalf` function (or `N`). It takes an argument `n` which specifies the number of significant digits.

```python
pi.evalf(n=50)
```

```python
y = (x + pi)**2
```

```python
N(y, 5) # same as evalf
```

When we numerically evaluate algebraic expressions we often want to substitute a symbol with a numerical value. In SymPy we do that using the `subs` function:

```python
y.subs(x, 1.5)
```

```python
N(y.subs(x, 1.5))
```

The `subs` function can of course also be used to substitute Symbols and expressions:

```python
y.subs(x, a+pi)
```

We can also combine numerical evolution of expressions with NumPy arrays:

```python
import numpy
```

```python
x_vec = numpy.arange(0, 10, 0.1)
```

```python
y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

```python
fig, ax = plt.subplots()
ax.plot(x_vec, y_vec);
```

However, this kind of numerical evolution can be very slow, and there is a much more efficient way to do it: Use the function `lambdify` to "compile" a Sympy expression into a function that is much more efficient to evaluate numerically:

```python
f = lambdify([x], (x + pi)**2, 'numpy')  # the first argument is a list of variables that
                                         # f will be a function of: in this case only x -> f(x)
```

```python
y_vec = f(x_vec)  # now we can directly pass a numpy array and f(x) is efficiently evaluated
```

The speedup when using "lambdified" functions instead of direct numerical evaluation can be significant, often several orders of magnitude. Even in this simple example we get a significant speed up:

```python
%%timeit

y_vec = numpy.array([N(((x + pi)**2).subs(x, xx)) for xx in x_vec])
```

    32.6 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit

y_vec = f(x_vec)
```

    3.3 µs ± 215 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Algebraic manipulations

One of the main uses of an CAS is to perform algebraic manipulations of expressions. For example, we might want to expand a product, factor an expression, or simply an expression. The functions for doing these basic operations in SymPy are demonstrated in this section.

### Expand and factor

The first steps in an algebraic manipulation

```python
(x+1)*(x+2)*(x+3)
```

```python
expand((x+1)*(x+2)*(x+3))
```

The `expand` function takes a number of keywords arguments which we can tell the functions what kind of expansions we want to have performed. For example, to expand trigonometric expressions, use the `trig=True` keyword argument:

```python
sin(a+b)
```

```python
expand(sin(a+b), trig=True)
```

See `help(expand)` for a detailed explanation of the various types of expansions the `expand` functions can perform.

The opposite a product expansion is of course factoring. The factor an expression in SymPy use the `factor` function:

```python
factor(x**3 + 6 * x**2 + 11*x + 6)
```

### Simplify

The `simplify` tries to simplify an expression into a nice looking expression, using various techniques. More specific alternatives to the `simplify` functions also exists: `trigsimp`, `powsimp`, `logcombine`, etc.

The basic usages of these functions are as follows:

```python
# simplify expands a product
simplify((x+1)*(x+2)*(x+3))
```

```python
# simplify uses trigonometric identities
simplify(sin(a)**2 + cos(a)**2)
```

```python
simplify(cos(x)/sin(x))
```

### apart and together

To manipulate symbolic expressions of fractions, we can use the `apart` and `together` functions:

```python
f1 = 1/((a+1)*(a+2))
```

```python
f1
```

```python
apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3)
```

```python
f2
```

```python
together(f2)
```

Simplify usually combines fractions but does not factor:

```python
simplify(f2)
```

## Calculus

In addition to algebraic manipulations, the other main use of CAS is to do calculus, like derivatives and integrals of algebraic expressions.

### Differentiation

Differentiation is usually simple. Use the `diff` function. The first argument is the expression to take the derivative of, and the second argument is the symbol by which to take the derivative:

```python
y
```

```python
diff(y**2, x)
```

For higher order derivatives we can do:

```python
diff(y**2, x, x)
```

```python
diff(y**2, x, 2) # same as above
```

To calculate the derivative of a multivariate expression, we can do:

```python
x, y, z = symbols("x,y,z")
```

```python
f = sin(x*y) + cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
diff(f, x, 1, y, 2)
```

## Integration

Integration is done in a similar fashion:

```python
f
```

```python
integrate(f, x)
```

$$x \cos{\left (y z \right )} + \begin{cases} 0 & \text{for}\: y = 0 \\- \frac{1}{y} \cos{\left (x y \right )} & \text{otherwise} \end{cases}$$

By providing limits for the integration variable we can evaluate definite integrals:

```python
integrate(f, (x, -1, 1))
```

and also improper integrals

```python
integrate(exp(-x**2), (x, -oo, oo))
```

Remember, `oo` is the SymPy notation for inifinity.

### Sums and products

We can evaluate sums and products using the functions: 'Sum'

```python
n = Symbol("n")
```

```python
Sum(1/n**2, (n, 1, 10))
```

```python
Sum(1/n**2, (n,1, 10)).evalf()
```

```python
Sum(1/n**2, (n, 1, oo)).evalf()
```

Products work much the same way:

```python
Product(n, (n, 1, 10)) # 10!
```

## Limits

Limits can be evaluated using the `limit` function. For example,

```python
limit(sin(x)/x, x, 0)
```

We can use 'limit' to check the result of derivation using the `diff` function:

```python
f
```

```python
diff(f, x)
```

$\displaystyle \frac{\mathrm{d}f(x,y)}{\mathrm{d}x} = \frac{f(x+h,y)-f(x,y)}{h}$

```python
h = Symbol("h")
```

```python
limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

We can change the direction from which we approach the limiting point using the `dir` keywork argument:

```python
limit(1/x, x, 0, dir="+")
```

```python
limit(1/x, x, 0, dir="-")
```

## Series

Series expansion is also one of the most useful features of a CAS. In SymPy we can perform a series expansion of an expression using the `series` function:

```python
series(exp(x), x)
```

By default it expands the expression around $x=0$, but we can expand around any value of $x$ by explicitly include a value in the function call:

```python
series(exp(x), x, 1)
```

And we can explicitly define to which order the series expansion should be carried out:

```python
series(exp(x), x, 1, 10)
```

The series expansion includes the order of the approximation, which is very useful for keeping track of the order of validity when we do calculations with series expansions of different order:

```python
s1 = cos(x).series(x, 0, 5)
s1
```

```python
s2 = sin(x).series(x, 0, 2)
s2
```

```python
expand(s1 * s2)
```

If we want to get rid of the order information we can use the `removeO` method:

```python
expand(s1.removeO() * s2.removeO())
```

But note that this is not the correct expansion of $\cos(x)\sin(x)$ to $5$th order:

```python
(cos(x)*sin(x)).series(x, 0, 6)
```

## Linear algebra

### Matrices

Matrices are defined using the `Matrix` class:

```python
m11, m12, m21, m22 = symbols("m11, m12, m21, m22")
b1, b2 = symbols("b1, b2")
```

```python
A = Matrix([[m11, m12],[m21, m22]])
A
```

$$\left[\begin{matrix}m_{11} & m_{12}\\m_{21} & m_{22}\end{matrix}\right]$$

```python
b = Matrix([[b1], [b2]])
b
```

$$\left[\begin{matrix}b_{1}\\b_{2}\end{matrix}\right]$$

With `Matrix` class instances we can do the usual matrix algebra operations:

```python
A**2
```

$$\left[\begin{matrix}m_{11}^{2} + m_{12} m_{21} & m_{11} m_{12} + m_{12} m_{22}\\m_{11} m_{21} + m_{21} m_{22} & m_{12} m_{21} + m_{22}^{2}\end{matrix}\right]$$

```python
A * b
```

$$\left[\begin{matrix}b_{1} m_{11} + b_{2} m_{12}\\b_{1} m_{21} + b_{2} m_{22}\end{matrix}\right]$$

And calculate determinants and inverses, and the like:

```python
A.det()
```

```python
A.inv()
```

$$\left[\begin{matrix}\frac{m_{22}}{m_{11} m_{22} - m_{12} m_{21}} & - \frac{m_{12}}{m_{11} m_{22} - m_{12} m_{21}}\\- \frac{m_{21}}{m_{11} m_{22} - m_{12} m_{21}} & \frac{m_{11}}{m_{11} m_{22} - m_{12} m_{21}}\end{matrix}\right]$$

## Solving equations

For solving equations and systems of equations we can use the `solve` function:

```python
solve(x**2 - 1, x)
```

```python
solve(x**4 - x**2 - 1, x)
```

System of equations:

```python
solve([x + y - 1, x - y - 1], [x,y])
```

In terms of other symbolic expressions:

```python
solve([x + y - a, x - y - c], [x,y])
```

## Further reading

* http://sympy.org/en/index.html - The SymPy projects web page.
* https://github.com/sympy/sympy - The source code of SymPy.
* http://live.sympy.org - Online version of SymPy for testing and demonstrations.

## Versions

```python
%reload_ext version_information

%version_information numpy, matplotlib, sympy
```

Software	Version

Python	3.6.3 64bit [GCC 7.2.0]

IPython	6.2.1

OS	Linux 4.4.86+ x86_64 with Ubuntu 17.10 artful

numpy	1.13.1

matplotlib	2.0.2

sympy	1.1.1

Sat Jan 20 12:39:59 2018 UTC

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

testing complete; Gopal

```python

```