#Training Neural Networks with Theano

**Training neural networks** involves quite a few tricky bits. We try to make everything clear and easy to understand, to get you training your neural networks as quickly as possible.

**Theano** allows us to write relatively concise code that follows the structure of the underlying maths. To run the code yourself, download the notebook at https://github.com/ASIDataScience/training-neural-networks-notebook

## Recognising hand-written digits

We will train a network to classify digits. More precisely, we want a network that when presented with an image of a hand-written digit will tell us what digit it is `('0', '1', ..., '9')`. The data we will use for this task is known as the MNIST dataset. It has a long tradition in neural networks research, as the dataset is quite small but still very tricky to classify correctly.

You can  download the MNIST dataset which we are using.

### Import Theano
You can install theano with the Python package manager `pip`. At the command line type

`pip install theano`

or check out the theano documentation if you run into trouble

The `theano.tensor` module contains useful functionality for manipulating vectors and matrices, like we will be doing here, so let's import it along with the full package.

```python
import theano
import theano.tensor as T
import numpy as np
```

    WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute optimized C-implementations (for both CPU and GPU) and will default to Python implementations. Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty string.

With the data loaded into a variable `mnist_dataset`, we split it into three part: a training set, used to teach the network to recognize digits, a validation set that could be used to tune and compare models, and finally a test set, used to see how well it has learned.

We then prepare the datasets, splitting each set into the images and their labels, and store them in Theano shared variables, a bit of theano magic that's explained later. Understanding this massaging of the data isn't crucial.

```python
import os
import gzip
import pickle

myPath = '' # mnist.pkl.gz is in this directory
f = gzip.open(os.path.join(myPath, 'mnist.pkl.gz'), 'rb')

try:        # for cross-platform, cross-version reasons, try two different pickle.load statements
    mnist_dataset = pickle.load(f, encoding='latin1')
except:
    mnist_dataset = pickle.load(f)
f.close()
```

```python
train_set, valid_set, test_set = mnist_dataset
prepare_data = lambda x: (theano.shared(x[0].astype('float64')),
                          theano.shared(x[1].astype('int32')))
(training_x, training_y), (valid_x, valid_y), (test_x, test_y) = map(prepare_data, (train_set, valid_set, test_set))
```

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

def plot_mnist_digit(image):
    '''Plot a single digit from the mnsist dataset'''
    image = np.reshape(image, [28,28])
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.matshow(image, cmap=matplotlib.cm.binary)
    plt.xticks(np.array([]))
    plt.yticks(np.array([]))
    plt.show()
```

Let's look at the first three images in the training set, then set about building a machine learning model that has the ability to recognise digits itself!

We've defined a function `plot_mnist_digit` exactly for printing out the training images - it's just for prettiness.

```python
for i in range(3):
    plot_mnist_digit(training_x.get_value()[i])             # training_x is a theano object containing *images*
    print ('Image class: '+ str(training_y.get_value()[i])) # training_y is a theano object containing *labels*
```

# The neural network model

The neural network model that we will build defines a probability distribution

$$P(Y = y \ |\ X = \boldsymbol{\textrm{x}} ; \theta),$$

where $Y$ represents the image class, which means it is a random variable that can take the values `0-9`, $X$ represents the image pixels and is a vector-valued random variable (we collapse the image matrix into a vector), and $\theta$ is the set of model parameters that we are going to learn.

In this tutorial we build two models: first implementing a logistic regression model, then extending it to a neural network model.

## Multi-class logistic regression

The equation for our first model is give by

$$P(Y = y \ |\ \boldsymbol{\textrm{x}} ; \theta) \propto \left[\sigma \left( \boldsymbol{\textrm{x}}^\boldsymbol{\textrm{T}} \boldsymbol{\textrm{W}}  + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}}\right)\right]_y,$$

where $[\boldsymbol{\textrm{x}}]_i$ is the $i$th entry of vector $\boldsymbol{\textrm{x}}$, and the use of the proportionality symbol $\propto$ means that the probability is equal to the expression on the right hand side times a constant chosen such that $\sum_y{P(Y = y \ |\ \boldsymbol{\textrm{x}} ; \theta)} = 1$

The parameter set $\theta$ for this model is $\theta = \{\boldsymbol{\textrm{W}}, \boldsymbol{\textrm{b}}\}$, where $\boldsymbol{\textrm{W}}$ is a matrix and $\boldsymbol{\textrm{b}}$ is a vector. We also use the non-linearity $\sigma$ given by

$$\sigma(t) = \frac{1}{1+e^{-t}}$$

When applied to a vector or matrix, the sigmoid function $\sigma(t)$ is applied entrywise.

## Understanding the logistic regression model

One way to think about the logistic regression model is that it takes the input ($\boldsymbol{\textrm{x}}$), puts it through a linear combination ($\boldsymbol{\textrm{x}}^\boldsymbol{\textrm{T}} \boldsymbol{\textrm{W}}  + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}}$) and then finally through a non-linearity: $\sigma(\boldsymbol{\textrm{x}}^\boldsymbol{\textrm{T}} \boldsymbol{\textrm{W}}  + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}})$.

The result of this operation is a vector representing the entire discrete distribution over all the possible classes - in our case the ten possible digits `0-9`. To get the probability of a particular class $y=6$ we extract the $6th$ entry of the probability vector: $\left[\sigma \left( \boldsymbol{\textrm{x}}^\boldsymbol{\textrm{T}} \boldsymbol{\textrm{W}}  + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}}\right)\right]_y$

Graphically the model looks like this:

Each indiviual entry of the vectors **x** and $P(Y)$ is shown as a circle -- known as the units (or artificial *neurons*) of the network. We have $D$ input units (the dimensionality of the input vector, which is the flattened matrix of pixels) and $C$ output units (the number of classes, which is the digits `0-9`). The model parameters **W** and **b** are represented as arrows. We also show the application of the sigmoid functions, but we do not represent the normalization that makes the probabilities sum up to $1$.

Another way to write the model above is using the $\textrm{SoftMax}$ function. A good exercise is deriving the $\textrm{SoftMax}$ function based on the fact that we can also write the same model using $\textrm{SoftMax}$ and the equal sign instead of the proportionality sign:

$$P(Y = y \ |\ \boldsymbol{\textrm{x}} ; \theta) = \left[\textrm{SoftMax} \left( \boldsymbol{\textrm{W}} \boldsymbol{\textrm{x}} + \boldsymbol{\textrm{b}}\right)\right]_y,$$

## Neural network with a single hidden layer

Our second model is given by

$$P(Y = y \ |\ \boldsymbol{\textrm{x}} ; \theta) = \left[ \textrm{SoftMax} \left( \boldsymbol{\textrm{h}} ^\boldsymbol{\textrm{T}}  \boldsymbol{\textrm{W}}_\boldsymbol{\textrm{hy}} + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}} _\boldsymbol{\textrm{hy}}\right)\right]_y, \\
\boldsymbol{\textrm{h}} = \tanh \left( \boldsymbol{\textrm{x}} ^\boldsymbol{\textrm{T}} \boldsymbol{\textrm{W}}_\boldsymbol{\textrm{xh}}  + \boldsymbol{\textrm{b}}^\boldsymbol{\textrm{T}}_\boldsymbol{\textrm{xh}}\right)$$

This is very similar to the logisitc regression model. Here we have introduced a new vector-valued random variable $h$. We call this a 'hidden' variable or 'latent' variable, as we do not have any data observations for it. This variable may not even correspond to any quantity in the real world, but we use it to increase the power of our statistical model.

We now also have more parameters: $\theta = \{\boldsymbol{\textrm{W}}_\boldsymbol{\textrm{xh}}, \boldsymbol{\textrm{W}}_\boldsymbol{\textrm{hy}}, \boldsymbol{\textrm{b}}_\boldsymbol{\textrm{xh}}, \boldsymbol{\textrm{b}}_\boldsymbol{\textrm{hy}}\}$.

$\tanh$ is the hyperbolic tangent function given by
$$\tanh(t) = \frac{e^t-e^{-t}}{e^t+e^{-t}}$$

Like with the sigmoid, when applied to a vector or matrix, $\tanh$ function is applied entrywise.

Graphically, this model looks like this:

Now our depiction shows a hidden layer with $M$ units (this number can be different from the number of input neurons and number of output neurons), and we have two different nonlinearities in the graph: $tanh$ and sigmoids (but again we are not graphically representing the SoftMax normalization).

## Teaching the network

The way we're going to make our network learn is by trying to find some values for our parameters $\theta$ so that the network is as likely as possible to guess the correct class. That is, given a data set of training images $\boldsymbol{\textrm{x}}_1, \boldsymbol{\textrm{x}}_2, \dots, \boldsymbol{\textrm{x}}_N$ and correct labels $y_1, y_2, \dots, y_N$, we want to find the parameters that maximize probability of the correct labels given the images.

This method of choosing parameters is called *maximum likelihood* (ML), and we can express it mathematically as finding the parameters $\theta$ which maximize the likelihood function:

$$\theta^* = {\arg\max}_\theta \ P( Y_1 = y_1, Y_2 = y_2, \dots, Y_N = y_N \ | \ \boldsymbol{\textrm{X}}_1 = \boldsymbol{\textrm{x}}_1,\boldsymbol{\textrm{X}}_2 = \boldsymbol{\textrm{x}}_2, \dots, \boldsymbol{\textrm{X}}_N = \boldsymbol{\textrm{x}}_N ; \theta)$$

And since our data points are independent, we can write this joint probability as a product of probabilities:

\begin{align}
P( Y_1 = y_1,\dots, Y_N = y_N \ | \ \boldsymbol{\textrm{X}}_1 = \boldsymbol{\textrm{x}}_1, \dots, \boldsymbol{\textrm{X}}_N = \boldsymbol{\textrm{x}}_N ; \theta) &= P( Y_1 = y_1 \ | \ \boldsymbol{\textrm{X}}_1 = \boldsymbol{\textrm{x}}_1 ; \theta) \times  \dots \times P( Y_N = y_N \ | \ \boldsymbol{\textrm{X}}_N = \boldsymbol{\textrm{x}}_N ; \theta) \\
&= \prod_{i=1}^N P( Y_i = y_i \ | \ \boldsymbol{\textrm{X}}_i = \boldsymbol{\textrm{x}}_i)
\end{align}

## Writing the likelihood function for an entire dataset

In our likelihood function above, each random variable pair $(X_1, Y_1), (X_2, Y_2), \dots, (X_N, Y_N)$ refers to a single data point. But since virtually all of our computations need to deal with multiple data points, we will find it both useful and computationally efficient to express both the mathematics and our Python code in terms of datasets. Thus, we will express the scalar random variables $Y_1, Y_2, \dots, Y_N$ as the vector-valued random variable $Y$, and the vector-valued random variables $X_1, X_2, \dots, X_N$ as the matrix-valued random variable $X$, where the matrix $X$ has as many **rows** as there are data points.

Using this notation, we rewrite the maximum likelihood equation above:

$$\theta^* = {\arg\max}_\theta \ P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)$$

Similarly, we can specify the logistic regression model it terms of multiple datapoints:

$$P(Y \ |\ X = \boldsymbol{\textrm{X}} ; \theta) = \textrm{SoftMax} \left(  \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)$$

Here the result is a matrix of probabilities with as many rows as there are data points, and as many columns as there are classes. We also consider the SoftMax to normalize the result of the linear combination $\left( \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)$ in such a way that each row of the result is a proper probability distribution summing to $1$. Note that we have had to multiply the bias vector $\boldsymbol{\textrm{b}}$ by the vertical vector of all ones $\boldsymbol{\textrm{1}}$ in order to add the bias term for every single data point.

The neural network model equations follow a similar pattern, which it would be a good exercise to write out for yourself.

## Computing the log-likelihood of a dataset

In most machine learning applications, it is better to maximize the log-likelihood rather than the likelihood. This is done because the log-likelihood tends to be simpler to compute and more numerically stable than the likelihood. In terms of the math, this doesn't make things much more complicated, as all we need to add is a $\log$ in front of the likelihood:

$$\theta^* = {\arg\max}_\theta \ \log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)$$

Since the logarithm of a product is the sum of the logarithms, we can write:

$$\log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta) = \sum_{i=1}^N \log P( Y_i = \boldsymbol{\textrm{y}}_i\ | \ X_i = \boldsymbol{\textrm{X}}_i ; \theta)$$

that is, the log joint probability is the sum of the log marginal probabilities.

Now let's plug in the probability of a dataset from above to obtain:

$$\log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta) = \sum \left[ \log\left( \textrm{SoftMax} \left( \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)\right)\right]_{\cdot,\boldsymbol{\textrm{y}}}$$

where we use the notation $[\boldsymbol{\textrm{M}}]_{\cdot,\boldsymbol{\textrm{y}}}$ to mean that from the matrix $\boldsymbol{\textrm{M}}$ we construct a new vector $(a_1, a_2, \dots, a_n)$ such that $a_i = M_{i, y_i} \forall i$. We use a slight abuse of notation and we use the sum symbol to indicate the summation of the entries in the vector; we need a summation because the log joint probability is the sum of the log marginal probabilities.

## Classifying a dataset

Once we have a set of parameters $\theta$ that we are happy with, we can use the model to classify new data.

We have built a model that gives us a distribution over classes given the data. How do we assign a class to a new data point? The simplest way is to choose the class with the highest probability under the model to be the class we assign. We can write this mathematically, again using vector notation:

$$\hat{\boldsymbol{\textrm{y}}} = {\arg\max} \ P(Y \ |\ \boldsymbol{\textrm{X}} ; \theta) $$

# Gradient ascent

Computing the maximum likelihood parameters is computationally unfeasible, so we're going to use a method called *gradient ascent* to find a set of parameters that are really good but perhaps not the absolute best.

The idea of gradient ascent is very simple. Given a function $f:\mathbb{R}^n \rightarrow \mathbb{R}$, we want to iteratively find points $f(\boldsymbol{\textrm{x}}_n)$ such that the value of the function gets progressively higher, that is: $f(\boldsymbol{\textrm{x}}_{n+1}) > f(\boldsymbol{\textrm{x}}_n) \forall n$. One way to do this is taking the direction of steepest ascent, which is just the gradient of the function $\frac{\partial f}{\partial \boldsymbol{\textrm{x}}}$, and taking a step in that direction times a constant $\lambda$ known as the *learning rate* that describes how big the step should be. We express this mathematically as:

$$ \boldsymbol{\textrm{x}}_{n+1} \leftarrow \boldsymbol{\textrm{x}}_n + \lambda \frac{\partial f}{\partial \boldsymbol{\textrm{x}}} $$

The last detail is choosing the starting point, $\boldsymbol{\textrm{x}}_0$, which we can arbitrarily choose by setting to zero or to some random value. Graphically, the algorithm looks like this, with each color representing the path from a different starting point:

Note that this method tends to find the top of the nearest hill ('local' maximum), and not the overall best point ('global' maximum).

It is also not guaranteed to increase the value of the function at each step; if the learning rate is too large, the algorithm could potentially jump across the top of the hill to a lower point on the other side of the hill. Much research goes into optimization methods, and many neural networks models are trained with methods that are more complicated than gradient ascent as it's presented here, but this same idea is at the base of all of those methods.

Finally, most people talk about and use gradient **descent** on the **negative** log-likelihood rather than gradient **ascent** on the log-likelihood; this is because gradient descent is the standard algorithm in the field of optimization. Gradient ascent in used in this tutorial to keep things a bit simpler.

## Optimizing our model with gradient ascent

This is extremely easy: we just apply the equation above to our parameters taking the gradient of the log-likelihood:

\begin{align}
\boldsymbol{\textrm{W}} &\leftarrow \boldsymbol{\textrm{W}} + \lambda \frac{\partial \log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)}{\partial \boldsymbol{\textrm{W}}} \\
\boldsymbol{\textrm{b}} &\leftarrow \boldsymbol{\textrm{b}} + \lambda \frac{\partial \log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)}{\partial \boldsymbol{\textrm{b}}} \\
\end{align}

# Coding with Theano

Coding with Theano is extremely simple: once we have the equations behind the model, we pretty much type them directly. Since we will be training on multiple data points, we are going to encode the model as we wrote it for datasets, using vectors and matrices.

We'll start with the logistic regression. Our model is:

$$P(Y \ |\ \boldsymbol{\textrm{X}} ; \theta) = \textrm{SoftMax} \left( \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)$$

The first thing we do is declare all of the variables ($\boldsymbol{\textrm{X}}, \boldsymbol{\textrm{y}}, \boldsymbol{\textrm{W}}, \boldsymbol{\textrm{b}}$) that we will be using and their types like you would in Java or C:

```python
n_classes = 10  # each digit is one of 0-9
dims = 28 * 28  # our input data is flattened 28x28 matrices of image pixels

X = T.dmatrix() # Theano double matrix
y = T.ivector() # Theano integer vector
W = theano.shared(np.zeros([dims,n_classes])) # Theano shared double matrix
b = theano.shared(np.zeros(n_classes))        # Theano shared double vector
```

As you can see above, we defined $W$ and $b$ to be *shared* variables. This means that the values of these variables are persistent -- their values live on after we have run Theano operations. This is opposed to regular Theano variables, which only take values when Theano runs, and otherwise only exist in the abstract.

The reason for making $W$ and $b$ shared variables is that we want to run multiple iterations of gradient descent, and to do that, we need their values to persist. Furthermore, we want to find good parameters through training, but we will then want to use the same parameters for prediction, so we need them to be persistent

Let's now write our statistical model in Theano. We basically copy the following equation into code:

$$P(Y \ |\ \boldsymbol{\textrm{X}} ; \theta) = \textrm{SoftMax} \left( \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)$$

```python
P = T.nnet.softmax(T.dot(X,W) + b) # the matrix of probabilities of all classes for all data points
```

Theano provides us with a **`T.nnet.softmax`** function to compute SoftMax, correctly normalizing the probabilities so that each row of the matrix **`P`** is a proper probability distribution that sums to $1$.

Note that we didn't need to multiply **`b`** by the **1** vector, Theano will do the correct addition for us automatically, just like numpy would do it. Here is a simple numpy example illustrating this:

```python
amatrix = np.zeros((3,2)) # 3x2 matrix of all zeros
avector = np.array((1,2)) # the vector [1,2]
amatrix + avector
```

    array([[ 1.,  2.],
           [ 1.,  2.],
           [ 1.,  2.]])

Our next equation is the log-likelihood (**LL**) of a dataset:

$$\log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta) = \sum \left[ \log\left( \textrm{SoftMax} \left( \boldsymbol{\textrm{X}} \boldsymbol{\textrm{W}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}\right)\right)\right]_{\cdot,\boldsymbol{\textrm{y}}}$$

```python
LL = T.mean(T.log(P)[T.arange(P.shape[0]), y]) # the log-likelihood (LL)
```

OK... there's a lot going on here.

We have used two important tricks:
- using mean instead of sum,
- using the strange indexing **`[T.arange(P.shape[0]), y]`**

Let's go over them one by one.

#### Use the mean instead of the sum
Imagine that we were to construct a new dataset that contains each data point in the original dataset twice. Then the log-likelihood of the new dataset will be double that of the original. More importantly, the gradient of the log-likelihood of the new dataset will also be double the gradient of the log-likelihood of the original dataset. But we would like the size of the gradient to not depend on the amount of duplication in our dataset, and the easiest way to accomplish that is to divide the gradient by the size of the dataset.

Since taking the mean is equivalent to taking the sum and then dividing by the number of data points, what we are computing here is a type of "normalized" log-likelihood that will cause our gradient descent algorithm to be robust to change in dataset size. The quantity we are computing in the code can be more precisely described as the average log-likelihood for a single datapoint.

#### Use indexing to create a vector from a matrix

The second thing we do is use **`[T.arange(P.shape[0]), y]`** to apply the mathematical operation denoted in the equation by $\left[ \cdot\right]_{\cdot,\boldsymbol{\textrm{y}}}$, that is, constructing a new vector $(a_1, a_2, \dots, a_n)$ from a matrix $\boldsymbol{\textrm{M}}$ such that $a_i = M_{i, y_i} \forall i$.

As cryptic as it may be, this is a peculiar, but standard numpy way to index. For example, given a matrix

**

M = np.array([[1,2,3,4],
              [5,6,7,8],
              [9,10,11,12]]

**

If we wanted to extract one element from each row, say the 1st element from the first row, and the last from the others

**
M[0,0], M[1,3], M[2,3]
**

We could write that as a single index expression by combining the indexes

**
M[(0,1,2), (0,3,3)]
**

But now the first index is just $(0 \dots \#\textrm{rows}-1)$, or, in code, **`np.arange(M.shape[0])`**.

So we can write:

```python
M = np.array([[1,  2,  3,  4],
              [5,  6,  7,  8],
              [9, 10, 11, 12]])

M[np.arange(M.shape[0]), (0,3,3)]
```

    array([ 1,  8, 12])

We're done with the model. There's one more thing we need to do, and that is specify the gradient updates

\begin{align}
\boldsymbol{\textrm{W}} &\leftarrow \boldsymbol{\textrm{W}} + \lambda \frac{\partial \log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)}{\partial \boldsymbol{\textrm{W}}} \\
\boldsymbol{\textrm{b}} &\leftarrow \boldsymbol{\textrm{b}} + \lambda \frac{\partial \log P( Y = \boldsymbol{\textrm{y}}\ | \ X = \boldsymbol{\textrm{X}} ; \theta)}{\partial \boldsymbol{\textrm{b}}} \\
\end{align}

```python
learning_rate = 0.5 # we tuned this parameter by hand
updates = [
        [W, W + learning_rate * T.grad(LL, W)],
        [b, b + learning_rate * T.grad(LL, b)]
        ]
```

## Theano functions

OK, we're done coding the model. What do we do next?

When working with Theano, the next step is to create a Theano function. A Theano function is the basic unit of Theano code that we call to do something. In our case, this something will be performing a single gradient ascent iteration.

We create a Theano function by calling the function **`theano.function`** (yes, we create a function by calling a function). **`theano.function`** has four important parameters that we provide in order to get a Theano function:

- `inputs`  -- the list of input variables of the Theano function, similar to the inputs of a Python function
- `outputs` -- the list of output variables of the Theano function, similar to the inputs of a Python function
- `updates` -- a list of updates for shared variables, in the format we used above when we defined the variable **`updates`**
- `givens` -- a dictionary that allows substituting some variables from the model with other variables

In our case, we want the input to be the training dataset, the updates to be the gradient ascent updates, and while we don't really need an output, it will be helpful to get the log-likelihood as an output to see that we are doing the right things.

However, we will use the **`givens`** instead of the **`input`** to provide the data to the function.

Doing it this way is more efficient, as we've already loaded up the training dataset into memory as a shared Theano function, when we first loaded the data.

Our Theano function will look like this:

```python
training_function = theano.function(
    inputs = [],             # use givens instead of the inputs as it's more efficient
    outputs = LL,            # output log-likelihood just to check that it is improving
    updates = updates,       # these are the gradient updates, the one part that's really important
    givens = {X: training_x, # we indicate that the model variables X and y defined in the abstract
              y: training_y} # should take the values in the shared variables training_x and training_y
    )
```

OK, let's run ten iterations of our code and see what the log-likelihood does

```python
for i in range(10):
    current_LL = training_function()
    print("Log-likelihood = " + str(current_LL) + "\t\t" +
          "Average probability of the correct class = " + str(np.exp(current_LL))
         )
```

    Log-likelihood = -2.3025850929940463		Average probability of the correct class = 0.1
    Log-likelihood = -1.8391209025064599		Average probability of the correct class = 0.158957103494
    Log-likelihood = -1.5250543679632202		Average probability of the correct class = 0.217609225573
    Log-likelihood = -1.3142440592188174		Average probability of the correct class = 0.268677350658
    Log-likelihood = -1.1682410159224519		Average probability of the correct class = 0.310913352197
    Log-likelihood = -1.0617673924666753		Average probability of the correct class = 0.34584402773
    Log-likelihood = -0.9820455376704271		Average probability of the correct class = 0.374544170518
    Log-likelihood = -0.9194230935951666		Average probability of the correct class = 0.398749015602
    Log-likelihood = -0.869394886438446		Average probability of the correct class = 0.419205139229
    Log-likelihood = -0.8276885884406652		Average probability of the correct class = 0.437058341404

## Using the model for classification

Great, it appears we're improving the log-likelihood of the model on the training data. Now to use it to classify. Recall our equation that expresses how to use the model to get the class:

$$\hat{\boldsymbol{\textrm{y}}} = {\arg\max} \ P(Y \ |\ \boldsymbol{\textrm{X}} ; \theta) $$

Let's put that in code:

```python
y_hat = T.argmax(P, axis=1)
```

Note that we had to specify **axis=1**, that is, we want to get the **argmax** for each **row**, as each row represents the distribution for one datapoint.

### Create a Theano function that classifies a dataset

Similarly to the training function, the classification function will use givens to pass in the test dataset, and output **`y_hat`** which we just defined

```python
classification_function = theano.function(
    inputs = [],
    outputs = y_hat,
    givens = {X:test_x} # don't need the true labels test_y here
    )
```

Now let's run the classification once, and print the first ten images, the true labels, and the labels assigned by the model

```python
test_y_hat = classification_function()
print ("Classification error: "+ str(100 * (1 - np.mean(test_y_hat == test_y.get_value()))) + "%")
for i in range(10):
    plot_mnist_digit(
        test_x.get_value()[i]                                # test_x is a theano object of images
    )
    print ('Image class: \t\t' + str(test_y.get_value()[i])) # test_y is a theano object of *true labels*
    print ('Model-assigned class: \t' + str(test_y_hat[i]))  # test_y_hat is a theano object of *predicted labels*
```

## Training a neural network

So far we have trained a logistic regression model. The neural network model is so similar that we can imlepement it with just a few changes to the code.

We need
- to declare the hidden layer variable
- to decide on the size of the hidden layer (we'll keep it small so it will run on your personal computer)
- new parameters

```python
H = T.dmatrix()  # Theano double matrix
hidden_layer_size = 20
W_xh = theano.shared(0.01 * np.random.randn(dims, hidden_layer_size))
W_hy = theano.shared(np.zeros([hidden_layer_size, n_classes]))
b_xh = theano.shared(np.zeros(hidden_layer_size))
b_hy = theano.shared(np.zeros(n_classes))
```

Remember our model? Let's write it using matrices and vector for an entire dataset:

$$\boldsymbol{\textrm{H}} = \tanh \left( \boldsymbol{\textrm{X}}  \boldsymbol{\textrm{W}}_\boldsymbol{\textrm{xh}}  + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}}_\boldsymbol{\textrm{xh}}\right), \\
P(Y \ |\ \boldsymbol{\textrm{x}} ; \theta) =  \textrm{SoftMax} \left( \boldsymbol{\textrm{H}} \boldsymbol{\textrm{W}}_\boldsymbol{\textrm{hy}} + \boldsymbol{\textrm{1b}}^\boldsymbol{\textrm{T}} _\boldsymbol{\textrm{hy}}\right)$$

Let's code it up:

```python
H =         T.tanh( T.dot(X, W_xh) + b_xh )
P = T.nnet.softmax( T.dot(H, W_hy) + b_hy )
```

Now let's add the gradient updates:

```python
LL = T.mean(T.log(P)[T.arange(P.shape[0]), y])  # the log-likelihood (LL)

updates = [
        [W_xh, W_xh + learning_rate * T.grad(LL, W_xh)],
        [W_hy, W_hy + learning_rate * T.grad(LL, W_hy)],
        [b_xh, b_xh + learning_rate * T.grad(LL, b_xh)],
        [b_hy, b_hy + learning_rate * T.grad(LL, b_hy)],
    ]
```

### Redefining Log-loss
The one extremely important thing we did here was redefined **`LL`**.

This is a crucial point about how Theano works:

Whenever we define a theano variable, like we did with **`P`**, we create a new object.
When we define a new theano variable in terms of another Theano variable, like we did with **`LL`**, using
**

`LL = T.mean(T.log(P)[T.arange(P.shape[0]), y])`

**
we implicitly create a new object for **`LL`** that has a reference to our variable **`P`** we just defined.

Now the crucial part: say we redefine **`P`**. Then our variable **`LL`** still has a reference to the *old* variable **`P`**, and we need to update the reference to **`LL`** by re-running the definition for **`LL`** for everything to work correctly.

Bugs in Theano are very commonly produced by exactly this. It is a good reason to always use Theano in scripts rather than in a notebook like we are here.

Phew, we are now ready to train!

```python
training_function = theano.function(
    inputs = [],
    outputs = LL,
    updates = updates,
    givens = {X: training_x[:5000],  # use only 10% of the data so model not too complicated
              y: training_y[:5000]}  # to train on a personal computer
    )
for i in range(60):                  # train more than for logistic regression as this model is more complex
    current_LL = training_function()
    print(
        "Log-likelihood = " + str(current_LL) + "\t\t" +
        "Average probability of the correct class = " + str(np.exp(current_LL))
    )
```

    Log-likelihood = -2.30258509299		Average probability of the correct class = 0.1
    Log-likelihood = -2.301260143		Average probability of the correct class = 0.100132582813
    Log-likelihood = -2.29985742336		Average probability of the correct class = 0.100273139311
    Log-likelihood = -2.29814933548		Average probability of the correct class = 0.100444561005
    Log-likelihood = -2.29585974449		Average probability of the correct class = 0.100674801444
    Log-likelihood = -2.2926097774		Average probability of the correct class = 0.101002523491
    Log-likelihood = -2.28785314718		Average probability of the correct class = 0.101484099578
    Log-likelihood = -2.28079743549		Average probability of the correct class = 0.102202674172
    Log-likelihood = -2.27031674023		Average probability of the correct class = 0.103279462141
    Log-likelihood = -2.25487812942		Average probability of the correct class = 0.104886325515
    Log-likelihood = -2.23253100489		Average probability of the correct class = 0.107256619276
    Log-likelihood = -2.20104023375		Average probability of the correct class = 0.110687957106
    Log-likelihood = -2.1582595724		Average probability of the correct class = 0.115526010828
    Log-likelihood = -2.10278232534		Average probability of the correct class = 0.122116188179
    Log-likelihood = -2.03470964692		Average probability of the correct class = 0.130718431493
    Log-likelihood = -1.95612616578		Average probability of the correct class = 0.141405141361
    Log-likelihood = -1.87084468946		Average probability of the correct class = 0.153993530151
    Log-likelihood = -1.78339741578		Average probability of the correct class = 0.168066185511
    Log-likelihood = -1.69781819817		Average probability of the correct class = 0.183082538427
    Log-likelihood = -1.61686684319		Average probability of the correct class = 0.19851971911
    Log-likelihood = -1.54193010867		Average probability of the correct class = 0.213967721666
    Log-likelihood = -1.47336522593		Average probability of the correct class = 0.229153034446
    Log-likelihood = -1.41092281327		Average probability of the correct class = 0.243918088414
    Log-likelihood = -1.35405500976		Average probability of the correct class = 0.258191167337
    Log-likelihood = -1.30209894369		Average probability of the correct class = 0.271960364056
    Log-likelihood = -1.25438406129		Average probability of the correct class = 0.285251491561
    Log-likelihood = -1.21029452586		Average probability of the correct class = 0.298109465553
    Log-likelihood = -1.16929950344		Average probability of the correct class = 0.310584428406
    Log-likelihood = -1.13096059734		Average probability of the correct class = 0.322723100526
    Log-likelihood = -1.09492559978		Average probability of the correct class = 0.334564497696
    Log-likelihood = -1.06091566234		Average probability of the correct class = 0.346138718988
    Log-likelihood = -1.02871055443		Average probability of the correct class = 0.357467598533
    Log-likelihood = -0.998135176742		Average probability of the correct class = 0.36856611137
    Log-likelihood = -0.969049151498		Average probability of the correct class = 0.379443660064
    Log-likelihood = -0.941339587098		Average probability of the correct class = 0.390104905683
    Log-likelihood = -0.914915833679		Average probability of the correct class = 0.40055033752
    Log-likelihood = -0.889704949206		Average probability of the correct class = 0.410776934935
    Log-likelihood = -0.86564729478		Average probability of the correct class = 0.420779096345
    Log-likelihood = -0.842692353352		Average probability of the correct class = 0.430549769435
    Log-likelihood = -0.820795127401		Average probability of the correct class = 0.440081594419
    Log-likelihood = -0.799913392634		Average probability of the correct class = 0.449367881001
    Log-likelihood = -0.780005880711		Average probability of the correct class = 0.45840331556
    Log-likelihood = -0.761031287056		Average probability of the correct class = 0.467184377286
    Log-likelihood = -0.742947906704		Average probability of the correct class = 0.475709499272
    Log-likelihood = -0.725713685377		Average probability of the correct class = 0.483979036908
    Log-likelihood = -0.709286504208		Average probability of the correct class = 0.491995108699
    Log-likelihood = -0.693624565104		Average probability of the correct class = 0.499761364693
    Log-likelihood = -0.678686790458		Average probability of the correct class = 0.507282723661
    Log-likelihood = -0.664433187553		Average probability of the correct class = 0.514565106971
    Log-likelihood = -0.650825153157		Average probability of the correct class = 0.521615186716
    Log-likelihood = -0.637825709613		Average probability of the correct class = 0.528440158193
    Log-likelihood = -0.625399672875		Average probability of the correct class = 0.53504754179
    Log-likelihood = -0.613513757875		Average probability of the correct class = 0.541445015994
    Log-likelihood = -0.602136629008		Average probability of the correct class = 0.547640281054
    Log-likelihood = -0.59123890443		Average probability of the correct class = 0.553640951444
    Log-likelihood = -0.580793122897		Average probability of the correct class = 0.559454474405
    Log-likelihood = -0.570773681465		Average probability of the correct class = 0.565088071363
    Log-likelihood = -0.561156751609		Average probability of the correct class = 0.570548698858
    Log-likelihood = -0.551920180443		Average probability of the correct class = 0.575843025592
    Log-likelihood = -0.543043382773		Average probability of the correct class = 0.580977422405

## Test the model

```python
y_hat = T.argmax(P, axis=1)
test_y_hat = classification_function()

print ("Classification error: " + str(100 * (1 - np.mean(test_y_hat == test_y.get_value()))) + "%")

for i in range(10):
    plot_mnist_digit(
        test_x.get_value()[i]                                # test_y is a theano object of *images*
    )
    print ('Image class: \t\t' + str(test_y.get_value()[i])) # test_y is a theano object of *true labels*
    print ('Model-assigned class: \t' + str(test_y_hat[i]))  # test_y_hat is a theano object of *predicted labels*
```

## We've trained a neural network with Theano!

Along the way we used a good chunk of mathematics, a hand-full of tricks but required very few lines of code.

Hopefully this leads nicely to using Theano more, in particular we would be excited to see you set theano to work with:
- smarter training algorithms,
- parallelized training, and
- training more popular network architectures

'Til next time!


# Shenfun - High-Performance Computing platform for the Spectral Galerkin method

# Shenfun - facts

1. Shenfun is named in honour of Professor Jie Shen for his seminal work on the spectral Galerkin method:-)
2. Shenfun is a high performance computing platform for solving partial differential equations (PDEs) with the spectral Galerkin method.
3. Shenfun has been run with 65,000 processors on a Cray XC40.
4. Shenfun is a high-level Python package originally developed for pseudo-spectral turbulence simulations.

# The Spectral Galerkin method

## Advantages

- Accuracy (spectral)
- Sparsity - very low memory use for coefficient matrices
- Efficiency - Very fast linear algebra solvers due to sparsity
- Robustness (round-off errors) - condition numbers much lower than for collocation methods

## Disadvantages

- No complex grids. Structured tensor product grids only
- Nonlinear terms must be implemented explicitly (pseudospectral)
- Generally considered more difficult to implement since we solve equations in spectral space

# The Spectral Galerkin method (in a nutshell)

approximates solutions $u(x)$ using global trial functions $\phi_k(x)$ and unknown expansion coefficients $\hat{u}_k$

$$
u(x) = \sum_{k=0}^{N-1}\hat{u}_k \phi_k(x)
$$

Multidimensional solutions are formed from outer (tensor) products of 1D bases

$$
u(x, y) = \sum_{k=0}^{N_0-1}\sum_{l=0}^{N_1-1}\hat{u}_{kl} \phi_{kl}(x, y)\quad \text{ or }\quad
u(x, y, z) = \sum_{k=0}^{N_0-1}\sum_{l=0}^{N_1-1} \sum_{m=0}^{N_2-1}\hat{u}_{klm} \phi_{klm}(x, y, z)
$$

where, for example

$$
\begin{align}
\phi_{kl}(x, y) &= T_k(x) L_l(y)\\
\phi_{klm}(x, y, z) &= T_k(x) L_l(y) \exp(\text{i}mz)
\end{align}
$$

$T_k$ and $L_k$ are Chebyshev and Legendre polynomials.

# The Spectral Galerkin method

solves PDEs, like Poisson's equation

\begin{align}
\nabla^2 u(x) &= f(x), \quad x \in [-1, 1] \\
u(\pm 1) &= 0
\end{align}

using variational forms by the method of weighted residuals. I.e., multiply PDE by a test function $v$ and integrate over the domain. For Poisson this leads to the problem:

Find $u \in H^1_0$ such that

$$(\nabla u, \nabla v)_w^N = -(f, v)_w^N \quad \forall v \in H^1_0$$

Here $(u, v)_w^{N}$ is a weighted inner product and $v(=\phi_j)$ is a test function.

# Weighted inner products

The weighted inner product is defined as

$$
 (u, v)_w = \int_{\Omega} u \overline{v} w \, d\Omega,
$$

where $w(\mathbf{x})$ is a weight associated with the chosen basis (different bases have different weights). The overline represents a complex conjugate (for Fourier).

$\Omega$ is a Cartesian product domain spanned by the chosen 1D bases.

# In Shenfun quadrature is used for the integrals

1D with Chebyshev basis:

$$
(u, v)_w ^N = \sum_{i=0}^{N-1} u(x_i) v(x_i) \omega_i \approx \int_{-1}^1 \frac{u v}{\sqrt{1-x^2}}  \, {dx},
$$

where $\{\omega_i\}_{i=0}^{N-1}$ are the quadrature weights associated with the chosen basis and quadrature rule. The associated quadrature points are denoted as $\{x_i\}_{i=0}^{N-1}$.

2D with mixed Chebyshev-Fourier:

$$
(u, v)_w^N = \int_{-1}^1\int_{0}^{2\pi} \frac{u \overline{v}}{2\pi\sqrt{1-x^2}} \, {dxdy} \approx \sum_{i=0}^{N_0-1}\sum_{j=0}^{N_1-1} u(x_i, y_j) \overline{v}(x_i, y_j) \omega^{(x)}_i \omega_j^{(y)} ,
$$

# Spectral Galerkin solution procedure

1. Choose function space(s) satisfying the correct boundary conditions
2. Transform PDEs to variational forms using inner products
3. Assemble variational forms and solve resulting linear algebra systems

# Orthogonal bases

| Family    | Basis                                    | Weight   | Domain    |
|  :---:    |         :---:                            | :---:    |  :---:   |
| Chebyshev | $$\{T_k\}_{k=0}^{N-1}$$                  | $$1/\sqrt{1-x^2}$$ | $$[-1, 1]$$ |
| Legendre  | $$\{L_k\}_{k=0}^{N-1}$$                  |  1       |$$[-1, 1]$$ |
| Jacobi    | $$\{J_k^{(\alpha,\beta)}\}_{k=0}^{N-1}$$ | $$(1-x)^{\alpha}(1+x)^{\beta}$$ |  $$[-1, 1]$$ |
| Fourier   | $$\{\exp(\text{i}kx)\}_{k=-N/2}^{N/2-1}$$| $$1/(2\pi)$$  |$$[0, 2\pi]$$ |
| Hermite   | $$\{H_k\}_{k=0}^{N-1}$$                  | $$e^{-x^2/2}$$        | $$[-\infty, \infty]$$|
| Laguerre  | $$\{La_k\}_{k=0}^{N-1}$$                 | $e^{-x/2}$        | $$[0, \infty]$$ |

```python
from shenfun import *
N = 8
C = FunctionSpace(N, 'Chebyshev', quad='GC', domain=[-2, 2])
L = FunctionSpace(N, 'Legendre')
x, w = C.points_and_weights()
print(C.points_and_weights())
C.sympy_basis(4)
```

# Jie Shen's bases with Dirichlet bcs

| family    | Basis                 | Boundary condition |
|-----------|-----------------------|----------|
| Chebyshev | $$\{T_k-T_{k+2}\}_{k=0}^{N-3}$$ | $$u(\pm 1) = 0$$ |
| Legendre  | $$\{L_k-L_{k+2}\}_{k=0}^{N-3}$$ | $$u(\pm 1) = 0$$ |
| Hermite   | $$\exp(-x^2)\{H_k\}_{k=0}^{N-1}$$ | $$u(\pm \infty) = 0$$ |
| Laguerre  | $$\exp(-x/2)\{La_k-La_{k+1}\}_{k=0}^{N-2}$$| $$u(0) = u(\infty) = 0$$ |

Composite bases are implemented using a stencil matrix, such that

$$
\phi_i = \sum_{j=0}^{N-1} K_{ij}Q_j
$$

where $K$ is the stencil matrix and $\{Q_j\}_{j=0}^{N-1}$ are the orthogonal polynomials, like $T_j$ or $L_j$.

```python
N = 8
C0 = FunctionSpace(N, 'Chebyshev', bc=(0, 0))
L0 = FunctionSpace(N, 'Legendre', bc=(0, 0))
H0 = FunctionSpace(N, 'Hermite')
La = FunctionSpace(N, 'Laguerre', bc=(0, 0))
L0.stencil_matrix().diags().toarray()
```

# Shen's bases with Neumann $u'(\pm 1) = 0$

| family    | Basis                 |
|-----------|-----------------------|
| Chebyshev | $$\left\{T_k-\frac{k^2}{(k+2)^2}T_{k+2}\right\}_{k=0}^{N-3}$$ |
| Legendre  | $$\left\{L_k-\frac{k(k+1)}{(k+2)(k+3)}L_{k+2}\right\}_{k=0}^{N-3}$$ |

```python
CN = FunctionSpace(N, 'Chebyshev', bc={'left': ('N', 0), 'right': ('N', 0)})
LN = FunctionSpace(N, 'Legendre', bc={'left': ('N', 0), 'right': ('N', 0)})
dict(CN.stencil_matrix())
```

# Shen's biharmonic bases $u(\pm 1) = u'(\pm 1) = 0$

| family    | Basis                 |
|-----------| :-----------------:   |
| Chebyshev | $$\left\{T_k-\frac{2(k+2)}{k+3}T_{k+2}+\frac{k+1}{k+3} T_{k+4}\right\}_{k=0}^{N-5}$$ |
| Legendre  | $$\left\{L_k-\frac{2(2k+5)}{(2k+7)}L_{k+2}+\frac{2k+3}{2k+7}L_{k+4}\right\}_{k=0}^{N-5}$$ |

```python
CB = FunctionSpace(N, 'Chebyshev', bc=(0, 0, 0, 0))
LB = FunctionSpace(N, 'Legendre', bc=(0, 0, 0, 0))
dict(CB.stencil_matrix())
```

# Multidimensional tensor product spaces

$$
\begin{align}
L_0 &= \{L_k(x)-L_{k+2}(x)\}_{k=0}^{N-3} \\
C_0 &= \{T_k(x)-T_{k+2}(x)\}_{k=0}^{N-3} \\
L_1 &= \{L_l(y)\}_{l=0}^{N-1} \\
LL(x, y) &= L_0(x) \otimes L_1(y) \\
CL(x, y) &= C_0(x) \otimes L_1(y)
\end{align}
$$

```python
L0 = FunctionSpace(N, 'Legendre', bc=(0, 0))
C0 = FunctionSpace(N, 'Chebyshev', bc=(0, 0))
L1 = FunctionSpace(N, 'Legendre')
LL = TensorProductSpace(comm, (L0, L1)) # comm is MPI.COMM_WORLD
CL = TensorProductSpace(comm, (C0, L1))
V = VectorSpace(LL) # For vector valued functions
f = Array(LL)
```

# Operators in shenfun

Act on instances of a `TestFunction`, `TrialFunction` or `Function`

- div
- grad
- curl
- Dx   (for a partial derivative)

# Assembly
- project
- inner

```python
L0 = FunctionSpace(N, 'Legendre', bc=(0, 0))
L1 = FunctionSpace(N, 'Legendre')
u = TrialFunction(L0)
v = TestFunction(L0)
uh = Function(L0)
g = Array(L0)
du = grad(u)  # vector valued expression
h = div(du)   # scalar valued expression

A = inner(Dx(u, 0, 2), v)
dict(A)
```

# The shenfun `Function` represents the solution

`uh = Function(L0)`

$$
u_h(x) = \sum_{k=0}^{N-1} \hat{u}_k \phi_{k}(x)
$$

The function evaluated for all quadrature points, $\{x_j\}_{j=0}^{N-1}$, is an `Array`

`uj = Array(L0)`

There is a (fast) `backward` transform for moving from `Function` to `Array`, and a `forward` transform to go the other way.

```python
uj = Array(L0)
uj = uh.backward(uj)
uh = uj.forward(uh)
```

# Projections

Project $g(\mathbf{x})$ to $V$:

Find $u$ in $V$ such that:

$$(u, v)_w = (I^Ng, v)_w \quad \text{for} \, v \in V $$

where $I^Ng$ is $\{g(x_j)\}_{j=0}^{N-1}$, i.e., $g(x)$ evaluated on the quadrature mesh.

Works if $g(x)$ is

- an `Array`, which is exactly a `Function` evaluated on the mesh
- an expression involving a `Function`, like `div(grad(uh))`
- a `sympy` expression, like `sin(x)`

```python
dudx = project(Dx(uh, 0, 1), L1) # Compute du/dx
wh = project(uj, L1)
import sympy as sp
x, y = sp.symbols('x,y')
ws = project(sp.sin(4*x), L1)
ws.eval(np.array([0.51]))
```

# Implementation matches mathematics

$$
A = (\nabla u, \nabla v)_w^N
$$

```python
A = inner(grad(u), grad(v))
```

```python
dict(A)
```

```python
print(A.diags().todense())
```

A diagonal stiffness matrix!

# Complete Poisson solver with error verification in 1D

```python
# Solve Poisson's equation
import matplotlib.pyplot as plt
from sympy import symbols, sin, cos, lambdify
from shenfun import *

# Use sympy to compute manufactured solution
x, y = symbols("x,y")
ue = sin(6*sp.pi*x)*(1-x**2) # `ue` is the manufactured solution
fe = ue.diff(x, 2) # `fe` is Poisson's right hand side for `ue`

SD = FunctionSpace(50, 'L', bc=(0, 0))
u = TrialFunction(SD)
v = TestFunction(SD)

b = inner(v, Array(SD, buffer=fe)) # Array is initialized with `fe`
A = inner(v, div(grad(u)))

uh = Function(SD)
uh = A.solve(b, uh)
ue = Array(SD, buffer=ue)
print("L2-error = ", np.sqrt(inner(1, (uh.backward()-ue)**2)))
```

```python
fe
```

```python
plt.plot(SD.mesh(), uh.backward(), SD.mesh(), ue)
```

# 2D - still closely matching mathematics

```python
L0 = FunctionSpace(N, 'Legendre', bc=(0, 0))
F1 = FunctionSpace(N, 'Fourier', dtype='d')
TP = TensorProductSpace(comm, (L0, F1))
u = TrialFunction(TP)
v = TestFunction(TP)
A = inner(grad(u), grad(v))
```

```python
print(A)
```

# ?

A is a list of two TPMatrix objects???

# `TPMatrix` is a Tensor Product matrix

A `TPMatrix` is the outer product of smaller matrices (2 in 2D, 3 in 3D etc).

Consider the inner product:

$$
\begin{align}
(\nabla u, \nabla v)_w &= \frac{1}{2\pi}\int_{-1}^{1}\int_{0}^{2\pi} \left(\frac{\partial u}{\partial x}, \frac{\partial u}{\partial y}\right) \cdot \left(\frac{\partial \overline{v}}{\partial x}, \frac{\partial \overline{v}}{\partial y}\right) {dxdy} \\
(\nabla u, \nabla v)_w &= \frac{1}{2\pi}\int_{-1}^1 \int_{0}^{2\pi} \frac{\partial u}{\partial x}\frac{\partial \overline{v}}{\partial x} {dxdy} + \int_{-1}^1 \int_{0}^{2\pi} \frac{\partial u}{\partial y}\frac{\partial \overline{v}}{\partial y} {dxdy}
\end{align}
$$

which, like `A`, is a sum of two terms. These two terms are the two `TPMatrix`es returned by `inner` above.

Now each one of these two terms can be written as the outer product of two smaller matrices.

Consider the first, inserting for test and trial functions

$$
\begin{align}
v &= \phi_{kl} = (L_k(x)-L_{k+2}(x))\exp(\text{i}ly) \\
u &= \phi_{mn}
\end{align}
$$

The first term becomes

$$
\small
\begin{align}
\int_{-1}^1 \int_{0}^{2\pi} \frac{\partial u}{\partial x}\frac{\partial \overline{v}}{\partial x} \frac{dxdy}{2\pi} &= \underbrace{\int_{-1}^1 \frac{\partial (L_m-L_{m+2})}{\partial x}\frac{\partial (L_k-L_{k+2})}{\partial x} {dx}}_{a_{km}} \underbrace{\int_{0}^{2\pi}  \exp(iny) \exp(-ily) \frac{dy}{2\pi}}_{\delta_{ln}} \\
   &= a_{km} \delta_{ln}
\end{align}
$$

and the second

$$
\small
\begin{align}
\int_{-1}^1 \int_{0}^{2\pi} \frac{\partial u}{\partial y}\frac{\partial \overline{v}}{\partial y} \frac{dxdy}{2\pi} &= \underbrace{\int_{-1}^1 (L_m-L_{m+2})(L_k-L_{k+2}) {dx}}_{b_{km}} \underbrace{\int_{0}^{2\pi} ln \exp(iny) \exp(-ily)\frac{dy}{2\pi}}_{l^2\delta_{ln}} \\
   &= l^2 b_{km} \delta_{ln}
\end{align}
$$

All in all:

$$
(\nabla u, \nabla v)_w = \left(a_{km} \delta_{ln} + l^2 b_{km} \delta_{ln}\right)
$$

The sum of two tensor product matrices!

```python
A = inner(grad(u), grad(v)) # <- list of two TPMatrices
for mat in A[1].mats:  # a_{km} and \delta_{ln}
    print(mat.diags().todense())
```

```python
plt.spy(A[1].diags(), markersize=1)
```

# 3D Poisson (with MPI and Fourier x 2)

```python
from sympy import symbols, sin, cos, lambdify
from shenfun import *

# Use sympy to compute manufactured solution
x, y, z = symbols("x,y,z")
ue = (cos(4*x) + sin(2*y) + sin(4*z))*(1-x**2)
fe = ue.diff(x, 2) + ue.diff(y, 2) + ue.diff(z, 2)

C0 = FunctionSpace(32, 'Chebyshev', bc=(0, 0))
F1 = FunctionSpace(32, 'Fourier', dtype='D')
F2 = FunctionSpace(32, 'Fourier', dtype='d')
T = TensorProductSpace(comm, (C0, F1, F2))
u = TrialFunction(T)
v = TestFunction(T)

# Assemble left and right hand
f_hat = inner(v, Array(T, buffer=fe))
A = inner(v, div(grad(u)))

# Solve
solver = chebyshev.la.Helmholtz(*A) # Very fast solver due to Jie Shen
u_hat = Function(T)
u_hat = solver(f_hat, u_hat)
assert np.linalg.norm(u_hat.backward()-Array(T, buffer=ue)) < 1e-12
print(u_hat.shape)
```

# Contour plot of slice with constant y

```python
X = T.local_mesh()
ua = u_hat.backward()
plt.contourf(X[2][0, 0, :], X[0][:, 0, 0], ua[:, 2], 100)
plt.colorbar()
```

# Run with MPI distribution of arrays

Here we would normally run from a bash shell

 [bash shell] mpirun -np 4 python poisson3D.py

Since we are in a Jupyter notebook, lets actually do this from python in a live cell:-) The exclamation mark '!' is a magic command to run bash scripts in Jupyter.

```python
!mpirun -np 8 python poisson3D.py
```

Note that Fourier bases are especially attractive because of features easily handled with MPI:

    - diagonal matrices
    - fast transforms

# Nonlinearities (convolution)
All treated with pseudo-spectral techniques

$$
\begin{align}
\hat{w}_k &= \widehat{u^2}_k
\end{align}
$$

That is, transform `Function`s to real space `Array`s, perform the nonlinear operation there and transform the nonlinear product back to spectral space (to a `Function`).

3/2-rule or 2/3-rule is possible for dealiasing with Fourier. Not for the remaining bases.

```python
uj = Array(SD)
#uj[:] = np.random.random(uj.shape)
uj = uh.backward(uj)
wh = Function(SD)
wh = SD.forward(uj*uj, wh)
```

# Mixed tensor product spaces

Solve several equations simultaneously

- Coupled equations
- Block matrices and vectors
- Tensor spaces of vectors, like velocity $u \in [\mathbb{R}^3]^3$

# Stokes equations
### lid-driven cavity - coupled solver

$$
\begin{align*}
\nabla^2 \mathbf{u} - \nabla p &= \mathbf{f} \quad \text{in }  \Omega, \quad \quad \Omega = [-1, 1]\times[-1, 1]\\
\nabla \cdot \mathbf{u} &= h \quad \text{in } \Omega  \\
\int_{\Omega} p dx &= 0 \\
\mathbf{u}(\pm 1, y) = \mathbf{u}(x, -1) = (0, 0) &\text{ and }\mathbf{u}(x, 1) = (1, 0) \text{ or } ((1-x^2)(1+x^2), 0)
\end{align*}
$$

Given appropriate spaces $V$ and $Q$ a variational form reads: find $(\mathbf{u}, p) \in V \times Q$ such that

$$
\begin{equation}
a((\mathbf{u}, p), (\mathbf{v}, q)) = L((\mathbf{v}, q)) \quad \forall (\mathbf{v}, q) \in V \times Q
\end{equation}
$$

where bilinear and linear forms are, respectively

$$
\begin{equation}
    a((\mathbf{u}, p), (\mathbf{v}, q)) = \int_{\Omega} (\nabla^2 \mathbf{u} - \nabla p) \cdot {\mathbf{v}} \, dx_w + \int_{\Omega} \nabla \cdot \mathbf{u} \, {q} \, dx_w,
\end{equation}
$$
$$
\begin{equation}
    L((\mathbf{v}, q)) = \int_{\Omega} \mathbf{f} \cdot {\mathbf{v}}\, dx_w + \int_{\Omega} h {q} \, dx_w
\end{equation}
$$

Using integration by parts for Legendre

$$
\begin{equation}
    a((\mathbf{u}, p), (\mathbf{v}, q)) = -\int_{\Omega} \nabla \mathbf{u} \cdot \nabla{\mathbf{v}} \, dx_w + \int_{\Omega} \nabla \cdot \mathbf{v} \, {p} \, dx_w + \int_{\Omega} \nabla \cdot \mathbf{u} \, {q} \, dx_w,
\end{equation}
$$

# Implementation of spaces, basis functions

```python
N = (40, 40)
family = 'Legendre'
D0X = FunctionSpace(N[0], 'Legendre', bc=(0, 0))
#D1Y = FunctionSpace(N[1], 'Legendre', bc=(1, 0)) # Regular lid
D1Y = FunctionSpace(N[1], 'Legendre', bc=(0, (1-x)**2*(1+x)**2)) # Regularized lid
D0Y = FunctionSpace(N[1], 'Legendre', bc=(0, 0))
PX = FunctionSpace(N[0], 'Legendre')
PY = FunctionSpace(N[1], 'Legendre')

# All required spaces
V1 = TensorProductSpace(comm, (D0X, D1Y))
V0 = TensorProductSpace(comm, (D0X, D0Y))
Q = TensorProductSpace(comm, (PX, PY), modify_spaces_inplace=True)
V = VectorSpace([V1, V0])
W = CompositeSpace([V0, V0])
VQ = CompositeSpace([V, Q])

# For inf-sup use P_N - P_{N-2} for velocity-pressure
PX.slice = lambda: slice(0, PX.N-2)
PY.slice = lambda: slice(0, PY.N-2)

# All required test and trial functions
up = TrialFunction(VQ)
vq = TestFunction(VQ)
u, p = up
v, q = vq
```

# Implementation Stokes - matrices and solve

```python
# Assemble matrices
A = inner(grad(v), -grad(u))
G = inner(div(v), p)
D = inner(q, div(u))

# Create Block matrix
sol = la.BlockMatrixSolver(A+G+D)

# Functions to hold solution and rhs
up_hat = Function(VQ)
fh_hat = Function(VQ)

# Solve Stokes problem. Note constraint for pressure
up_hat = sol(fh_hat, u=up_hat, constraints=((2, 0, 0),))

# Move solution to Array in real space
up = up_hat.backward()
u_, p_ = up
```

```python
X = Q.local_mesh(True)
plt.quiver(X[0], X[1], u_[0], u_[1])
```

# Sparsity pattern of block matrix

$$
M =
  \begin{bmatrix}
      A[0]+A[1] & 0 & G[0] \\
      0 & A[2]+A[3] & G[1] \\
      D[0] & D[1] & 0
  \end{bmatrix}
$$

```python
%matplotlib notebook
plt.figure(figsize=(6,4))
plt.spy(sol.mat.diags(), markersize=0.5)
```

# Block matrix

$$
M =
  \begin{bmatrix}
      A[0]+A[1] & 0 & G[0] \\
      0 & A[2]+A[3] & G[1] \\
      D[0] & D[1] & 0
  \end{bmatrix}
$$

where $D = G^T$ for the Legendre basis, making $M$ symmetric. For Chebyshev $M$ will not be symmetric.

Solver through [scipy.sparse.linalg](https://docs.scipy.org/doc/scipy/reference/sparse.linalg.html)

For Navier-Stokes of the lid-driven cavity, see https://github.com/spectralDNS/shenfun/blob/master/demo/NavierStokesDrivenCavity.py


# What have we learned about Navier-Stokes and CFD?
**by Jose Alvarez and Mahmud Nobe**

So far in this unit, this is what we have covered:

* **Session 7**.
During the first half of session 7, we familiarized ourselves with the physical concepts relevant to fluid dynamics, such as the **conservation of mass** and the **balance of momentum**. We reviewed calculus concepts such as the **divergence operator** and clarified doubts about the notation used in the Navier-Stokes equations.   During the second half, we discussed how to **discretize the partial differential equations** in the model.

* **Session 8**
In a way, session 8 was about algorithms. We wrote pseudocode and practiced how to
  - initialize the data structures and set up the global variables for the numerical solver;
  - set up the initial and boundary conditions;
  - proceed to compute the "starred" velocity, solve for the pressure Poisson equation, and update the velocity field; and
  - how to produce and interpret a visualization.

* **Session 9**.
The learning goal of session 9 is to make you capable of implementing your first CFD solver. To achieve this, we will assign the following three main resources:
  * Deshmukh, G. (2021). _Computational Fluid Dynamics using Python: Modeling Laminar Flow_. Towards Data Science. Retrieved from https://towardsdatascience.com/computational-fluid-dynamics-using-python-modeling-laminar-flow-272dad1ebec
  * Owkes, M. (2020). _A guide to writing your first CFD solver_. Retrieved from https://www.montana.edu/mowkes/research/source-codes/GuideToCFD_2020_02_28_v2.pdf
  * Barba, Lorena A., and Forsyth, Gilbert F. (2018). _CFD Python: the 12 steps to Navier-Stokes equations [Step 11: Cavity Flow with Navier–Stokes]_. Journal of Open Source Education. Retrieved from https://github.com/barbagroup/CFDPython/blob/master/lessons/14_Step_11.ipynb

Barba et. al. (2018) will be our main source of reference for sesion 9. We liked this implementation because it is the most straightforward, and we considered their approach to discretize the Laplacian of the pressure and solve the Poisson equation to be very pedagogically effective.

## The ultimate guide to write your first CFD solver by Alvarez & Mahmud (2021)

Nonetheless, one of the best ways to learn is to contrast two different ways to do the same task. In this guide, we took inspiration from the approach by [Owkes (2020)](https://www.montana.edu/mowkes/research/source-codes/GuideToCFD_2020_02_28_v2.pdf), but make a few simplifying assumptions:

* Accodring to Owkes (2020, p. 2), "_A staggered grid is used to store the variables where the pressure is stored at the cell center and the velocities are stored at the cell faces. This, **possibly odd**, choice is made since it allows for the solution to
have a tight coupling between pressure and the velocity and has been found to be the preferred methodology._" [emphasis mine] **We strongly disagree with this.** We believe this to be the wrong way of thinking about the problem. We prefer to think of a mesh as a net of points, i.e., we do not distinguish between the the border of a cell and its center.
* As a consequence, we do not need to overcomplicate the formulas as Owkes (2020) does in equations (11), (15), and (23/24) of his paper.

Furthermore, we also have our differences with Barba et. al. (2012):
  * You may notice that one important consideration we have for incompressible CFD solver is how to incorporate the mass conservation with the momentum conservation equation. Both Owekes and Barba connects them using a pressure poison equation with a source term. As we went over last class, Owekes separates the momentum equation in predictor and corrector steps. Then taking the divergence of the corrector steps, they found the source term in pressure poisson equation as, $b = \frac{\rho}{\Delta t} \nabla \cdot \mathbf{u}^*$. In this study guide we would follow this approch.
  * On the other hand, Barba et al. first write down the momentum equation separately for two component (u and v). Then they multiply both side of u-momentum equation with $\frac{\partial}{\partial x}$ and similarly both side of v-momentum equation with $\frac{\partial}{\partial y}$. Then they added these two equations and use mass conservation equation to simplify everything. Finally the source term they found for the pressure poisson equation is,
$$b = -\rho \left( \frac{\partial u}{\partial x}^2 + 2 \frac{\partial u}{\partial y} \frac{\partial v}{\partial x} + \frac{\partial v}{\partial y}^2\right)$$
One advantage in this approach is we do not need to consider the predictor, corrector steps, (i.e. no extra calculation of $u^*, v^*$) and we can only use the two momentum equation and this pressure poisson equation to solve for CFD. But the calculation to get the source term becomes very messier (if you checked the optional video, then you know that).

  * Which approach do you think is better? **For this guide, you'll realize that we took much of the mathematical model of Owkes (2020) and much of the numerical algorithm of Barba et. al. (2012).**

  * We did love the use of **vectorized operations** in Barba et. al. (2012) and the way they define boundary conditions. Our only addition to their approach is the use of **reusable functions** for finite differences of 1st-order and 2nd-order. (We also preferred to use **centered differences** as opposed to forward  differences whenever possible.) We realized  this leads to much more cleaner code that is more readable without sacrificing efficiency in most instances.

# Overview of mathematical model

A partial differential equation is an equation of the form:

$$f(u, \frac{\partial u}{\partial x}, ..., \partial^n_x u, \frac{\partial u}{\partial y}, ..., \partial^n_y u, x, y)=0$$
It is important to remember that there is nothing different about Navier-Stokes. We'll try to express the model in this basic form whenever possible. We hope that this will help to clarify any remaining doubts about the notation.

## Continuity equation (conservation of mass)

$$\nabla \mathbf{u}=0$$

What this is actually saying:

$$\frac{\partial u}{\partial x} + \frac{\partial u}{\partial y} = 0$$

## Conservation (balance) of momentum

$$\frac{\partial \mathbf{u}}{\partial t} + \mathbf{u}\cdot \nabla \mathbf{u} = - \frac{1}{\rho} \nabla p + \nu \nabla^2\mathbf{u}$$

What this is actually saying:

$$\frac{\partial u}{\partial t} +  \left( u \frac{\partial u}{\partial x}+ v\frac{\partial u}{\partial y} \right) = - \frac{1}{\rho} \frac{\partial p}{\partial x} + \nu \left(\frac{\partial^2 u}{\partial^2 x} + \frac{\partial^2 u}{\partial^2 y}\right)$$

$$\frac{\partial v}{\partial t} + \left( u \frac{\partial v}{\partial x}+ v\frac{\partial v}{\partial y} \right) = - \frac{1}{\rho} \frac{\partial p}{\partial y} + \nu \left(\frac{\partial^2 v}{\partial^2 x} + \frac{\partial^2 v}{\partial^2 y}\right)$$

# Initialize global variables and data structures

```python
import numpy as np
from matplotlib import pyplot as plt, cm
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline
import sys
```

```python
nx = 41
x_max = 2
x_min = 0

ny = 41
y_max = 2
y_min = 0

nt = 500
dx = (x_max - x_min) / (nx - 1)
dy = (y_max - y_min) / (ny - 1)

x = np.linspace(x_min, x_max, nx)
y = np.linspace(y_min, x_max, ny)
X, Y = np.meshgrid(x,y)

rho = 1 # density
nu = .1 # viscosity coefficient
dt = .001 # timestep

u = np.zeros((ny, nx))
v = np.zeros((ny, nx))
p = np.zeros((ny, nx))

```

# Define boundary conditions

There are two main usual types of boundary conditions: **Dirichlet (or "fixed")** boundary conditions and **Neumann (or "second-type")** boundary conditions. The former specifies the values of the solution at the boundaries of the domain, and the latter the values of the derivative at the boundaries of the domain.

If you read section 8 in Owkes (2020), you'll realize that his goal is to achieve a $u_{top}=0$ boundary condition. (He means a Dirichlet boundary condition, i.e., an specific value of the function.) Owkes (2020) explains how he achieved this in section 8, but in our opinion, his approach is unnecessarily cumbersome. At least, he needs to account for the fact that his discretization of the velocity field does not coincide with the boundaries of the continuous domain. Because Barba et. al. (2012) did things differently (their discretization coincides with the boundaries of the continuous domain), they don't have that problem: they can go ahead and define the (Dirichlet) boundary conditions "directly."

```python
def set_velocity_boundary(u,v):
    # Velocidty boundary conditions (Dirichlet or "fixed")
    u[0, :]  = 0
    u[:, 0]  = 0
    u[:, -1] = 0
    u[-1, :] = 1    # set velocity on cavity lid equal to 1
    v[0, :]  = 0
    v[-1, :] = 0
    v[:, 0]  = 0
    v[:, -1] = 0

    return u, v
```

The following implementation of Neumann boundary conditions for the pressure comes from Barba et. al. (2012). You'll realize that is also very straightforward:

```python
def set_pressure_boundary(p):
    # Pressure boundary conditions (Neumann or "second-type")
    '''
    Trick question: Would the approach below break depending on
    whether you use forward, backward, or centered differences?
    Why or why not?
    '''
    p[:, -1] = p[:, -2] # dp/dx = 0 at x = 2
    p[0, :] = p[1, :]   # dp/dy = 0 at y = 0
    p[:, 0] = p[:, 1]   # dp/dx = 0 at x = 0
    # Pressure boundary conditions (Dirichlet or "fixed")
    p[-1, :] = 0        # p = 0 at y = 2

    return p
```

The intuition for the previous is nothing out of this world: If you want to set up Dirichlet (or "fixed") boundary conditions, define the values directly on the boundaries (no joke, Einstein); if you want to set up Neumann (or "second-type") boundary conditions, make sure that the rate of change across the boundaries is such that the finite difference approximation matches the desired value.

**Note that for Neumann boundary condition, the boundary value of pressure depends on the pressure of the nearest column/row. So it changes in every step.** Thus it is always a best practice to include the boundary conditions inside of the final update function. But note that if we have a Dirichlet or fixed boundary condition, then it will work even if we just define the boundary condition once and not have that inside of our update function.

# Discretize the domain: Approximate derivatives with finite differences

```python
def diff_1st_x(f,dx):
  return (f[1:-1, 2:] - f[1:-1,0:-2])/(2*dx)

def diff_1st_y(f,dy):
  return (f[2:, 1:-1] - f[0:-2,1:-1])/(2*dy)

def diff_2nd_x(f,dx):
  return (
      # f(i-1,j) + 2*f(i,j) + f(i+1,j)
      (f[1:-1,0:-2] - 2*f[1:-1,1:-1] + f[1:-1,2:])
      /dx**2
      )

def diff_2nd_y(f,dy):
  return (
      # f(i,j-1) + 2*f(i,j) + f(i,j+1)
      (f[0:-2,1:-1] - 2*f[1:-1,1:-1] + f[2:,1:-1])
      /dy**2
      )

def laplacian(f, dx, dy):
  return (diff_2nd_x(f, dx) +
          diff_2nd_y(f, dy))
```

# Temporal discretization

Appreciating the partial derivatives explicitly is wonderful, but it can get pretty hard to express the ideas without the compact notation:

$$\frac{\mathbf{u}^{n+1}-\mathbf{u}^n}{\Delta t}=-\frac{1}{\rho}\nabla p^{n+1} - \mathbf{u}^n \cdot \nabla \mathbf{u}^n + \nu \nabla ^2 \mathbf{u}^n$$

Which we can rewrite as:

$$\mathbf{u}^{n+1}=\left( -\frac{1}{\rho}\nabla p^{n+1}  - \mathbf{u}^n \cdot \nabla \mathbf{u}^n + \nu \nabla ^2 \mathbf{u}^n \right)\Delta t + \mathbf{u}^n $$

Which is actually also easy to breakdown (take note of the the slight abuse of notation, $\frac{\partial u^n}{\partial x}=\frac{u^n(i+1,j)-u^n(i-1,j}{2\Delta x}$, which conveys the finite difference approximation at time $n$):

$$u^{n+1} = \left( - \frac{1}{\rho} \frac{p^{n+1}-p^n}{\Delta x} -\left( u^n \frac{\partial u^n}{\partial x}+ v^n\frac{\partial u^n}{\partial y} \right) + \nu \left(\frac{\partial^2 u^n}{\partial^2 x} + \frac{\partial^2 u^n}{\partial^2 y}\right) \right) \Delta t + u^n$$

$$v^{n+1} = \left( - \frac{1}{\rho} \frac{p^{n+1}-p^n}{\Delta y} -\left( u^n \frac{\partial v^n}{\partial x}+ v^n\frac{\partial v^n}{\partial y} \right) + \nu \left(\frac{\partial^2 v^n}{\partial^2 x} + \frac{\partial^2 v^n}{\partial^2 y}\right) \right) \Delta t + v^n$$

```python
def compute_vel_star(un, vn, dx, dy, dt):
  u_star = un.copy()
  v_star = vn.copy()

  # Equation (7), Owkes (2020)
  u_star[1:-1,1:-1] = (nu * laplacian(un, dx, dy) -
                         (un[1:-1, 1:-1]*diff_1st_x(un,dx) +
                          vn[1:-1, 1:-1]*diff_1st_y(un,dy))
                      )*dt + un[1:-1, 1:-1]
  '''
  Pre-class challenge:
  * Can you rewrite this function
  to avoid the repetitive lines of code?
  * If so, do your changes sacrifice
  efficiency for readibility? Why or why not?
  '''

  # Equation (12), Owkes (2020)
  v_star[1:-1,1:-1] = (nu * laplacian(vn, dx, dy) -
                         (un[1:-1, 1:-1]*diff_1st_x(vn,dx) +
                          vn[1:-1, 1:-1]*diff_1st_y(vn,dy))
                      )*dt + vn[1:-1, 1:-1]

  return u_star, v_star
```

## What about the pressure term?

### $\mathbf{u}^*$ and the _predictor step_.

You may have noticed that we dropped the pressure term in the previous cell, and instead calculated $u^*$ and $v^*$. This is known as the **_predictor step_**:

$$\mathbf{u}^*=\left(- \mathbf{u}^n \cdot \nabla \mathbf{u}^n + \nu \nabla ^2 \mathbf{u}^n \right)\Delta t + \mathbf{u}^n $$

This is what we actually implemented in our code above, the preditor step or $\mathbf{u}^*$. It's really nothing more than the **_value of the velocity without the influence of pressure_**.

We will update $\mathbf{u}^{n+1}$ from $\mathbf{u}^*$ later (the **corrector step**). Before, we need to compute the pressure with the Poisson Equation.

# The pressure Poisson Equation

The **pressure Poisson Equation** is (see Owkes (2020)):

$$ \nabla^2 p^{n+1} = -\frac{\rho}{\Delta t} \nabla \cdot \mathbf{u}^*$$

Except that is not true. As it turns, there is a typo in equation (6) of Owkes (2020). The right hand side should be positive, as follows:

$$\nabla ^2 p^{n+1} = \frac{\rho}{\Delta t} \nabla \cdot \mathbf{u}^*$$

This took us a while to catch in our the code (try to change the sign and you'll see what happens). Our program is more similar to Barba et. al. (2012), but this bit is really based on the model that Owkes (2020) follows for the **source term**, which he calls $\mathbf{R}$ instead of $b$. Appendix A shows how to derive the pressure Poisson Equation in terms of the starred velocity correctly.

Anyways, here's how it continues to break down:

$$\frac{\partial ^2 p^{n+1}}{\partial x^2}+ \frac{\partial ^2 p^{n+1}}{\partial y^2} = \frac{\rho}{\Delta t} \left( \frac{\partial u^*}{\partial x} + \frac{\partial v^*}{\partial y} \right)$$

We will combine the approaches of Owkes (2020) and Barba et. al. (2012). The divergence operator applied to the predictor step $\mathbf{u}^*$ allows for the code to follow the math as in Owkes (2020) more closely, while the discretization of the Laplacian of the pressure as in Barba et. al. (2012) allows for a more pedagogically intuitive way to solve the Poisson Equation:

Let $b = \frac{\rho}{\Delta t} \left( \frac{\partial u^*}{\partial x} + \frac{\partial v^*}{\partial y} \right)$

```python
def get_b(u_star, v_star, dx, dy, dt):
  '''
  Right hand side of equation (6), Owkes (2020)
  '''
  # b comes from starred velocities, which have boundary
  # conditions (the starred velocity is just the velocity
  # without the influence of pressure, so it still maintains
  # the boundary conditions of the velocity),
  # but b does not really have boundary conditions of its own.
  # (P.S. Important point is how the value is used in pressure_poisson function,
  # i.e., `b[1:-1, 1:-1]` or simply `b`)
  # b = np.zeros((ny, nx))

  # (Divergence of vector-valued function)
  divergence_vel_star = (diff_1st_x(u_star,dx) +
                         diff_1st_y(v_star,dy))
  # b[1:-1,1:-1] = (1*rho/dt) * divergence_vel_star
  b = (rho/dt) * divergence_vel_star

  return b
```

then

$$\frac{\partial ^2 p^{n+1}}{\Delta x^2}+ \frac{\partial ^2 p^{n+1}}{\Delta y^2} = b$$

$$\frac{p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j)}{\Delta x^2}+ \frac{p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j)}{\Delta y^2} = b$$

$$\frac{\left(p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j)\right) \Delta y^2 + \left( p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j) \right) \Delta x^2}{\Delta x^2 \Delta y^2} = b$$

$$\left(p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j)\right) \Delta y^2 + \left( p^{n+1}(i-1,j) - 2p^{n+1}(i,j) + p^{n+1}(i+1,j) \right) \Delta x^2 = b^n(i,j) \Delta x^2 \Delta y^2$$

$$p^{n+1}(i-1,j)\Delta y^2 - 2p^{n+1}(i,j)\Delta y^2 + p^{n+1}(i+1,j)\Delta y^2  + p^{n+1}(i-1,j)\Delta x^2 - 2p^{n+1}(i,j)\Delta x^2 + p^{n+1}(i+1,j)\Delta x^2   = b^n(i,j) \Delta x^2 \Delta y^2$$

$$ - 2\left( \Delta y^2 + \Delta x^2 \right) p^{n+1}(i,j)    = - \Delta y^2 \left(p^{n+1}(i-1,j) + p^{n+1}(i+1,j) \right) - \Delta x^2 \left( p^{n+1}(i-1,j) + p^{n+1}(i+1,j) \right) + b^n(i,j) \Delta x^2 \Delta y^2$$

finally
$$ p^{n+1}(i,j)    = \frac{\Delta y^2 \left(p^{n+1}(i-1,j) + p^{n+1}(i+1,j) \right) + \Delta x^2 \left( p^{n+1}(i-1,j) + p^{n+1}(i+1,j) \right) - b^n(i,j) \Delta x^2 \Delta y^2}{2\left( \Delta y^2 + \Delta x^2 \right) }$$

```python
def pressure_poisson(p, b, dx, dy):
  pn = p.copy()

  # (pn(i-1,j) + pn(i+1,j)) * dy**2
  term1 = (pn[1:-1,0:-2] + pn[1:-1,2:]) * dy**2
  # (pn(i,j-1) + pn(i,j+1)) * dx**2
  term2 = (pn[0:-2,1:-1] + pn[2:,1:-1]) * dx**2
  # term 3 is -b * dx^2  * dy^2
  term3 = -(b * (dx**2 * dy**2))

  p[1:-1,1:-1] = (
      (term1 + term2 + term3)
      / (2*(dx**2+dy**2))
  )

  # reset boundary condition
  p = set_pressure_boundary(p)

  return p
```

# Update velocities: _**Corrector step**_

Finally, we have the pressure we were missing. Now we can update the velocity from $\mathbf{u}^*$ into $\mathbf{u}^{n+1}$.

Recall that the corrector step formula from Owkes (2020) is:

$$\mathbf{u}^{n+1}=\Delta t\left( - \frac{1}{\rho}  \nabla p^{n+1}\right) + \mathbf{u}^*$$

which can be broken down as:

$$u^{n+1}=-\frac{\Delta t}{\rho}\left(\frac{\partial p^{n+1}}{\partial x} \right) + u^*$$
$$v^{n+1}=-\frac{\Delta t}{\rho}\left( \frac{\partial p^{n+1}}{\partial y}\right) + v^*$$

```python
def corrector_step(u_star, v_star, p, dx, dy, dt, rho):
  u = np.empty_like(u_star)
  v = np.empty_like(u_star)

  u[1:-1,1:-1] = (-dt/rho)*(diff_1st_x(p, dx)) + u_star[1:-1,1:-1]
  v[1:-1,1:-1] = (-dt/rho)*(diff_1st_y(p, dy)) + v_star[1:-1,1:-1]

  # reset velocity boundary
  u, v = set_velocity_boundary(u, v)

  return u, v
```

# Run simulation

```python
def simulate_cavity_flow(nt, u, v, dt, dx, dy, p, rho, nu):
  for n in range(nt):
    un = u.copy()
    vn = v.copy()

    # Predictor step (starred velocities)
    u_star, v_star = compute_vel_star(un, vn, dx, dy, dt)

    # Solve pressure Poisson Equation
    b = get_b(u_star, v_star, dx, dy, dt)
    p = pressure_poisson(p, b, dx, dy)

    # Corrector step (update velocities)
    u, v = corrector_step(u_star, v_star, p, dx, dy, dt, rho)

  return u, v, p
```

Recall that all of the necessary variables were initialized in the top of the notebook.

```python
u, v, p = simulate_cavity_flow(1000, u, v, dt, dx, dy, p, rho, nu)
```

# Visualize results

```python
fig = plt.figure(figsize=(11,7), dpi=100)
# plotting the pressure field as a contour
plt.contourf(X, Y, p, alpha=0.5, cmap=cm.viridis)
plt.colorbar()
# plotting the pressure field outlines
plt.contour(X, Y, p, cmap=cm.viridis)
# plotting velocity field
plt.quiver(X[::2, ::2], Y[::2, ::2], u[::2, ::2], v[::2, ::2])
plt.xlabel('X')
plt.ylabel('Y');
```

```python
fig = plt.figure(figsize=(11, 7), dpi=100)
plt.contourf(X, Y, p, alpha=0.5, cmap=cm.viridis)
plt.colorbar()
plt.contour(X, Y, p, cmap=cm.viridis)
plt.streamplot(X, Y, u, v)
plt.xlabel('X')
plt.ylabel('Y');
```

# References:
Owkes, M. (2020). A guide to writing your first CFD solver.

Barba, Lorena A., and Forsyth, Gilbert F. (2018). CFD Python: the 12 steps to Navier-Stokes equations [Step 11: Cavity Flow with Navier–Stokes]. Journal of Open Source Education.

# Appendix A:
Pressure Poisson equation by Owekes

Corrector Term:

\begin{align}
    \mathbf{u}^{n+1} - \mathbf{u}^*=\Delta t\left( - \frac{1}{\rho}  \nabla p^{n+1}\right)\\
    \mathbf{u}^{n+1} - \mathbf{u}^*= - \frac{\Delta t}{\rho}  \nabla p^{n+1}
\end{align}

Taking the divergence of both side,
\begin{align}
    \nabla \cdot \mathbf{u}^{n+1} - \nabla \cdot \mathbf{u}^*= - \frac{\Delta t}{\rho}  \nabla^2 p^{n+1}
\end{align}

From mass conservation, we know that,

\begin{align}
    \nabla \cdot \mathbf{u}^{n+1} &= 0\\
    \therefore \nabla \cdot \mathbf{u}^* &= \frac{\Delta t}{\rho}  \nabla^2 p^{n+1}\\
    \nabla^2 p^{n+1} &= \frac{\rho}{\Delta t} \nabla \cdot \mathbf{u}^*
\end{align}

```python

```


```
%%capture
!pip install --upgrade --quiet pip
!pip install  --quiet cirq==0.7
```

# QAOA Experiment
In this experiment, you are going to implement a hardware compatible QAOA algorithm for determining the Max-Cut of the processor's hardware graph (where you assign random edge weights). You will need to:

1. Define a random set of weights over the hardware graph.
2. Construct a QAOA circuit using Cirq
3. Calculate the expectated value of the QAOA cost function.
4. Create an outer loop optimization to minimize your cost function.
5. Compare cuts found from QAOA with the optimal cut.

## 1. Defining a random set of weights over the hardware graph
In order to make the problem easily embedable on a quantum device you are going to look at the problem of Max-Cut on the same graph that the device's qubit connectivity defines, but with random valued edge weights.

```
import cirq
import sympy
import numpy as np
import matplotlib.pyplot as plt
working_device = cirq.google.Bristlecone
print(working_device)
```

                                                 (0, 5)────(0, 6)
                                                 │         │
                                                 │         │
                                        (1, 4)───(1, 5)────(1, 6)────(1, 7)
                                        │        │         │         │
                                        │        │         │         │
                               (2, 3)───(2, 4)───(2, 5)────(2, 6)────(2, 7)───(2, 8)
                               │        │        │         │         │        │
                               │        │        │         │         │        │
                      (3, 2)───(3, 3)───(3, 4)───(3, 5)────(3, 6)────(3, 7)───(3, 8)───(3, 9)
                      │        │        │        │         │         │        │        │
                      │        │        │        │         │         │        │        │
             (4, 1)───(4, 2)───(4, 3)───(4, 4)───(4, 5)────(4, 6)────(4, 7)───(4, 8)───(4, 9)───(4, 10)
             │        │        │        │        │         │         │        │        │        │
             │        │        │        │        │         │         │        │        │        │
    (5, 0)───(5, 1)───(5, 2)───(5, 3)───(5, 4)───(5, 5)────(5, 6)────(5, 7)───(5, 8)───(5, 9)───(5, 10)───(5, 11)
             │        │        │        │        │         │         │        │        │        │
             │        │        │        │        │         │         │        │        │        │
             (6, 1)───(6, 2)───(6, 3)───(6, 4)───(6, 5)────(6, 6)────(6, 7)───(6, 8)───(6, 9)───(6, 10)
                      │        │        │        │         │         │        │        │
                      │        │        │        │         │         │        │        │
                      (7, 2)───(7, 3)───(7, 4)───(7, 5)────(7, 6)────(7, 7)───(7, 8)───(7, 9)
                               │        │        │         │         │        │
                               │        │        │         │         │        │
                               (8, 3)───(8, 4)───(8, 5)────(8, 6)────(8, 7)───(8, 8)
                                        │        │         │         │
                                        │        │         │         │
                                        (9, 4)───(9, 5)────(9, 6)────(9, 7)
                                                 │         │
                                                 │         │
                                                 (10, 5)───(10, 6)

Since a circuit covering the entire Bristlecone device cannot be easily simulated, a small subset of the device graph will be used instead.

```
import networkx as nx

# Identify working qubits from the device.
device_qubits = working_device.qubits
working_qubits = sorted(device_qubits)[:12]

# Populate a networkx graph with working_qubits as nodes.
working_graph = nx.Graph()
for qubit in working_qubits:
  working_graph.add_node(qubit)

# Pair up all neighbors with random weights in working_graph.
for qubit in working_qubits:
  for neighbor in working_device.neighbors_of(qubit):
    if neighbor in working_graph:
      # Generate a randomly weighted edge between them. Here the weighting
      # is a random 2 decimal floating point between 0 and 5.
      working_graph.add_edge(
          qubit, neighbor, weight=np.random.randint(0,500) / 100)

nx.draw_circular(working_graph, node_size=1000, with_labels=True)
plt.show()
```

## 2. Construct the QAOA circuit
Now that you have created a Max-Cut problem graph, it's time to generate the QAOA circuit following [Farhi et al.](https://arxiv.org/abs/1411.4028). For simplicity $p=1$ is chosen.

```
from cirq.contrib.svg import SVGCircuit

alpha = sympy.Symbol('alpha')
beta = sympy.Symbol('beta')

qaoa_circuit = cirq.Circuit(
    # Prepare uniform superposition on working_qubits == working_graph.nodes
    cirq.H.on_each(working_graph.nodes()),

    # Do ZZ operations between neighbors u,v in the graph. u is a qubit v is
    # its neighbor qubit and w is the weight between these qubits.
    (cirq.ZZ(u, v) ** (alpha * w['weight']) for (u, v, w) in working_graph.edges(data=True)),

    # Apply X operations along all nodes of the graph. Again working_graph's
    # nodes are the working_qubits. Not here we use a moment
    # which will force all of the gates into the same line.
    cirq.Moment(cirq.X(qubit) ** beta for qubit in working_graph.nodes()),

    # All relevant things can be computed in computational basis.
    (cirq.measure(qubit) for qubit in working_graph.nodes()),
)
SVGCircuit(qaoa_circuit)
```

## 3. Calculating the expected value of the QAOA cost function
Now that you have created your parameterized QAOA circuit, you are going to need a way to calculate the expectation values of the cost function. Since the cost function is the (weighted) sum of all $ZZ$ pairs in the graph you can compute all of these values simultaneously.

First you need a way to estimate the expectation value given the samples from each qubit.

```
def estimate_cost(graph, samples):
    """Estimate the cost function of the QAOA on the given graph using the
    provided computational basis bitstrings."""
    cost_value = 0.0

    # Loop over edge pairs and compute contribution.
    for u, v, w in graph.edges(data=True):
      u_samples = samples[str(u)]
      v_samples = samples[str(v)]

      # Determine if it was a +1 or -1 eigenvalue.
      u_signs = (-1)**u_samples
      v_signs = (-1)**v_samples
      term_signs = u_signs * v_signs

      # Add scaled term to total cost .
      term_val = np.mean(term_signs) * w['weight']
      cost_value += term_val

    return -cost_value
```

Now if you run `qaoa_circuit` with a certain set of values for your placeholders you can use `estimate_expectation` to calculate the expectation value of the cost function for the circuit:

```
alpha_value = np.pi / 4
beta_value = np.pi / 2
sim = cirq.Simulator()
sample_results = sim.sample(qaoa_circuit, params={alpha: alpha_value, beta: beta_value}, repetitions=20000)
print(f'Alpha={alpha_value} Beta={beta_value}')
print(f'Estimated cost: {estimate_cost(working_graph, sample_results)}')
```

    Alpha=0.7853981633974483 Beta=1.5707963267948966
    Estimated cost: -1.2681320000000003

## 4. Outer loop optimization
There are lots of different techniques to choose parameters for `qaoa_circuit`. When there are only two parameters you can keep things simple and sweep over incremental pairings using `np.linspace` and track the minimum value you've found along the way.

```
grid_size = 5
exp_values = np.empty((grid_size, grid_size))
par_values = np.empty((grid_size, grid_size, 2))
for i, alpha_value in enumerate(np.linspace(0, 2*np.pi, grid_size)):
  for j, beta_value in enumerate(np.linspace(0, 2*np.pi, grid_size)):
    samples = sim.sample(
        qaoa_circuit,
        params={alpha: alpha_value, beta: beta_value},
        repetitions=20000)
    exp_values[i][j] = estimate_cost(working_graph, samples)
    par_values[i][j] = alpha_value, beta_value
```

You can also examine the cost function values w.r.t $\alpha$ and $\beta$:

```
plt.title('Heamap of QAOA Cost Function Value')
plt.xlabel(r'$\alpha$')
plt.ylabel(r'$\beta$')
plt.imshow(exp_values)
```

## 5. Compare cuts
Since you are going to be comparing graph cuts in `working_graph` a helper function to draw them and print their the size would be useful:

```
def output_cut(S_partition):
  """Plot and output the graph cut information."""

  # Generate the colors.
  coloring = []
  for node in working_graph:
    if node in S_partition:
      coloring.append('blue')
    else:
      coloring.append('red')

  # Get the weights
  edges = working_graph.edges(data=True)
  weights = [w['weight'] for (u,v, w) in edges]

  nx.draw_circular(
      working_graph,
      node_color=coloring,
      node_size=1000,
      with_labels=True,
      width=weights)
  plt.show()
  size = nx.cut_size(working_graph, S_partition, weight='weight')
  print(f'Cut size:{size}')

# Test with the empty S and all nodes placed in T.
output_cut([])
```

To get cuts using the QAOA you will first need to extract the best control parameters you found during your sweep:

```
best_exp_index = np.unravel_index(np.argmax(exp_values), exp_values.shape)
best_parameters = par_values[best_exp_index]
print(f'Best control parameters:{best_parameters}')
```

    Best control parameters:[3.14159265 6.28318531]

Each bitstring can be seen as a candidate cut in your graph. The qubits that measured 0 correspond to that qubit being in one cut partition and a qubit that measured to 1 corresponds to that qubit being in the other cut partition. Now that you've found good parameters for your circuit, you can just sample some bistrings, iterate over them and pick the one that gives the best cut:

```
# Number of candidate cuts to sample.
num_cuts = 100
candidate_cuts = sim.sample(
    qaoa_circuit,
    params={alpha: best_parameters[0], beta: best_parameters[1]},
    repetitions=num_cuts)

# Variables to store best cut partitions and cut size.
best_qaoa_S_partition = set()
best_qaoa_T_partition = set()
best_qaoa_cut_size = -9999

# Analyze each candidate cut.
for i in range(num_cuts):
  candidate = candidate_cuts.iloc[i]
  one_qubits = set(candidate[candidate==1].index)
  S_partition = set()
  T_partition = set()
  for node in working_graph:
    if str(node) in one_qubits:
      # If a one was measured add node to S partition.
      S_partition.add(node)
    else:
      # Otherwise a zero was measured so add to T partition.
      T_partition.add(node)

  cut_size = nx.cut_size(
      working_graph, S_partition, T_partition, weight='weight')

  # If you found a better cut update best_qaoa_cut variables.
  if cut_size > best_qaoa_cut_size:
    best_qaoa_cut_size = cut_size
    best_qaoa_S_partition = S_partition
    best_qaoa_T_partition = T_partition
```

The QAOA is known to do just a little better better than random guessing for Max-Cut on 3-regular graphs at `p=1`. You can use very similar logic to the code above, but now instead of relying on the QAOA to decied your `S_partition` and `T_partition` you can just pick then randomly:

```
import random

best_random_S_partition = set()
best_random_T_partition = set()
best_random_cut_size = -9999

# Randomly build candidate sets.
for i in range(num_cuts):
  S_partition = set()
  T_partition = set()
  for node in working_graph:
    if random.random() > 0.5:
      # If we flip heads add to S.
      S_partition.add(node)
    else:
      # Otherwise add to T.
      T_partition.add(node)

  cut_size = nx.cut_size(
      working_graph, S_partition, T_partition, weight='weight')

  # If you found a better cut update best_random_cut variables.
  if cut_size > best_random_cut_size:
    best_random_cut_size = cut_size
    best_random_S_partition = S_partition
    best_random_T_partition = T_partition
```

```
print('-----QAOA-----')
output_cut(best_qaoa_S_partition)

print('-----RANDOM-----')
output_cut(best_random_S_partition)
```

```

```


#  ¿Cómo se mueve un péndulo?

> Se dice que un sistema cualquiera, mecánico, eléctrico, neumático, etc., es un oscilador armónico si, cuando se deja en libertad fuera de su posición de equilibrio, vuelve hacia ella describiendo oscilaciones sinusoidales, o sinusoidales amortiguadas en torno a dicha posición estable.
- https://es.wikipedia.org/wiki/Oscilador_armónico

Referencias:
   - http://matplotlib.org
   - https://seaborn.pydata.org
   - http://www.numpy.org
   - http://ipywidgets.readthedocs.io/en/latest/index.html

**En realidad esto es el estudio de oscilaciones. **
___

```python
from IPython.display import YouTubeVideo
YouTubeVideo('k5yTVHr6V14')
```

Los sistemas mas sencillos a estudiar en oscilaciones son el sistema ` masa-resorte` y el `péndulo simple`.

\begin{align}
\frac{d^2 x}{dt^2} +  \omega_{0}^2 x &= 0, \quad \omega_{0} = \sqrt{\frac{k}{m}}\notag\\
\frac{d^2 \theta}{dt^2} + \omega_{0}^{2}\, \theta &= 0, \quad\mbox{donde}\quad \omega_{0}^2 = \frac{g}{l}
\end{align}

___
## Sistema `masa-resorte`

La solución a este sistema `masa-resorte` se explica en términos de la segunda ley de Newton. Para este caso, si la masa permanece constante y solo consideramos la dirección en $x$. Entonces,
\begin{equation}
F = m \frac{d^2x}{dt^2}.
\end{equation}

¿Cuál es la fuerza? **Ley de Hooke!**
\begin{equation}
F = -k x, \quad k > 0.
\end{equation}

Vemos que la fuerza se opone al desplazamiento y su intensidad es proporcional al mismo.  Y $k$ es la constante elástica o recuperadora del resorte.

Entonces, un modelo del sistema `masa-resorte` está descrito por la siguiente **ecuación diferencial**:

\begin{equation}
\frac{d^2x}{dt^2} + \frac{k}{m}x = 0,
\end{equation}

cuya solución se escribe como
\begin{equation}
x(t) = A \cos(\omega_{o} t) + B \sin(\omega_{o} t)
\end{equation}
Y su primera derivada (velocidad) sería
\begin{equation}
\frac{dx(t)}{dt} = \omega_{0}[- A \sin(\omega_{0} t) + B\cos(\omega_{0}t)]
\end{equation}

 Ver en el tablero que significa solución de la ecuación diferencial.

### **¿Cómo se ven las gráficas de $x$ vs $t$ y $\frac{dx}{dt}$ vs $t$?**

_Esta instrucción es para que las gráficas aparezcan dentro de este entorno._

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Podemos cambiar qué tan grandes son los números de los ejes
import matplotlib as mpl
label_size = 14
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
```

```python
import numpy as np
```

```python
# Definición de funciones a graficar
t = np.linspace(0, 10, 1000)
A, B = 2, 3
w0 = 2
x = A * np.cos(w0 * t) + B * np.sin(w0 * t)
dx = w0 * (-A * np.sin(w0 * t) + B * np.cos(w0 * t))
# Gráfico
plt.figure(figsize=(6, 4))
plt.plot(t, x, label="$x(t)$")
plt.plot(t, dx, label="$x'(t)$")
plt.xlabel("Tiempo")
plt.grid()
plt.legend()
```

Y si consideramos un conjunto de frecuencias de oscilación, entonces

```python
frecuencias = np.array([.1, .2 , .5, .6])   # Vector de diferentes frecuencias
plt.figure(figsize = (7, 4))                # Ventana de gráfica con tamaño

# Graficamos para cada frecuencia
t = np.linspace(0, 5, 100)
for f in frecuencias:
    w0 = 2 * np.pi * f
    x = A * np.cos(w0 * t) + B * np.sin(w0 * t)
    plt.plot(t, x, label=f"f={f}")

plt.xlabel("Tiempo")
plt.grid()
plt.legend()
```

Si queremos manipular un poco mas las cosas, hacemos uso de lo siguiente:

```python
from ipywidgets import *
```

```python
def masa_resorte(t=0):
    A, B, w0 = .5, .1, .5                            # Parámetros
    x = A * np.cos(w0 * t) + B * np.sin(w0 * t)      # Función de posición

    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(x, [0],  'ko', ms = 10)
    ax.set_xlim(xmin = -0.6, xmax = .6)
    ax.axvline(x=0, color = 'r')
    ax.axhline(y=0, color = 'grey', lw = 1)
    fig.canvas.draw()
```

```python
interact(masa_resorte, t=(0, 50,.01));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=50.0, step=0.01), Output()), _dom_classes=('…

La opción de arriba generalmente será lenta, así que lo recomendable es usar `interact_manual`.

```python
interact_manual(masa_resorte, t=(0, 50,.01));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=50.0, step=0.01), Button(description='Run In…

___
## Péndulo simple
Ahora, si fijamos nuestra atención al movimiento de un péndulo simple _(oscilaciones pequeñas)_, la ecuación diferencial a resolver tiene la misma forma:

\begin{equation}
\frac{d^2 \theta}{dt^2} + \omega_{0}^{2}\, \theta = 0, \quad\mbox{donde}\quad \omega_{0}^2 = \frac{g}{l}.
\end{equation}

La diferencia más evidente es como hemos definido a $\omega_{0}$. Esto quiere decir que,

\begin{equation}
\theta(t) = A\cos(\omega_{0} t) + B\sin(\omega_{0}t)
\end{equation}

Si graficamos la ecuación de arriba vamos a encontrar un comportamiento muy similar al ya discutido anteriormente. Es por ello que ahora veremos el movimiento en el plano $xy$. Es decir,

\begin{align}
x &= l \sin(\theta), \quad
y = l \cos(\theta)
\end{align}

```python
# Podemos definir una función que nos entregue theta dados los parámetros y el tiempo
def theta_t(a, b, g, l, t):
    omega_0 = np.sqrt(g / l)
    return a * np.cos(omega_0 * t) + b * np.sin(omega_0 * t)
```

```python
# Hacemos un gráfico interactivo del péndulo
def pendulo_simple(t=0):
    a, b = .5, .1                            # Parámetros
    g, l = 9.8, 0.5
    theta = theta_t(a, b, g, l, t)      # Función de posición
    x = l * np.sin(theta)
    y = -l * np.cos(theta)

    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.plot([0], [0],  'dr', ms=10)
    ax.plot(x, y, 'ok', ms=20)
    ax.plot([0, x], [0, y], 'k')
    ax.axvline(x=0, color='grey')
    ax.set_xlim(xmin=-0.5, xmax=0.5)
    ax.set_ylim(ymin=-0.5, ymax=0.05)
    fig.canvas.draw()
```

```python
interact_manual(pendulo_simple, t=(0, 10,.01));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=10.0, step=0.01), Button(description='Run In…

### Condiciones iniciales

Realmente lo que se tiene que resolver es,

\begin{equation}
\theta(t) = \theta(0) \cos(\omega_{0} t) + \frac{\dot{\theta}(0)}{\omega_{0}} \sin(\omega_{0} t)
\end{equation}

> **Actividad.** Modificar el programa anterior para incorporar las condiciones iniciales.

```python
# Solución:
def theta_t(theta0, dtheta0, g, l, t):
    omega_0 = np.sqrt(g / l)
    return theta0 * np.cos(omega_0 * t) + dtheta0 / w0 * np.sin(omega_0 * t)
```

```python
def pendulo_simple(t=0):
    fig = plt.figure(figsize = (5,5))
    ax = fig.add_subplot(1, 1, 1)
    l = 2
    g = 9.8
    theta0 = np.pi / 4
    dtheta0 = 0
    x = l * np.sin(theta_t(np.pi / 4, 0, 9.8, 2, t))
    y =  - l * np.cos(theta_t(np.pi / 4, 0, 9.8, 2, t))
    ax.plot(x, y, 'ko', ms = 10)
    ax.plot([0], [0], 'rD')
    ax.plot([0, x ], [0, y], 'k-', lw = 1)
    ax.set_xlim(xmin = -2.2, xmax = 2.2)
    ax.set_ylim(ymin = -2.2, ymax = .2)
    fig.canvas.draw()

interact_manual(pendulo_simple, t=(0, 10,.01));
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=10.0, step=0.01), Button(description='Run In…

### Plano fase  $(x, \frac{dx}{dt})$

La posición y velocidad para el sistema `masa-resorte` se escriben como:

\begin{align}
x(t) &= x(0) \cos(\omega_{o} t) + \frac{\dot{x}(0)}{\omega_{0}} \sin(\omega_{o} t)\\
\dot{x}(t) &= -\omega_{0}x(0) \sin(\omega_{0} t) + \dot{x}(0)\cos(\omega_{0}t)]
\end{align}

```python
# Parámetros
k = 3  #constante elástica [N]/[m]
m = 1 # [kg]
omega_0 = np.sqrt(k / m)
x_0 = .5
dx_0 = .1
```

```python
# Vector de tiempo
t = np.linspace(0, 100, 1000)
```

```python
# Posición y velocidad
x = x_0 * np.cos(omega_0 * t) + dx_0 / omega_0 * np.sin(omega_0 * t)
dx = -omega_0 * x_0 * np.sin(omega_0 * t) + dx_0 * np.cos(omega_0 * t)
```

```python
# Gráficas en el tiempo
plt.figure(figsize=(6, 4))
plt.plot(t, x, label="$x(t)$")
plt.plot(t, dx, label="$x'(t)$")
plt.xlabel("Tiempo")
plt.grid()
plt.legend()
```

```python
# posición vs. velocidad
# Gráfico
plt.figure(figsize=(5, 8))
plt.plot(x, dx, label="$x(t)$")
plt.xlabel("Posición")
plt.ylabel("Velocidad")
plt.grid()
plt.legend()
```

#### Multiples condiciones iniciales

```python
k = 3  #constante elástica [N]/[m]
m = 1 # [kg]
omega_0 = np.sqrt(k/m)
```

```python
# Condiciones iniciales
x_0s = np.array([.7, .5, .25, .1])
dx_0s = np.array([.2, .1, .05, .01])
```

```python
# Vector de tiempo
t = np.linspace(0, 100, 1000)
```

```python
# Gráficas
plt.figure(figsize=(5, 8))
for i, x_0 in enumerate(x_0s):
    x = x_0 * np.cos(omega_0 * t) + dx_0s[i] / omega_0 * np.sin(omega_0 * t)
    dx = -omega_0 * x_0 * np.sin(omega_0 * t) + dx_0s[i] * np.cos(omega_0 * t)
    plt.plot(x, dx / omega_0)

plt.xlabel("Posición")
plt.ylabel("Velocidad")
plt.grid()
```

Trayectorias del oscilador armónico simple en el espacio fase $(x,\, \dot{x}\,/\omega_0)$ para diferentes valores de la energía.

# Anuncios parroquiales

## 1. Quiz la próxima clase

Created with Jupyter by Lázaro Alonso. Modified by Esteban Jiménez Rodríguez