# Multiple layer models

In this notebook we will explore network models with multiple layers of transformations. This will build upon the single-layer affine model we looked at in the previous notebook and use material covered in the [second](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp02-sln.pdf) and [third](http://www.inf.ed.ac.uk/teaching/courses/mlp/2016/mlp03-mlp.pdf) lectures.

You will need to use these models for the experiments you will be running in the first coursework so part of the aim of this lab will be to get you familiar with how to construct multiple layer models in our framework and how to train them.

## What is a layer?

Often when discussing (neural) network models, a network layer is taken to mean an input to output transformation of the form

\begin{equation}
    \boldsymbol{y} = \boldsymbol{f}(\mathbf{W} \boldsymbol{x} + \boldsymbol{b})
    \qquad
    \Leftrightarrow
    \qquad
    y_k = f\left(\sum_{d=1}^D \left( W_{kd} x_d \right) + b_k \right)
\end{equation}

where $\mathbf{W}$ and $\boldsymbol{b}$ parameterise an affine transformation as discussed in the previous notebook, and $f$ is a function applied elementwise to the result of the affine transformation. For example a common choice for $f$ is the logistic sigmoid function
\begin{equation}
  f(u) = \frac{1}{1 + \exp(-u)}.
\end{equation}

In the second lecture slides you were shown how to train a model consisting of an affine transformation followed by the elementwise logistic sigmoid using gradient descent. This was referred to as a 'sigmoid single-layer network'.

In the previous notebook we also referred to single-layer models, where in that case the layer was an affine transformation, with you implementing the various necessary methods for the `AffineLayer` class before using an instance of that class within a `SingleLayerModel` on a regression problem. We could in that case consider the function $f$ to simply be the identity function $f(u) = u$. In the code for the labs we will however use a slightly different convention. Here we will consider the affine transformations and subsequent elementwise function $f$ to each be a separate transformation layer.

This will mean we can combine our already implemented `AffineLayer` class with any non-linear function applied to the outputs just by implementing a layer object for the relevant non-linearity and then stacking the two layers together. An alternative would be to have our new layer objects inherit from `AffineLayer` and then call the relevant parent class methods in the child class however this would mean we need to include a lot of the same boilerplate code in every new class.

To give a concrete example, in the `mlp.layers` module there is a definition for a `SigmoidLayer` equivalent to the following (documentation strings have been removed here for brevity)

```python
class SigmoidLayer(Layer):

    def fprop(self, inputs):
        return 1. / (1. + np.exp(-inputs))

    def bprop(self, inputs, outputs, grads_wrt_outputs):
        return grads_wrt_outputs * outputs * (1. - outputs)
```

As you can see this `SigmoidLayer` class has a very lightweight definition, defining just two key methods:

  * `fprop` which takes a batch of activations at the input to the layer and forward propagates them to produce activates at the outputs (directly equivalently to the `fprop` method you implemented for then `AffineLayer` in the previous notebook),
  * `brop` which takes a batch of gradients with respect to the outputs of the layer and backward propagates them to calculate gradients with respect to the inputs of the layer (explained in more detail below).

This `SigmoidLayer` class only implements the logistic sigmoid non-linearity transformation and so does not have any parameters. Therefore unlike `AffineLayer` it is derived directly from the base `Layer` class rather than `LayerWithParameters` and does not need to implement `grads_wrt_params` or `params` methods.

To create a model consisting of an affine transformation followed by applying an elementwise logistic sigmoid transformation we first create a list of the two layer objects (in the order they are applied from inputs to outputs) and then use this to instantiate a new `MultipleLayerModel` object:

```python
from mlp.layers import AffineLayer, SigmoidLayer
from mlp.models import MultipleLayerModel

layers = [AffineLayer(input_dim, output_dim), SigmoidLayer()]
model = MultipleLayerModel(layers)
```

Because of the modular way in which the layers are defined we can also stack an arbitrarily long sequence of layers together to produce deeper models. For instance the following would define a model consisting of three pairs of affine and logistic sigmoid transformations.

```python
model = MultipleLayerModel([
    AffineLayer(input_dim, hidden_dim), SigmoidLayer(),
    AffineLayer(hidden_dim, hidden_dim), SigmoidLayer(),
    AffineLayer(hidden_dim, output_dim), SigmoidLayer(),
])
```

## Back-propagation of gradients

To allow training models consisting of a stack of multiple layers, all layers need to implement a `bprop` method in addition to the `fprop` we encountered in the previous week.

The `bprop` method takes gradients of an error function with respect to the *outputs* of a layer and uses these gradients to calculate gradients of the error function with respect to the *inputs* of a layer. As the inputs to a non-input layer in a multiple-layer model consist of the outputs of the previous layer, this means we can calculate the gradients of the error function with respect to the outputs of every layer in the model by iteratively propagating the gradients backwards through the layers of the model (i.e. from the last to first layer), hence the term 'back-propagation' or 'bprop' for short. A block diagram illustrating this is shown for a three layer model below.

For a layer with parameters, the gradients with respect to the layer outputs are required to calculate gradients with respect to the layer parameters. Therefore by combining backward propagation of gradients through the model with computing the gradients with respect to parameters in the relevant layers we can calculate gradients of the error function with respect to all of the parameters of a multiple-layer model in a very efficient manner (in fact the computational cost of computing gradients with respect to all of the parameters of the model using this method will only be a constant factor times the cost of calculating the model outputs in the forwards pass).

We so far have abstractly talked about calculating gradients with respect to the inputs of a layer using gradients with respect to the layer outputs. More concretely we will be using the chain rule for derivatives to do this, similarly to how we used the chain rule in exercise 4 of the previous notebook to calculate gradients with respect to the parameters of an affine layer given gradients with respect to the outputs of the layer.

In particular if our layer has a batch of $B$ vector inputs each of dimension $D$, $\left\lbrace \boldsymbol{x}^{(b)} \right\rbrace_{b=1}^B$, and produces a batch of $B$ vector outputs each of dimension $K$, $\left\lbrace \boldsymbol{y}^{(b)}\right\rbrace_{b=1}^B$,  then we can calculate the gradient with respect to the $d^\textrm{th}$ dimension of the $b^{\textrm{th}}$ input given the gradients with respect to the $b^{\textrm{th}}$ output using

\begin{equation}
    \frac{\partial \bar{E}}{\partial x^{(b)}_d} =
    \sum_{k=1}^K \left(
        \frac{\partial \bar{E}}{\partial y^{(b)}_k} \frac{\partial y^{(b)}_k}{\partial x^{(b)}_d}
    \right).
\end{equation}

Mathematically therefore the `bprop` method takes an array of gradients with respect to the outputs $\frac{\partial y^{(b)}_k}{\partial x^{(b)}_d}$ and applies a sum-product operation with the partial derivatives of each output with respect to each input $\frac{\partial y^{(b)}_k}{\partial x^{(b)}_d}$ to produce gradients with respect to the inputs of the layer $\frac{\partial \bar{E}}{\partial x^{(b)}_d}$.

For the affine transformation used in the `AffineLayer` implemented last week, i.e a forwards propagation corresponding to

\begin{equation}
    y^{(b)}_k = \sum_{d=1}^D \left( W_{kd} x^{(b)}_d \right) + b_k
\end{equation}

then the corresponding partial derivatives of layer outputs with respect to inputs are

\begin{equation}
    \frac{\partial y^{(b)}_k}{\partial x^{(b)}_d} = W_{kd}
\end{equation}

and so the backwards-propagation method for the `AffineLayer` takes the following form

\begin{equation}
    \frac{\partial \bar{E}}{\partial x^{(b)}_d} =
    \sum_{k=1}^K \left( \frac{\partial \bar{E}}{\partial y^{(b)}_k} W_{kd} \right).
\end{equation}

This can be efficiently implemented in NumPy using the `dot` function

```python
class AffineLayer(LayerWithParameters):

    # ... [implementation of remaining methods from previous week] ...

    def bprop(self, inputs, outputs, grads_wrt_outputs):
        return grads_wrt_outputs.dot(self.weights)
```

An important special case applies when the outputs of a layer are an elementwise function of the inputs such that $y^{(b)}_k$ only depends on $x^{(b)}_d$ when $d = k$. In this case the partial derivatives $\frac{\partial y^{(b)}_k}{\partial x^{(b)}_d}$ will be zero for $k \neq d$ and so the above summation collapses to a single term, giving

\begin{equation}
    \frac{\partial \bar{E}}{\partial x^{(b)}_d} =
    \frac{\partial \bar{E}}{\partial y^{(b)}_d} \frac{\partial y^{(b)}_d}{\partial x^{(b)}_d}
\end{equation}

i.e. to calculate the gradient with respect to the $b^{\textrm{th}}$ input vector we just perform an elementwise multiplication of the gradient with respect to the $b^{\textrm{th}}$ output vector with the vector of derivatives of the outputs with respect to the inputs. This case applies to the `SigmoidLayer` and to all other layers applying an elementwise function to their inputs.

For the logistic sigmoid layer we have that

\begin{equation}
    y^{(b)}_d = \frac{1}{1 + \exp(-x^{(b)}_d)}
    \qquad
    \Rightarrow
    \qquad
    \frac{\partial y^{(b)}_d}{\partial x^{(b)}_d} =
    \frac{\exp(-x^{(b)}_d)}{\left[ 1 + \exp(-x^{(b)}_d) \right]^2} =
     y^{(b)}_d \left[ 1 -  y^{(b)}_d  \right]
\end{equation}

which you should now be able relate to the implementation of `SigmoidLayer.bprop` given earlier:

```python
class SigmoidLayer(Layer):

    def fprop(self, inputs):
        return 1. / (1. + np.exp(-inputs))

    def bprop(self, inputs, outputs, grads_wrt_outputs):
        return grads_wrt_outputs * outputs * (1. - outputs)
```

## Exercise 1: training a softmax model on MNIST

For this first exercise we will train a model consisting of an affine transformation plus softmax on a multiclass classification task: classifying the digit labels for handwritten digit images from the MNIST data set introduced in the first notebook.

First run the cell below to import the necessary modules and classes and to load the MNIST data provider objects. As it takes a little while to load the MNIST data from disk into memory it is worth loading the data providers just once in a separate cell like this rather than recreating the objects for every training run.

We are loading two data provider objects here - one corresponding to the training data set and a second to use as a *validation* data set. This is data we do not train the model on but measure the performance of the trained model on to assess its ability to *generalise* to unseen data.

If you are in the Monday or Tuesday lab sessions you will not yet have had the lecture introducing the concepts of generalisation and validation data sets (though those doing MLPR alongside this course should already be familiar with these ideas). As you will need to report both training and validation set performances in your experiments for the first coursework assignment we are providing code here to give an example of how to do this.

```python
import sys
mlpdir = '/home/student/Dropbox/msc_Artificial_Intelligence/mlp_Machine_Learning_Practical/mlpractical'
sys.path.append(mlpdir)
```

```python
import numpy as np
import matplotlib.pyplot as plt
import logging
from mlp.layers import AffineLayer, SoftmaxLayer, SigmoidLayer
from mlp.errors import CrossEntropyError, CrossEntropySoftmaxError
from mlp.models import SingleLayerModel, MultipleLayerModel
from mlp.initialisers import UniformInit
from mlp.learning_rules import GradientDescentLearningRule
from mlp.data_providers import MNISTDataProvider
from mlp.optimisers import Optimiser
%matplotlib inline
plt.style.use('ggplot')

# Seed a random number generator
seed = 6102016
rng = np.random.RandomState(seed)

# Set up a logger object to print info about the training run to stdout
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.handlers = [logging.StreamHandler()]

# Create data provider objects for the MNIST data set
train_data = MNISTDataProvider('train', rng=rng)
valid_data = MNISTDataProvider('valid', rng=rng)
input_dim, output_dim = 784, 10
```

To minimise replication of code and allow you to run experiments more quickly a helper function is provided below which trains a model and plots the evolution of the error and classification accuracy of the model (on both training and validation sets) over training.

```python
def train_model_and_plot_stats(
        model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval):

    # As well as monitoring the error over training also monitor classification
    # accuracy i.e. proportion of most-probable predicted classes being equal to targets
    data_monitors={'acc': lambda y, t: (y.argmax(-1) == t.argmax(-1)).mean()}

    # Use the created objects to initialise a new Optimiser instance.
    optimiser = Optimiser(
        model, error, learning_rule, train_data, valid_data, data_monitors)

    # Run the optimiser for 5 epochs (full passes through the training set)
    # printing statistics every epoch.
    stats, keys = optimiser.train(num_epochs=num_epochs, stats_interval=stats_interval)

    # Plot the change in the validation and training set error over training.
    fig_1 = plt.figure(figsize=(8, 4))
    ax_1 = fig_1.add_subplot(111)
    for k in ['error(train)', 'error(valid)']:
        ax_1.plot(np.arange(1, stats.shape[0]) * stats_interval,
                  stats[1:, keys[k]], label=k)
    ax_1.legend(loc=0)
    ax_1.set_xlabel('Epoch number')

    # Plot the change in the validation and training set accuracy over training.
    fig_2 = plt.figure(figsize=(8, 4))
    ax_2 = fig_2.add_subplot(111)
    for k in ['acc(train)', 'acc(valid)']:
        ax_2.plot(np.arange(1, stats.shape[0]) * stats_interval,
                  stats[1:, keys[k]], label=k)
    ax_2.legend(loc=0)
    ax_2.set_xlabel('Epoch number')

    return stats, keys, fig_1, ax_1, fig_2, ax_2
```

Running the cell below will create a model consisting of an affine layer follower by a softmax transformation and train it on the MNIST data set by minimising the multi-class cross entropy error function using a basic gradient descent learning rule. By using the helper function defined above, at the end of training curves of the evolution of the error function and also classification accuracy of the model over the training epochs will be plotted.

You should try running the code for various settings of the training hyperparameters defined at the beginning of the cell to get a feel for how these affect how training proceeds. You may wish to create multiple copies of the cell below to allow you to keep track of and compare the results across different hyperparameter settings.

```python
# Set training run hyperparameters
batch_size = 100  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 0.1  # learning rate for gradient descent
num_epochs = 100  # number of training epochs to perform
stats_interval = 5  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)

# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

#### Less epochs than before

```python
# Set training run hyperparameters
batch_size = 100  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 0.1  # learning rate for gradient descent
num_epochs = 20  # number of training epochs to perform
stats_interval = 5  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)

# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

Learning Rate from 0.1 now is much bigger at value 2

```python
# Set training run hyperparameters
batch_size = 100  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 2  # learning rate for gradient descent
num_epochs = 20  # number of training epochs to perform
stats_interval = 5  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)

# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

### Optional extra: more efficient softmax gradient evaluation

In the lectures you were shown that for certain combinations of error function and final output layers, that the expressions for the gradients take particularly simple forms.

In particular it can be shown that the combinations of

  * logistic sigmoid output layer and binary cross entropy error function
  * softmax output layer and cross entropy error function

lead to particularly simple forms for the gradients of the error function with respect to the inputs to the final layer. In particular for the latter softmax and cross entropy error function case we have that

\begin{equation}
    y^{(b)}_k = \textrm{Softmax}_k\left(\boldsymbol{x}^{(b)}\right) = \frac{\exp(x^{(b)}_k)}{\sum_{d=1}^D \left\lbrace \exp(x^{(b)}_d) \right\rbrace}
    \qquad
    E^{(b)} = \textrm{CrossEntropy}\left(\boldsymbol{y}^{(b)},\,\boldsymbol{t}^{(b)}\right) = -\sum_{d=1}^D \left\lbrace t^{(b)}_k \log(y^{(b)}_k) \right\rbrace
\end{equation}

and it can be shown (this is an instructive mathematical exercise if you want a challenge!) that

\begin{equation}
    \frac{\partial E^{(b)}}{\partial x^{(b)}_d} = y^{(b)}_d - t^{(b)}_d.
\end{equation}

The combination of `CrossEntropyError` and `SoftmaxLayer` used to train the model above calculate this gradient less directly by first calculating the gradient of the error with respect to the model outputs in `CrossEntropyError.grad` and then back-propagating this gradient to the inputs of the softmax layer using `SoftmaxLayer.bprop`.

Rather than computing the gradient in two steps like this we can instead wrap the softmax transformation in to the definition of the error function and make use of the simpler gradient expression above. More explicitly we define an error function as follows

\begin{equation}
    E^{(b)} = \textrm{CrossEntropySoftmax}\left(\boldsymbol{y}^{(b)},\,\boldsymbol{t}^{(b)}\right) = -\sum_{d=1}^D \left\lbrace t^{(b)}_k \log\left[\textrm{Softmax}_k\left( \boldsymbol{y}^{(b)}\right)\right]\right\rbrace
\end{equation}

with corresponding gradient

\begin{equation}
    \frac{\partial E^{(b)}}{\partial y^{(b)}_d} = \textrm{Softmax}_d\left( \boldsymbol{y}^{(b)}\right) - t^{(b)}_d.
\end{equation}

The final layer of the model will then be an affine transformation which produces unbounded output values corresponding to the logarithms of the unnormalised predicted class probabilities. An implementation of this error function is provided in `CrossEntropySoftmaxError`. The cell below sets up a model with a single affine transformation layer and trains it on MNIST using this new cost. If you run it with equivalent hyperparameters to one of your runs with the alternative formulation above you should get identical error and classification curves (other than floating point error) but with a minor improvement in training speed.

```python
# Set training run hyperparameters
batch_size = 100  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 0.1  # learning rate for gradient descent
num_epochs = 100  # number of training epochs to perform
stats_interval = 5  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)

# Create affine model (outputs are logs of unnormalised class probabilities)
model = SingleLayerModel(
    AffineLayer(input_dim, output_dim, param_init, param_init)
)

# Initialise the error object
error = CrossEntropySoftmaxError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

```python
# Set training run hyperparameters
batch_size = 10000  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 0.1  # learning rate for gradient descent
num_epochs = 100  # number of training epochs to perform
stats_interval = 10  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)

# Create affine model (outputs are logs of unnormalised class probabilities)
model = SingleLayerModel(
    AffineLayer(input_dim, output_dim, param_init, param_init)
)

# Initialise the error object
error = CrossEntropySoftmaxError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

## Exercise 2: training deeper models on MNIST

We are now going to investigate using deeper multiple-layer model archictures for the MNIST classification task. You should experiment with training models with two to five `AffineLayer` transformations interleaved with `SigmoidLayer` nonlinear transformations. Intermediate hidden layers between the input and output should have a dimension of 100. For example the `layers` definition of a model with two `AffineLayer` transformations would be

```python
layers = [
    AffineLayer(input_dim, 100),
    SigmoidLayer(),
    AffineLayer(100, output_dim),
    SoftmaxLayer()
]
```

If you read through the extension to the first exercise you may wish to use the `CrossEntropySoftmaxError` without the final `SoftmaxLayer`.

Use the code from the first exercise as a starting point and start with training hyperparameters which gave reasonable performance for the shallow architecture trained previously.

Some questions to investigate:

  * How does increasing the number of layers affect the model's performance on the training data set? And on the validation data set?
  * Do deeper models seem to be harder or easier to train (e.g. in terms of ease of choosing training hyperparameters to give good final performance and/or quick convergence)?
  * Do the models seem to be sensitive to the choice of the parameter initialisation range? Can you think of any reasons for why setting individual parameter initialisation scales for each `AffineLayer` in a model might be useful? Can you come up with (or find) any heuristics for setting the parameter initialisation scales?

You do not need to come up with explanations for all of these (though if you can that's great!), they are meant as prompts to get you thinking about the various issues involved in training multiple-layer models.

You may wish to start with shorter pilot training runs (by decreasing the number of training epochs) for each of the model architectures to get an initial idea of appropriate hyperparameter settings before doing one or two longer training runs to assess the final performance of the architectures.

### Models with two affine layers

```python
# Set training run hyperparameters
batch_size = 100  # number of data points in a batch
init_scale = 0.1  # scale for random parameter initialisation
learning_rate = 0.1  # learning rate for gradient descent
num_epochs = 100  # number of training epochs to perform
stats_interval = 5  # epoch interval between recording and printing stats

# Reset random number generator and data provider states on each run
# to ensure reproducibility of results
rng.seed(seed)
train_data.reset()
valid_data.reset()

# Alter data-provider batch size
train_data.batch_size = batch_size
valid_data.batch_size = batch_size

# Create a parameter initialiser which will sample random uniform values
# from [-init_scale, init_scale]
param_init = UniformInit(-init_scale, init_scale, rng=rng)
```

```python
# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

### Models with three affine layers

```python
# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

### Models with four affine layers

```python
# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

### Models with five affine layers

```python
# Create affine + softmax model
model = MultipleLayerModel([
    AffineLayer(input_dim, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, 100, param_init, param_init),
    SigmoidLayer(),
    AffineLayer(100, output_dim, param_init, param_init),
    SoftmaxLayer()
])

# Initialise a cross entropy error object
error = CrossEntropyError()

# Use a basic gradient descent learning rule
learning_rule = GradientDescentLearningRule(learning_rate=learning_rate)

_ = train_model_and_plot_stats(
    model, error, learning_rule, train_data, valid_data, num_epochs, stats_interval)
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import math
from numpy.linalg import inv
from numpy import matmul as mm
import scipy
from scipy.linalg import lu_factor,lu_solve
from scipy.linalg import cho_factor,cho_solve
from scipy.stats import multivariate_normal
```

```python
x = np.linspace(-4,2,20)
y = (x+1)**3+10*np.random.normal(0,1,20)
```

### Posterior distribution

$$p(\mathbf{w|x,y},\mathcal{M})=\frac{p(\mathbf{w}|\mathcal{M})p(\mathbf{y|x,w},\mathcal{M})}{p(\mathbf{y|x},\mathcal{M})}=\mathcal{N}(\mathbf{w|\mu,\Sigma})$$
where

\begin{align}
\mathbf{\Sigma} &= (\sigma^{-2}_{noise}\mathbf{\Phi^\top\Phi}+\sigma^{-2}_{\mathbf{w}}\mathbf{I})^{-1}\\
\mathbf{\mu} &= (\mathbf{\Phi^\top\Phi}+\frac{\sigma^2_{noise}}{\sigma^2_\mathbf{w}}\mathbf{I})^{-1}\mathbf{\Phi^\top y}
\end{align}

### Predictive distribution

\begin{align}
p(y_*|x_*,\mathbf{x},\mathbf{y},\mathcal{M}) &= \int p(y_*|x_*,\mathbf{w},\mathcal{M})p(\mathbf{y}|\mathbf{x},\mathbf{w},\mathcal{M})d\mathbf{w}\\
&= \mathcal{N}(y_*|\mathbf{\phi}(x_*)^\top\mathbf{\mu},\mathbf{\phi}(x_*)^\top\mathbf{\Sigma}\mathbf{\phi}(x_*)+\sigma^2_{noise}\mathbf{I})
\end{align}

### Marginal likelihood (Evidence)

\begin{align}
p(\mathbf{y}|\mathbf{x},\mathcal{M})&= \int p(\mathbf{w}|\mathbf{x},\mathcal{M})p(\mathbf{y}|\mathbf{x},\mathbf{w},\mathcal{M})d\mathbf{w}\\
&= \mathcal{N}(\mathbf{y};\mathbf{0},\sigma^2_\mathbf{w}\mathbf{\Phi\Phi^\top}+\sigma^2_{noise}\mathbf{I})
\end{align}

```python
def BayesFitLU(x,y,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    margcov = mm(Phi,Phi.T)+100*np.eye(Phi.shape[0])
    lu,piv = lu_factor(mm(Phi.T,Phi)+100*np.eye(Phi.shape[1]))
    tmp = lu_solve((lu,piv),Phi.T)
    w = mm(tmp,y)
    f = mm(Phi,w)
    logmarglik = multivariate_normal.logpdf(y,None,margcov)

    covlu,covpiv = lu_factor(mm(Phi.T,Phi)/100+np.eye(Phi.shape[1]))
    return f,covlu,covpiv,w,logmarglik
```

```python
mllistLU = []
plt.figure(figsize=(20,20))
for i in range(10):
    plt.subplot(5,2,i+1)
    f,lu,piv,w,lml = BayesFitLU(x,y,i)
    if i == 3:
        print(w)
    mllistLU.append(lml)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')
    plt.title('Order '+str(i)+': '+str(lml))
```

```python
plt.plot(np.arange(10),mllistLU)
plt.title('Log-Marginal Evidence')
```

```python
def BayesPredictLU(x,mu,lu,piv,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    tmu = mm(Phi,mu)
    tcov = mm(Phi,lu_solve((lu,piv),Phi.T))+100*np.eye(Phi.shape[0])
    return tmu,tcov
```

```python
mu,lu,piv,w,lml = BayesFitLU(x,y,3)
xt = np.linspace(-6,4,100)
ytmu,ytcov = BayesPredictLU(xt,w,lu,piv,3)
```

```python
unc = np.sqrt(np.diag(ytcov))
plt.fill_between(xt,ytmu-unc,ytmu+unc,color='lime')
plt.plot(xt,ytmu,'r-')
plt.plot(x,y,'r.')
```

### Summary

```python
mllistLU = []
plt.figure(figsize=(10,15))
plt.subplot(3,1,1)
for i in range(10):
    f,lu,piv,w,lml = BayesFitLU(x,y,i)
    mllistLU.append(lml)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')

plt.subplot(3,1,2)
plt.plot(np.arange(10),mllistLU)
plt.title('Log-Marginal Evidence')

mu,lu,piv,w,lml = BayesFitLU(x,y,3)
xt = np.linspace(-6,4,100)
ytmu,ytcov = BayesPredictLU(xt,w,lu,piv,3)
plt.subplot(3,1,3)
unc = np.sqrt(np.diag(ytcov))
plt.fill_between(xt,ytmu-unc,ytmu+unc,color='lime')
plt.plot(xt,ytmu,'r-')
plt.plot(x,y,'r.')
plt.title('Order 3 log-evidence: '+str(lml))
```

```python

```

Probabilistic Machine Learning/Introduction to Probabilistic Machine Learning/Bayesian inference and prediction with finite regression models.ipynb

Probabilistic Machine Learning/Introduction to Probabilistic Machine Learning/.ipynb_checkpoints/Bayesian inference and prediction with finite regression models-checkpoint.ipynb


# Espacios de fases de la ecuación con difusión

```python
from sympy.solvers import solve
from sympy import Symbol
import matplotlib.pyplot as plt
import numpy as np
```

```python
u1 = Symbol('u1')
u2 = Symbol('u2')
```

```python
# TABLE II paper diffusion

r1 = 0.0001 # en este caso, ambas poblaciones tienen una interacción
r2 = 0.6 # positiva con el entorno, aunque los depredadores son casi autónomos
b11 = 0.0019 # las presas son cooperativas entre sí
b12 = -0.00075 # la población 1 corresponde a las presas
b21 = 0.00091 # la población 2 a los depredadores
b22 = -0.0019 # los depredadores son competitivos entre sí
a1 = 0.0005
a2 = 0.000625
c1 = 0.001251
c2 = 0.001
s = r2/r1 # estos son los parámetros del sistema adimensionalizado, s es la proporción de rs
q1 = a1/(c1*r1) # los qs son las proporciones entre as y cs
q2 = a2/(c2*r1) # ambos tienen a r1 de denominador, por la factorización que se hizo para s
p11 = b11/(c1*r1) # los ps equivalen a los bs
p12 = b12/(c2*r1) # de modo que incorporan tanto la interacción
p21 = b21/(c1*r1) # como el límite de las cs
p22 = b22/(c2*r1) # y tienen el factor de r1 en el denominador, por la factorización de s
g = 1 # es el gamma que propone Murray
d = 19 # estoy escogiendo un valor de difusión mayor a la d crítica, que es 18.algo
```

```python
# Functionals. In the first step, we define the functionals.

f1 = g*u1*(1-q1*u1+(p11*u1+p12*u2)*(1-u1))
f2 = g*u2*(s-q2*u2+(p21*u1+p22*u2)*(1-u2))
```

```python
# Calculate the solutions and print them
solucion = solve([f1,f2],[u1,u2])
print(solucion)
```

    [(-8.93463091074237e-5, 0.0), (0.0, 0.0), (0.0, 0.309881237415477), (0.0, 1.01906613100558), (0.424693552787801, 0.466861893539141), (0.441900679728548, 0.473154874821710), (0.736931451572265, 0.0), (0.619932130000084 - 0.348403740313958*I, 1.02658982019912 - 0.00728446411237864*I), (0.619932130000084 + 0.348403740313958*I, 1.02658982019912 + 0.00728446411237864*I)]

```python
xlist = [i[0] for i in solucion if not (np.iscomplex(complex(i[0])))]
ylist = [i[1] for i in solucion if not(np.iscomplex(complex(i[1]))) ]

plt.axvline(0, c='grey')
plt.axhline(0, c='grey')
plt.plot(xlist,ylist,'ko')

y, x = np.mgrid[min(ylist)-1:max(ylist)+1:150j, min(xlist)-1:max(xlist)+1:150j]
eq1 = g*x*(1-q1*x+(p11*x+p12*y)*(1-x))
eq2 = g*y*(s-q2*y+(p21*x+p22*y)*(1-y))
plt.streamplot(x, y, eq1, eq2, color=eq1, cmap='Greys',linewidth=1.5,density = 4.0)
plt.xlim(-0.1, 1.0)
plt.ylim(-0.1, 1.0)
plt.xlabel(r'$U_{1}$',size = 14)
plt.ylabel(r'$U_{2}$',size= 14)

#plt.tight_layout()
#plt.savefig('figure5_1.eps',dpi= 1500)
#plt.savefig('figure5_1.pdf',dpi= 1500)
plt.savefig('figure5_1.jpg',dpi= 1500)
plt.show()
```

```python

```


### Programación lineal

>La programación lineal es el campo de la optimización matemática dedicado a maximizar o minimizar (optimizar) funciones lineales, denominada función objetivo, de tal forma que las variables de dicha función estén sujetas a una serie de restricciones expresadas mediante un sistema de ecuaciones o inecuaciones también lineales.

## 4. Ejemplo básico
Una compañía produce dos productos ($X_1$ y $X_2$) usando dos máquinas ($A$ y $B$). Cada unidad de **$X_1$ que se produce requiere 50 minutos en la máquina $A$** y 30 minutos en la máquina $B$. Cada unidad de **$X_2$ que se produce requiere 24 minutos en la máquina $A$** y 33 minutos en la máquina $B$.

Al comienzo de la semana hay 30 unidades de $X_1$ y 90 unidades de $X_2$ en inventario. El tiempo de uso disponible de la **máquina $A$ es de 40 horas** y el de la máquina $B$ es de 35 horas.

La demanda para $X_1$ en la semana actual es de 75 unidades y de $X_2$ es de 95 unidades. La política de la compañía es maximizar la suma combinada de unidades de $X_1$ e $X_2$ en inventario al finalizar la semana.

Formular el problema de decidir cuánto hacer de cada producto en la semana como un problema de programación lineal.

### Solución

Sean:
- $x_1$ la cantidad de unidades de $X_1$ a ser producidas en la semana, y
- $x_2$ la cantidad de unidades de $X_2$ a ser producidas en la semana.

Notar que lo que se quiere es maximizar $x_1+x_2$.

Restricciones:
1. El tiempo de uso disponible de la máquina $A$ es de 40 horas: $50x_1+24x_2\leq 40(60)\Rightarrow 50x_1+24x_2\leq 2400$.
2. El tiempo de uso disponible de la máquina $B$ es de 35 horas: $30x_1+33x_2\leq 35(60)\Rightarrow 30x_1+33x_2\leq 2100$.
3. La demanda para $X_1$ en la semana actual es de 75 unidades: $x_1+30\geq 75\Rightarrow x_1\geq 45\Rightarrow -x_1\leq -45$.
4. La demanda para $X_2$ en la semana actual es de 95 unidades: $x_2+90\geq 95\Rightarrow x_2\geq 5\Rightarrow -x_2\leq -5$.

$ 50x_1+24x_2\leq 2400$

Finalmente, el problema puede ser expresado en la forma explicada como:
\begin{equation}
\begin{array}{ll}
\min_{x_1,x_2} & -x_1-x_2 \\
\text{s. a. }  & 50x_1+24x_2\leq 2400 \\
               & 30x_1+33x_2\leq 2100 \\
               & -x_1\leq -45 \\
               & -x_2\leq -5,
\end{array}
\end{equation}

o, eqivalentemente
\begin{equation}
\begin{array}{ll}
\min_{\boldsymbol{x}} & \boldsymbol{c}^T\boldsymbol{x} \\
\text{s. a. }        & \boldsymbol{A}_{eq}\boldsymbol{x}=\boldsymbol{b}_{eq} \\
                     & \boldsymbol{A}\boldsymbol{x}\leq\boldsymbol{b},
\end{array}
\end{equation}
con
- $\boldsymbol{c}=\left[-1 \quad -1\right]^T$,
- $\boldsymbol{A}=\left[\begin{array}{cc}50 & 24 \\ 30 & 33\\ -1 & 0\\ 0 & -1\end{array}\right]$, y
- $\boldsymbol{b}=\left[2400\quad 2100\quad -45\quad -5\right]^T$.

Preferiremos, en adelante, la notación vectorial/matricial.

### Solución Gráfica:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

Se tienen las desigualdades:
$$ 50x_1 + 24x_2\leq 2400 $$
$$ 30x_1 + 33x_2\leq 2100 $$

Para hacer el plot las pasamos a igualdades y despejamos para una variable:

$$ 50x_1 +24x_2 = 2400 $$
$$ 30x_1 +33x_2 = 2100 $$

El despeje entonces queda:

$$ x_2 = (2400-50x_1)/ 24 $$
$$ x_2 = (2100-30x_1)/ 33 $$

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

```python
def x2_v1(x1):                 #Maquina A
    return (2400 - 50*x1)/24
def x2_v2(x1):                 #Maquina B
    return (2100 - 30*x1)/33
```

```python
x1 = np.linspace(0, 100)
```

```python
plt.plot(x1, x2_v1(x1), 'magenta')         #Desigualdad 1
plt.plot(x1, x2_v2(x1))                    #Desigualdad 2
plt.plot([45, 45], [0, 25],"k")
plt.scatter([45, 45], [0, 25], color = "red", s = 100)
plt.plot([43, 47], [5, 5], 'r')
plt.scatter([43, 47], [5, 5], color = "red", s = 100)
plt.xlim(xmin = 44.6, xmax = 46)           #Zoom en x
plt.ylim(ymin = 4, ymax = 7)               #Zoom en y
plt.scatter([45, 45, 45.6], [5, 6.25, 5], color = 'red')   #Punto en el cruce de las lineas
plt.fill_between(np.array([45, 45.6]), x2_v1(np.array([45, 45.6])),
                5*np.ones(2), alpha = .2, color = 'blue')
plt.xlabel(r"$x_1$", fontsize = 18)
plt.ylabel(r"$x_2$", fontsize = 18)
plt.show()
```

```python
### GRAFICA SIN ZOOM:
plt.plot(x1, x2_v1(x1), 'magenta')        #Desigualdad 1
plt.plot(x1, x2_v2(x1))                   #Desigualdad 2
plt.plot([45, 45], [0, 25], 'k')
plt.plot([43, 47], [5, 5], 'r')
plt.xlim(xmin = 0, xmax = 80)
plt.ylim(ymin = -30, ymax = 80)
#plt.scatter([45], [6.25], color = 'red')
#plt.fill_between(np.array([45, 45.6]), x2_v1(np.array([45, 45.6])),
#                5*np.ones(2), alpha = .2, color = 'orange')
plt.show()
```

YA TENEMOS UN PUNTO POSIBLE PARA LA SOLUCIÓN, $(45, 5)$. LOS OTROS SON $(45, 6.25)$ $(45.6, 5)$

### Solucion con Algoritmos:

```python
import scipy.optimize as opt
```

```python
#res = opt.linprog(c, A_ub=A, b_ub=b, options={'disp': True})
```

```python
c = np.array([-1, -1])               # min(-x1 -x2)
A = np.array([[50, 24], [30, 33]])
b = np.array([2400, 2100])
x1_bound = (45, None)
x2_bound = (5, None)
```

```python
res = opt.linprog(c, A_ub=A, b_ub=b, bounds=(x1_bound, x2_bound), options={'disp': True})
```

    Optimization terminated successfully.
             Current function value: -51.250000
             Iterations: 4

```python
res
```

         fun: -51.25
     message: 'Optimization terminated successfully.'
         nit: 4
       slack: array([  0.  , 543.75,   0.  ,   1.25])
      status: 0
     success: True
           x: array([45.  ,  6.25])

En inventario teníamos $90$ de $x_2$ y $30$ de $x_1$

## Tarea

**Actividad.** Mónica hace aretes y cadenitas de joyería. Es tan buena, que todo lo que hace lo vende.

Le toma 30 minutos hacer un par de aretes y una hora hacer una cadenita, y como Mónica también es estudihambre, solo dispone de 10 horas a la semana para hacer las joyas. Por otra parte, el material que compra solo le alcanza para hacer 15 unidades (el par de aretes cuenta como unidad) de joyas por semana.

La utilidad que le deja la venta de las joyas es \$15 en cada par de aretes y \$20 en cada cadenita.

¿Cuántos pares de aretes y cuántas cadenitas debería hacer Mónica para maximizar su utilidad?

#### Solucion con Algoritmos:

```python
import numpy as np
```

```python
c = np.array([-15, -20])           # min(-x1 -x2)
A = np.array([[0.5, 1], [1, 1]])
b = np.array([10, 15])
x1_cota = (0, None)
x2_cota = (0, None)
```

```python
res_mon = opt.linprog(c=c, A_ub=A, b_ub=b, bounds = (x1_cota,x2_cota))
```

```python
res_mon
```

         fun: -250.0
     message: 'Optimization terminated successfully.'
         nit: 2
       slack: array([0., 0.])
      status: 0
     success: True
           x: array([10.,  5.])

Esto nos dice que Monica deberia hacer 10 pares de aretes y 5 cadenas para lograr maximizar su utiliddad.

#### Solucion Geométrica:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

Par de aretes: $30$ min.

Cadena:        $1$ hora.

$$ 10x_1 + 5x_2 = 250 $$

Donde:

$ x_1 = Par de aretes $

$ x_2 = Cadena $
$$ 10(15) + 5(20) = 250 $$

Despejando los Aretes ($x_1$):

$$ x_1 = (250 - 5x_2)/10 $$

Despejando las Cadenas ($x_2$):

$$ x_2 = (250 - 10x_1)/5 $$

```python
def x2_v1(x1):                 #Despejando Aretes
    return (250 - 5*x2)/10
def x2_v2(x1):                 #Despejando Cadenas
    return (250 - 10*x1)/5
```

```python
x1 = np.linspace(0,100)
```

```python
plt.plot(x1, x2_v1(x1), 'magenta')         #Desigualdad 1
plt.plot(x1, x2_v2(x1))                    #Desigualdad 2
plt.plot([20, 20], [10, 15.1],"k")
plt.scatter([20, 20], [10, 15.1], color = "red", s = 50)
#plt.plot([43, 47], [5, 5], 'r')
#plt.scatter([43, 47], [5, 5], color = "red", s = 100)
plt.xlim(xmin = 10, xmax = 40)           #Zoom en x
plt.ylim(ymin = 0, ymax = 35)               #Zoom en y
#plt.scatter([45, 45, 45.6], [5, 6.25, 5], color = 'red')   #Punto en el cruce de las lineas
#plt.fill_between(np.array([45, 45.6]), x2_v1(np.array([45, 45.6])),
#                5*np.ones(2), alpha = .2, color = 'blue')
#plt.xlabel(r"$x_1$", fontsize = 18)
#plt.ylabel(r"$x_2$", fontsize = 18)
#plt.show()
```


Remarques de analyse fonctionnelle.

---
Auteur: [André ROCHA](https://github.com/rochamatcomp)

---

# Les espaces $\ell^{p}(n)$

#### Définition.
Soit $1 \le p \le \infty$. L'espace $\ell^{p}(n)$ est défini comme étant l'espace $\mathbb{R}^{n}$ muni de la norme :

\begin{equation}
    ||x||_{p} = \left( \sum\limits_{i=1}^{n} |x_{i}|^{p} \right)^{1/p}, \quad si 1 \le p < \infty
\end{equation}

\begin{equation}
    ||x||_{\infty} = \max\limits_{1 \le i \le n} |x_{i}|. \quad (p = \infty).
\end{equation}

#### Propositions à preuver.
L'espace $\ell^{p}(n)$ est un espace vectoriel normé complet (espace de Banach).

##### Remarque.
> Regarde que $\ell^{p}(n)$ est l'espace $\mathbb{R}^{n}$ muni de la norme euclidienne, la quelle est derivé de un produit intérieur.

<!--

F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx
\end{equation}
-->
<!--
 !-- $\mathscr{L}$
 !-- $\mathscr{l}$.
-->

<!--
 !--\begin{equation})
 !--   \llbracket     1 \rrbracket       \quad
 !--   \llparenthesis 2 \rrparenthesis   \quad
 !--   \llceil        3 \rrceil          \quad
 !--   \llfloor       4 \rrfloor         \quad
 !--\end{equation}
-->

```python

```


=============================

Moist Adiabats on a Tephigram

=============================

Sept 2015

nmoisseeva@eos.ubc.ca

This notebook is intended to summarize the work on moist adiabats on a tephigram.

```python
#import necessary packages
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 10, 7  # that's default image size for this
```

Set up thermodynamical constants:

```python
a = 0.285611 #Rd/Cp: for dry air
b = 1.35e7   #K^2: Lv^2*Eps/Cp*Rd
c = 2488.4   #K: Lv/Cp
LvRv = 5422. #K: Lv/Rv
T0 = 273.15  #0C

Eps = 0.622  #gv/gd: ratio of gas constants dry vs vapour
e0 = 0.611   #kPa: Clausius-Clayperon constant
```

Next, set up a test values for moist adiabat labels and pressure, and some storage arrays. Note that the range of pressures is defined from 100 to 1kP with 0.01 integration step, to ensure the potential temperature formula doesn't blow up for values less than 1. Also the tested temperature range is limited to +30C, as past this value the specific heat of air begins to change, which can introduce additional error into the equations.

```python
Prange = np.arange(100,1, -0.01)
ThetaW = np.arange(-50,30)
adiabats = np.empty((len(Prange),len(ThetaW)))
dry_adiabats = np.empty_like(adiabats)
```

Now, define formulas for Clausius-Clayperon, saturation mixing ratio and dP/dT as follows:

\begin{equation}
e_s = e_0 * exp(\frac{L_v}{R_v} * (\frac{1}{T_0}-\frac{1}{T}))
\end{equation}

\begin{equation}
r_s = \frac{\varepsilon * e_s}{P - e_s}
\end{equation}

\begin{equation}
\frac{\Delta T}{\Delta P} = \frac{a * T + c * r_s}{P * (1 + \frac{b * r_s}{T^2})}
\end{equation}

```python
def f_es(T):
	return  e0*np.exp(LvRv*(1./T0 - 1./T))

def f_rs(P,es):
	return  Eps*es / (P - es)

def dTdP(P,T):
	return (a*T + c*rs)/(P*(1+(b*rs/T**2)))
```

Manually integrate to calculate moist adiabats (for each pressure and temperature):

```python
for nT, Temp in enumerate(ThetaW):
	T = Temp + T0
	for nP,Pres in enumerate(Prange):
		es = f_es(T)
		rs = f_rs(Pres,es)
		grad = dTdP(Pres,T)
		T = T - grad*0.01
		adiabats[nP,nT] = T                       #store moist adiabat
		dry_adiabats[nP,nT] = T*((100./Pres)**a)  #store dry adiabat
```

Plot some adiabats (sanity check) every 10C:

```python
plt.plot(adiabats[:,0::10]-T0,Prange);
plt.gca().invert_yaxis();
plt.xlabel("moist adiabats [C]");
plt.ylabel("pressure [kPa]");
```

Define a function to calculate equivalent potential temperature:

\begin{equation}
    \theta_e = \theta_w * exp(\frac{a_3 * r_{s0}}{\theta_w})
\end{equation}

```python
a3 = 2490.  #K*kg_air/kg_vapour
def f_thE(ThetaW,rs0):
	return ThetaW * np.exp(a3*rs0/ThetaW)
```

Note, that the value of $a_3$ will be discussed further below.

Next, calculate equivalent temperature for each moist adiabat:

```python
ThetaE = np.empty((len(ThetaW)))
for nT, Temp in enumerate(ThetaW):
	T = Temp + T0
	es0 = f_es(T)
	rs0 = f_rs(100,es0)
	ThetaE[nT] = f_thE(T,rs0)
```

Finally, we can create an array of normalized adiabats, by using the following expression:

\begin{equation}
\theta_{norm} = \frac{\theta - \theta_{dry}}{\theta_e}
\end{equation}

This should ensure that all adiabats start at 0 at the surface and approach -1 at the top of the atmosphere. Since we did not test the pressures close to the top of the atmosphere (we stopped at 1kPa) the end of the normalized adiabat should reach values of ~-0.8

```python
norm_adiabats = (adiabats - dry_adiabats)/ThetaE
```

If successful, all adiabats should collapse into a single shape.

```python
plt.plot(norm_adiabats, Prange);
plt.gca().invert_yaxis();
plt.ylabel("pressure [kPa]");
plt.xlabel(r"$\theta_{norm}$");
```

This appears to work reasonably well for the given range of temperatures and pressures. To evaluate the error we can calculate the standard deviation for each $\theta_{norm}$ value and plot the resultant curve:

```python
spread = np.std(norm_adiabats, axis = 1)
plt.plot(spread, Prange);
plt.gca().invert_yaxis();
plt.xlabel("standard deviation");
plt.ylabel("pressure [kPa]");
```

Note that the standard deviation values are generally small, remaining below 0.0045 (or relatively less than half a percent of $\theta_{norm}$. By varying the value of $a_3$ the distribution of error can be modified. For example, by setting $a_3 = 2300$ K kg air /kg vapour the relative maximum near 60 kPa drops to 0.003, while the value at the top of the atmosphere reaches 0.006.

Moreover, simply reducing the tested temperature range to extend to the maximum of 25C reduces maximum standard deviation to 0.002.

Generally, the remaining error is likely caused by the dependance of constants on temperature (i.e. they are not truly constants). The mid-range bulge in the normalized graph is driven by the contribution from the saturated mixing ration term. With larger moisture content at higher temperatures the error grows.

To use the above relationship, $\theta_{norm}$ can be averaged to produce a single similarity relationship and fit with a simple exponential function. Based on $\theta_w$, the relationship between $\theta_{norm}$ and $P$, calculated values of $e_s$, $r_s$ and $\theta_{dry}$ can be used to determine the temperature.


# Correlation function

In this notebook we'll explore the correlation function $g(r, p; L)$ for various values of $p$. The correlation function is a measure of the probability that two sites a distance $r$ apart are part of the same connected cluster.

```python
import numpy as np
import matplotlib.pyplot as plt
import tqdm
import sklearn.linear_model
import seaborn as sns
from correlation_function import correlation_function
```

```python
sns.set()
```

```python
p_c = 0.59275
L = 400
```

```python
g_c = correlation_function(p_c, L)
r_c = np.arange(len(g_c))
```

```python
fig = plt.figure(figsize=(14, 10))

p_below = np.linspace(0.55, p_c, 11)[:-1]
S_below = np.zeros_like(p_below)
i = 0

for p in tqdm.tqdm_notebook(p_below):
    g = correlation_function(p, L)
    S_below[i] = (np.trapz(g))
    r = np.arange(len(g))
    plt.loglog(r, g, label=fr"$p = {p:.3f}$")
    i += 1

plt.loglog(r_c, g_c, "--", label=fr"$p = p_c = {p}$", lw=2)

plt.legend(loc="best")
plt.xlabel(r"$r$")
plt.ylabel(r"$g(r, p)$")
plt.title(fr"Correlation function $g(r, p; L)$ as a function of distance $r$ for $L = {L}$ and $p \to p_c$ from below.")
plt.show()
```

```python
fig = plt.figure(figsize=(14, 10))

p_above = np.linspace(0.55, p_c, 11)[:-1]
S_above = np.zeros_like(p_above)
i = 0

for p in tqdm.tqdm_notebook(p_above):
    g = correlation_function(p, L)
    S_above[i] = (np.trapz(g))
    r = np.arange(len(g))
    plt.loglog(r, g, label=fr"$p = {p:.3f}$")
    i += 1

plt.loglog(r_c, g_c, "--", label=fr"$p = p_c = {p}$", lw=2)

plt.legend(loc="best")
plt.xlabel(r"$r$")
plt.ylabel(r"$g(r, p)$")
plt.title(fr"Correlation function $g(r, p; L)$ as a function of distance $r$ for $L = {L}$ and $p \to p_c$ from above.")
plt.show()
```

Here we have plotted the correlation function $g(r, p; L)$ as a function of inter-site distance $r$ (taxi-cab metric). We see how the correlation function falls rapidly to zero when reach a threshold.

We now wish to find the correlation length $\xi(p, L)$ and plot it as a function of $|p - p_c|$. We know that the average cluster size, $S$, is given by
\begin{align}
    S = \sum_{j} g(r_j, p)
    = \int g(r, p) dr.
\end{align}
Furthermore, as $p \to p_c$ we have the scaling
\begin{align}
    S \propto |p - p_c|^{-\gamma} \propto \xi^{2 - \eta}.
\end{align}
Rewriting, we find the relation
\begin{align}
    \xi \propto |p - p_c|^{-\nu},
\end{align}
where we introduced the exponent $\nu = \gamma / (2 - \eta)$.

```python
# Computing the difference, |p - p_c|
p_min_pc_below = np.abs(p_below - p_c)
p_min_pc_above = np.abs(p_above - p_c)
```

```python
plt.loglog(p_min_pc_below, S_below)
plt.show()
```

```python
plt.loglog(p_min_pc_above, S_above)
plt.show()
```


---

## Item XX

Explain how to obtain the nodes and weights for the Gaussian quadrature for from the Jacobi matrices.

---

The gaussian quadrature consists on approximating:

$$
\int_a^b f(x) \, dx \approx \sum_{j=1}^n w_j f(x_j)
$$

choosing the right $x_j$ and $w_j$ so that the approximation is perfect if $f$ is any polynomial of degree $2n-1$ or smaller.

**Lemma 1**: Being $\{p_j(x)\}_{j=0}^n$ a set of $n$ orthonormal polynomials where $\text{deg}(p_j) = j$. Then there exist a quadrature with nodes $\{x_k\}_{k=1}^n$ as the $n$ roots of $p_n$ that approximates perfectly any polynomial $f(x)$ of degree $2n-1$ or smaller.

*Proof*: By performing a [polynomial long division](https://en.wikipedia.org/wiki/Polynomial_long_division) of $f(x)$ over $p_n(x)$, we have that:
$$
f(x) = p_n(x) q(x) + r(x) \,,
$$
with $\text{deg}(r),\text{deg}(q) < n$.

So we have that:
\begin{align}
\int_a^b f(x) \, dx &= \underbrace{\int_a^b p_n(x) q(x) \,dx}_{= 0} + \int_a^b r(x) \,dx
\end{align}
We see that:
$$
\int_a^b p_n(x) q(x) \ = 0
$$
because $p_n \in \mathbb{P}^{n-1}$ which is part of the subspace generated by the base $\{p_j(x)\}_{j=0}^{n-1}$, and thus, orthogonal to $p_n(x)$.

By defining:
$$
L_j(x) = \prod_{i=1\\i\neq j}^n \frac{x-x_i}{x_j-x_i} \,,
$$
we have that:
\begin{align}
\int_a^b r(x) \,dx
&= \int_a^b \sum_{k=1}^n r(x_j) L_j(x) \,dx
\\ &= \sum_{k=1}^n \left( r(x_j) \underbrace{\int_a^b L_j(x) \,dx}_{=w_j} \right)
\\ &= \sum_{k=1}^n r(x_j) w_j
\end{align}
which corresponds with the quadrature. $\square$

Now, in order to build the orthonormal polynomials $\{p_j(x)\}_{j=0}^n$, so that $\text{deg}(p_j) = j$ we can use the following [recurrence relation](https://en.wikipedia.org/wiki/Gaussian_quadrature#Recurrence_relation):

\begin{align}
p_{r+1}(x) &= (x-a_{r})p_{r}(x)-b_{r}p_{r-1}(x)
\\ p_{-1}(x) &= 0 \,,
\end{align}
where:
$$
a_{r}:={\frac {\langle xp_{r},p_{r} \rangle}{ \langle p_{r},p_{r} \rangle}},\qquad b_{r}:={\frac {\langle xp_{r},p_{r-1} \rangle}{\langle p_{r-1},p_{r-1} \rangle}}={\frac {\langle p_{r},p_{r} \rangle}{\langle p_{r-1},p_{r-1} \rangle}} \,.
$$

Which can be rewritten as:

\begin{align}
x p_r(x) &= \beta_{r-1} p_{r-1}(x) + \alpha_r p_r(x) + \beta_r p_{r+1}(x)
\\ p_{-1}(x) &= 0
\end{align}

So, we have that:

\begin{align}
x \underbrace{\left( \begin{matrix}
p_0(x) \\
p_1(x) \\
p_2(x) \\
\vdots \\
p_{n-2}(x) \\
p_{n-1}(x) \\
\end{matrix} \right)}_{\mathbf{p}(x)} &= \underbrace{ \left( \begin{matrix}
a_0 & b_0 \\
b_0 & a_1 & b_1 \\
 & b_1 & a_2 & b_2 \\
 & & \ddots & \ddots & \ddots \\
 & & & b_{n-3} & a_{n-2} & b_{n-2} \\
 & & & & b_{n-2} & a_{n-1} \\
\end{matrix} \right)}_{T_n} \underbrace{\left( \begin{matrix}
p_0(x) \\
p_1(x) \\
p_2(x) \\
\vdots \\
p_{n-2}(x) \\
p_{n-1}(x) \\
\end{matrix} \right)}_{\mathbf{p}(x)} + b_{n-1} p_n(x) \underbrace{\left( \begin{matrix}
0 \\
0 \\
0 \\
\vdots \\
0 \\
1 \\
\end{matrix} \right)}_{\mathbf{e}_{n}} \\
x \mathbf{p}(x) &= T_n \mathbf{p}(x)+ b_{n-1} p_n(x) \mathbf{e}_n \,.
\end{align}

Making $x=x_j$ where $\{x_j\}_{j=1}^{n}$ are the roots of $p_n(x)$:
\begin{align}
x_j \mathbf{p}(x_j) &= T_n \mathbf{p}(x_j)+ b_{n-1} \underbrace{p_n(x_j)}_{=0} \mathbf{e}_n \\
x_j \mathbf{p}(x_j) &= T_n \mathbf{p}(x_j) \,,
\end{align}

which means that the roots of $p_n$ are the eigenvalues of this system, where $T_n$ is a tridiagonal (Jacobi) matrix. These roots also are the nodes of the Guassian quadrature due to the proven lemma.

```python

```


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
from scipy import stats
```

## Calculus in SymPy ##

Working with densities involves calculus which can sometimes be time-consuming. This course gives you two ways of reducing the amount of calculus involved.

- Probabilistic methods can help reduce algebra and calculus. You've seen this with algebra in the discrete case. You'll see it with calculus as we learn more about densities.
- Python has a symbolic math module called `SymPy` that does algebra, calculus, and much other symbolic math. In this section we will show you how to do calculus using `SymPy`.

We will demonstrate the methods in the context of an example. Suppose $X$ has density given by

$$
f(x) =
\begin{cases}
105x^2(1-x)^4 ~~~ \text{if } 0 \le x \le 1 \\
0 ~~~~~~~~~ \text{otherwise}
\end{cases}
$$

As you can see from its graph below, $f$ could be used to model the distribution of a random proportion that you think is likely to be somewhere between 0.2 and 0.4.

```python
# HIDDEN
x_vals = np.arange(0, 1.01, 0.01)
f = 105 * x_vals**2 * (1 - x_vals)**4
plt.plot(x_vals, f, color='darkblue', lw=2)
plt.xlabel('$x$')
plt.ylabel('$f(x)$', rotation=0);
```

The density $f$ is a polynomial on the unit interval, and in principle the algebra and calculus involved in integrating it are straightforward. But they are tedious. So let's get `SymPy` to do the work.

First, we will import all the functions in `SymPy` and set up some printing methods that make the output look nicer than the retro typewritten pgf output you saw in a previous section. In future sections of this text, you can assume that this importing and initialization will have been done at the start.

```python
from sympy import *
init_printing()
```

Next, we have to tell Python which variables are symbolic and what their possible values are. The function `declare` lets us do this. It takes as its arguments the string representing the variable, and an option `interval` specifying the interval of possible values of the variable. In our example, the variable `x` takes values in the unit interval. In later examples we will show you how to declare infinite intervals of possible values.

```python
declare('x', interval=(0, 1))
```

Now we will assign the name `density` to the expression that defines $f$. The expression looks just like a numerical calculation, but the output is algebraic!

```python
density = 105 * x**2 * (1-x)**4
density
```

That's the density $f$ defined by the equation at the start of the section. Notice that what we naturally think of as $1 - x$ is expressed as $-x + 1$. That's because `SymPy` is writing the polynomial leading with the term of highest degree.

Let's not simply accept that this function is a density. Let's check that it is a density by integrating it from 0 to 1. To do this, we use the method `Integral` that takes the name of a function and a *tuple* (a sequence in parentheses) consisting of the variable of integration and the lower and upper limits of integration. We have assigned this integral to the name `total_area`

```python
total_area = Integral(density, (x, 0, 1))
total_area
```

The output of displays the integral, which is nice, but what we really want is its numerical value. In `SymPy`, this is achieved by rather rudely instructing the method to `doit()`.

```python
total_area.doit()
```

This confirms that the function $f$ is a density.

We can use `Integral` again to find the chance of any interval. Here is $P(0.2 < X < 0.4)$.

```python
p_02_04 = Integral(density, (x, 0.2, 0.4)).doit()
p_02_04
```

For $x$ in the unit interval, the cdf of $X$ is
$$
F(x) ~ = ~ P(X \le x) ~ = ~ \int_0^x f(s)ds ~ = ~ I(s)~ \Big{\rvert}_0^x ~ = ~ I(x) - I(0)
$$

where $I$ is the indefinite integral of $f$.

To get the indefinite integral, simply ask `SymPy` to integrate the density; there are no limits of integration.

```python
indefinite = Integral(density).doit()
indefinite
```

Now $F(x) = I(x) - I(0)$. You can see at a glance that $I(0) = 0$ but here is how `SymPy` would figure that out.

To evaluate $I(0)$, `SymPy` must substitute $x$ with 0 in the expression for $I$. This is achieved by the method `subs` that takes the variable as its first argument and the specified value as the second.

```python
I_0 = indefinite.subs(x, 0)
I_0
```

```python
cdf = indefinite - I_0
cdf
```

To find the value of the cdf at a specified point, say 0.4, we have to substitute $x$ with 0.4 in the formula for the cdf.

```python
cdf_04 = cdf.subs(x, 0.4)
cdf_04
```

Thus $P(X \le 0.4)$ is roughly 58%. Earlier we calulated $P(0.2 < X < 0.4) = 43.2\%$, which we can confirm by using the cdf:

```python
cdf_02 = cdf.subs(x, 0.2)
cdf_04 - cdf_02
```

The expectation $E(X)$ is a definite integral from 0 to 1:

```python
expectation = Integral(x*density, (x, 0, 1)).doit()
expectation
```

Notice how simple the answer is. Later in the course, you will see why.

Here is $E(X^2)$, which turns out to be another simple fraction. Clearly, the density $f$ has interesting properties. We will study them later. For now, let's just get the numerical answers.

```python
expected_square = Integral((x**2)*density, (x, 0, 1)).doit()
expected_square
```

Now you can find $SD(X)$.

```python
sd = (expected_square - expectation**2)**0.5
sd
```

### SymPy and the Exponential Density ###
One of the primary distributions in probability theory, the exponential distribution has a positive parameter $\lambda$ known as the "rate", and density given by

$$
f(t) ~ = \lambda e^{-\lambda t}, ~~~ t \ge 0
$$

The density is 0 on the negative numbers. Here is its graph when $\lambda = 3$.

```python
# HIDDEN

lam = 3
tt = np.arange(0, 2.01, 0.01)
f = lam * np.exp(-1*lam*tt)
plt.plot(tt, f, color='darkblue', lw=2)
plt.xlabel('$t$')
plt.ylabel('$f(t)$', rotation=0)
plt.title('Exponential Density, Rate $\lambda = 3$');
```

To check that $f$ is a density, we have to confirm that its integral is 1. So we will declare two positive symbolic variables `t` and `lamda`. Notice the incorrectly spelled `lamda`. That is because `lambda` has another meaning in Python, as some of you might know.

In fact `lamda` is a constant, not a variable. But `SymPy` needs to know that it's an algebraic object, so we have to declare it as such.

Note the use of `positive=True` to declare positive variables.

```python
declare('lamda', positive=True)
declare('t', positive=True)
```

Now we will define the density function. Notice the use of `exp` for the exponential function. Notice also that the form of the answer looks different from the way we have written it above, though it's algebraically the same.

```python
expon_density = lamda * exp(-lamda * t)
expon_density
```

This is an unavoidable aspect of computer aided algebra, and it is the reason we will use `SymPy` purely for computation, not for display.

To see that the function is a density, we can check that its integral from 0 to $\infty$ is 1. The symbol that `SymPy` uses for $\infty$ is `oo`, a double lower case o. It looks very much like $\infty$.

```python
Integral(expon_density, (t, 0, oo)).doit()
```

Suppose $T$ has the exponential $(\lambda)$ density. Then for $t \ge 0$ the cdf of $T$ is

$$
F_T(t) ~ = ~ P(T \le t) ~ = ~ \int_0^t \lambda e^{-\lambda s}ds
$$

This is a straightforward integral that you can probably do in your head. However, let's get some more practice using `SymPy` to find cdf's.
We will use the same method that we used to find the cdf in the previous example.

$$
\int_0^t \lambda e^{-\lambda s}ds ~ = ~ I(t) - I(0)
$$

where $I$ is the indefinite integral of the density. To get this indefinite integral we will use `Integral` as before, except that this time we must specify `t` as the variable of integration. That is because `SymPy` sees two algebraic quantities `t` and `lamda` in the density, and doesn't know which one is the variable unless we tell it.

```python
indefinite = Integral(expon_density, t).doit()
indefinite
```

Now use $F_T(t) = I(t) - I(0)$:

```python
I_0 = indefinite.subs(t, 0)
I_0
```

```python
cdf = indefinite - I_0
cdf
```

Thus the cdf of the exponential $(\lambda)$ density is
$$
F_T(t) ~ = ~ 1 - e^{-\lambda t}
$$
Also,
$$
E(T) ~ = ~ \int_0^\infty t \lambda e^{-\lambda t} dt ~ = ~ \frac{1}{\lambda}
$$

which you can check by integration by parts. But `SymPy` is faster:

```python
expectation = Integral(t*expon_density, (t, 0, oo)).doit()
expectation
```

And
$$
E(T^2) = \frac{2}{\lambda^2}
$$

```python
expected_square = Integral(t**2 * expon_density, (t, 0, oo)).doit()
expected_square
```

So
$$
Var(T) ~ = ~ \frac{2}{\lambda^2} - \frac{1}{\lambda^2} ~ = ~ \frac{1}{\lambda^2}
$$
and hence
$$
SD(T) ~ = ~ \frac{1}{\lambda}
$$

The purpose of this section has been to give you a workout in `SymPy`. We will take a closer look at the exponential distribution in the next section.

```python

```


# Лаба Дмитро

## Лабораторна робота №2
## Варіант 2

```python
import numpy as np
import sympy as sp
from scipy.linalg import eig
from sympy.matrices import Matrix
from IPython.display import display, Math, Latex

def bmatrix(a):
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{bmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{bmatrix}']
    return Math('\n'.join(rv))
```

# Завдання №1

```python
industry = (600, 450, 1200)
industry_added_value = 0.3

agriculture = (450, 400, 700)
agriculture_added_value = 0.5

matrix = np.array([industry, agriculture])
added_value = np.array([industry_added_value, agriculture_added_value])
```

```python
gross_output = np.sum(matrix, axis=1)
direct_expenses = matrix[0:2, 0:2]/gross_output
print('Прямі витрати: ')
display(bmatrix(direct_expenses))
```

    Прямі витрати:

$$\begin{bmatrix}
  0.26666667 & 0.29032258\\
  0.2 & 0.25806452\\
\end{bmatrix}$$

```python
full_expenses = np.linalg.inv(np.eye(len(direct_expenses)) - direct_expenses)
print('Повні витрати: ')
display(bmatrix(full_expenses))
```

    Повні витрати:

$$\begin{bmatrix}
  1.52654867 & 0.59734513\\
  0.41150442 & 1.50884956\\
\end{bmatrix}$$

```python
prices = added_value @ full_expenses
print('Ціни на промислову та сільськогосподарську продукцію: ')
display(bmatrix(prices))
```

    Ціни на промислову та сільськогосподарську продукцію:

$$\begin{bmatrix}
  0.66371681 & 0.93362832\\
\end{bmatrix}$$

# Завдання №2

```python
A = np.array([
    [0.4, 0.4, 0.2],
    [0.2, 0.5, 0.4],
    [0.1, 0.1, 0.2]
])
n = len(A)
```

## Власні числа

```python
eigvals = np.linalg.eigvals(A)
print('\n'.join(['({0:.2f} {1} {2:.2f}i)'.format(n.real, '+-'[n.imag < 0], abs(n.imag)) for n in eigvals]))
```

    (0.84 + 0.00i)
    (0.13 + 0.07i)
    (0.13 - 0.07i)

## Коефіцієнти характеристичного поліному

```python
M = Matrix(A)
lamda = sp.symbols('lamda')
characteristic_polynomial = M.charpoly(lamda)
sp.init_printing(use_latex='mathjax')
display(sp.factor(characteristic_polynomial))
```

$$1.0 \left(1.0 \lambda^{3} - 1.1 \lambda^{2} + 0.24 \lambda - 0.018\right)$$

## Число Фробеніуса

```python
frobenius_number = np.real(np.max(eigvals))
print('{0:.2f}'.format(frobenius_number))
```

    0.84

## Правий та лівий вектори Фробеніуса

```python
a, left, right = eig(A, left=True)
print('Лівий вектор: ')
left_frobenius = np.real(left[:, 0])
display(bmatrix(left_frobenius))

print('Перевірка: ')
display(bmatrix(left_frobenius @ A))
display(bmatrix(frobenius_number * left_frobenius))

print('Правий вектор: ')
right_frobenius = np.real(right[:, 0])
display(bmatrix(right_frobenius))

print('Перевірка: ')
display(bmatrix(A @ right_frobenius))
display(bmatrix(frobenius_number * right_frobenius))

```

    Лівий вектор:

$$\begin{bmatrix}
  0.44397979 & 0.69076468 & 0.57072419\\
\end{bmatrix}$$

    Перевірка:

$$\begin{bmatrix}
  0.37281727 & 0.58004668 & 0.47924667\\
\end{bmatrix}$$

$$\begin{bmatrix}
  0.37281727 & 0.58004668 & 0.47924667\\
\end{bmatrix}$$

    Правий вектор:

$$\begin{bmatrix}
  0.7089431 & 0.67144485 & 0.21578112\\
\end{bmatrix}$$

    Перевірка:

$$\begin{bmatrix}
  0.5953114 & 0.56382349 & 0.18119502\\
\end{bmatrix}$$

$$\begin{bmatrix}
  0.5953114 & 0.56382349 & 0.18119502\\
\end{bmatrix}$$

## Продуктивність матриці

За критерієм Леонтьєва, необхідня і достатня умова $\lambda_A < 1$ виконується, тому матриця є продуктивною

## Матриця повних витрат

```python
B = np.linalg.inv(np.eye(len(A)) - A)
display(bmatrix(B))
```

$$\begin{bmatrix}
  2.95081967 & 2.78688525 & 2.13114754\\
  1.63934426 & 3.7704918 & 2.29508197\\
  0.57377049 & 0.81967213 & 1.80327869\\
\end{bmatrix}$$

## Збіжність до матриці повних витрат

```python
matrix_sum, A_power = np.eye(n), np.eye(n)
for i in range(200):
    A_power = A_power @ A
    matrix_sum += A_power
    if np.all(B - matrix_sum) < 0.01:
        print('Матриця збіжна до матриці повних витрат на кроці {}'.format(i))
        break
display(bmatrix(matrix_sum))
```

    Матриця збіжна до матриці повних витрат на кроці 193

$$\begin{bmatrix}
  2.95081967 & 2.78688525 & 2.13114754\\
  1.63934426 & 3.7704918 & 2.29508197\\
  0.57377049 & 0.81967213 & 1.80327869\\
\end{bmatrix}$$

## Вектор кінцевого випуску

```python
y = np.array([100, 70, 80])
end_output = np.linalg.inv(np.eye(n) - A) @ y
display(bmatrix(end_output))
```

$$\begin{bmatrix}
  660.6557377 & 611.47540984 & 259.01639344\\
\end{bmatrix}$$


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

| Mathematical notation | Python code | Requires SymPy? |
|--|--|--|
| $x+y$ | `x+y` | no |
| $x-y$ | `x-y` | no |
| $xy$ | `x*y` | no |
| $\frac xy$ | `x/y` | no |
| $\left\lfloor\frac xy\right\rfloor$ | `x//y` | no |
| remainder of $x\div y$ | `x%y` | no |
| $x^y$ | `x**y` | no |
| $\vert x\vert$ | `abs(x)` | no |
| $\ln x$ | `log(x)` | yes |
| $\log_a b$ | `log(b,a)` | yes |
| $e^x$ | `E` | yes |
| $\pi$ | `pi` | yes |
| $\sin x$ | `sin(x)` | yes |
| $\sin^{-1} x$ | `asin(x)` | yes |
| $\sqrt x$ | `sqrt(x)` | yes |

Other trigonometric functions are also available besides just `sin`,
including `cos`, `tan`, etc.

Note that SymPy gives precise answers to mathematical queries,
which may not be what you want.

```python
sqrt(2)
```

$\displaystyle \sqrt{2}$

If you want a decimal approximation instead, you can use the `N` function.

```python
N(sqrt(2))
```

$\displaystyle 1.4142135623731$

Or you can use the `evalf` function.

```python
sqrt(2).evalf()
```

$\displaystyle 1.4142135623731$

By contrast, if you need an exact rational number when Python gives you
an approximation, you can use the `Rational` function to build one.
Note the differences below:

```python
1/3
```

$\displaystyle 0.333333333333333$

```python
Rational(1,3)
```

$\displaystyle \frac{1}{3}$

database/tasks/How to do basic mathematical computations/Python, using SymPy.ipynb


   # Muon decay experiment (v_101817_1)

We use this code to:
* simulate the muon decay experiment,
* obtain a best fit value for the lifetime, and
* the corresponding error using Monte Carlo methods.

```python
%matplotlib notebook

from __future__ import division    #for division operator
import numpy as np
from scipy.optimize import minimize, curve_fit
import scipy.stats as stats
from pynverse import inversefunc
import matplotlib.pylab as plt
outfilepath = '/Users/Jatan/Google Drive/PHYS2010/Muon_decay/'   #modify this path to your plot directory
```

We first simulate the muon data set using Monte carlo methods. Initializing,

```python
sig_samples = 1500
bkg_samples = 600
start_t = 0
stop_t = 20 #micro seconds
bw = 0.01 #10 ns
data_bins = np.arange(0, stop_t + bw, bw)
tau = 2.2 # microseconds
def exp_func(t, lifetime):
 	return (0.714/(lifetime*(1 - np.exp(-stop_t/lifetime))))*np.exp(-t/lifetime) + (0.286/20)
```

## Method 1: Rejection sampling

1800 samples were generated using rejection sampling. Also shown is a histogram of events per 0.5 $\mu s$ bins for reference.

```python
signal = []
count = 1
count_tot = 1

while count <= (sig_samples+bkg_samples):
	count_tot += 1
	x = stop_t*np.random.random()
	y = np.random.random()
	if y <= exp_func(x, tau):
		signal.append(x)
		count += 1

events, bins = np.histogram(signal, data_bins)
rescaled_bins = (bins[:-1] + bins[1:]) / 2  #rescaling bins to find midpoints for each bin

print events[0]

fig, ax = plt.subplots(1, 1)
plt.hist(signal, 50, normed='True', histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
plt.xlim(0, 20)
plt.xlabel(r'$t \; [ \mathrm{in \; s} ]$')
plt.ylim(0, 0.35)
plt.setp(ax.get_yticklabels()[0], visible=False)
plt.ylabel(r'Events')
plt.legend(loc='upper right', frameon=False)
plt.grid()
plt.show()
```

    7

    <IPython.core.display.Javascript object>

As there is data available in every bin, we can minimize the chi-squared statistic to obtain best fit parameters ($\,A, \tau$ and $B\,$) for the function,
\begin{equation}
f_\text{fit}(t) = A\,e^{\,(\frac{-t}{\tau})} \, + \, B
\end{equation}

where the chi-squared statistic is calculated the usual way,
\begin{equation}
\chi^2  = \sum_{i=1}^{N} \, \frac{(\, y_i - f_\text{fit}(t_i)\,)^2}{\sigma^2_{\text{th},\, i}}
\end{equation}

We use $\sigma_{\text{th},\, i} = \sqrt{f_\text{fit}(t_i)}$ instead of the experimental error to prevent the value of $\chi^2$ from diverging for bins with no/very few events.

```python
init_params_chisq = [1, 1, 1]
def func_chisq(params):
	A, lifetime, B = params
	return np.sum([pow((events[i] - (A*np.exp(-rescaled_bins[i]/lifetime) + B)), 2)/(A*np.exp(-rescaled_bins[i]/lifetime) + B) for i in range(len(events))])
fit_param = minimize(func_chisq, init_params_chisq, method='nelder-mead')
print fit_param.x

chi_sq = func_chisq(fit_param.x)
print chi_sq #/(len(rescaled_bins)-3) for chi_sq/dof
```

    [ 7.42865364  2.14190264  0.62008802]
    1463.579367

```python
result_bestfit = [(fit_param.x[0]*np.exp(-rescaled_bins[i]/fit_param.x[1]) + fit_param.x[2]) for i in range(len(events))]
fig, ax = plt.subplots(1, 1)
plt.hist(signal, len(rescaled_bins), histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
plt.plot(rescaled_bins, result_bestfit, color='g', lw=2, label='pdf')
plt.xlim(0, 20)
plt.xlabel(r'$t \; [ \mathrm{in \ s} ]$')
plt.ylabel(r'Events')
plt.suptitle('Chi-square fit')
plt.legend(loc='upper right', frameon=False)
plt.grid()
plt.savefig(outfilepath + 'chisq_fit.pdf')
```

    <IPython.core.display.Javascript object>

To avoid binning errors, we perform an unbinned maximum likelihood fit on the data. Since the signal for the muon decay is given by:
\begin{equation}
N(t) = N_\text{sig} \, e^{\,(\frac{-t}{\tau})} + N_\text{bkg}
\end{equation}

it's preferable to use an exponential pdf for the fit. The normalized pdf we consider is
\begin{equation}
f(x) = A\,e^{\,(\frac{-t}{\tau})} + B
\end{equation}
where
\begin{equation}
A = \frac{0.7}{\tau\,(1 - e^{\,(\frac{-20}{\tau})})}, \;\;\;
B = \frac{0.286}{20}
\end{equation}

We can define the likelihood function as
\begin{equation}
\mathcal{L}\,(\tau \,|\, t_i) = \prod_{i=1}^{N} \left( A \, e^{\,(\frac{-t_i}{\tau})} + B \right)
\end{equation}

Or equivalently the log likelihood,
\begin{equation}
\log \mathcal{L}\,(t_i \,|\, \tau) = \sum_{i=1}^{N} \log \left( A\,e^{\,(\frac{-t_i}{\tau})} + B\right)
\end{equation}

We maximize the log likelihood to obtain the best fit lifetime $\tau$ and the maximum likelihood value.

```python
def likelihood_fit(params):
	lifetime = params
	logll = np.sum([np.log((0.714/(lifetime*(1 - np.exp(-20/lifetime))))*np.exp(-signal[i]/lifetime) + (0.286/20)) for i in range(len(signal))])
	return -logll

init_params = [1,]
result = minimize(likelihood_fit, init_params, method='nelder-mead')
tau_fit = result.x[0]   #best fit estimator = np.sum(signal)/len(signal)

print likelihood_fit(tau_fit), tau_fit
```

    4981.4079094 2.17705078125

```python
fig, ax = plt.subplots(1, 1)
plt.hist(signal, 30, normed='True', histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
plt.plot(data_bins, exp_func(data_bins, tau_fit), color='g', lw='2', label='pdf')
plt.xlim(0, 20)
plt.xlabel(r'$t \; [ \mathrm{in \ s} ]$')
#plt.ylim(0, 300)
#plt.setp(ax.get_yticklabels()[0], visible=False)
plt.ylabel(r'Events')
plt.suptitle('Maximum Likelihood fit')
plt.legend(loc='upper right', frameon=False)
plt.grid()
plt.savefig(outfilepath + 'ml_fit.pdf')
```

    <IPython.core.display.Javascript object>

To estimate the error in the lifetime parameter $\tau$, we generate $10^3$ 'copies' of our experiment and calculate the standard error in the best fit values obtained for each copy.

```python
tau_fit = []
likhood = []
no_of_samples = 1000
for i in range(no_of_samples):

	signal = []
	count = 1
	count_tot = 1
	while count <= (sig_samples+bkg_samples):
		count_tot += 1
		x = stop_t*np.random.random()
		y = np.random.random()
		if y <= exp_func(x, tau):
			signal.append(x)
			count += 1

#perform likelihood fit to data
	def likelihood_fit(params):
		lifetime = params
		logll = np.sum([np.log((0.714/(lifetime*(1 - np.exp(-stop_t/lifetime))))*np.exp(-signal[i]/lifetime) + (0.286/20)) for i in range(len(signal))])
		return -logll

	init_params = [1,]
	result = minimize(likelihood_fit, init_params, method='nelder-mead')
	tau_fit.append(result.x[0])
	likhood.append(-likelihood_fit(result.x[0]))

print np.average(tau_fit), np.sqrt(np.var(tau_fit)/len(tau_fit))  #mean and variance in lifetime calculation
```

    2.19739169922 0.00243410500107

```python
fig, ax = plt.subplots(1, 2)
ax[0].hist(tau_fit, histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
ax[1].hist(likhood, histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
# plt.xlim(0, 20)
ax[0].set_xlabel(r'Lifetime $\; [ \mathrm{in \ s} ]$')
ax[0].set_ylabel(r'Events')
ax[1].set_xlabel(r'Max. likelihood')
ax[1].set_ylabel(r'Events')
#ax[0].set_suptitle('Best fit (lifetime)')
# plt.legend(loc='upper right', frameon=False)
ax[0].grid()
ax[1].grid()
plt.savefig(outfilepath + 'lifetime_fit.pdf')
```

    <IPython.core.display.Javascript object>

## Method 2: Inverse cdf method

```python
signal = []
cdf = (lambda t: (0.714/(1 - np.exp(-20/tau)))*(1 - np.exp(-t/tau)) + (0.286/20)*t)
icdf = inversefunc(cdf)
for i in range(sig_samples+bkg_samples):
	signal.append(icdf(np.random.random()))

events, bins = np.histogram(signal, 50)
rescaled_bins = (bins[:-1] + bins[1:]) / 2  #rescaling bins to find midpoints for each bin

fig, ax = plt.subplots(1, 1)
plt.scatter(rescaled_bins, events)
plt.xlim(0, 20)
plt.xlabel(r'$t \; [ \mathrm{in \; s} ]$')
#plt.ylim(0, 16)
plt.setp(ax.get_yticklabels()[0], visible=False)
plt.ylabel(r'Events')
plt.grid()
plt.show()
```

As there is data available in every bin, we can minimize the chi-squared to obtain best fit parameters ($\,A, \tau$ and $B\,$) for the function,
\begin{equation}
f_\text{fit}(t) = A\,e^{\,(\frac{-t}{\tau})} \, + \, B
\end{equation}

where chi-squared is calculated the usual way,
\begin{equation}
\chi^2  = \sum_{i=1}^{N} \, \frac{(\, y_i - f_\text{fit}(t_i)\,)^2}{\sigma^2_{\text{th},\, i}}
\end{equation}

We use $\sigma_{\text{th},\, i} = \sqrt{f_\text{fit}(t_i)}$ instead of the experimental error to prevent the value of $\chi^2$ from blowing up for bins with no/very few events.

```python
init_params_chisq = [1, 1, 1]
def func_chisq(arg, A, lifetime, B):
	return (A*np.exp(-arg/lifetime) + B)
sigma = [np.sqrt(exp_func(rescaled_bins[i], tau)) for i in range(len(events))]
fit_param = curve_fit(func_chisq, rescaled_bins, events, init_params_chisq, sigma)[0]
print fit_param

chi_sq = np.sum([pow((events[i] - func_chisq(rescaled_bins[i], *fit_param)), 2)/pow(sigma[i], 2) for i in range(len(events))])
print (chi_sq/(len(rescaled_bins)-3))
```

```python
fig, ax = plt.subplots(1, 1)
plt.hist(signal, len(rescaled_bins), histtype='stepfilled', color='blue', alpha=0.2, label='histogram')
plt.plot(rescaled_bins, func_chisq(rescaled_bins, *fit_param), color='g', lw=2, label='pdf')
plt.xlim(0, 20)
plt.xlabel(r'$t \; [ \mathrm{in \ s} ]$')
plt.ylabel(r'Events')
plt.suptitle('Chi-square fit')
plt.legend(loc='upper right', frameon=False)
plt.grid()
plt.savefig(outfilepath + 'chisq_fit.pdf')
```


# Intro to SymPy

Here we give a (quick) introduction to SymPy. This tutorial assumes you are already familiar with SymPy expressions, so this notebook should serve as a refresher.

## Expressions

The function `init_printing()` will enable LaTeX pretty printing in the notebook for SymPy expressions. We recommend calling it at the top of any notebook that uses SymPy.

```python
from sympy import *
init_printing()
```

SymPy symbols are created with the `symbols()` function. SymPy expressions are built up from symbols, numbers, and SymPy functions,

```python
x, y, z = symbols('x y z')
```

SymPy automatically pretty prints symbols with greek letters and subscripts.

```python
alpha1, omega_2 = symbols('alpha1 omega_2')
alpha1, omega_2
```

```python
sin(x + 1) - cos(y)
```

### Exercise

Write a symbolic expression for $$\frac{1}{\sqrt{2\pi\sigma^2} } \; e^{ -\frac{(x-\mu)^2}{2\sigma^2} }.$$ Remember that the function for $e^x$ is `exp(x)`. You will need to create symbols for `sigma` and `mu`. Square root is `sqrt`.

```python
# Write your answer here

```

## Some reminders on common gotchas:

Dividing two integers in Python creates a float, like `1/2` -> `0.5`. If you want a rational number, use `Rational(1, 2)` or `S(1)/2`.

```python
x + 1/2
```

```python
x + S(1)/2
```

`^` is the XOR operator. Use `**` for powers.

```python
x ^ y
```

```python
x ** y
```

All SymPy expressions are immutable. Functions that operate on an expression return a new expression.

```python
expr = x + 1
expr
```

```python
expr.subs(x, 2)
```

```python
expr
```

## Floating point representation

Since most languages targeted will not support symbolic representation it is useful to let SymPy evaluate a floating point approximation (up to a user specified number of digits).

```python
sqrt(2)
```

```python
sqrt(2).evalf(7)
```

### Exercise
Compute 100 digits of $\pi$.

```python
# Write your answer here

```

## Undefined functions

Undefined functions are created with `Function()`. Undefined are useful to state that one variable depends on another (for the purposes of differentiation).

```python
f = Function('f')
```

```python
f(x) + 1
```

## Derivatives

Derivatives are computed with the `diff()` function, using the syntax `diff(expr, var1, var2, ...)`.

```python
diff(sin(x + 1)*cos(y), x, y)
```

```python
diff(f(x) + 1, x)
```

### Exercise

Write an expression representing the wave equation in one dimension: $${\partial^2 u\over \partial t^2 } = c^2 { \partial^2 u\over \partial  x^2}.$$ Remember that $u$ is a function in two variables. You can represent an equation using `Eq`, like

```python
Eq(x, y)
```

```python
# Write your answer here

```

## Matrices
Matrices are created with `Matrix`. A matrix can contain any symbolic expression. See `Matrix?` for different ways to create a Matrix.

```python
Matrix([[1, 2], [3, 4]])
```

```python
# An unnested list will create a column vector
Matrix([1, 2, 3])
```

```python
Matrix([x, y, z])
```

Matrices support all common operations, and have many methods for performing operations.

```python
Matrix([[1, 2], [3, 4]])*Matrix([x, y])
```

for example, calculating the Jacobian matrix is as easy as:

```python
Matrix([sin(x) + y, cos(y) + x, z]).jacobian([x, y, z])
```

and for those of you who don't remember, the Jacobian is defined as:

$$
J = \begin{bmatrix}
\frac{\partial f_1}{\partial x_1} &  \frac{\partial f_1}{\partial x_2} & \cdots \\
\frac{\partial f_2}{\partial x_1} &  \frac{\partial f_2}{\partial x_2} & ~ \\
\vdots                            &  ~ & \ddots \\
\end{bmatrix}
$$

you will come across this mathematical entity in later notebooks in this tutorial.

### Exercise

Create the following matrix $$\left[\begin{matrix}1 & 0 & 1\\-1 & 2 & 3\\1 & 2 & 3\end{matrix}\right]$$

```python
# Write your answer here

```

Now create a matrix representing $$\left[\begin{matrix}x\\y\\z\end{matrix}\right]$$ and multiply it with the previous matrix to get $$\left[\begin{matrix}x + z\\- x + 2 y + 3 z\\x + 2 y + 3 z\end{matrix}\right].$$

```python
# Write your answer here

```

Now take the Jacobian of that matrix with respect to your column vector, to get the original matrix back.

```python
# Write your answer here

```

## Matrix symbols

SymPy can also operate on matrices of symbolic dimension ($n \times m$). `MatrixSymbol("M", n, m)` creates a matrix $M$ of shape $n \times m$.

```python
n, m = symbols('n m', integer=True)
M = MatrixSymbol("M", n, m)
b = MatrixSymbol("b", m, 1)
```

```python
M*b
```

```python
(M*b).shape
```

Some matrix expression functions do not evaluate unless you call `doit`.

```python
Transpose(M*b)
```

```python
Transpose(M*b).doit()
```

### Exercise

Write a matrix expression representing $$Au + Bv,$$ where $A$ and $B$ are $100\times 100$ and $u$ and $v$ are $100 \times 1$.

```python
# Write your answer here

```

## Indexed
Indexed symbols can be created with `IndexedBase` and `Idx`. `IndexedBase("A")` represents an array `A` and `Idx('i')` represents an index `i`. You can also use `symbols('i')` instead of `Idx('i')`.

```python
A = IndexedBase("A")
i = Idx('i')
```

```python
A[i]
```

### Exercise

Write an Indexed expression for $$A[i, j, k]$$

```python
# Write your answer here

```

## Solving systems of equations

A useful tool in your toolbelt when manipulating expressions is the ``solve`` function. `solve` solves equations symbolically (not numerically). The return value is a list of solutions.

You can give `solve` an `Eq`, or if you give it an expression, it automatically assumes that it is equal to 0.

```python
solve(Eq(x**2, 4), x)
```

```python
solve(x**2 + 3*x - 3, x)
```

It can also handle systems of equations. The return is a list of dictionaries, mapping symbols to solutions.

```python
eq1 = x**2 + y**2 - 4  # circle of radius 2
eq2 = 2*x + y - 1  # straight line: y(x) = -2*x + 1
solve([eq1, eq2], [x, y])
```

### Exercises

Solve the following system of equations: $$\begin{align}z &= x^2 - y^2\\z^2 &= x^2 + y^2 + 4\\z &= x + y\end{align}$$

```python
# Write your answer here

```

## Solving differential equations
If you are dealing with a differential equation, say:

$$
f''(x) = -f(x)
$$

SymPy's dsolve can (sometimes) produce an exact symbolic solution. Like `solve`, `dsolve` assumes that expressions are equal to 0.

```python
f = Function('f')
```

```python
dsolve(f(x).diff(x, 2) + f(x))
```

### Exercise

Solve the following ODE: $$f''(x) + 2f'(x) + f(x) = \sin(x)$$

```python
# Write your answer here

```