###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2014 L.A. Barba, G.F. Forsyth, C.D. Cooper [CFD Python](https://github.com/barbagroup/CFDPython), (c)2013 L.A. Barba, also under CC-BY.

```python
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, 'r').read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 1D (non)linear Advection

In the [last lecture](https://github.com/daniel-koehn/Differential-equations-earth-system/tree/master/03_Lorenz_equations#ordinary-differential-equations-in-earth-sciences-the-lorenz-equations), we looked into numerical integration methods for the solution of ordinary differential equations (ODEs), using a simplified thermal convection problem - the Lorenz equations. In this module, we will study the numerical solution of **partial differential equations (PDEs)**, where the unknown is a multi-variate function. The problem could depend on time, $t$, and one spatial dimension $x$ (or more), which means we need to build a discretization grid with each independent variable.

We will start our discussion of numerical PDEs with 1-D linear and non-linear advection equations.

## 1D Linear Advection

The *one-dimensional linear advection equation* is the simplest, most basic model that can be used to learn something about numerical solution of PDEs. It's surprising that this little equation can teach us so much! Here it is:

$$
\begin{equation}
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0 \tag{1}
\end{equation}
$$

The equation represents a *wave* propagating with speed $c$ in the $x$ direction, without change of shape. For that reason, it's sometimes called the *one-way wave equation*.

With an initial condition $u(x,0)=u_0(x)$, the equation has an exact solution given by:

$$
\begin{equation}
u(x,t)=u_0(x-ct) \tag{2}
\end{equation}
$$

##### Exercise 1

Prove that eq. (2) solves the 1D linear convection equation (1). Take the time and space derivative and stick them into the equation to see that it holds!

Look at the exact solution for a moment ... we know two things about it:

1. its shape does not change, being always the same as the initial waveform $u_0$, only shifted in the $x$-direction; and
2. it's constant along so-called **characteristic curves**, $x-ct=$constant. This means that for any point in space and time, you can move back along the characteristic curve to $t=0$ to know the value of the solution.

#### Characteristic curves for positive wave speed.

Why do we call the equations *linear*? PDEs can be either linear or non-linear. In a linear equation, the unknown function $u$ and its derivatives appear only in linear terms, in other words, there are no products, powers, or transcendental functions applied on them.

What is the most important feature of linear equations? Do you remember? In case you forgot: solutions can be superposed to generate new solutions that still satisfy the original equation. This is super useful!

## Finite-differences

In the previous lessons, we discretized time derivatives; now we have derivatives in both space *and* time, so we need to discretize with respect to *both* these variables.

Imagine a *space-time* plot, where the coordinates in the vertical direction represent advancing in time—for example, from $t^n$ to $t^{n+1}$—and the coordinates in the horizontal direction move in space: consecutive points are $x_{i-1}$, $x_i$, and $x_{i+1}$.  This creates a grid where a point has both a temporal and spatial index. Here is a graphical representation of the space-time grid:

$$
\begin{matrix}
t^{n+1} & \rightarrow & \bullet  && \bullet  && \bullet  \\
t^n & \rightarrow & \bullet  && \bullet  && \bullet  \\
& &  x_{i-1} && x_i && x_{i+1}
\end{matrix}
$$

For the numerical solution of $u(x,t)$, we'll use subscripts to denote the spatial position, like $u_i$, and superscripts to denote the temporal instant, like $u^n$.  We would then label the solution at the top-middle point in the grid above as follows:
$u^{n+1}_{i}$.

Each grid point below has an index $i$, corresponding to the spatial position and increasing to the right, and an index $n$, corresponding to the time instant and increasing upwards.  A small grid segment would have the following values of the numerical solution at each point:

$$
\begin{matrix}
& &\bullet & & \bullet & &  \bullet \\
& &u^{n+1}_{i-1} & & u^{n+1}_i & & u^{n+1}_{i+1} \\
& &\bullet & & \bullet & &  \bullet \\
& &u^n_{i-1} & & u^n_i & & u^n_{i+1} \\
& &\bullet & & \bullet & &  \bullet \\
& &u^{n-1}_{i-1} & & u^{n-1}_i & & u^{n-1}_{i+1} \\
\end{matrix}
$$

Another way to explain our discretization grid is to say that it is built with constant steps in time and space, $\Delta t$ and $\Delta x$, as follows:

$$
\begin{eqnarray}
x_i &=& i\, \Delta x \quad \text{and} \quad t^n= n\, \Delta t \nonumber \\
u_i^n &=& u(i\, \Delta x, n\, \Delta t) \notag
\end{eqnarray}
$$

### Discretizing our model equation

Let's see how to discretize the 1-D linear advection equation in both space and time.  By definition, the partial derivative with respect to time changes only with time and not with space; its discretized form changes only the $n$ indices.  Similarly, the partial derivative with respect to $x$ changes with space not time, and only the $i$ indices are affected.

We'll discretize the spatial coordinate $x$ into points indexed from $i=0$ to $N$, and then step in discrete time intervals of size $\Delta t$.

From the definition of a derivative (and simply removing the limit), we know that for $\Delta x$ sufficiently small:

$$
\begin{equation}
\frac{\partial u}{\partial x}\approx \frac{u(x+\Delta x)-u(x)}{\Delta x} \notag
\end{equation}
$$

This formula could be applied at any point $x_i$. But note that it's not the only way that we can estimate the derivative. The geometrical interpretation of the first derivative $\partial u/ \partial x$ at any point is that it represents the slope of the tangent to the curve $u(x)$. In the sketch below, we show a slope line at $x_i$ and mark it as "exact." If the formula written above is applied at $x_i$, it approximates the derivative using the next spatial grid point: it is then called a _forward difference_ formula.

But as shown in the sketch below, we could also estimate the spatial derivative using the point behind $x_i$, in which case it is called a _backward difference_. We could even use the two points on each side of $x_i$, and obtain what's called a _central difference_ (but in that case the denominator would be $2\Delta x$).

#### Three finite-difference approximations at $x_i$.

We have three possible ways to represent a discrete form of $\partial u/ \partial x$:

* Forward difference: uses $x_i$ and $x_i + \Delta x$,
* Backward difference: uses $x_i$ and $x_i- \Delta x$,
* Central difference: uses two points on either side of $x_i$.

The sketch above also suggests that some finite-difference formulas might be better than others: it looks like the *central difference* approximation is closer to the slope of the "exact" derivative. Curious if this is just an effect of our exaggerated picture? We'll show you later how to make this observation rigorous!

The three formulas are:

$$
\begin{eqnarray}
\frac{\partial u}{\partial x} & \approx & \frac{u(x_{i+1})-u(x_i)}{\Delta x} \quad\text{Forward}\notag\\
\frac{\partial u}{\partial x} & \approx & \frac{u(x_i)-u(x_{i-1})}{\Delta x} \quad\text{Backward}\notag\\
\frac{\partial u}{\partial x} & \approx & \frac{u(x_{i+1})-u(x_{i-1})}{2\Delta x} \quad\text{Central}\notag
\end{eqnarray}
$$

**Euler's method** is equivalent to using a forward-difference scheme for the time derivative. Let's stick with that, and choose the backward-difference scheme for the space derivative (forward in time & backward in space **FTBS Method**). Our discrete equation is then:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + c \frac{u_i^n - u_{i-1}^n}{\Delta x} = 0 \notag
\end{equation}
$$

where $n$ and $n+1$ are two consecutive steps in time, while $i-1$ and $i$ are two neighboring points of the discretized $x$ coordinate. With given initial conditions, the only unknown in this discretization is $u_i^{n+1}$.  We solve for this unknown to get an equation that lets us step in time, as follows:

$$
\begin{equation}
u_i^{n+1} = u_i^n - c \frac{\Delta t}{\Delta x}(u_i^n-u_{i-1}^n) \notag
\end{equation}
$$

We like to make drawings of a grid segment, showing the grid points that influence our numerical solution. This is called a **stencil**. Below is the stencil for solving our model equation with the finite-difference formula we wrote above.

#### Stencil for the "forward-time/backward-space" scheme.

## And compute!

Alright. Let's get a little Python on the road. First: we need to load our array and plotting libraries, as usual.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

We also set notebook-wide plotting parameters for the font family and the font size by modifying entries of the `rcParams` dictionary.

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

As a first exercise, we'll solve the 1D linear advection equation with a *square wave* initial condition, defined as follows:

$$
\begin{equation}
u(x,0)=\begin{cases}2 & \text{where } 0.5\leq x \leq 1,\\
1 & \text{everywhere else in } (0, 2)
\end{cases}
\notag
\end{equation}
$$

We also need a boundary condition on $x$: let $u=1$ at $x=0$. Our spatial domain for the numerical solution will only cover the range $x\in (0, 2)$.

#### Square wave initial condition.

Now let's define a few variables; we want to make an evenly spaced grid of points within our spatial domain. In the code below, we define a variable called `nx` that will be the number of spatial grid points, and a variable `dx` that will be the distance between any pair of adjacent grid points. We also can define a step in time, `dt`, a number of steps, `nt`, and a value for the wave speed: we like to keep things simple and make $c=1$.

```python
# Set parameters.
nx = 41  # number of spatial discrete points
L = 2.0  # length of the 1D domain
dx = L / (nx - 1)  # spatial grid size
nt = 25  # number of time steps
dt = 0.02  # time-step size
c = 1.0  # advection speed

# Define the grid point coordinates.
x = numpy.linspace(0.0, L, num=nx)
```

We also need to set up our initial conditions. Here, we use the NumPy function `numpy.ones()` defining an array which is `nx`-element long with every value equal to $1$. How useful! We then *change a slice* of that array to the value $u=2$, to get the square wave, and we print out the initial array just to admire it. But which values should we change?  The problem states that we need to change the indices of `u` such that the square wave begins at $x = 0.5$ and ends at $x = 1$.

We can use the [`numpy.where()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html) function to return a list of indices where the vector $x$ meets some conditions.
The function [`numpy.logical_and()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html) computes the truth value of `x >= 0.5` **and** `x <= 1.0`, element-wise.

```python
# Set initial conditions with 1.0 everywhere (for now).
u0 = numpy.ones(nx)

# Get a list of indices where 0.5 <= x <= 1.0.
mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
print(mask)
```

    (array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], dtype=int64),)

With the list of indices, we can now update our initial conditions to get a square-wave shape.

```python
# Set initial condition u = 2.0 where 0.5 <= x <= 1.0.
u0[mask] = 2.0
print(u0)
```

    [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 1. 1. 1.
     1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]

Now let's take a look at those initial conditions we've built with a handy plot.

```python
# Plot the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='--', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 2.5);
```

It does look pretty close to what we expected. But it looks like the sides of the square wave are not perfectly vertical. Is that right? Think for a bit.

Now it's time to write some code for the discrete form of the advection equation using our chosen FTBS finite-difference scheme.

For every element of our array `u`, we need to perform the operation:

$$
u_i^{n+1} = u_i^n - c \frac{\Delta t}{\Delta x}(u_i^n-u_{i-1}^n)
$$

We'll store the result in a new (temporary) array `un`, which will be the solution $u$ for the next time-step.  We will repeat this operation for as many time-steps as we specify and then we can see how far the wave has traveled.

We first initialize the placeholder array `un` to hold the values we calculate for the $n+1$ time step, using once again the NumPy function `ones()`.

Then, we may think we have two iterative operations: one in space and one in time, so we may start by nesting a spatial loop inside the time loop, as shown below. You see that the code for the finite-difference scheme is a direct expression of the discrete equation:

```python
u = u0.copy()
for n in range(1, nt):
    un = u.copy()
    for i in range(1, nx):
        u[i] = un[i] - c * dt / dx * (un[i] - un[i - 1])
```

**Note 1**—We stressed above that our physical problem needs a boundary condition at $x=0$. Here we do not need to impose it at every iteration because our discretization does not change the value of u[0]: it remains equal to one and our boundary condition is therefore satisfied during the whole computation!

Now let's inspect our solution array after advancing in time with a line plot.

```python
# Plot the solution after nt time steps
# along with the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, label='Initial',
            color='C0', linestyle='--', linewidth=2)
pyplot.plot(x, u, label='nt = {}'.format(nt),
            color='C1', linestyle='-', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 2.5);
```

That's funny. Our square wave has definitely moved to the right, but it's no longer in the shape of a top-hat. **What's going on?**

##### Dig deeper

The solution differs from the expected square wave because the discretized equation is an approximation of the continuous differential equation that we want to solve. There are errors: we knew that. But the modified shape of the initial wave is something curious. Maybe it can be improved by making the grid spacing finer. Why don't you try it? Does it help?

## Spatial truncation error

Recall the finite-difference approximation we are using for the spatial derivative:

$$
\begin{equation}
\frac{\partial u}{\partial x}\approx \frac{u(x+\Delta x)-u(x)}{\Delta x}\notag
\end{equation}
$$

We obtain it by using the definition of the derivative at a point, and simply removing the limit, in the assumption that $\Delta x$ is very small. But we already learned with Euler's method that this introduces an error, called the *truncation error*.

Using a Taylor series expansion for the spatial terms now, we see that the backward-difference scheme produces a first-order method, in space.

$$
\begin{equation}
\frac{\partial u}{\partial x}(x_i) = \frac{u(x_i)-u(x_{i-1})}{\Delta x} + \frac{\Delta x}{2} \frac{\partial^2 u}{\partial x^2}(x_i) - \frac{\Delta x^2}{6} \frac{\partial^3 u}{\partial x^3}(x_i)+ \cdots
\notag
\end{equation}
$$

The dominant term that is neglected in the finite-difference approximation is of $\mathcal{O}(\Delta x)$. We also see that the approximation *converges* to the exact derivative as $\Delta x \rightarrow 0$. That's good news!

In summary, the chosen "forward-time/backward space" difference scheme is first-order in both space and time: the truncation errors are $\mathcal{O}(\Delta t, \Delta x)$. We'll come back to this!

## Non-linear advection

Let's move on to the non-linear advection equation, using the same methods as before. The 1-D convection equation is:

$$
\begin{equation}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = 0 \tag{3}
\end{equation}
$$

The only difference with the linear case is that we've replaced the constant wave speed $c$ by the variable speed $u$. The equation is non-linear because now we have a product of the solution and one of its derivatives: the product $u\,\partial u/\partial x$. This changes everything!

We're going to use the same discretization as for linear convection: forward difference in time and backward difference in space. Here is the discretized equation:

$$
\begin{equation}
\frac{u_i^{n+1}-u_i^n}{\Delta t} + u_i^n \frac{u_i^n-u_{i-1}^n}{\Delta x} = 0 \notag
\end{equation}
$$

Solving for the only unknown term, $u_i^{n+1}$, gives an equation that can be used to advance in time:

$$
\begin{equation}
u_i^{n+1} = u_i^n - u_i^n \frac{\Delta t}{\Delta x} (u_i^n - u_{i-1}^n) \tag{4}
\end{equation}
$$

There is very little that needs to change from the code written so far. In fact, we'll even use the same square-wave initial condition. But let's re-initialize the variable `u` with the initial values, and re-enter the numerical parameters here

```python
# Set parameters.
nx = 41  # number of spatial discrete points
L = 2.0  # length of the 1D domain
dx = L / (nx - 1)  # spatial grid size
nt = 10  # number of time steps
dt = 0.02  # time-step size

x = numpy.linspace(0.0, L, num=nx)

# Re-initialization of u0
u0 = numpy.ones(nx)
mask = numpy.where(numpy.logical_and(x >= 0.5, x <= 1.0))
u0[mask] = 2.0
```

 How does it look?

```python
# Plot the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.title('Initial conditions')
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, color='C0', linestyle='--', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 2.5);
```

##### Exercise 2

By changing just one line of code in the solution of the linear advection, we are able to get the non-linear solution: the line that corresponds to the discrete equation now has `un[i]`, replacing the constant velocity `c`. Implement the FTBS-FD approach of the non-linear advection problem eq.(4) in the cell below, run the code ...

```python
# MODIFY THE LINEAR ADVECTION CODE BELOW TO SOLVE THE NONLINEAR ADVECTION PROBLEM!
u = u0.copy()
for n in range(1, nt):
    un = u.copy()
    for i in range(1, nx):
        u[i] = un[i] - c * dt / dx * (un[i] - un[i - 1])
```

... and plot the solution along the initial condition

```python
# Plot the solution after nt time steps
# along with the initial conditions.
pyplot.figure(figsize=(4.0, 4.0))
pyplot.xlabel('x')
pyplot.ylabel('u')
pyplot.grid()
pyplot.plot(x, u0, label='Initial',
            color='C0', linestyle='--', linewidth=2)
pyplot.plot(x, u, label='nt = {}'.format(nt),
            color='C1', linestyle='-', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(0.0, 2.5);
```

Hmm. That's quite interesting: like in the linear case, we see that we have lost the sharp sides of our initial square wave, but there's more. Now, the wave has also lost symmetry! It seems to be lagging on the rear side, while the front of the wave is steepening. Is this another form of numerical error, do you ask? No! It's physics!

##### Dig deeper

Think about the effect of having replaced the constant wave speed $c$ by the variable speed given by the solution $u$. It means that different parts of the wave move at different speeds. Make a sketch of an initial wave and think about where the speed is higher and where it is lower ...

## What we learned:

- How to solve the 1D (non)linear advection problem by the FTBS finite difference method

- While the analytical solution of the linear advection problem should only shift the initial waveform to the right, the FD solution on a coarse spatial grid also changes the shape of the initial waveform

- In case of the 1D non-linear advection problem we lose the symmetry of the initial waveform


# Teorema de Bayes, distribuciones previas y previas conjugadas

> Justo como ilustra la imagen de arriba, dadas dos VA $A$ y $B$, el Teorema de Bayes se puede escribir como:
>
> $$
  P(A | B) = \frac{P(B | A) P(A)}{P(B)}
  $$

> En este notebook recapitularemos qué significa cada uno de esos elementos, estudiaremos porqué es complejo el cálculo exacto de la distribución posterior, y veremos un método analítico para hacer este cálculo exacto.

> **Objetivos:**
> - Explicar porqué el cálculo exacto de la distribución posterior es una tarea intratable en muchas ocasiones.
> - Principio de Maximum Aposteriori (MAP).
> - Comprender el concepto de distribución previa conjugada.

> **Referencias:**
> - Bayesian Methods for Machine Learning course, HSE University, Coursera.
> - Statistical Rethinking, Richard McElreath, 2018.

## 1. Teorema de Bayes

Como ya vimos, el enfoque Bayesiano considera que los parámetros $\theta$ son los que se consideran aleatorios, y los datos $X$ están fijos como evidencia.

En ese sentido, nos interesa modelar la distribución de los parámetros dada la evidencia que estamos observando $P(\theta | X)$, lo cual, por el Teorema de Bayes podemos escribir como:

$$
P(\theta | X) = \frac{P(X | \theta) P(\theta)}{P(X)},
$$

donde:
- $P(\theta | X)$ se conoce como **distribución posterior** (en inglés *posterior*). La posterior nos dice la probabilidad de los parámetros después de haber observado los datos.
- $P(X | \theta)$ se conoce como **función de verosimilitud** (en inglés *likelihood*). La verosimilitud nos dice que tan bien los parámetros explican los datos.
- $P(\theta)$ se conoce como **distribución previa** (en inglés *prior*). En la previa, incluimos todo el conocimiento que podamos tener acerca de los parámetros.
- $P(X)$ se conoce como **distribución de evidencia** (en inglés *evidence*).

#### ¿Qué es la (distribución de) evidencia $P(X)$?

Ejemplos:

1. Imaginemos que estamos trabajando con visión por computadora de imágenes de diferentes artistas (por ejemplo Van Gogh,  DaVinci, Monet). ¿Qué es $P(X)$?

   **Respuesta**

2. Imaginemos que estamos estimando la probabilidad de que un cliente compre en nuestra e-commerce en las siguientes dos semanas (clasificador binario). En este caso:

   $$
   P(\theta | X, y) = \frac{P(y | X, \theta) P(\theta | X)}{P(y | X)},
   $$

   ¿Qué es $P(y | X)$?

   **Respuesta**

Por tanto, modelar la distribución de evidencia es una tarea bastante compleja que en la mayoría de los casos no somos capaces de realizar.

Por tanto, en esta clase, estaremos estudiando metodologías analíticas para evitar el cálculo de la distribucion de evicencia.

## 2. Maximum Aposteriori (MAP)

Recordemos la clase de [mínimos cuadrados desde una perspectiva probabilística](../tema3/3_regresion_lineal.ipynb).

En ciertas ocasiones no nos interesa modelar toda la distribución de los parámetros, sino simplemente encontrar el valor *más probable* de los parámetros dadas las observaciones. Esto da lugar al estimador MAP:

$$
\theta_{MAP} = \arg \max_{\theta} P(\theta | X),
$$

el cual, usando el Teorema de Bayes se puede escribir como:

$$
\theta_{MAP} = \arg \max_{\theta} \frac{P(X | \theta) P(\theta)}{P(X)}.
$$

Sin embargo, como la evidencia $P(X)$ no depende de los parámetros $\theta$:

$$
\theta_{MAP} = \arg \max_{\theta} P(X | \theta) P(\theta).
$$

De esta manera, observamos que para el cálculo del estimador MAP, evitamos completamente el cálculo de la distribución de evidencia. El problema de estimación resultante lo podemos resolver de manera eficiente usando métodos numéricos.

#### Y si siempre nos quedamos con el estimador MAP, ¿Cuál es el problema?

Si solamente nos quedáramos con el estimador MAP, el enfoque Bayesiano pierde el sentido. Recordamos que bajo el enfoque Bayesiano nos interesa modelar la incertidumbre de los parámetros ante las observaciones que hacemos del mundo, y con un estimador MAP obtenemos los valores **fijos** más probables de los parámetros.

Esto tiene varias implicaciones:

1. Habíamos dicho que una de las ventajas del enfoque Bayesiano era que nos permitía hacer aprendizaje on-line, solo haciendo los cálculos relativos a cada paso:

   $$
   P_k(\theta) = P(\theta | x_k) = \frac{P(x_k | \theta) P_{k-1}(\theta)}{P(x_k)}.
   $$

   Sin embargo, no podemos usar la estimación MAP como nueva previa en el próximo paso, debido a que sería una función impulso:

   $$
   P_{k-1}(\theta) = \delta(\theta - \theta_{MAP}) = \left\{\begin{array}{cc}
   1 & \text{si } \theta=\theta_{MAP} \\
   0 & \text{en otro caso}
   \end{array}\right.,
   $$

   y esto no nos aportaría información al siguiente paso:

   $$
   P_k(\theta) = P(\theta | x_k) = \frac{P(x_k | \theta) \delta(\theta - \theta_{MAP})}{P(x_k)} = \delta(\theta - \theta_{MAP}).
   $$

2. Podríamos caer en un caso donde $\theta_{MAP}$ sea un valor atípico, y aunque sea el "más probable", la mayoría de los datos estén concentrados en otra región. Por ejemplo

```python
# Importar matplotlib.pyplot
from matplotlib import pyplot as plt
# Importar scipy.stats.norm
from scipy.stats import norm
# Importar numpy
import numpy as np
```

```python
# Definir VA X, Y
X = norm(loc=0.5, scale=0.1)
Y = norm(loc=0.7, scale=0.01)
```

```python
# Suma ponderada de densidades
x = np.linspace(0, 1, 1000)
fz = 0.8 * X.pdf(x) + 0.2 * Y.pdf(x)
```

```python
# Graficar densidad de Z
plt.plot(x, fz)
```

3. Si solo estimamos un punto, no podríamos estimar regiones de credibildiad (intervalos de confianza).

## 3. Distribuciones conjugadas

De manera que los estimadores MAP son bastante útiles en ciertas aplicaciones, pero si queremos aún más flexibilidad en cuanto a la estimación de la distribución de los parámetros, no nos es útil.

Si queremos estimar esta distribucion dadas las observaciones debemos buscar otros caminos. Uno de ellos son las **distribuciones conjugadas**.

Recapitulando, la distribución posterior la podemos escribir como:

$$
P(\theta | X) = \frac{P(X | \theta) P(\theta)}{P(X)},
$$

donde:

- La función de verosimilitud $P(X | \theta)$ la fija el modelo.
- La distribución de evidencia $P(X)$ la fijan los datos.

Peeeero, la distribución previa es algo que podemos elegir de acuerdo a **nuestra experiencia, conocimiento previo o simplemente para acomodar los cálculos**.

Es decir, podríamos elegir la distribución previa $P(\theta)$ con el único fin de evitar el cálculo de la la distribución de evidencia $P(X)$.

> *Definición.* (**Previa conjugada**) Decimos que la distribución previa $P(\theta)$ es conjugada a la función de verosimilitud $P(X | \theta)$, si la distribución posterior pertenece a la misma familia de distribuciones que la distribución previa.

¿Cómo es esto posible?

Supongamos que tenemos dos dos densidades normales distintas.

```python
# Definir VA X, Y
X = norm(loc=0.5, scale=0.1)
Y = norm(loc=0.7, scale=0.15)
```

```python
# Graficar densidades
x = np.linspace(0, 1.4, 1000)
plt.plot(x, X.pdf(x), label="Densidad 1")
plt.plot(x, Y.pdf(x), label="Densidad 2")
plt.legend()
```

Si multiplicamos estas dos densidades, y renormalizamos para que la integral de $1$, obtenemos de nuevo una distribución normal:

```python
from scipy.integrate import quad
```

```python
def pdf_z(x):
    return X.pdf(x) * Y.pdf(x)
```

```python
val, err = quad(pdf_z, -10, 10)
val, err
```

    (1.1959423430718399, 1.4430475689561403e-11)

```python
# Graficar densidad de Z
x = np.linspace(0, 1.4, 1000)
plt.plot(x, X.pdf(x), label="Densidad 1")
plt.plot(x, Y.pdf(x), label="Densidad 2")
plt.plot(x, X.pdf(x) * Y.pdf(x) / val, label="Densidad del producto")
plt.legend()
```

Matemáticamente, sean $X_1 \sim \mathcal{N}(\mu_1, \sigma_1^2)$ y $X_2 \sim \mathcal{N}(\mu_2, \sigma_2^2)$. Entonces:

\begin{align}
\mathcal{N}(x | \mu_1, \sigma_1^2) \mathcal{N}(x | \mu_2, \sigma_2^2) & \propto \exp\left\{-\frac{(x - \mu_1)^2}{2 \sigma_1^2}\right\} \exp\left\{-\frac{(x - \mu_2)^2}{2 \sigma_2^2}\right\} \\
& = \exp\left\{-\frac{(x - \mu_1)^2}{2 \sigma_1^2} -\frac{(x - \mu_2)^2}{2 \sigma_2^2} \right\} \\
& = \exp\left\{-\frac{\sigma_2^2(x - \mu_1)^2 + \sigma_1^2(x - \mu_2)^2}{2 \sigma_1^2 \sigma_2^2}\right\} \\
& = \exp\left\{-\frac{(x - \bar{\mu})^2}{2 \bar{\sigma}^2} + const\right\} \\
& \propto \exp\left\{-\frac{(x - \bar{\mu})^2}{2 \bar{\sigma}^2}\right\}
\end{align}

De modo que el producto vuelve a ser una densidad normal. Ver [el siguiente enlace](https://www.johndcook.com/blog/2012/10/29/product-of-normal-pdfs/).

Con base en lo anterior, tomando la posterior:

$$
P(\theta | X) = \frac{P(X | \theta) P(\theta)}{P(X)},
$$

Si la función de verosimilitud $P(X | \theta) = \mathcal{N}(X | \theta, \sigma^2)$, y la previa se selecciona como:

$$
P(\theta) = \mathcal{N}(\theta | m, s^2),
$$

obtenemos:

$$
P(\theta | X) = \mathcal{N}(\theta | a, b^2).
$$

Es decir, para una distribución normal sobre los datos con media $\theta$, la previa conjugada es una distribución normal sobre $\theta$.

#### ¿Y qué pasa con la evidencia?

Recordemos que la distribución posterior es una distribución **sobre los parámetros**. Debido a que la distribución de evidencia no depende de los parámetros, la podemos pensar como una constante de normalización para que la distribución resultante integre (sume) uno.

Esto es una buena noticia, porque operacionalmente, no nos debemos preocupar mucho por las constantes que hacen que la distribución integre a uno. Debemos tener especial cuidado con que **la densidad resultante tenga la misma forma funcional que la previa**.

**Ejercicio:** supongamos que la función de verosimilitud y la distribución previa son:

$$
p(x | \theta) = \mathcal{N}(x |\theta, 1) \qquad p(\theta) = \mathcal{N}(\theta |0, 1)
$$

Encontrar completamente la densidad de probabilidad posterior

$$
p(\theta | x) = \frac{p(x | \theta) p(\theta)}{p(x)}.
$$

En clase ...

Veremos algunos ejemplos adicionales, pero antes debemos aprender un par de distribuciones más.

## 4. Algunas distribuciones de probabilidad importantes

### 4.1. Distribución Gamma

Es una distribución continua, cuyo soporte son los reales positivos ($x\in\mathbb{R}_+$), de dos parámetros $a, b >0$, cuya función de densidad de probabilidad es:

$$
\Gamma(x | a, b) = \frac{b^a}{\Gamma(a)} x^{a - 1} \exp\{-b x\}
$$

```python
# Importar scipy.stats.gamma
from scipy.stats import gamma
```

```python
gamma?
```

```python
# Graficar para diferentes valores de a y b
x = np.linspace(0, 5, 100)
plt.plot(x, gamma.pdf(x, a=1, scale=1 / 2), label="$a=1, b=2$")
plt.plot(x, gamma.pdf(x, a=2, scale=1 / 2), label="$a=2, b=2$")
plt.plot(x, gamma.pdf(x, a=3, scale=1 / 2), label="$a=3, b=2$")
plt.plot(x, gamma.pdf(x, a=0.5, scale=1 / 1), label="$a=0.5, b=1$")
plt.legend()
```

Si $X \sim \Gamma(a, b)$, algunos estadísticos importantes son:

$$
E[X] = \frac{a}{b}
$$

$$
Mode[X] = \frac{a - 1}{b}, \quad \text{para } a > 1
$$

$$
Var[X] = \frac{a}{b^2}
$$

La distribución Gamma es importante porque es la previa conjugada para la **precisión** en una verosimilitud normal.

¿Qué es la precisión? Es el recíproco de la varianza. Es decir, la densidad

$$
\mathcal{N}(x | \mu, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left\{-\frac{(x - \mu)^2}{2 \sigma^2}\right\},
$$

en términos de la precisión $\gamma = \frac{1}{\sigma^2}$ es

$$
\mathcal{N}\left(x | \mu, \frac{1}{\gamma}\right) = \frac{\sqrt{\gamma}}{\sqrt{2 \pi}} \exp\left\{-\gamma\frac{(x - \mu)^2}{2}\right\}.
$$

¿Cuál es la previa conjugada respecto a la precisión? La respuesta es la distribución Gamma. Veamos

$$
p(\gamma | x) = \frac{p(x | \gamma) p(\gamma)}{p(x)} \propto p(x | \gamma) p(\gamma)
$$

Si la verosimilitud es normal con parámetro la precisión: $p(x | \gamma) = \mathcal{N}\left(x | \mu, \frac{1}{\gamma}\right) \propto \gamma^{1/2} \exp\left\{-\gamma s\right\}$, con $s=\frac{(x - \mu)^2}{2}$, y la previa es una distribución gamma: $p(\gamma) = \Gamma(\gamma|a,b) \propto \gamma^{a-1} \exp\{-b \gamma\}$, entonces

$$
p(\gamma | x) \propto \gamma^{1/2} \exp\left\{-\gamma s\right\} \gamma^{a-1} \exp\{-b \gamma\} = \gamma^{a - 1/2} \exp\{-(s+b) \gamma\},
$$

esto es:

$$
p(\gamma | x) = \Gamma(\gamma|a+1/2,b+s)
$$

### 4.2. Distribución Beta

Es una distribución continua, cuyo soporte es el intervalo $[0, 1]$, de dos parámetros $a, b>0$, cuya función de densidad de probabilidad es:

$$
B(x | a, b) = \frac{\Gamma(a + b)}{\Gamma(a)\Gamma(b)} x^{a - 1} (1 - x)^{b - 1}
$$

```python
# Importar scipy.stats.gamma
from scipy.stats import beta
```

```python
# Graficar para diferentes valores de a y b
x = np.linspace(0, 1, 100)
plt.plot(x, beta.pdf(x, a=0.5, b=0.5), label="$a=0.5, b=0.5$")
plt.plot(x, beta.pdf(x, a=1, b=1), label="$a=1, b=1$")
plt.plot(x, beta.pdf(x, a=2, b=2), label="$a=2, b=2$")
plt.plot(x, beta.pdf(x, a=2, b=5), label="$a=2, b=5$")
plt.plot(x, beta.pdf(x, a=10, b=5), label="$a=10, b=5$")
plt.legend(loc="upper left", bbox_to_anchor=(1.05, 1))
```

Si $X \sim Beta(a, b)$, algunos estadísticos importantes son:

$$
E[X] = \frac{a}{a + b}
$$

$$
Mode[X] = \frac{a - 1}{a + b - 2}, \quad \text{para } a > 1, \text{ y } b > 1
$$

$$
Var[X] = \frac{ab}{(a + b)^2(a + b - 1)}
$$

La distribución Beta es importante porque es la previa conjugada para la verosimilitud de Bernoulli.

___
#### Distribución de Bernoulli

Supongamos que tenemos el experimento trillado de tirar una moneda al aire, y sabemos que cae cara con probabilidad $\theta$ y sello con probabilidad $1 - \theta$.

Asignamos la VA $X$ de la siguiente manera:

$$
X = \left\{\begin{array}{cc}
1 & \text{si cara} \\
0 & \text{si sello} \\
\end{array}\right.
$$

Una suposición plausible es que cada tiro de la moneda es independiente.

Supongamos que hemos tirado $5$ veces la moneda obteniendo los siguientes datos:

$$
D = \{ 1, 1, 0, 1, 0\}.
$$

En este sentido, la verosimilitud de los datos es:

$$
P(D | \theta) = \prod_{i=1}^{5} P(X_i | \theta) = \theta^{3}(1 - \theta)^{2}.
$$

En general, si la moneda cae $N_1$ veces cara y $N_0$ veces sello, la verosimilitud sería:

$$
P(D | \theta) = \theta^{N_1}(1 - \theta)^{N_0},
$$

donde $N=N_0+N_1$ es el número total de tiros.
___

Ya que tenemos todo claro, veamos que la distribución Beta es la previa conjugada para la verosimilitud de Bernoulli:

$$
p(\theta | X) = \frac{p(X | \theta) p(\theta)}{p(X)} \propto p(X | \theta) p(\theta)
$$

Si la verosimilitud es de Bernoulli con probabilidad $\theta$: $p(X | \theta) = \theta^{N_1}(1 - \theta)^{N_0}$, y la previa es una distribución Beta: $p(\theta) = B(\theta | a, b) \propto \theta^{a - 1} (1 - \theta)^{b - 1}$, entonces

$$
p(\theta | X) \propto \theta^{N_1}(1 - \theta)^{N_0} \theta^{a - 1} (1 - \theta)^{b - 1} = \theta^{a + N_1 - 1} (1 - \theta)^{b + N_0 -1},
$$

esto es:

$$
p(\theta | X) = B(\theta | a + N_1, b + N_0).
$$

**Ilustración:**

Definimos la variable aleatoria (discreta) binomial:

$$
X = \left\{ \begin{array}{ccc} 1 & \text{si} & \text{la moneda cae cara} \\
                               0 & \text{si} & \text{la moneda cae sello} \end{array} \right.
$$

con función de masa de probabilidad dada por:

- $P(X=1) = P(X=1 | \theta) = \theta$;
- $P(X=0) = P(X=0 | \theta) = 1 - P(X=1) = 1 - \theta$.

De esta manera, la función de verosimilitud satisface:

$$
P(\mathcal{D} | \theta) = \mathcal{L}(\theta) = \theta^{N_1} (1 - \theta)^{N_0}.
$$

Acabamos de demostrar que la distribución Beta sobre el parámetro $\theta$ es la previa conjugada:

$$
P(\theta) = Beta(\theta | a, b) \propto \theta^{a-1} (1 - \theta)^{b - 1}
$$

```python
from scipy.stats import bernoulli
```

```python
# VA real
theta_true = 0.7
X = bernoulli(p=theta_true)
```

```python
# Datos
data = X.rvs(size=100)
data
```

    array([1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,
           1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,
           0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1,
           1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1,
           1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0])

```python
# Verosimilitud
N1 = (data == 1).sum()
N0 = (data == 0).sum()
theta = np.linspace(0, 1, 101)
likelihood = theta**N1 * (1 - theta)**N0
```

```python
plt.plot(theta, likelihood)
plt.legend(loc="upper left", bbox_to_anchor=(1.05, 1))
```

```python
# Previa
a0 = 5
b0 = 5
prior = beta(a=a0, b=b0)
```

```python
plt.plot(theta, prior.pdf(theta))
```

```python
# Posterior
a1 = a0+N1
b1 = b0+N0
posterior = beta(a=a0+N1, b=b0+N0)
```

```python
plt.plot(theta, posterior.pdf(theta))
```

```python
posterior.mean()
```

    0.7

**Recibimos datos de otros 100 tiros de la moneda**

```python
from copy import copy
```

```python
prior = copy(posterior)
```

```python
# Datos nuevos
data_new = X.rvs(size=100)
data_new
```

    array([1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1,
           0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1,
           1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0,
           1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1,
           1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0])

```python
N1 = (data_new == 1).sum()
N0 = (data_new == 0).sum()
```

```python
# Posterior con los nuevos datos
a2 = a1 + N1
b2 = b1 + N0
posterior = beta(a=a2, b=b2)
```

```python
plt.plot(theta, posterior.pdf(theta))
plt.plot(theta, prior.pdf(theta))
```

```python
prior.a
```

    0.0

## Conclusión

Para evitar el cálculo de la distribución de evidencia vimos dos posibilidades:

1. Estimador MAP:
   - Obtenemos el parámetro más probable dada la evidencia.
   - No se modela la distribución; obtenemos un valor fijo.
   - No se puede usar para aprendizaje on-line.
   - Podríamos estimar un valor atípico.

2. Previas conjugadas:
   - Seleccionamos la distribución previa para evitar el cálculo de la evidencia.
   - Modelamos la distribución posterior.
   - Se puede usar para aprendizaje on-line.
   - Al selecciónar la previa conjugada con el fin de evitar el cálculo de la evidencia, es posible que esto vaya en contra de la intuición o conocimiento previo que tengamos acerca del problema. No siempre la previa conjugada es adecuada.

Created with Jupyter by Esteban Jiménez Rodríguez.


```python
from IPython.core.display import HTML
from IPython.display import Image
import ipywidgets as widgets
import IPython.display as display

import sympy as sp
import numpy as np
from utils import symplot, symdisp, round_expr

HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

# *Circuitos Elétricos I - Semana 8*

### Circuitos RL e RC de primeira ordem

Os quatro tipos possíveis de circuitos de primeira ordem estão ilustrados na figura abaixo.

Lembrando que um circuito de primeira ordem qualquer, com vários resistores e fontes, por exemplo, pode ser reduzido a um dos quatro circuitos acima fazendo $R=R_{th}$, $v_s=v_{th}$ e $i_s=i_{N}$. Logo, desde que o circuito contenha apenas um elemento indutor ou capacitor, a análise de um circuito de primeira ordem deve ser feita primeiramente determinando-se o circuito equivalente de Thévenin ou de Norton conectado aos terminais do elemento em questão.

### EDO linear homogênea

Seja para a corrente passando pelo indutor (circuito RL) ou para a tensão aplicada aos terminais do capacitor (circuito RC), a aplicação das leis de Kirchhoff e da **convenção passiva** nos circuitos RL e RC sempre levarão a uma EDO homogênea separável de primeira ordem do tipo

$$ \begin{equation}\label{eq1} \large \frac{dx(t)}{dt} = -\frac{1}{\tau}x(t), \end{equation}$$

com $\tau$ sendo a constante de tempo do circuito. A solução de (\ref{eq1}) pode ser obtida via integração fazendo

$$ \begin{equation}  \large \int_{x\left(t_{0}^+\right)}^{x(t)} \frac{d u}{u}=-\frac{1}{\tau} \int_{t_{0}^+}^{t} d v \end{equation}$$.

Logo, a solução da EDO homogênea será dada por  $$ \begin{equation} \large x(t) = x(t_0^+)e^{-\frac{(t-t_0^+)}{\tau}}. \end{equation} $$

### Resposta natural

A resposta natural de circuitos RL/RC corresponderá a solução da EDO homogênea, ou seja,

$$ \begin{equation} \large i_L(t) = i_L(t_0^+)e^{-\frac{(t-t_0^+)}{\tau}}, \end{equation}$$

com $\tau = L/R$ para o circuito RL e

$$\begin{equation}  \large v_C(t) = v_C(t_0^+)e^{-\frac{(t-t_0^+)}{\tau}}, \end{equation} $$

com $\tau = RC$ para o circuito RC.

### Resposta ao degrau

A resposta ao degrau de circuitos RL/RC corresponderá a solução da EDO homogênea adicionada da solução particular (ou solução de regime estacionário). Logo,

$$\large i_L(t) = i_L(\infty) + A_0e^{-\frac{(t-t_0^+)}{\tau}}, $$

com $\tau = L/R$ para o circuito RL e

$$\large v_C(t) = v_C(\infty) + A_0e^{-\frac{(t-t_0^+)}{\tau}}, $$

com $\tau = RC$ para o circuito RC.

A constante $A_o$ da resposta ao degrau pode ser determinada utilizando as condições iniciais de corrente no indutor ou tensão no capacitor.

Desse modo, para o circuito RL

$$\large{\begin{align} i_L(t_0^+) &= i_L(\infty) + A_0e^{-\frac{(t_0^+-t_0^+)}{\tau}}\nonumber\\ &= i_L(\infty) + A_0 \nonumber\\ \Rightarrow A_0 &= i_L(t_0^+)- i_L(\infty)\nonumber
\end{align} }$$

e para o circuito RC

$$\large{\begin{align} v_C(t_0^+) &= v_C(\infty) + A_0e^{-\frac{(t_0^+-t_0^+)}{\tau}}\nonumber\\ &= v_C(\infty) + A_0 \nonumber\\ \Rightarrow A_0 &= v_C(t_0^+)- v_C(\infty)\nonumber
\end{align} }$$

### Resposta geral

A resposta geral de circuitos RL/RC corresponderá às expressões,

$$ \begin{equation} \large i_L(t) = i_L(\infty) + \left[i_L(t_0^+)- i_L(\infty)\right]e^{-\frac{(t-t_0^+)}{\tau}},\end{equation} $$

com $\tau = L/R$ para o circuito RL e

$$ \begin{equation} \large v_C(t) = v_C(\infty) + \left[v_C(t_0^+)- v_C(\infty)\right]e^{-\frac{(t-t_0^+)}{\tau}},\end{equation} $$

com $\tau = RC$ para o circuito RC.

Finalmente, para qualquer circuito de primeira ordem,

$$\begin{equation}  \large x(t) = x(\infty) + \left[x(t_0^+)-x(\infty)\right]e^{-\frac{(t-t_0^+)}{\tau}},\end{equation} $$

com $\tau$ sendo a constante de tempo associada a este circuito.

### Problema 1

Para circuito da figura abaixo, a chave encontra-se conectada ao terminal $a$ há muito tempo. Em $t=0s$, posição da chave muda do ponto $a$ para o ponto $b$. Em $t=20 ms$, a chave é desconectada do ponto $b$, permanecendo aberta. Determine:

a. A corrente no indutor para $0^+s\leq t \leq 20^-ms$.\
b. A tensão nos terminais indutor em $t=0^-s$ e $t=0^+s$.\
c. A tensão nos terminais do resistor de 40 $\Omega$ em $t=0^-s$ e $t=0^+s$.\
d. A corrente no indutor para $t\geq20^+ms$.\
e. A tensão nos terminais do indutor em $t=20^+ms$.

Simulação do circuito disponível no link: https://tinyurl.com/yfs69qqu

#### Visualização das curvas $i_L(t)$, $v_L(t)$ e $p_L(t)$

```python
# parâmetros do circuito
L   = 1
vth = -80e-3
Rth = 18

# informações obtidas pela análise do circuito
iL_inf = vth/Rth # iL(infinito)
iL_t0  = 2.9e-3  # iL(t0+)

τ = L/Rth  # constante de tempo do circuito RL
t0  = 0    # instante inicial

t = sp.symbols('t', real=True) # define a variável tempo

iL = sp.Piecewise( (iL_t0, t<0),                                              # iL(t) para t<t0-
                   (iL_inf + (iL_t0 - iL_inf)*sp.exp(-(t-t0)/τ), t>=0) )*1000 # iL(t) para t>t0+

symdisp('i_L(t) =', round_expr(iL.simplify(), 3), 'mA')

intervalo  = np.linspace(t0-2*τ, t0+8*τ, 400)
symplot(t, iL, intervalo, funLabel = '$i_L(t)$ [mA]')
```

```python
vL = L*sp.diff(iL,t)
vL = vL.simplify()

symdisp('v_L(t) = ', vL, ' mV')

symplot(t, vL, intervalo, funLabel = '$v_L(t)$ [mV]')
```

```python
pL = vL*iL
pL = pL.simplify()

symdisp('p_L(t) = ', round_expr(pL,3), ' μW')

symplot(t, pL, intervalo, funLabel = '$p_L(t)$ [μW]')
```

```python
import ipywidgets as widgets
import IPython.display as display

img1 = open('./figures/J11C2a.gif', 'rb').read()
img2 = open('./figures/J11C2b.gif', 'rb').read()

wi1 = widgets.Image(value=img1, format='gif', width=600, height=400)
wi2 = widgets.Image(value=img2, format='gif', width=600, height=400)

sidebyside = widgets.HBox([wi1, wi2])
display.display(sidebyside)
```

### Problema 2

No circuito da figura abaixo, antes de conectado ao circuito, o capacitor possui uma tensão inicial de $v_C=10~V$. Em $t=0s$, o capacitor é conectado ao circuito. Determine:

a. O circuito equivalente de Thévenin do ponto de vista dos terminais do capacitor.\
b. A tensão no capacitor $v_C(t)$ para $t\geq 0^+s$.\
c. A tensão $v_x(t)$ para $t\geq 0^+s$.

Simulação do circuito disponível no link: https://tinyurl.com/yzhty8w3

Jupyter notebooks/Circuitos Eletricos I - Semana 8.1.ipynb


## Data-driven Design and Analyses of Structures and Materials (3dasm)

## Lecture 9

### Miguel A. Bessa | M.A.Bessa@tudelft.nl  | Associate Professor

**What:** A lecture of the "3dasm" course

**Where:** This notebook comes from this [repository](https://github.com/bessagroup/3dasm_course)

**Reference for entire course:** Murphy, Kevin P. *Probabilistic machine learning: an introduction*. MIT press, 2022. Available online [here](https://probml.github.io/pml-book/book1.html)

**How:** We try to follow Murphy's book closely, but the sequence of Chapters and Sections is different. The intention is to use notebooks as an introduction to the topic and Murphy's book as a resource.
* If working offline: Go through this notebook and read the book.
* If attending class in person: listen to me (!) but also go through the notebook in your laptop at the same time. Read the book.
* If attending lectures remotely: listen to me (!) via Zoom and (ideally) use two screens where you have the notebook open in 1 screen and you see the lectures on the other. Read the book.

**Optional reference (the "bible" by the "bishop"... pun intended 😆) :** Bishop, Christopher M. *Pattern recognition and machine learning*. Springer Verlag, 2006.

**References/resources to create this notebook:**
* [Car figure](https://korkortonline.se/en/theory/reaction-braking-stopping/)

Apologies in advance if I missed some reference used in this notebook. Please contact me if that is the case, and I will gladly include it here.

## **OPTION 1**. Run this notebook **locally in your computer**:
1. Confirm that you have the 3dasm conda environment (see Lecture 1).

2. Go to the 3dasm_course folder in your computer and pull the last updates of the [repository](https://github.com/bessagroup/3dasm_course):
```
git pull
```
3. Open command window and load jupyter notebook (it will open in your internet browser):
```
conda activate 3dasm
jupyter notebook
```
4. Open notebook of this Lecture.

## **OPTION 2**. Use **Google's Colab** (no installation required, but times out if idle):

1. go to https://colab.research.google.com
2. login
3. File > Open notebook
4. click on Github (no need to login or authorize anything)
5. paste the git link: https://github.com/bessagroup/3dasm_course
6. click search and then click on the notebook for this Lecture.

```python
# Basic plotting tools needed in Python.

import matplotlib.pyplot as plt # import plotting tools to create figures
import numpy as np # import numpy to handle a lot of things!
from IPython.display import display, Math # to print with Latex math

%config InlineBackend.figure_format = "retina" # render higher resolution images in the notebook
plt.style.use("seaborn") # style for plotting that comes from seaborn
plt.rcParams["figure.figsize"] = (8,4) # rescale figure size appropriately for slides
```

## Outline for today

* Linear models for regression
    - A practical session on how to train linear regression models

**Reading material**: This notebook + Chapter 11

## Today's lecture is going to be more practical

Since we covered the fundamentals of Bayesian and non-Bayesian machine learning...

* Today we will focus on how to train **linear regression models** using [scikit-learn](https://scikit-learn.org)

* In a later lecture we will derive the models that we are going to cover today.

As we learned in Lecture 2, let's load the pandas dataframe that is in the "docs" folder of Lecture 9:

```python
import pandas as pd
# read csv data provided by someone else (this time I also specify that the first column provides the indices)
car_prob_df = pd.read_csv("docs/data_for_car_prob.csv", index_col=0)
print(car_prob_df)
```

                x           y
    0    9.516939   29.749036
    1   72.398757  642.132203
    2   17.950326   36.648484
    3    9.440853   18.604106
    4   78.791008  769.656168
    5   16.961121   57.971010
    6   65.410368  559.093313
    7   58.671099  463.686613
    8   21.550603   92.242676
    9   36.866913  197.688573
    10  15.728748   56.885233
    11  58.511494  388.753795
    12  57.419190  399.807488
    13  38.459157  213.181519
    14   8.841742   20.387384
    15  60.733051  516.341724
    16  49.256663  307.931956
    17  35.895121  181.123049
    18  79.195652  750.178284
    19  69.156669  553.153541
    20  77.634896  746.031880
    21   9.254011   20.810698
    22  15.451468   39.872527
    23  14.438247   42.118771
    24  13.410999   44.775122
    25  53.747057  375.013937
    26  10.283719   19.438868
    27  82.005477  742.336845
    28  81.805562  706.620282
    29  51.837742  345.212876
    30  20.283785   65.303165
    31  28.359647  155.185137
    32  74.993715  676.628982
    33  21.827564   81.150935
    34  70.519111  700.520033
    35  74.208532  622.453560
    36  14.518958   40.927570
    37  13.357644   39.770922
    38  75.346253  707.973754
    39  44.923956  251.300805
    40  26.801159  124.098654
    41  29.906265  118.100900
    42  40.226356  215.082100
    43  66.282662  537.845048
    44  47.342777  308.558833
    45   3.087674    5.947997
    46  21.254611  101.295276
    47  46.939484  345.778352
    48  38.875692  219.095582
    49  76.705452  742.720134

As before, we can separate the data into inputs (features) $x$ and outputs $y$ (targets)

```python
Data_x = car_prob_df['x'].values # select the input VALUES from your dataframe into Data_x
Data_y = car_prob_df['y'].values # select the output VALUES from your dataframe inta Data_y
print("Data_x is:\n",Data_x)
print("\nData_y is:\n",Data_y)
```

    Data_x is:
     [ 9.51693942 72.39875748 17.95032583  9.44085299 78.79100778 16.96112056
     65.4103675  58.67109927 21.55060313 36.86691294 15.72874781 58.51149357
     57.41918959 38.45915667  8.84174221 60.73305107 49.25666345 35.89512052
     79.19565172 69.15666925 77.63489641  9.25401128 15.45146824 14.43824684
     13.41099874 53.74705712 10.28371886 82.00547705 81.80556249 51.8377421
     20.28378484 28.35964692 74.99371524 21.82756352 70.51911096 74.20853195
     14.51895792 13.35764354 75.34625316 44.92395642 26.80115926 29.90626522
     40.22635624 66.28266205 47.34277718  3.08767411 21.25461134 46.93948443
     38.87569199 76.70545196]

    Data_y is:
     [ 29.74903647 642.13220315  36.64848446  18.60410602 769.65616843
      57.97101034 559.09331318 463.68661322  92.24267632 197.68857288
      56.88523327 388.75379474 399.80748803 213.18151905  20.38738432
     516.34172363 307.93195589 181.12304936 750.17828361 553.15354059
     746.03187971  20.81069833  39.87252654  42.11877078  44.77512244
     375.01393668  19.43886782 742.33684483 706.62028237 345.21287569
      65.30316533 155.18513747 676.62898211  81.15093549 700.52003305
     622.45356019  40.92757044  39.77092163 707.97375405 251.30080489
     124.09865438 118.10089977 215.08209978 537.84504756 308.55883254
       5.94799685 101.29527607 345.77835213 219.09558165 742.72013356]

And we can plot the data:

```python
fig_car_data, ax_car_data = plt.subplots() # create a plot
ax_car_data.plot(Data_x, Data_y, 'b.')
ax_car_data.set_xlabel("x (m/s)", fontsize=20) # create x-axis label with font size 20
ax_car_data.set_ylabel("y (m)", fontsize=20) # create y-axis label with font size 20
ax_car_data.set_title("Car stopping distance problem", fontsize=20); # create title with font size 20
```

## Supervised learning: regression models

As we have been discussing, when we do regression via supervised learning we want to:

* create a machine learning model
* train it on known data (known inputs $x$ and outputs $y$)
* predict for new (unseen) data points, i.e. predict $y^*$ for a new value of $x$.

Today we will talk about the simplest models: **linear regression**.

## Linear regression models

Linear regression models encompass a class of machine learning methods that is larger than you might think...

As we will see, despite being called "linear" these models can do more than fitting a simple "line" to our data.

For now, let's consider 1d datasets, i.e. where we have one input $x$ and one output $y$.

### Simplest 1d linear regression model: fitting a line to your data

1. Observation distribution:

Usually, assumed as a Gaussian distribution,

$$
p(y|x, \mathbf{z}) = \mathcal{N}(y| \mu_{y|z} = \mathbf{w}^T \boldsymbol{\phi}(x), \sigma_{y|z}^2 = \sigma^2)
$$

where $\mathbf{z} = (\mathbf{w}, \sigma)$ are all the hidden rv's of the model, i.e. the model parameters.
* the vector $\mathbf{w} = [w_0, w_1]^T$ includes the **bias** term $w_0$ and the **weight** $w_1$.
* the vector $\boldsymbol{\phi}(x) = [1, x]^T$ includes the **basis functions**.

2. A chosen Prior distribution on each hidden rv of $\mathbf{z}$:

Usually, the prior on $w_0$ and $\sigma$ is the Uniform distribution.

However, the prior on the weight $w_1$ is often chosen as something else (but it can also be Uniform).

Does this model remind you of something we did?

* Car stopping distance problem when we knew one of the rv's and fixed $x$!

Note that,

$$
\begin{align}
p(y|x, \mathbf{z}) &= \mathcal{N}(y| \mathbf{w}^T \boldsymbol{\phi}(x), \sigma^2) \\
&= \mathcal{N}(y| w_0 + w_1 x, \sigma^2) \\
\end{align}
$$

where we previously called $w_0 \equiv b$, $w_1 \equiv z$ and $\sigma^2 \equiv \sigma_{y|z}^2$.

Therefore, the only difference is that we now start to consider more than one rv, so we group them into vector $\mathbf{z} = (\mathbf{w}, \sigma)$.

About notation:

* We can also write $\mathbf{z}^T = [\mathbf{w}^T, \sigma]$.

### 1d linear regression models with different basis functions: fitting a polynomial to your data

However, we know that usually a straight line does not provide a good fit to most data sets.

A very important realization:

* The basis functions vector $\boldsymbol{\phi}(x)$ does not need to be a *linear transformation*.

* It could be a polynomial or any other **nonlinear** transformation of the feature (input) $x$.

As long as the parameters of the basis functions vector $\boldsymbol{\phi}(x)$ are **fixed**, the model remains **linear in the parameters**, even if is not linear in the input (feature). That's why we still call this a **linear model**.

Here's how our **linear** regression model looks like for a polynomial basis functions vector $\boldsymbol{\phi}(x)$.

1. Observation distribution:

$$
p(y|x, \mathbf{z}) = \mathcal{N}(y| \mu_{y|z} = \mathbf{w}^T \boldsymbol{\phi}(x), \sigma_{y|z}^2 = \sigma^2)
$$

where $\mathbf{z} = (\mathbf{w}, \sigma)$ are all the hidden rv's of the model, i.e. the model parameters.
* the vector $\mathbf{w} = [w_0, w_1, w_2 ..., w_d]^T$ includes the **bias** term $w_0$ and the remaining **weights** $w_i$ with $i=1,..., d$.
* the vector $\boldsymbol{\phi}(x) = [1, x, x^2, ..., x^d]^T$ includes the **basis functions**, which now correspond to a polynomial of degree $d$.

2. A chosen Prior distribution for each hidden rv of $\mathbf{z}$, as mentioned previously.

## Linear regression models from a Bayesian perspective

The choice of likelihood and prior determines what is the linear regression model that you are choosing!

| Likelihood | Prior (on the weights)    | Posterior      | Name of the model | Book section  |
|---        |---         |---             |---              |---            |
| Gaussian  | Uniform    | Point estimate | Least Squares regression  | 11.2.2  |
| Gaussian  | Gaussian    | Point estimate | Ridge regression   | 11.3  |
| Gaussian  | Laplace    | Point estimate | Lasso regression  | 11.4  |
| Student-$t$  | Uniform    | Point estimate | Robust regression   | 11.6.1  |
| Laplace  | Uniform    | Point estimate | Robust regression   | 11.6.2  |
| Gaussian  | Gaussian    | Gaussian | Bayesian linear regression   | 11.7 |

We will derive some of these models in a later class. Today, we focus on the practical aspects!

## Training (fitting) a linear model with Scikit-learn

Let's see how to use [scikit-learn](https://scikit-learn.org) to train linear regression models.

* [scikit-learn](https://scikit-learn.org) is a well-documented and user-friendly library that is great for introducing machine learning.

* You should really read the documentation.
    - It includes many useful examples.
    - It provides a short introduction to common machine learning algorithms.

## Example 1: training a linear model for the car stopping distance problem

Let's start by importing from scikit-learn the simplest linear regression model:

* Least Squares Regression.

And let's consider the simplest basis function:

* A line (polynomial of degree 1).

```python
from sklearn.linear_model import LinearRegression # For Least Squares Regression
from sklearn.preprocessing import PolynomialFeatures # For Polynomial basis functions
from sklearn.pipeline import make_pipeline # to link different objects
```

Now let's define the model (Least Squares Regression + polynomial of degree 1)

```python
# We start by defining the model (polynomial basis + Least Squares Regression)
degree = 1 # degree of polynomial we want to fit
poly_model = make_pipeline(PolynomialFeatures(degree),LinearRegression())
```

Then we train the model for our data (input Data_x and output Data_y that were loaded with pandas)

```python
# Uncomment line below (this is just for students to understand: don't panic when encountering an error!)
#poly_model.fit(Data_x,Data_y) # but it gives an ERROR!
```

This gives an error! Fortunately, scikit-learn tells us what happened...

Scikit-learn expects the inputs to be formatted as a 2D array (matrix), instead of a 1D array (vector).

This happens because usually we fit machine learning models for multidimensional inputs.

```python
# Reshape the input vector into a 2D array:
Data_X = np.reshape(Data_x, (-1, 1)) # we use capital letters for matrices and lower case for vectors
```

```python
print(Data_X)
```

    [[ 9.51693942]
     [72.39875748]
     [17.95032583]
     [ 9.44085299]
     [78.79100778]
     [16.96112056]
     [65.4103675 ]
     [58.67109927]
     [21.55060313]
     [36.86691294]
     [15.72874781]
     [58.51149357]
     [57.41918959]
     [38.45915667]
     [ 8.84174221]
     [60.73305107]
     [49.25666345]
     [35.89512052]
     [79.19565172]
     [69.15666925]
     [77.63489641]
     [ 9.25401128]
     [15.45146824]
     [14.43824684]
     [13.41099874]
     [53.74705712]
     [10.28371886]
     [82.00547705]
     [81.80556249]
     [51.8377421 ]
     [20.28378484]
     [28.35964692]
     [74.99371524]
     [21.82756352]
     [70.51911096]
     [74.20853195]
     [14.51895792]
     [13.35764354]
     [75.34625316]
     [44.92395642]
     [26.80115926]
     [29.90626522]
     [40.22635624]
     [66.28266205]
     [47.34277718]
     [ 3.08767411]
     [21.25461134]
     [46.93948443]
     [38.87569199]
     [76.70545196]]

After reshaping the input as a 2D array, we see that we can fit the model!

```python
poly_model.fit(Data_X,Data_y) # now we were able to train (fit) our linear model to the data!
```

    Pipeline(steps=[('polynomialfeatures', PolynomialFeatures(degree=1)),
                    ('linearregression', LinearRegression())])

That's it! Here's your first ML model: fitting a straight line 😆

Now that we have a model, we can predict the output $y^*$ for any new input point $x^*$.

In particular, we can predict the output for each of the input points $x$ that we used for training the model (i.e. at Data_X).

```python
y_pred = poly_model.predict(Data_X) # In scikit-learn, predicting from a model is a one-liner
```

Done! These are the predictions for all your training points.

But we can also predict the output $y^*$ for other points.

This enables us to visualize the model by predicting the output for a uniformly spaced set of points.

```python
# Now create linearly spaced points for plotting our linear model
x_plot = np.linspace(0, 90, 200) # 200 points uniformly spaced
y_plot = poly_model.predict(np.reshape(x_plot, (-1, 1))) # prediction of those points (note the reshape again)
```

Finally, we can just plot the data, the predictions for each input training point, and the linear model.

```python
# The usual plotting style for this model and the data.
fig_poly, ax_poly = plt.subplots() # create a plot
ax_poly.plot(Data_x, Data_y, 'b.', markersize=12,label="Data") # Markers locating data points)
ax_poly.plot(Data_x, y_pred, 'm*', markersize=12,label="Predictions") # Markers locating prediction points)
legend_str = "Linear regression with Polynomial of degree " + str(degree)
ax_poly.plot(x_plot, y_plot, 'm-', linewidth=2,label=legend_str) # polynomial interpolation plotted
ax_poly.set_xlabel("x (m/s)", fontsize=20) # create x-axis label with font size 20
ax_poly.set_ylabel("y (m)", fontsize=20) # create y-axis label with font size 20
ax_poly.set_title("Car stopping distance problem", fontsize=20) # create title with font size 20
ax_poly.legend(loc='upper left', fontsize=15) # replot legend
plt.close(fig_poly) # do not plot the figure now. We will show it in a later cell
```

```python
fig_poly # just call the figure that we created in the previous cell.
```

Now we can see all the things that we have done in a single plot!

## Exercise 1

1. Put it all together and create a linear model using a polynomial of degree 2 for the same data.

2. Compare your plot with the one obtained for the straight line (polynomial of degree 1).

3. Play a bit with your code by changing the degree of the polynomial. What happens?

```python
# Write your code for Exercise 1:

# until here.
```

If you keep increasing the polynomial degree, you see that the prediction gets worse!

This is called **overfitting**.

- Overfitting is a natural consequence of having a model that is more complex than it should be!

- As we know, the mean of the data that originates from the car stopping distance problem is generated with a quadratic model: $y = z_1 x + z_2 x^2$
    * Therefore, we don't need additional complexity to describe the *mean* of the data!

Also, the fit is better **within the domain** that we used for training ($x\in [3, 83]$) than away from it.

In other words, we **interpolate** better than we **extrapolate**.

* Nevertheless, overfitting is also an issue when we are interpolating...
    - See this by making the degree very high, e.g. 30, and plot for $x\in [5, 80]$.

It might be surprising, but this is an important issue in ML.

It is very common to use models that are complicated (have many parameters) and that perform poorly even when interpolating, but especially when extrapolating!

## Example 2: linear model for a noiseless problem

Consider now a problem not governed by a polynomial law and without uncertainty.

Let's consider the function $x\sin(x)$ in the domain $x\in[0,10]$. We did this in Lecture 2!

```python
# 1. Define the function f(x) = x sin(x)
def f(x):
    return x * np.sin(x)
# 2. Create a vector of 50 points that are uniformly spaced between 0 and 10
n_data = 50 # number of points for plotting the function
x_data = np.linspace(0, 10, n_data) # uniformly spaced points
# 3. Compute the output vector:
y_data = f(x_data)
# 4. Plot the function and the data
fig1, ax1 = plt.subplots() # This opens a new figure
# Plot points and interpolate them:
ax1.plot(x_data, y_data, 'ko:', markersize=6, linewidth=2,label=u'ground truth: $f(x) = x\,\sin(x)$')
ax1.set_xlabel('$x$', fontsize=20) # label of the x axis
ax1.set_ylabel('$f(x)$', fontsize=20) # label of the y axis
ax1.legend(loc='upper left', fontsize=15) # plot legend in the upper left corner
```

With lots of data, even linear interpolation between the points can approximate the function well.

However, what if we use just a few points from our dataset x_data?

```python
n_train = 5 # points to train the algorithm
x_train = np.linspace(0, 10, n_train) # 5 points uniformly distributed
y_train = f(x_train)
ax1.plot(x_train, y_train, 'r*', markersize=12,label="Training points") # Markers locating training points
ax1.plot(x_train, y_train, 'g-', linewidth=2, label=u'local linear interpolation') # linear interpolation
ax1.legend(loc='upper left', fontsize=15) # replot legend
fig1 # replot fig1 now overlaying the plot in the previous cell
```

This is called local interpolation because each line only depends on the two points it is connecting (not on the other points).

* This is not an ML model! It's different from the linear model we created for the car stopping distance problem.

So, let's train a linear ML model using a polynomial of degree 4 as basis function for this example.

```python
degree = 4 # degree of polynomial

poly_model = make_pipeline(PolynomialFeatures(degree), LinearRegression()) # model

X_train = np.reshape(x_train, (-1, 1)) # convert input vector into 2d array

poly_model.fit(X_train,y_train) # fit the polynomial to our 5 training points
#y_pred = poly_model.predict(X_train) # prediction of our polynomial

X_data = np.reshape(x_data, (-1, 1)) # Don't forget to convert to a 2d array
y_pred = poly_model.predict(X_data) # prediction our model for all 50 data points

# Plot x_data and prediction as a blue line:
ax1.plot(x_data, y_pred, 'b-', linewidth=2, label="Polynomial of degree %d prediction" % degree)

# Replot figure and legend:
ax1.legend(loc='upper left', fontsize=15)
fig1
```

Our polynomial (blue) is clearly different to the function that we want to "learn", i.e. $x \sin(x)$.

How do we evaluate the quality of our approximation?

* By evaluating the error of our polynomial model in the points that we didn't use in the fit.

Two common metrics are $R^2$ and $\text{MSE}$ (you will have to search for them and explain them!)

```python
# Import error metrics:
from sklearn.metrics import mean_squared_error, r2_score

# Compute MSE and R2 for the polynomial model we fitted
mse_value = mean_squared_error(y_data, y_pred)
r2_value = r2_score(y_data, y_pred)

print('MSE for polynomial = ', mse_value)
print('R2 score for polynomial = ', r2_value)
```

    MSE for polynomial =  5.826961574280784
    R2 score for polynomial =  0.578691860751741

As expected, these predictions are not great because:

* We want $\text{MSE}$ to be as low as possible

* The closer $R^2$ is to 1.0 the better

You will dive deeper into this when solving the **Midterm Project**.

## Example 3: linear model for noisy datasets

Let's consider one last case where we perturb the function of Example 2 $f(x) = x\sin(x)$ with noise.

* This is similar to what happens in the car stopping distance problem.

* Here, the difference is that at every point the noise is random (not as predictable as before)

Let's "fabricate" such dataset.

```python
seed = 1987 # set a random seed so that everyone gets the same result
np.random.seed(seed)

# Let's perturb every y_data point with Gaussian noise
random_std = 0.5 + 1.0 * np.random.random(y_data.shape)

# Then, take the random value for STD from 0.5 to 1.5 for each
# data point and create noise following a Gaussian distribution with
# that STD at that point:
noise = np.random.normal(0, random_std)

# The perturbed data becomes:
y_noisy_data = y_data + noise
```

For comparison, we plot the noisy data with the noiseless function that we would like to discover $x \sin(x)$:

```python
fig2, ax2 = plt.subplots() # This opens a new figure

# Plot the noiseless function ("the ground thruth")
ax2.plot(x_data, y_data, 'r:', linewidth=2,
         label=u'ground truth: $f(x) = x\,\sin(x)$')

# Plot the noisy dataset that we are given:
plt.errorbar(x_data, y_noisy_data, random_std, fmt='kX',
             markersize=6, label=u'noisy dataset')

ax2.set_xlabel('$x$', fontsize=20) # label of the x axis
ax2.set_ylabel('$f(x)$', fontsize=20) # label of the y axis
ax2.legend(loc='upper left', fontsize=15) # plot legend in the upper left corner
```

Note a couple of things:

* The black "x" marks the actual measured value.

* The black bars indicate the noise in each data point (each data point has a different noise value). Formally, we call this aleatoric uncertainty.
    - In the plot, I centered the error bars around the measured value. Note that the bars do not have the same length (different standard deviation).

### A note on data preprocessing

Usually, when we are given a dataset we need to find a way to **train** and **test** our model using that data.

However, to test our model we have to use data that we have not used in training, otherwise we would be cheating!

This is done by splitting the dataset (in this case x_data) into two sets:

1. **Training** set (for example: 75% of the dataset)

2. **Test** set with the remaining points of the dataset

Scikit-learn has a very easy way of doing this:

```python
from sklearn.model_selection import train_test_split

X_data = np.reshape(x_data,(-1,1)) # a 2D array that scikit-learn likes
# Let's split the data points into 10% for the training set (5 points)
# and the rest for the test set:
X_train, X_test, y_train, y_test = train_test_split(X_data,
                                    y_noisy_data, test_size=0.90,
                                    random_state=seed)
```

Note that the train_test_split module of scikit-learn picks points pseudo-randomly according to the random_state seed value.

Let's visualize the training and testing sets:

```python
x_train = X_train[:] # converting back to 1D array for plotting
x_test = X_test[:] # converting back to 1D array for plotting
# Plot the noisy training dataset:
ax2.plot(x_train, y_train, 'g*', markersize=18, label="Training points") # Markers locating training points
ax2.plot(x_test, y_test, 'bs', markersize=6,label="Testing points") # Markers locating training points
ax2.set_xlabel('$x$', fontsize=20) # label of the x axis
ax2.set_ylabel('$f(x)$', fontsize=20) # label of the y axis
ax2.legend(loc='upper left', fontsize=15) # plot legend in the upper left corner
fig2
```

Let's create a new figure with less clutter by just plotting the ground truth function and the training points.

```python
fig3, ax3 = plt.subplots() # This opens a new figure

# Plot the noiseless function ("the ground thruth")
ax3.plot(x_data, y_data, 'r:', linewidth=2, label=u'ground truth: $f(x) = x\,\sin(x)$')
ax3.plot(x_train, y_train, 'g*', markersize=18, label="Training points") # Markers locating training points
ax3.set_xlabel('$x$', fontsize=20) # label of the x axis
ax3.set_ylabel('$f(x)$', fontsize=20) # label of the y axis
ax3.legend(loc='upper left', fontsize=15) # plot legend in the upper left corner
```

## Exercise 2

Fit a polynomial of degree 4 to this training data and calculate the $R^2$ and $\text{MSE}$ metrics for the testing data.

```python
# Write your code for Exercise 2:

# until here.
```

Well done...

Yet, this does not seem like a great result, does it?

The $R^2$ value is so bad that it is even negative!

* What explains this result?

* Can we do something to fix this while still using polynomials?

* If we used more points would that help?

* What if we increased the degree of the polynomial?

### You will explore these and other things in Part 1 of the Midterm Project...

Have fun!

## Solution for Exercise 1

```python
degree = 2
poly_model = make_pipeline(PolynomialFeatures(degree),LinearRegression()) # model
poly_model.fit(np.reshape(Data_x, (-1, 1)),Data_y) # training (fitting)
y_pred = poly_model.predict(Data_X) # predict y for Data_x
x_plot = np.linspace(0, 90, 200) # 200 points uniformly spaced
y_plot = poly_model.predict(np.reshape(x_plot, (-1, 1))) # prediction for those points
fig_poly, ax_poly = plt.subplots() # create a plot
ax_poly.plot(Data_x, Data_y, 'b.', markersize=12,label="Data") # Markers locating data points)
ax_poly.plot(Data_x, y_pred, 'm*', markersize=12,label="Predictions") # Markers locating prediction points)
legend_str = "Linear regression with Polynomial of degree " + str(degree)
ax_poly.plot(x_plot, y_plot, 'm-', linewidth=2,label=legend_str) # polynomial interpolation plotted
ax_poly.set_xlabel("x (m/s)", fontsize=20) # create x-axis label with font size 20
ax_poly.set_ylabel("y (m)", fontsize=20) # create y-axis label with font size 20
ax_poly.set_title("Car stopping distance problem", fontsize=20) # create title with font size 20
ax_poly.legend(loc='upper left') # replot legend
```

### Solution to Exercise 2

``` python
degree = 4 # degree of polynomial we want to fit
poly_model2 = make_pipeline(PolynomialFeatures(degree),LinearRegression())
poly_model2.fit(X_train,y_train) # fit the polynomial to our 5 points in x_train
y_pred = poly_model2.predict(X_data) # prediction of our polynomial
# Compute MSE and R2 for the polynomial model we fitted
mse_value = mean_squared_error(y_noisy_data, y_pred)
r2_value = r2_score(y_noisy_data, y_pred)
print('MSE for polynomial = ', mse_value)
print('R2 score for polynomial = ', r2_value)
# Then, plot the polynomial prediction on top of fig1:
ax3.plot(x_data, y_pred, 'bo-', markersize=6, linewidth=2, label="Polynomial of degree %d prediction" % degree)
# Replot figure and legend:
ax3.legend(loc='upper right')
fig3
```