# Large Deviation Principle - Binomial Distribution

Use Large Deviation Theory to calculate the distribution of sample means sampled from a binomial distribution.

\begin{equation}
p(x) = 1/2 \delta(x-1/2) + 1/2 \delta(x+1/2)
\end{equation}

Which means, scaled cumulative distribution function $\lambda$ is:

\begin{equation}
\lambda(k) = \lim_{n\rightarrow \inf} \frac{1}{n} \ln \left< \exp nks  \right> = \ln cosh(k/2)
\end{equation}

With $s = \sum_i^n x_i$.

The rate function is given by the supremum of:

\begin{equation}
ks - \lambda(k) = ks - \ln cosh(k/2)
\end{equation}

Subject to $k$, which occurs for $k = 2\tanh^{-1}(2s)$

Which gives:

\begin{equation}
I(s) = 2s \tanh^{-1}(2s) - \ln\cosh(\tanh^{-1}(2s)/2)
\end{equation}

The probability distribution of the sample mean should then be approximately $p \asymp \exp(-nI(s))$.

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import random as rnd

s = np.linspace(-0.5,0.5,100)

def rate_function(s):
    return 2*s*np.arctanh(s) - np.log(np.cosh(np.arctanh(s)/2))

def rate_function_to_second_order(s):
    return (15/8)*s**2

plt.plot(s,rate_function(s))
plt.plot(s,rate_function_to_second_order(s))
plt.hlines(xmin=min(s),xmax=max(s),y=0)

plt.legend(['Rate Function','Rate Function (up to 2nd order Taylor Expansion)'])
```

```python
def sample_mean_probability(n):
    s = np.linspace(-0.5,0.5,10000)
    p = np.exp(-n*rate_function(s))
    p = p/np.sum(p)
    return s,p

def sample_mean_probability_approx(n):
    s = np.linspace(-0.5,0.5,10000)
    p = np.exp(-n*rate_function_to_second_order(s))
    p = p/np.sum(p)
    return s,p

N = [2,10,50]
for n in N:
    s, p = sample_mean_probability(n)
    plt.plot(s,p)

plt.legend(['n = %i' % n for n in N])
_ = plt.title("Probability Distribution of the Sample Mean")
```

```python
fig = plt.figure(figsize=(14,5))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

N = [2,10,50]
for n in N:
    s, p = sample_mean_probability(n)
    s, p_a = sample_mean_probability_approx(n)
    ax1.plot(s,p-p_a)
    ax2.plot(s,p,'.')
    ax2.plot(s,p_a)

ax2.set_xlim(0,0.2)
#ax2.set_ylim(-0.0001,0.001)
ax1.legend(['n = %i' % n for n in N])
_ = ax1.set_title("Error on Second Order Approximation")
```

```python
# The Experiment!
import scipy.stats

def assign(x):
    if x < 0.5:
        return -1/2
    else:
        return 1/2

def sample_mean(n):
    s = np.mean([assign(x) for x in rnd.uniform(0,1,n)])
    return s

def distribution(n,N = 10000):

    data = [sample_mean(n) for i in range(N)]
    hist = np.histogram(data,bins=100,range=(-0.5,0.5))
    return scipy.stats.rv_histogram(hist)

N = [20,1000]
for n in N:
    s, p = sample_mean_probability(n)
    dist = distribution(n,N=10000)
    plt.plot(s,p)
    plt.plot(s,max(p)*dist.pdf(s)/max(dist.pdf(s)))

```

In the experiment, the histograms had to be artifially rescaled because the discretization concentrates data in spikes, while the large deviation results gives a continuous function.

The shape of the distribution is correct, so that is encouraging.

The normalization has to then be done by taking discretization into account -- i.e., calculate the possible values that can result, evaluate the PDF on them, and let those values sum to 1.

# Correctly Normalizing the Large Deviation Result:

This is done by evaluating the continuous LD result only at the discrete, possible values of the sample mean, and then dividing by the sum of the values.

```python
from collections import Counter

def possible_sample_means(n):
    s = [-0.5*(n-i) + i*0.5 for i in range(0,n+1)]
    return np.array(s)/n

def sample_mean_probability(n):
    s = possible_sample_means(n)
    p = np.exp(-n*rate_function(s))
    p = p/np.sum(p)
    return s,p

N = [1,2,5,10]
for n in N:
    s, p = sample_mean_probability(n)
    data = Counter([sample_mean(n) for i in range(10000)])
    s_exp = list(data.keys())
    p_exp = np.array(list(data.values()))/np.sum(list(data.values()))

    plt.plot(s,p,'-*')
    plt.plot(s_exp,p_exp,'o')

```

```python
N = [100,1000]
for n in N:
    s, p = sample_mean_probability(n)
    data = Counter([sample_mean(n) for i in range(10000)])
    s_exp = list(data.keys())
    p_exp = np.array(list(data.values()))/np.sum(list(data.values()))

    plt.plot(s,p,'-*')
    plt.plot(s_exp,p_exp,'o')
```

Large Deviation Principle - Binomial Distribution.ipynb


```python
from sympy import *
init_printing(use_latex='mathjax')
```

```python
def get_diff(expression, *symbols):
    print("Expression:")
    display(expression)
    results = []
    for symbol in symbols:
        print("derivated by ", symbol)
        df = diff(expression, symbol)
        display(df)
        results.append(df)
    return results
```

```python
x, y, z = symbols('x y z')
expr = sin(x) * exp(y * z ** 2)
get_diff(expr, z)
```

    Expression:

$$e^{y z^{2}} \sin{\left (x \right )}$$

    derivated by  z

$$2 y z e^{y z^{2}} \sin{\left (x \right )}$$

$$\left [ 2 y z e^{y z^{2}} \sin{\left (x \right )}\right ]$$

```python
x, y, m = symbols('x y m')
expr = pi * x ** 3 + x * y ** 2 + m * y ** 4
get_diff(expr, y)
```

    Expression:

$$m y^{4} + \pi x^{3} + x y^{2}$$

    derivated by  y

$$4 m y^{3} + 2 x y$$

$$\left [ 4 m y^{3} + 2 x y\right ]$$

```python
x, y, m = symbols('x y m')
expr = pi * x ** 3 + x * y ** 2 + m * y ** 4
get_diff(expr, y)
```

    Expression:

$$m y^{4} + \pi x^{3} + x y^{2}$$

    derivated by  y

$$4 m y^{3} + 2 x y$$

$$\left [ 4 m y^{3} + 2 x y\right ]$$

```python
x, y, z = symbols('x y z')
expr = y * x ** 2 + z * y ** 2 + x * z ** 2
get_diff(expr, x, y, z)
```

    Expression:

$$x^{2} y + x z^{2} + y^{2} z$$

    derivated by  x

$$2 x y + z^{2}$$

    derivated by  y

$$x^{2} + 2 y z$$

    derivated by  z

$$2 x z + y^{2}$$

$$\left [ 2 x y + z^{2}, \quad x^{2} + 2 y z, \quad 2 x z + y^{2}\right ]$$

```python
x, y, z = symbols('x y z')
expr = exp(2 * x) * sin(y) * z ** 2 + cos(z) * exp(x) * exp(y)
get_diff(expr, x,y,z)
```

    Expression:

$$z^{2} e^{2 x} \sin{\left (y \right )} + e^{x} e^{y} \cos{\left (z \right )}$$

    derivated by  x

$$2 z^{2} e^{2 x} \sin{\left (y \right )} + e^{x} e^{y} \cos{\left (z \right )}$$

    derivated by  y

$$z^{2} e^{2 x} \cos{\left (y \right )} + e^{x} e^{y} \cos{\left (z \right )}$$

    derivated by  z

$$2 z e^{2 x} \sin{\left (y \right )} - e^{x} e^{y} \sin{\left (z \right )}$$

$$\left [ 2 z^{2} e^{2 x} \sin{\left (y \right )} + e^{x} e^{y} \cos{\left (z \right )}, \quad z^{2} e^{2 x} \cos{\left (y \right )} + e^{x} e^{y} \cos{\left (z \right )}, \quad 2 z e^{2 x} \sin{\left (y \right )} - e^{x} e^{y} \sin{\left (z \right )}\right ]$$

## Total derivative

```python
def total_derivative(expr, replacements, symbol):
    print("Original expression: ")
    display(expr)
    expr = expr.subs(replacements)
    return get_diff(expr, symbol)[0]
```

```python
x, y, t = symbols('x y t')
expr = pi * y * x ** 2
df = total_derivative(expr, {y: t ** 2 - 1, x : t ** 2 + 1}, t)
expand(df)
```

    Original expression:

$$\pi x^{2} y$$

    Expression:

$$\pi \left(t^{2} - 1\right) \left(t^{2} + 1\right)^{2}$$

    derivated by  t

$$4 \pi t \left(t^{2} - 1\right) \left(t^{2} + 1\right) + 2 \pi t \left(t^{2} + 1\right)^{2}$$

$$6 \pi t^{5} + 4 \pi t^{3} - 2 \pi t$$

```python
expr = 4 * pi * t * ( t ** 2 + 1 ) * (t ** 2 - 1 ) + 2 * pi * t * ( t ** 2 + 1 ) ** 2
display(expr)
expand(expr)
# Good one!
```

$$4 \pi t \left(t^{2} - 1\right) \left(t^{2} + 1\right) + 2 \pi t \left(t^{2} + 1\right)^{2}$$

$$6 \pi t^{5} + 4 \pi t^{3} - 2 \pi t$$

```python
expand(2 * pi * (t ** 2 + 1 ) ** 2 * (t ** 2 - 1) + pi * (t ** 2 + 1 ) ** 2 * ( t * 2 - 1))
# Not good one
```

$$2 \pi t^{6} + 2 \pi t^{5} + \pi t^{4} + 4 \pi t^{3} - 4 \pi t^{2} + 2 \pi t - 3 \pi$$

```python
expand(4 * pi * t * ((t ** 2) + 1 ) ** 2 + 2 * pi * t * ( (t ** 2) + 1 ) ** 2)
# Not good one
```

$$6 \pi t^{5} + 12 \pi t^{3} + 6 \pi t$$

```python
x, y, z, t = symbols('x y z t')
expr = cos(x) * sin(y) * exp(2 * z)
df = total_derivative(expr, {x : t + 1, y : t - 1, z: t ** 2}, t)
factor(df)
```

    Original expression:

$$e^{2 z} \sin{\left (y \right )} \cos{\left (x \right )}$$

    Expression:

$$e^{2 t^{2}} \sin{\left (t - 1 \right )} \cos{\left (t + 1 \right )}$$

    derivated by  t

$$4 t e^{2 t^{2}} \sin{\left (t - 1 \right )} \cos{\left (t + 1 \right )} - e^{2 t^{2}} \sin{\left (t - 1 \right )} \sin{\left (t + 1 \right )} + e^{2 t^{2}} \cos{\left (t - 1 \right )} \cos{\left (t + 1 \right )}$$

$$\left(4 t \sin{\left (t - 1 \right )} \cos{\left (t + 1 \right )} - \sin{\left (t - 1 \right )} \sin{\left (t + 1 \right )} + \cos{\left (t - 1 \right )} \cos{\left (t + 1 \right )}\right) e^{2 t^{2}}$$

## Jacobian

```python
x, y = symbols('x y')
expr = y * x ** 2 + x * y * 3 / 4 + 10
get_diff(expr, x,y)
```

    Expression:

$$x^{2} y + \frac{3 x}{4} y + 10$$

    derivated by  x

$$2 x y + \frac{3 y}{4}$$

    derivated by  y

$$x^{2} + \frac{3 x}{4}$$

$$\left [ 2 x y + \frac{3 y}{4}, \quad x^{2} + \frac{3 x}{4}\right ]$$

```python
x, y = symbols('x y')
expr = exp(x) * cos(y) + x * exp(3 * y) - 2
get_diff(expr, x,y)
```

    Expression:

$$x e^{3 y} + e^{x} \cos{\left (y \right )} - 2$$

    derivated by  x

$$e^{x} \cos{\left (y \right )} + e^{3 y}$$

    derivated by  y

$$3 x e^{3 y} - e^{x} \sin{\left (y \right )}$$

$$\left [ e^{x} \cos{\left (y \right )} + e^{3 y}, \quad 3 x e^{3 y} - e^{x} \sin{\left (y \right )}\right ]$$

```python
x, y, z = symbols('x y z')
expr = exp(x) * cos(y) + x**2 * y**2 * z**2
get_diff(expr, x,y,z)
```

    Expression:

$$x^{2} y^{2} z^{2} + e^{x} \cos{\left (y \right )}$$

    derivated by  x

$$2 x y^{2} z^{2} + e^{x} \cos{\left (y \right )}$$

    derivated by  y

$$2 x^{2} y z^{2} - e^{x} \sin{\left (y \right )}$$

    derivated by  z

$$2 x^{2} y^{2} z$$

$$\left [ 2 x y^{2} z^{2} + e^{x} \cos{\left (y \right )}, \quad 2 x^{2} y z^{2} - e^{x} \sin{\left (y \right )}, \quad 2 x^{2} y^{2} z\right ]$$

```python
x, y, z = symbols('x y z')
expr = x **2 + 3 * exp(y) * exp(z) + cos(x) * sin(z)
J = get_diff(expr, x,y,z)
list(map(lambda j : j.evalf(subs={x: 0, y:0, z:0}), J))
```

    Expression:

$$x^{2} + 3 e^{y} e^{z} + \sin{\left (z \right )} \cos{\left (x \right )}$$

    derivated by  x

$$2 x - \sin{\left (x \right )} \sin{\left (z \right )}$$

    derivated by  y

$$3 e^{y} e^{z}$$

    derivated by  z

$$3 e^{y} e^{z} + \cos{\left (x \right )} \cos{\left (z \right )}$$

$$\left [ 0, \quad 3.0, \quad 4.0\right ]$$

```python
x, y, z = symbols('x y z')
expr = x * exp(y) * cos(z) + 5 * x ** 2 * sin(y) * exp(z)
J = get_diff(expr, x,y,z)
list(map(lambda j : j.evalf(subs={x: 0, y:0, z:0}), J))
```

    Expression:

$$5 x^{2} e^{z} \sin{\left (y \right )} + x e^{y} \cos{\left (z \right )}$$

    derivated by  x

$$10 x e^{z} \sin{\left (y \right )} + e^{y} \cos{\left (z \right )}$$

    derivated by  y

$$5 x^{2} e^{z} \cos{\left (y \right )} + x e^{y} \cos{\left (z \right )}$$

    derivated by  z

$$5 x^{2} e^{z} \sin{\left (y \right )} - x e^{y} \sin{\left (z \right )}$$

$$\left [ 1.0, \quad 0, \quad 0\right ]$$

```python

```

Certification 2/Week2.1 - Differential with multiple variable.ipynb


# Bayes (Part II)

```python
import numpy as np
import matplotlib.pyplot as plt
```

## The Metropolis-Hastings algorithm

In the lecture this week, we have seen the _Metropolis-Hastings_ algorithm, which allows us to get samples from an _unnormalized_ distribution function, i.e., a function that returns a probability multiplied by some constant factor. Note that this can be any distribution (although in practice we will need it to get samples from an unnormalized posterior).

First of all, have a look at the description of the algorithm and try to implement it as described here.
> __**NOTE**__ You can find the implemented algo at the bottom of this notebook, but please try by yourself before looking at it!

```python
def mcmc(nsamples, unnorm_prob_f, proposalf, burnin=100, initial=0.):
    """
    Parameters
    ----------
    nsamples: int
        Number of samples to draw (includes burnin)
    unnorm_prob_f: func
        Function taking a point in the support and
        returning its unnormalized probability
    proposalf: func
        Function that takes current position
        and returns a proposal for where to move next
    burnin: int
        Number of initial samples to exclude
    initial: float or func
        If float, starts from that point.
        If func, starts from the output of initial()
    Returns
    -------
    list
        A list of samples
    """

    # set initial point in variable 'current' and
    # calculate current probability ('curr_prob')
    # ADD CODE HERE

    # we put the samples in list 'states'
    states = []
    for i in range(nsamples):

        # append current position to states
        # ADD CODE HERE

        # proposes a new point
        # ADD CODE HERE

        # calculate unnormalized probability
        # at proposed point
        # ADD CODE HERE

        # calculates probability of acceptance
        # in variable 'acceptance'
        # ADD CODE HERE

        if np.random.random_sample() < acceptance:
            # if acceptance is higher than random btw 0 and 1,
            # accept move, else stay where you are (i.e. do nothing)
            # ADD CODE HERE

    # return list of samples
    return states[burnin:]
```

This algorithm has a lot of pros:
- Suprisingly simple to implement
- If we get enough samples it is guaranteed to converge to the true posterior!
- Works for both continuous and discrete parameter spaces
But it also has some cons:
- Can be very slow to converge
- Does not work well with highly dimensional spaces
- There is no way to tell if it converged to the true posterior

In general, we should get as many samples as possible and run multiple chains with different initial points, to see if they converged to the same distribution.

## Getting samples from an unnormalized normal distribution

The simplest application is to get samples from a distribution that we can calculate exactly, and see if the algorithm converges to it. Let's take the standard normal distribution, which has density function:

$$
\varphi (z)= \frac {1}{\sqrt {2\pi }} e^{-{\frac {z^{2}}{2}}}
$$

let's write a function to calculate this probability density:

```python
def normal(x,mu=0,sigma=1):
    """
    Probability density of normal distribution at x
    """
    numerator = np.exp((-(x-mu)**2)/(2*sigma**2))
    denominator = sigma * np.sqrt(2*np.pi)
    return numerator/denominator
```

And let's test this function to make sure it does what we expect:

```python
xs = np.linspace(-3,3,1000)
plt.plot(xs, normal(xs))
plt.xlabel('x')
plt.ylabel('density')
plt.show()
```

Now suppose that we didn't have the explicit formula for the distribution, but rather some black-box function where $\varphi$ is multiplied by an unknown constant $K$:

$$
\varphi' (z)= K \frac {1}{\sqrt {2\pi }} e^{-{\frac {z^{2}}{2}}}
$$

> __**NOTE**__: If $K$ is equal to the normalization constant $\sqrt {2\pi }$, $\varphi' (z)$ becomes just $e^{-{\frac {z^{2}}{2}}}$. This is essentially the trick we will use to sample from the posterior, where we don't have the posterior as such, but we can calculate the posterior multiplied by $P(D)$.

Now we can use our mcmc function above to take samples from the distribution:

```python
samples = mcmc(
    # Number of samples
    100000,
    # Unnormalized density function
    lambda x: np.exp(-x**2 / 2),
    # Proposal function from point x
    lambda x: np.random.normal(loc=x, scale=0.1),
)
```

We can plot a histogram of the samples and compare it to the true distribution, and if you wrote the MCMC function correctly, they should look very close:

```python
# plot a histogram of the samples
plt.hist(samples, bins=100, density=True)
# also plot the true distribution, just to be sure!
xs = np.linspace(-3,3,1000)
plt.plot(xs, normal(xs))
plt.show()
```

## Using MHMC for Bayesian inference: categorization in continuous space

Now recall the categorization case that we saw in class: we observe some samples from an unknown category, and we have to form a posterior over the position of the unseen category's boundaries. The case we have seen in class has discrete boundaries, and therefore we could in theory calculate the posterior directly: consider all the possible hypotheses (categories) $H$, calculate $P(H) P(D \mid H)$, and sum it all to find $P(D)$.

However, now imagine the case where we have a _continuous_ space, and so infinitely many categories. The probability density of sampling a certain observation from a category is still 0 if the observation lies outside the category and $1/|H|$ otherwise, but now the category $H$ can have a float size (e.g., category [-1, 0.5] has size 1.5). Now it becomes difficult to calculate the $P(D)$, because we can't easily integrate over all categories!

However, if we define some prior over hypotheses (i.e. the categories), we can still easily calculate the numerator of the posterior, namely $P(H) P(D \mid H)$, for specific categories. A simple prior can be defined as follows:

$$
p(H) = \phi(x=l_H, \mu=0, \sigma=10) \; \phi(x=u_H, \mu=0, \sigma=10)
$$

where $\phi$ is the probability density function of a normal distribution, $l_H$ is the lower bound, and $u_H$ is the upper bound.

```python
def prior_f(boundaries):
    lb, ub = boundaries
    return normal(lb, sigma=10) * normal(ub, sigma=10)
```

Likelihood can be defined as discussed above:

$$
p(D \mid H) = \prod_{d \in D} \left( |H|^{-1} \text{ if } d \in H \text{ else } 0 \right)
$$

where $D$ is a set of observations (points from the unobserved category).

```python
def likelihood_f(data, boundaries):
    lb, ub = boundaries
    compatible = all([lb < i < ub for i in data])
    if compatible:
        return 1/(ub-lb)
    else:
        return 0
```

With the prior and the likelihood as defined above, we can write a function that given some data returns the unnormalized posterior function (i.e. a function form a category to the unnormalized posterior probability of the category given the data):

```python
def define_unnorm_posterior(data):
    def unnorm_posterior(boundaries):
        return (
            likelihood_f(data, boundaries) *
            prior_f(boundaries)
        )
    return unnorm_posterior
```

Finally, we just need a proposal function that takes a current category and proposes a new one. The proposal function here does this by moving the category up or down and stretching it by some amount:

```python
def proposal_f(current_boundaries):
    # transform boundaries linearly
    # (i.e., move up and down and stretch)
    shift = np.random.normal()
    # we want the stretch to be always positive
    # to preserve order of upper and lower bound
    # (otherwise all samples with wrong order would
    # have prob 0 in the likelihood function)
    # Moreover, a stretching by some value x
    # should be as likely as a contraction by x
    # To preserve symmetry!
    stretch = np.exp(np.random.normal())
    return shift + stretch*current_boundaries
```

Let's check the distribution of the upper and lower bounds of the proposals starting from category [-1,1]:

```python
proposed = [proposal_f(np.array([-1,1])) for _ in range(10000)]
lbs, ubs = np.array(proposed).T
plt.hist(lbs, bins=100, alpha=0.2)
plt.hist(ubs, bins=100, alpha=0.2)
plt.xlim(-15, 15)
plt.show()
```

Now we are ready to run approximate Bayesian inference on some data! Let's define some data first:

```python
observations = [0.1, 2., 1.1]
```

Take samples from the posterior distribution:

```python
samples = mcmc(
    # Number of samples
    500000,
    # Unnormalized density function
    define_unnorm_posterior(observations),
    # Proposal function from point x
    proposal_f,
    initial=np.array([-0.5, 0.5])
)
```

    /tmp/ipykernel_2721/2883728898.py:42: RuntimeWarning: invalid value encountered in double_scalars
      acceptance = min(move_prob/curr_prob,1)
    /tmp/ipykernel_2721/2883728898.py:42: RuntimeWarning: divide by zero encountered in double_scalars
      acceptance = min(move_prob/curr_prob,1)

Let's plot the posterior distributions of upper and lower bound:

```python
lbs, ubs = np.array(samples).T
```

```python
plt.hist(lbs, bins=100, density=True, label='Lower bound')
plt.hist(ubs, bins=100, density=True, label='Upper bound')
plt.xlim(-12, 12)
plt.scatter(
    observations,
    [1.]*len(observations),
    color='red',
    label='observations'
)
plt.legend()
plt.show()
```

> __QUESTION__ What happens if we add more observations? Do you see the size effect in continuous space?

> __EXERCISE__ (if there is time left at the end) Write a version of this inference model for 2-d space, where categories are squares and observations are points in 2d space.

## Using MHMC in discrete spaces

We can use Metropolis-Hastings for discrete hypotheses spaces too! The basic idea is the same: we move around the space of hypotheses, calculating the unnormalized posterior for each point, and accepting new moves according to the rule described in the algo. The only complication is that we can't use the normal distribution centered at the current position as a proposal distribution, but we must come up with something different.

As an example which will be useful in the future, consider the space of sentences generated by a PCFG as the space of hypotheses. For instance, the following PCFG:

$$
S \rightarrow a | b | Sa | Sb
$$

with uniform substitution probabilities (each $1/3$). The space of hypotheses then would be: 'a', 'b', 'aa', 'ab', 'ba', 'bb', and so on.

```python
def complete(incomplete_sentence):
    result = incomplete_sentence
    options = ['a', 'b', 'Sa', 'Sb']
    while 'S' in result:
        result = np.random.choice(options) + result[1:]
    return result
```

```python
# test function
complete('Saa')
```

    'aaa'

The probability of a specific sentence of length $n$ then is $4^{-n}$ (can you see why?). Let's verify by sampling a bunch of sentences and finding the proportion of times we sample each one:

```python
n = 10000
samples = [complete('S') for _ in range(n)]
sentences, counts = np.unique(samples, return_counts=True)
argsort = np.argsort(counts)[::-1]
sentences = sentences[argsort]
counts = counts[argsort] / n
```

```python
plt.bar(x=sentences[:20], height=counts[:20])
plt.xticks(rotation=80)
plt.show()
```

Based on this, it is easy to calculate the probability of each sentence:

```python
def prior_f(sentence):
    return 4**(-len(sentence))
```

```python
# example of prior calculation
prior_f('a')
```

    0.25

We want to learn the sentence that produces some data. If we can see the sentence directly, this is easy: just look at the sentence and we're done. However, suppose that we can't see the sentence directly, but we see a 'noisy' version of it, where possibly some characters have been added at the end, where:
- nothing is added with probability 1/3.
- 'a' and 'b' are added with probability 1/3 each.

We can calculate the resulting observation probability:

```python
def likelihood_f(observations, sentence):
    likelihood = 1
    for observation in observations:
        # Calculates probability of observation given true sentence
        if observation == sentence:
            likelihood *= 1/3
        elif observation.startswith(sentence):
            likelihood *= 1/3**(len(observation)-len(sentence))
        else:
            return 0
    return likelihood
```

```python
# test the function
print(likelihood_f(['abaa', 'abaaa', 'aba'], 'ab'))
print(likelihood_f(['abaa', 'abaaa', 'abab'], 'ab'))
print(likelihood_f(['abaa', 'abaaa', 'abab'], 'abb'))
```

    0.001371742112482853
    0.00045724737082761767
    0

Like in the example above, we can now calculate the unnormalized posterior:

```python
def define_unnorm_posterior(data):
    def unnorm_posterior(sentence):
        return (
            likelihood_f(data, sentence) *
            prior_f(sentence)
        )
    return unnorm_posterior
```

Finally, we need a proposal distribution, which brings us from a current sentence to a possible next step. For MHMC to work as written above, it has to be the case that the probability of going from $x$ to $x'$ is the same as the probability of going from $x'$ to $x$. If this is not the case, we need to multiply the value of `acceptance` in `mcmc` by $\frac{P(x' \rightarrow x) }{ P(x \rightarrow x')}$ to balance things out.

In practice, we will use the _subtree-regeneration_ transition distribution proposed at p.153 in Goodman, Noah D., Joshua B. Tenenbaum, Jacob Feldman, and Thomas L. Griffiths. “A Rational Analysis of Rule-Based Concept Learning.” Cognitive Science 32, no. 1 (2008): 108–54. https://doi.org/10.1080/03640210701802071. It works as follows, starting from current sentence $x$:
- Sample a node $n$ at random from the parse tree of $x$
- Remove everything below $n$ and replace $n$ with appropriate non-terminal, generating a tree $y$ containing non-terminals.
- Complete $y$ with the PCFG, obtaining new sentence $x'$.
- The acceptance probability from `mcmc` then gets additionally multiplied by:

\begin{align}
\frac{p(x' \rightarrow x)}{p(x \rightarrow x')}
&=
\frac{ P(\text{sampling node $n$ from $x'$}) }{ P(\text{sampling node $n$ from $x'$}) }
\frac{ P(x \mid \text{PCFG}) }{ P(x' \mid \text{PCFG}) } \\
&= \frac{|x'|^{-1}}{|x|^{-1}} \frac{4^{-|x|}}{4^{-|x'|}} \\
&= \frac{|x|}{|x'|} \frac{4^{-|x|}}{4^{-|x'|}}
\end{align}

```python
def transition_f(current):
    n = np.random.randint(len(current))
    truncated = 'S' + current[:n]
    return complete(truncated)
```

```python
# test the function
transition_f('aba')
```

    'bbb'

```python
def acceptance_f(current, proposed, unnorm_prob_f):
    move_when_lower = (
        (unnorm_prob_f(proposed) * len(current) * 0.25**len(current))
        /
        (unnorm_prob_f(current) * len(proposed) * 0.25**len(proposed))
    )
    return min(
        move_when_lower,
        1
    )
```

```python
def mcmc(nsamples, unnorm_prob_f, proposalf, burnin=100, initial=0.):
    """
    Same as mcmc above but with different acceptance_f
    """

    current = initial() if callable(initial) else initial
    curr_prob = unnorm_prob_f(current)
    states = []
    for i in range(nsamples):
        states.append(current)
        movement = proposalf(current)
        move_prob = unnorm_prob_f(movement)
        acceptance = acceptance_f(
            current,
            movement,
            unnorm_prob_f
        )
        if np.random.random_sample() < acceptance:
            current = movement
            curr_prob = move_prob
    return states[burnin:]
```

Define some observations:

```python
observations = ['aabaab', 'aaba', 'aabaaaa']
```

Take samples with our algorithm:

```python
samples = mcmc(
    100000,
    define_unnorm_posterior(observations),
    transition_f,
    initial='a'
)
```

Plot the posterior distribution:

```python
sentences, counts = np.unique(samples, return_counts=True)
argsort = np.argsort(counts)[::-1]
sentences = sentences[argsort]
counts = counts[argsort] / len(samples)

plt.bar(x=sentences, height=counts)
plt.xticks(rotation=80)
plt.show()
```

Various things to be noticed:
- The most likely sentence is the longest sentence that all the observations share, namely 'aaba'
- The second most likely is 'aab', because it would be a 'suspicious' coincidence if all observations independently produced an 'a' right after 'aab', if the true sentence is indeed 'aab'.
- Same, but even more so, applies to 'aa' and 'a'.
- So even though 'aaba' is a priori less likely than 'a', it get a higher posterior probability because of the likelihood!

## Homework

Third (and last) homeworks set! Consider the following CFG (interpret it as a PCFG with uniform probabilities), where each sentence from this grammar is an integer:
$$
S \rightarrow 1 | S+1 | S-1
$$
Note that there are multiple way to arrive at the same integer. E.g. $I(1+1-1)=I(1)$.

1. Write a function `prior_f` that takes a sentence produced by the PCFG and returns its probability in the PCFG.

Assume that there is a true (unknown) sentence $H$ that we are trying to infer based on some observations ${o_1, \dots, o_n}$. The observations aren't just the integer defined by the sentence, but rather each observation is equal to the integer defined by the sentence plus some normally distributed noise. The likelihood function therefore is as follows:
$$
p(o \mid H) = P(H \mid \text{PCFG}) \; \phi(x= o-I(H), \mu=0, \sigma=2.)
$$
as usual, $I$ is the interpretation function, $\phi$ is the density function of the normal distribution. For instance if the true sentence was `1+1`, we might observe 1.94.

2. Write a function `likelihood_f` that calculates the probability of a list of observations given a sentence.
> __HINT__ First find the integer $i$ defined by the sentence, then for each observation find the difference between $i$ and the observation, then find the probability of those differences (given that they are samples from a normal with $\mu=0$ and $\sigma=2$), and multiply those probabilities together.
3. Write a function to calculate the unnormalized posterior. This is essentially what the `define_unnorm_posterior` function above does.

4. Write a transition function: a function that takes a current sentence $x$ and returns a randomly generated proposal $x'$. You can modify the subtree-regeneration above to adapt it to this case.
5. Using mcmc, draw 100000 samples from $P(H \mid D)$.

## Implemented MHMC

```python
def mcmc(nsamples, unnorm_prob_f, proposalf, burnin=100, initial=0.):
    """
    Parameters
    ----------
    nsamples: int
        Number of samples to draw (includes burnin)
    unnorm_prob_f: func
        Function taking a point in the support and
        returning its unnormalized probability
    proposalf: func
        Function that takes current position
        and returns a proposal for where to move next
    burnin: int
        Number of initial samples to exclude
    initial: float or func
        If float, starts from that point.
        If func, starts from the output of initial()
    Returns
    -------
    list
        A list of samples
    """

    current = initial() if callable(initial) else initial
    curr_prob = unnorm_prob_f(current)

    # we put the samples in list 'states'
    states = []
    for i in range(nsamples):

        # append current position to states
        states.append(current)

        # proposes a new point
        movement = proposalf(current)

        # calculate unnormalized probability
        # at proposed point
        move_prob = unnorm_prob_f(movement)

        # calculates probability of acceptance
        acceptance = min(move_prob/curr_prob,1)

        # if acceptance is higher than random btw 0 and 1,
        # accept move, else stay where you are (i.e. do nothing)
        if np.random.random_sample() < acceptance:
            current = movement
            curr_prob = move_prob

    # return list of samples
    return states[burnin:]
```


In this notebook, you will implement the forward longitudinal vehicle model. The model accepts throttle inputs and steps through the longitudinal dynamic equations. Once implemented, you will be given a set of inputs that drives over a small road slope to test your model.

The input to the model is a throttle percentage $x_\theta \in [0,1]$ which provides torque to the engine and subsequently accelerates the vehicle for forward motion.

The dynamic equations consist of many stages to convert throttle inputs to wheel speed (engine -> torque converter -> transmission -> wheel). These stages are bundled together in a single inertia term $J_e$ which is used in the following combined engine dynamic equations.

\begin{align}
    J_e \dot{\omega}_e &= T_e - (GR)(r_{eff} F_{load}) \\ m\ddot{x} &= F_x - F_{load}
\end{align}

Where $T_e$ is the engine torque, $GR$ is the gear ratio, $r_{eff}$ is the effective radius, $m$ is the vehicle mass, $x$ is the vehicle position, $F_x$ is the tire force, and $F_{load}$ is the total load force.

The engine torque is computed from the throttle input and the engine angular velocity $\omega_e$ using a simplified quadratic model.

\begin{align}
    T_e = x_{\theta}(a_0 + a_1 \omega_e + a_2 \omega_e^2)
\end{align}

The load forces consist of aerodynamic drag $F_{aero}$, rolling friction $R_x$, and gravitational force $F_g$ from an incline at angle $\alpha$. The aerodynamic drag is a quadratic model and the friction is a linear model.

\begin{align}
    F_{load} &= F_{aero} + R_x + F_g \\
    F_{aero} &= \frac{1}{2} C_a \rho A \dot{x}^2 = c_a \dot{x}^2\\
    R_x &= N(\hat{c}_{r,0} + \hat{c}_{r,1}|\dot{x}| + \hat{c}_{r,2}\dot{x}^2) \approx c_{r,1} \dot{x}\\
    F_g &= mg\sin{\alpha}
\end{align}

Note that the absolute value is ignored for friction since the model is used for only forward motion ($\dot{x} \ge 0$).

The tire force is computed using the engine speed and wheel slip equations.

\begin{align}
    \omega_w &= (GR)\omega_e \\
    s &= \frac{\omega_w r_e - \dot{x}}{\dot{x}}\\
    F_x &= \left\{\begin{array}{lr}
        cs, &  |s| < 1\\
        F_{max}, & \text{otherwise}
        \end{array}\right\}
\end{align}

Where $\omega_w$ is the wheel angular velocity and $s$ is the slip ratio.

We setup the longitudinal model inside a Python class below. The vehicle begins with an initial velocity of 5 m/s and engine speed of 100 rad/s. All the relevant parameters are defined and like the bicycle model, a sampling time of 10ms is used for numerical integration.

```python
import sys
import math
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

class Vehicle():
    def __init__(self):

        # ==================================
        #  Parameters
        # ==================================

        #Throttle to engine torque
        self.a_0 = 400
        self.a_1 = 0.1
        self.a_2 = -0.0002

        # Gear ratio, effective radius, mass + inertia
        self.GR = 0.35
        self.r_e = 0.3
        self.J_e = 10 # inertia
        self.m = 2000
        self.g = 9.81

        # Aerodynamic and friction coefficients
        self.c_a = 1.36
        self.c_r1 = 0.01

        # Tire force
        self.c = 10000
        self.F_max = 10000

        # State variables
        self.x = 0 # position
        self.v = 5 # velocity
        self.a = 0 # acceleration
        self.w_e = 100 # engine angular velocity
        self.w_e_dot = 0 # engine angular acceleration

        self.sample_time = 0.01

    def reset(self):
        # reset state variables
        self.x = 0
        self.v = 5
        self.a = 0
        self.w_e = 100
        self.w_e_dot = 0
```

Implement the combined engine dynamic equations along with the force equations in the cell below. The function $\textit{step}$ takes the throttle $x_\theta$ and incline angle $\alpha$ as inputs and performs numerical integration over one timestep to update the state variables. Hint: Integrate to find the current position, velocity, and engine speed first, then propagate those values into the set of equations.

```python
class Vehicle(Vehicle):
    def step(self, throttle, alpha):
        # ==================================
        #  Implement vehicle model here
        # Integrate to find the current position, velocity, and engine speed first,
        # then propagate those values into the set of equations.
        # x theta is throttle
        # alpha is incline angle in radian

        x_dot = self.v # velocity
        v_dot = self.a # acceleration
        w_e_dot = self.w_e_dot # engine angular acceleration
        self.x += x_dot * self.sample_time # position
        self.v += v_dot * self.sample_time # velocity
        self.w_e += w_e_dot * self.sample_time # engine speed
        omega_w = self.GR * self.w_e # Wheel speed = geer ratio * engine speed
        s = omega_w * self.r_e / self.v - 1 # Slip ratio formula
        F_x = self.c * s if abs(s) < 1 else self.F_max # tire force
        F_g = self.m * self.g * np.sin(alpha) # gravitational force
        R_x = self.c_r1 * x_dot # rolling friction
        F_aero = self.c_a * x_dot**2 # aerodynamic drag
        F_load = F_aero + R_x + F_g # total load force
        T_e = throttle * (self.a_0 + self.a_1 * self.w_e + self.a_2 * self.w_e**2) # engine torque
        self.a = (F_x - F_load) / self.m # resulted acceleration
        self.w_e_dot = (T_e - self.GR * self.r_e * F_load) / self.J_e # resulted angular acceleration
        # ==================================
        pass
```

Using the model, you can send constant throttle inputs to the vehicle in the cell below. You will observe that the velocity converges to a fixed value based on the throttle input due to the aerodynamic drag and tire force limit. A similar velocity profile can be seen by setting a negative incline angle $\alpha$. In this case, gravity accelerates the vehicle to a terminal velocity where it is balanced by the drag force.

```python
sample_time = 0.01
time_end = 100
model = Vehicle()

t_data = np.arange(0,time_end,sample_time)
v_data = np.zeros_like(t_data)

# throttle percentage between 0 and 1
throttle = 0.2

# incline angle (in radians)
alpha = 0

for i in range(t_data.shape[0]):
    v_data[i] = model.v
    model.step(throttle, alpha)

plt.plot(t_data, v_data)
plt.show()
```

We will now drive the vehicle over a slope as shown in the diagram below.

To climb the slope, a trapezoidal throttle input is provided for the next 20 seconds as shown in the figure below.

The vehicle begins at 20% throttle and gradually increases to 50% throttle. This is maintained for 10 seconds as the vehicle climbs the steeper slope. Afterwards, the vehicle reduces the throttle to 0.

In the cell below, implement the ramp angle profile $\alpha (x)$ and throttle profile $x_\theta (t)$ and step them through the vehicle dynamics. The vehicle position $x(t)$ is saved in the array $\textit{x_data}$. This will be used to grade your solution.

```python
time_end = 20
t_data = np.arange(0,time_end,sample_time)
x_data = np.zeros_like(t_data)

# reset the states
model.reset()

# ==================================
#  Learner solution begins here
# ==================================

ramp_angle_data = np.zeros_like(t_data)
throttle_data = np.zeros_like(t_data)

n = t_data.shape[0]

for i in range(n//4):
    throttle_data[i] = 0.2 + (0.3/(n//4)) * i
for i in range(n//4, 3*n//4):
    throttle_data[i] = 0.5
for i in range(3*n//4, n):
    throttle_data[i] = 0.5 - (i-3*n//4) * (0.5/(n//4))

for i in range(n):
    x_data[i] = model.x # store current position
    v_data[i] = model.v # store current velocity
    if model.x < 60:
        ramp_angle_data[i] = math.atan2(3, 60) # return arc tangent in radian
    elif model.x < 150:
        ramp_angle_data[i] = math.atan2(12-3, 90) # return arc tangent in radian
    else:
        ramp_angle_data[i] = 0
    model.step(throttle_data[i], ramp_angle_data[i]) # update dynamic model with new throttle and ramp angle

# ==================================
#  Learner solution ends here
# ==================================

# Plot x vs t for visualization
plt.plot(t_data, x_data)
plt.show()
```

If you have implemented the vehicle model and inputs correctly, you should see that the vehicle crosses the ramp at ~15s where the throttle input begins to decrease.

The cell below will save the time and vehicle inputs as text file named $\textit{xdata.txt}$. To locate the file, change the end of your web directory to $\textit{/notebooks/Course_1_Module_4/xdata.txt}$

Once you are there, you can download the file and submit to the Coursera grader to complete this assessment.

```python
data = np.vstack([t_data, x_data]).T
np.savetxt('xdata.txt', data, delimiter=', ')
```

Congratulations! You have now completed the assessment! Feel free to test the vehicle model with different inputs in the cell below, and see what trajectories they form. In the next module, you will see the longitudinal model being used for speed control. See you there!

```python
sample_time = 0.01
time_end = 30
model.reset()

t_data = np.arange(0,time_end,sample_time)
x_data = np.zeros_like(t_data)

# ==================================
#  Test various inputs here
# ==================================
for i in range(t_data.shape[0]):

    model.step(0,0)

plt.axis('equal')
plt.plot(t_data, x_data)
plt.show()
```


## One Step Method
Applying the Euler formula to the first order equation
\begin{equation}
y^{'} = sin(x)
\end{equation}
is approximated by

\begin{equation}
\frac{w_{i+1}-w_i}{h}=sin(x_i)
\end{equation}

Rearranging
\begin{equation}
w_{i+1}=w_i+hsin(x_i)
\end{equation}

## DECLARING LIBRARIES

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
warnings.filterwarnings("ignore")
```

## Setting up X

```python
h=0.1
x_end=10
INITIALCONDITION=1
N=int(x_end/h)
Numerical_Solution=np.zeros(N)
x=np.zeros(N)
Analytic_Solution=np.zeros(N)
```

## NUMERICAL SOLUTION

```python
Numerical_Solution[0]=INITIALCONDITION
x[0]=0
Analytic_Solution[0]=INITIALCONDITION
for i in range (1,N):
    Numerical_Solution[i]=Numerical_Solution[i-1]+h*math.sin(x[i-1])
    x[i]=x[i-1]+h
    Analytic_Solution[i]=2.0-math.cos(x[i])
```

## Plotting

```python
fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(1,3,1)
plt.plot(x,Numerical_Solution,color='red')
#ax.legend(loc='best')
plt.title('Numerical Solution')
# --- right hand plot
ax = fig.add_subplot(1,3,2)
plt.plot(x,Analytic_Solution,color='blue')
plt.title('Analytic Solution')

#ax.legend(loc='best')
ax = fig.add_subplot(1,3,3)
plt.plot(x,Analytic_Solution-Numerical_Solution,color='blue')
plt.title('Error')

# --- title, explanatory text and save
fig.suptitle('Sine Solution', fontsize=20)
plt.tight_layout()
plt.subplots_adjust(top=0.85)
```

```python

```

Chapter 01 - Euler Methods/Supplementary/102_Euler Method Sinewave.ipynb


```python
# obligatory import statements
import numpy as np
from matplotlib import pyplot as pl
from scipy.optimize import minimize
from scipy.ndimage import median_filter
from scipy.stats.kde import gaussian_kde
from sklearn.metrics import mean_squared_error, mean_squared_log_error, mean_absolute_error
from sympy import symbols, log, sqrt, Piecewise
from sympy.functions import Abs
from sympy.plotting import plot
```

# Choosing a loss function for regression

---
**TL;DR**

In short, the workflow can be summarized as follows:
1. Choose MSE
2. Check distribution of residual errors
3. If this is symmetric with heavy tails consider Huber
4. Else if the values are positive (although offset can deal with negative values) and the errors are positively skewed with a heavy right tail consider MSLE

---

With the regression data frame analytics jobs we now support three different loss functions: `mse`, `msle`, and `huber`. While all three of them can be used to train a model to predict real-valued data by minimizing average loss between actual values $a_i$ and predictions $p_i$
$$\frac{1}{N}\sum_{i=1}^{N}L(a_i,p_i),$$
they would work best in different scenarios. Let's look at what loss function works best for which case.

## Mean squared error (MSE)

[MSE](https://en.wikipedia.org/wiki/Mean_squared_error) is the most commonly used loss function. It works well in many scenarios and you should try it if you are unsure which loss function to use or you don't know much about your data.

As the name suggests, MSE minimizes the quadratic difference between the prediction and the actual value:
$$L(a_i,p_i) = \tfrac{1}{2}(a_i-p_i)^2$$

```python
a = symbols('a') #actual value
p = symbols('p') #predicted value
mse = lambda a,p: (a-p)**2
mse_plot = plot(mse(0, p),(p, -3, 3), show=False, legend=True, line_color="red")
mse_plot[0].label='MSE'
mse_plot.show()
```

The problem with the squared error is that it makes small errors (< 1.0) even smaller and large large errors (>1.0) disproportionately larger.

This means that if you have a few outliers in your data, for example, due to a large measurement error, the model will put useless effort to fit these data points often degrading the accuracy of the majority of predictions as a result.

```python
# Let's see how to identify
np.random.seed(1000)
N = 10000
w = np.ones(5)
X = np.random.randn(N, 5)
y = X.dot(w)

# The last 1000 target values get some serious noise
y[-1000:] += np.random.normal(0,100.0, 1000)

mse = lambda x: mean_squared_error(y, X.dot(x))

x0 = np.zeros(X.shape[1]) #initial guess
result_w = minimize(mse, x0, tol=1e-5)['x'] #MSE result
residuals_mse = y - X.dot(result_w)

# Plot the distribution of the residuals
kde = gaussian_kde(residuals_mse)
dist_space = np.linspace(min(residuals_mse), max(residuals_mse), 100)
_ = pl.plot(dist_space, kde(dist_space))
```

Datasets with large outliers show heavy tails in the residual distribution when trained using MSE as shown above.

In these cases what you would usually want to do is instead of fitting the mean **squared** error to fit the mean **absolute** error (MAE)
$$L(a_i,p_i) = \vert a_i - p_i\vert^2$$

```python
mae = lambda a,p: Abs(a-p)
mae_plot = plot(mae(0, p),(p, -3, 3), show=False, line_color="blue")
mae_plot[0].label = "MAE"
mse_plot.extend(mae_plot)
mse_plot.show()
```

MAE does not over-prioritize outliers and it is easy to interpret: mean absolute error of 20 USD on price prediction means that you are on average 20 USD off your target mark in one way or another.

Unfortunately, it is not straightforward to train models on MAE using conventional algorithms because of the sharp kink around `0.0`. For this reason, you may want to use a loss function that behaves as MAE for errors larger than 1 and as MSE for errors smaller than 1. This loss function is called [Huber loss](https://en.wikipedia.org/wiki/Huber_loss).

## Huber loss

Generally, Huber loss uses a parameter $\delta$ to define the transition point between MSE and MAE:
$$L(a, p) = \left\{\begin{aligned}
\tfrac{1}{2}(a-p)^2 \quad\mathrm{for}\ |a-p| \le \delta \\
\delta\vert a - p \vert - \tfrac{1}{2}\delta^2 \quad\mathrm{otherwise}
\end{aligned}\right.
$$

```python
huber = lambda delta,a,p: Piecewise((0.5*(a-p)**2, Abs(a-p) <= delta), (delta*Abs(a-p)-0.5*delta**2, True))
huber1_plot = plot(huber(1.0, 0, p),(p, -3, 3), show=False, line_color="green")
huber1_plot[0].label = "Huber $\delta=1.0$"
huber2_plot = plot(huber(2.0, 0, p),(p, -3, 3), show=False, line_color="turquoise")
huber2_plot[0].label = "Huber $\delta=2.0$"
mse_plot.extend(huber1_plot)
mse_plot.extend(huber2_plot)
mse_plot.show()
```

Particularly interesting is that for $\delta=1.0$ Huber loss is asymptotically similar to MAE and behaves as MSE around 0. For other values of $\delta$, Huber loss still increases linearly for large errors and $\delta$ controls simply the slope of this growth.

In machine learning data frame analytics, we implemented the so-called [Pseudo-Huber loss](https://en.wikipedia.org/wiki/Huber_loss#Pseudo-Huber_loss_function)
$$L(a_i,p_i) = \delta^2 \left(\sqrt{1+((a_i-p_i)/\delta)^2}-1\right).$$

This results in faster code while keeping the properties of the regular Huber loss.

```python
pseudo_huber = lambda delta,a,p: delta**2*(sqrt(1+((a-p)/delta)**2)-1)
pseudo_huber1 = plot(pseudo_huber(1.0, 0,p),(p, -3, 3), show=False, legend=True, line_color='lightgreen')
pseudo_huber1[0].label='Pseud-Huber $\delta=1.0$'
huber1_plot.extend(pseudo_huber1)
huber1_plot.legend = True
huber1_plot.show()
```

### Conclusion

To use Huber loss with a certain parameter $\delta$ in your data frame analytics regression job, you can specify parameter `loss_function` (set it to `huber`) and `loss_function_parameter`. Use Huber loss if you have outliers in your data coming from measurement errors. This is often the case when you can see a symmetric heavy-tailed distribution of residuals after training your model with MSE loss function.

You can experiment with different values of $\delta$, also the default value 1.0 is fine for most cases.

## Mean squared logarithmic error (MSLE)

This loss function is used when you have positive targets distributed with a long tail such as log-normal distribution. There are [numerous examples](https://en.wikipedia.org/wiki/Log-normal_distribution#Occurrence_and_applications) of such data: house pricing, income of private households, city population, etc.

```python
from sympy.stats import LogNormal, density
z = symbols('z')
log_norm_1 = LogNormal("x", 0, 1.0)
log_norm_2 = LogNormal("x", 0, 0.5)
log_norm_3 = LogNormal("x", 0, 0.25)
pdf_plot1= plot(density(log_norm_1)(z), (z,0.01, 5), show=False, line_color="blue")
pdf_plot2= plot(density(log_norm_2)(z), (z,0.01, 5), show=False, line_color="green")
pdf_plot3= plot(density(log_norm_3)(z), (z,0.01, 5), show=False, line_color="red")
pdf_plot1.extend(pdf_plot2)
pdf_plot1.extend(pdf_plot3)
pdf_plot1.show()

```

The distribution of the values has a "bump" for smaller values and then a long tail with a small number of very large values.

Naturally, when you want to predict the target values, you want to allow for an error depending on the magnitude of the target. It’s ok to miss a population of a metropolitan area by 10 000, but for a small town it is unforgivable. In this case MSLE can be more suitable for the regression job than MSE. As the name suggests, MSLE minimizes the quadratic difference between the logarithms of the actual value and the prediction
$$L(a_i, p_i) = (\log(a_i + t) - \log(p_i + t))^2$$

For example, $\log\left((1 + 1100) / (1 + 1000)\right)^2$ is about the same as $\log((1 + 11000) / (1 + 10000))^2$ so missing a value by 10\% is penalised equivalently whether the actual error is 100 or 1000. At the same time, with MSE the latter case costs 100x more.

Qualitatively speaking, MSLE tends to lead to lower error for small targets and for higher error on large targets than MSE. This trade-off can be controlled by adjusting an offset parameter $t$ which is set to 1.0 by default. By increasing this parameter, you influence the “transition point” at which you go from minimizing quadratic error to minimizing quadratic log error.

Let's take a look how this offset affects quantitative results.

```python
# We generate random data and train a simple linear model to minimize
# MSE and MSLE loss functions.
np.random.seed(1000)
N = 10000
w = np.ones(5)
X = np.random.randn(N, 5)
y = X.dot(w)
yexp = np.exp(y)

mse = lambda x: mean_squared_error(yexp, X.dot(x))
def msle(x, t=1):
    pred = X.dot(x)+t
    pred[pred<0] = 0
    return mean_squared_log_error(yexp+t, pred)

x0 = np.zeros(X.shape[1]) #initial guess
res_mse = minimize(mse, x0, tol=1e-5) #MSE residuals
results = {} #MSLE residuals for different offsets
T = [1.0, 10.0, 100.0]
for t in T:
    results[t] = minimize(lambda x: msle(x, t), x0, tol=1e-5)

sorted_idx = np.argsort(yexp)
label_sorted = yexp[sorted_idx]
def get_sorted_residuals(results):
    residuals = yexp - X.dot(results['x'])
    return residuals[sorted_idx]

f, (axTop, axMiddle, axBottom) = pl.subplots(3, 1, sharex=True, figsize=(10, 15))

# plot the density distribution of the target values on top
kde = gaussian_kde(yexp)
yrange = np.logspace(min(y), max(y), 100, base=np.exp(1))
axTop.plot(yrange, kde(yrange))
axTop.set_ylabel('density')

# plot the magnitudes of residuals for MSE and MSLE with different offsets
for t in T:
    # use a median filter to smooth residuals and see the trend.
    axMiddle.plot(label_sorted, median_filter(get_sorted_residuals(results[t]),  size=50), label='t={}'.format(t))
    axBottom.plot(label_sorted, median_filter(get_sorted_residuals(results[t]),  size=50), label='t={}'.format(t))
axMiddle.plot(label_sorted, median_filter(get_sorted_residuals(res_mse),  size=50), label='mse')
axBottom.plot(label_sorted, median_filter(get_sorted_residuals(res_mse),  size=50), label='mse')

axBottom.set_xscale('log')
axBottom.set_xlabel('target')
axBottom.set_xticks(np.logspace(-3,3,7))
axMiddle.set_ylabel('residual')
axBottom.set_ylabel('residual')
axMiddle.set_ylim(800,1000)
axBottom.set_ylim(-20,75)
axMiddle.legend()

pl.show()
```

Note that the scale of the y-axis in the bottom has been changed to be able to see those small residuals!

First, MSE tends to result in larger errors for small target values despite the fact that most of the targets are rather small. It rather focuses on producing smaller errors for large target values.

Second, if you look at the density of the target values (top diagram), you see that qualitatively the "transition point" happens somewhere around 10: we have a high concentration of small targets <10 and low number of targets >>10. Quantitatively, this "transition point" is somewhere between 80th and 90th percentile.

```python
quantiles = np.linspace(0, 1.0, 11)
qvalues = np.quantile(yexp, np.linspace(0, 1.0, 11))
for q, val in zip(quantiles, qvalues):
    print("Percentile {:.0f}\tvalue {:.2f}".format(q*100, val))
```

    Percentile 0	value 0.00
    Percentile 10	value 0.06
    Percentile 20	value 0.15
    Percentile 30	value 0.31
    Percentile 40	value 0.54
    Percentile 50	value 0.96
    Percentile 60	value 1.74
    Percentile 70	value 3.10
    Percentile 80	value 6.18
    Percentile 90	value 16.46
    Percentile 100	value 4891.30

Setting the offset parameter to 1.0 or 100.0 undershoots and overshoots this "transition point". Both lead to similar results with greater residuals for small targets and lower residuals for large targets compared to the offset parameter of 10.0. However, trying to "nail" the correct transition point super exactly is not worth it: setting offset parameter to 6.0 or 16.0  won't change the results significantly compared to 10.0.

### Conclusion

To use MSLE with a certain parameter offset $t$ in your data frame analytics regression job, you can specify parameter `loss_function` (set it to `msle`) and `loss_function_parameter`. Try it, if you have one of the [numerous examples](https://en.wikipedia.org/wiki/Log-normal_distribution#Occurrence_and_applications) of positive target values which scale over a large scale.

If you are unsure, train the model using MSE loss function and [test residual for normality](https://en.wikipedia.org/wiki/Normality_test). If normality cannot be assumed, then you should try MSLE as well.

You can leave the `loss_function_parameter` as it is, but if you want to experiment with different offset values, it is recommended to set the value somewhere at the 80th percentile of the target distribution and then increase by an order of magnitude and look at the results qualitatively.

Machine Learning/Regression Loss Functions/regression-loss-functions.ipynb


```python
import math
import numpy as np
from sympy import Matrix
```

```python
a = np.array([[1,-7],[-2,-3]])
mtx = Matrix(a)
```

```python
mtx
```

$\displaystyle \left[\begin{matrix}1 & -7\\-2 & -3\end{matrix}\right]$

L1 norm

```python
mtx.norm(ord=1)
```

$\displaystyle 10$

```python
np.linalg.norm(a, ord=1)
```

    10.0

Infinite norm

```python
np.linalg.norm(a,ord=math.inf)
```

    8.0

Froibinous norm or Eucledian norm

```python
mtx.norm(ord='fro')
```

$\displaystyle 3 \sqrt{7}$

```python

```


# Numerical Integration
#### Preliminaries
We have to import the array library `numpy` and the plotting library `matplotlib.pyplot`, note that we define shorter aliases for these.

Next we import from `numpy` some of the functions that we will use more frequently and from an utility library functions to format conveniently our results.

Eventually we tell to the plotting library how we want to format our plots (`fivethirtyeight` is a predefined style that mimics a popular site of statistical analysis and yes, statistical analysis and popular apply both to a single item... while `00_mplrc` refers to some tweaks I prefer to use)

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from numpy import cos, exp, pi, sin, sqrt
from IPython.display import Latex, display
plt.style.use(['fivethirtyeight', './00_mplrc'])
```

## Plot of the load

We start analyzing the loading process.

Substituting $t_1=1.2$s in the expression of $p(t)$ we have

$$p(t) = \left({15625 \over 27} {t^3 \over s^3} - {8750 \over 9} {t^2 \over s^2}
      + {1100 \over 3} {t \over s} + 400 \right)\text{N}$$

```python
N = 6000
def it(t): return int((t+1)*N/3)

t = np.linspace(-1,2,N+1)
p = np.where(t<0, 0, np.where(
             t<=1.2, 15625./27*t**3 - 8750./9*t**2 + 1100./3*t + 400, 0))
```

We will use the above expression everywhere in the following.

Let's visualize this loading with the help of a plot:

```python
plt.plot(t,p)
plt.ylim((-20,520))
plt.xticks((-1,0,1,1.2,2))
plt.xlabel('t [second]')
plt.ylabel('p(t) [Newton]')
plt.title(r'The load $p(t)$')
plt.show()
```

From the plot it is apparent that the loading can be approximated by a constant rectangle plus a sine, of relatively small amplitude, with a period of about 1 second.

I expect that the forced response will be a damped oscillation about the static displacement $400/k$, followed by a free response in which we will observe a damped oscillation about the zero displacement.

## The Particular Integral

Writing the particuar integral as

$$\xi(t) = \left(A {t^3 \over s^3} + B {t^2 \over s^2}
      + C {t \over s} + D \right)$$

where $A, B, C, D$ all have the dimension of a length, deriving with respect to time and substituting in the equation of motion we have

$$\left( 6A\frac{t}{s} + 2 B\right)\frac{m}{s^2}
      + \left( 3A\frac{t^2}{s^2} + 2B\frac{t}{s} + C \right) \frac{c}{s}
      + \left( A\frac{t^3}{s^3} + B\frac{t^2}{s^2} + C\frac{t}{s} + D \right) k=\left({15625 \over 27} {t^3 \over s^3} - {8750 \over 9} {t^2 \over s^2}
      + {1100 \over 3} {t \over s} + 400 \right)\text{N}$$

Collecting the powers of $t/s$ in the left member we have

$$A k \frac{t^3}{s^3} + \left( B k + \frac{3 A c}{s} \right) \frac{t^2}{s^2} +  \left( C k + \frac{2 B c}{s} + \frac{6 A m}{s^2} \right) \frac{t}{s}
      + \left(D k + \frac{C c}{s} + \frac{2 B m}{s^2}\right)=\left({15625 \over 27} {t^3 \over s^3} - {8750 \over 9} {t^2 \over s^2}
      + {1100 \over 3} {t \over s} + 400 \right)\text{N}$$

and equating the coefficients of the different powers, starting from the higher to the lower ones, we have a sequence of equations of a single unknown, solving this sequence of equations gives, upon substitution of the numerical values of the system parameters,

\begin{align*}
  A& = \frac{625}{14256}\text{m}=0.0438411896745
\text{m}\\
  B& = \left(\frac{-175}{2376} - \frac{19}{209088} \sqrt{11}\right)\text{m}=-0.0739545830991
\text{m}\\
  C& = \left(\frac{133}{1306800} \sqrt{11} + \frac{218117}{7920000}\right)\text{m}=0.0278775758319\text{m}\\
  D& = \left(\frac{3013181}{99000000} - \frac{135571859}{7187400000000} \sqrt{11}\right)\text{m}=0.0303736121006\text{m}\\
\end{align*}

Substituting in $\xi(t)$ and taking the time derivative we have

\begin{align*}
  \xi(t) &= \frac{625}{14256} t^3 -
            \left( \frac{19}{209088} \sqrt{11} + \frac{175}{2376} \right) t^2 +
            \left( \frac{133}{1306800} \sqrt{11} + \frac{218117}{7920000} \right) t -
            \frac{135571859}{7187400000000} \sqrt{11} + \frac{3013181}{99000000},\\
  \dot\xi(t) &= \frac{625}{4752} t^2 -
              2 \left( \frac{19}{209088} \sqrt{11} + \frac{175}{2376} \right) t +
              \frac{133}{1306800} \sqrt{11} + \frac{218117}{7920000}
\end{align*}

```python
xi = np.where(t<0, 0,
     np.where(t<=1.2001,
              625./14256*t**3 - (19./209088*sqrt(11) + 175./2376)*t**2
          + t*(133./1306800*sqrt(11) + 218117./7920000)
          - 135571859./7187400000000*sqrt(11) + 3013181./99000000, 0))
dot_xi = np.where(t<0, 0,
         np.where(t<=1.2001,
                  625./4752*t**2 - 2*(t*(19./209088*sqrt(11) + 175./2376))
                + 133./1306800*sqrt(11) + 218117./7920000, 0))
xi_0 = - 135571859./7187400000000*sqrt(11) + 3013181./99000000
dot_xi_0 = + 133./1306800*sqrt(11) + 218117./7920000
```

### Plot of the particular integral

```python
plt.plot(t,xi)
plt.title('Particular integral')
plt.xlabel('s')
plt.ylabel('m')
plt.show()
plt.plot(t, dot_xi)
plt.title('Time derivative of particular integral')
plt.xlabel('s')
plt.ylabel('m/s')
plt.show()
```

## System Response

### Forced Response $0\le t\le t_1$

The response in terms of displacement and velocity is

\begin{align}
  x(t) &= \exp(-\zeta\omega_nt) \left(
    R\cos(\omega_Dt) + S\sin(\omega_Dt)\right) + \xi(t),\\
  v(t) &= \exp(-\zeta\omega_nt) \left(
    \left(S\cos(\omega_Dt)-R\sin(\omega_Dt)\right)\omega_D -
    \left(R\cos(\omega_Dt)+S\sin(\omega_Dt)\right)\zeta\omega_n
  \right) + \dot\xi(t)
\end{align}

and we can write the following initial conditions, taking into account that at time $t=0$ the system is at rest,

\begin{align}
  x(0) &= 1 \cdot \left(
    R\cdot1 + S\cdot0\right) + \xi_0\\&=R+\xi_0=0,\\
  \dot x(0) &= 1 \cdot \left(
    \left(S\cdot1-R\cdot0\right)\omega_D -
    \left(R\cdot1+S\cdot0\right)\zeta\omega_n
  \right) + \dot\xi_0\\&=S\omega_D-R\zeta\omega_n+\dot\xi_0=0
\end{align}

The constants of integration are

\begin{align}
  R &= -\xi_0,\\
  S &= \frac{R\zeta\omega_n-\dot\xi_0}{\omega_D}
\end{align}

```python
mass = 12.0
stif = 13200.0
zeta = 0.038

w2_n = stif/mass
w1_n = sqrt(w2_n)
w1_D = w1_n*sqrt(1-zeta**2)

damp = 2*zeta*w1_n*mass

R = -xi_0
S = (R*zeta*w1_n-dot_xi_0)/w1_D

display(HTML("<center><h3>Forced Displacements</h3></center>"))
display(Latex(r"""
    $x(t) = \exp(-%g\cdot%g\,t) \left(
    %+g\cos(%g\,t)%+g\sin(%g\,t)
    \right)+\xi(t)$
    """%(zeta,w1_n,R,w1_D,S,w1_D)))

t1  = 1.2
it1 = it(t1)

x_t1 = exp(-zeta*w1_n*t1)*(R*cos(w1_D*t1)+S*sin(w1_D*t1))+xi[it1]
v_t1 =(exp(-zeta*w1_n*t1)*(R*cos(w1_D*t1)+S*sin(w1_D*t1))*(-zeta)*w1_n
      +exp(-zeta*w1_n*t1)*(S*cos(w1_D*t1)-R*sin(w1_D*t1))*w1_D
      +dot_xi[it1])
display(Latex(
        r'$$x(t_1)=x_1=%+g\,\text{m},\qquad v(t_1)=v_1=%+g\,\text{m/s}$$'%
        (x_t1,v_t1)))
```

Forced Displacements

    $x(t) = \exp(-0.038\cdot33.1662\,t) \left(
    -0.0303736\cos(33.1423\,t)-0.00199618\sin(33.1423\,t)
    \right)+\xi(t)$

$$x(t_1)=x_1=+0.035918\,\text{m},\qquad v(t_1)=v_1=+0.237819\,\text{m/s}$$

### Free Response

For $t\ge t_1$ the response (no external force) is given by

\begin{align}
  x^*(t) &= \exp(-\zeta\omega_nt) \left(R^*\cos(\omega_Dt) + S^*\sin(\omega_Dt)\right),\\
  v^*(t) &= \exp(-\zeta\omega_nt) \left(
    \left(S^*\cos(\omega_Dt)-R^*\sin(\omega_Dt)\right)\omega_D -
    \left(R^*\cos(\omega_Dt)+S^*\sin(\omega_Dt)\right)\zeta\omega_n
  \right).
\end{align}

By the new initial conditions,

$$x^*(t_1) = x(t_1) = x_1, \qquad v^*(t_1) = v(t_1) = v_1,$$

we have, with
$e_1 = \exp(-\zeta\omega_{n}t_1)$,
$c_1 = \cos(\omega_Dt_1)$ and
$s_1 = \sin(\omega_Dt_1)$

$$ e_1 \begin{bmatrix}
     c_1 & s_1 \\
     -\omega_D s_1 - \zeta\omega_n c_1 & \omega_D c_1 - \zeta\omega_n s_1 \end{bmatrix} \,
     \begin{Bmatrix}R^*\\S^*\end{Bmatrix} = \begin{Bmatrix}x_1\\v_1\end{Bmatrix}
$$

that gives

$$
\begin{Bmatrix}R^*\\S^*\end{Bmatrix} =
\frac{1}{\omega_D\,e_1}\,
\begin{bmatrix}
  \omega_D c_1 - \zeta\omega_n s_1 & -s_1 \\
  \zeta\omega_n c_1 + \omega_D s_1 & c_1
\end{bmatrix}\,
\begin{Bmatrix}x_1\\v_1\end{Bmatrix}.
$$

```python
e_t1 = exp(-zeta*w1_n*t1)
c_t1 = cos(w1_D*t1)
s_t1 = sin(w1_D*t1)

Rs = (w1_D*c_t1*x_t1 - zeta*w1_n*s_t1*x_t1 - s_t1*v_t1) /w1_D /e_t1
Ss = (w1_D*s_t1*x_t1 + zeta*w1_n*c_t1*x_t1 + c_t1*v_t1) /w1_D /e_t1

display(HTML("<center><h3>Free Displacements</h3></center>"))
display(Latex(r"""
    $$x^*(t) = \exp(-%g\cdot%g\,t) \left(
    %+g\cos(%g\,t)%+g\sin(%g\,t)
    \right)$$
    """%(zeta,w1_n,Rs,w1_D,Ss,w1_D)))

xs_t1 = exp(-zeta*w1_n*t1)*(Rs*cos(w1_D*t1)+Ss*sin(w1_D*t1))
vs_t1 = ((exp(-zeta*w1_n*t1)*(Rs*cos(w1_D*t1)+Ss*sin(w1_D*t1))*(-zeta)*w1_n
      +exp(-zeta*w1_n*t1)*(Ss*cos(w1_D*t1)-Rs*sin(w1_D*t1))*w1_D))

display(Latex(r'$$x^*(t_1)=%+g\,\text{m},\qquad v^*(t_1) = %+g\,\text{m/s}$$'%
              (xs_t1,vs_t1)))
```

Free Displacements

    $$x^*(t) = \exp(-0.038\cdot33.1662\,t) \left(
    -0.112254\cos(33.1423\,t)+0.124351\sin(33.1423\,t)
    \right)$$

$$x^*(t_1)=+0.035918\,\text{m},\qquad v^*(t_1) = +0.237819\,\text{m/s}$$

#### Putting it all together

First, the homogeneous response

```python
x_hom = np.where(t<0, 0,
        np.where(t<1.2001, exp(-zeta*w1_n*t) * (R *cos(w1_D*t) + S *sin(w1_D*t)),
                           exp(-zeta*w1_n*t) * (Rs*cos(w1_D*t) + Ss*sin(w1_D*t))))
v_hom = np.where(t<0, 0,
        np.where(t<1.2001,
                 exp(-t*zeta*w1_n) * (
                    ( S*w1_D- R*zeta*w1_n)*cos(t*w1_D) -
                    ( S*zeta*w1_n+ R*w1_D)*sin(t*w1_D)),
                 exp(-t*zeta*w1_n) * (
                    (Ss*w1_D-Rs*zeta*w1_n)*cos(t*w1_D) -
                    (Ss*zeta*w1_n+Rs*w1_D)*sin(t*w1_D))))
```

then, we put together the homogeneous response and the particular integral, in the different intervals

```python
x = x_hom+xi
v = v_hom+dot_xi
```

### Plot of the response

```python
plt.plot(t,x)
plt.title('Displacement')
plt.show()

plt.plot(t,v)
plt.title('Velocity')
plt.show()

plt.plot(t,x)
plt.title('Displacement, zoom around t_1')
plt.xlim((1.16,1.24))
plt.show()

plt.plot(t,v)
plt.title('Velocity, zoom around t_1')
plt.xlim((1.16,1.24))
plt.show()
```

## Numerical Integration

The time step we are going to use is specified in terms of the natural period of vibration, $h=T_n/12$:

```python
t_n = 2*pi/w1_n
h = t_n/12.0
```

We need a function that returns the value of the load,

```python
def load(t):
    return 0 if t > 1.20001 else (
        + 578.703703703703705*t**3
        - 972.22222222222222*t**2
        + 366.666666666666667*t
        + 400.)
```

### Initialization

The final time for the computation, the factors that modify the increment of the load to take into account the initial conditions, the modified stiffness, the containers for the results, the initial values of the results.

```python
stop = 2.0 + h/2

a0fac = 3.0*mass + damp*h/2.0
v0fac = 6.0*mass/h + 3.0*damp

ks = stif + 3.0*damp/h + 6.0*mass/h**2

T, X, V, A = [], [], [], []

x0 = 0
v0 = 0
t0 = 0
p0 = load(t0)
```

### Computational Loop

```python
while t0 < stop:
    a0 = (p0-stif*x0-damp*v0)/mass
    for current, vec in zip((t0,x0,v0,a0),(T,X,V,A)):
        vec.append(current)
    t1 = t0+h
    p1 = load(t1)
    dp = p1-p0
    dx = (dp+a0fac*a0+v0fac*v0)/ks
    dv = 3*dx/h -a0*h/2 -3*v0
    t0 = t1 ; p0 = p1 ; x0 = x0+dx ; v0 = v0+dv
```

### The Plot

```python
plt.plot(T,X,label='numerical')
plt.plot(t,x,label='analytical')
plt.xlim((0,2))
plt.legend(loc=3);
```

```python
plt.plot(T,X,'-o',label='numerical')
plt.plot(t,x,'-x',label='analytical')
plt.title('Displacement, zoom around t_1')
plt.xlim((1.16,1.24))
plt.ylim((0,0.04))
plt.legend(loc=3)
plt.show()
```

```python
# an IPython incantation that properly formats this notebook
from IPython.display import HTML
HTML(open("00_custom.sav.css", "r").read())
```


cf. David Darmofal. 16.901 Computational Methods in Aerospace Engineering, Spring 2005. (Massachusetts Institute of Technology: MIT OpenCourseWare), http://ocw.mit.edu (Accessed 22 May, 2016). License: Creative Commons BY-NC-SA

# Lecture 1 Numerical Integration of Ordinary Differential Equations: An Introduction

```python
import sympy
m_p, g = sympy.symbols('m_p g', real=True)
```

```python
t = sympy.Symbol('t',real=True)
```

```python
u = sympy.Function('u')(t)
D = sympy.Function('D')(u)
```

```python
SphFreeFall = sympy.Eq(m_p*sympy.diff(u,t),m_p*g - D)
sympy.pprint( SphFreeFall )
```

        d
    m_p⋅──(u(t)) = g⋅m_p - D(u(t))
        dt

For low speeds,

```python
rho_g, mu_g = sympy.symbols("rho_g mu_g",real=True)
# density and dynamic viscosity of the atmosphere,
```

```python
a = sympy.symbols("a",real=True)
```

```python
Re = 2*rho_g*u*a/mu_g
```

```python
C_D = 24./Re + 6./(1. + sympy.sqrt(Re)) + 0.4
```

```python
Dlow = 0.5*rho_g*sympy.pi*a**2*u**2*C_D
```

```python
Dlow

```

    0.5*pi*a**2*rho_g*(0.4 + 6.0/(sqrt(2)*sqrt(a*rho_g*u(t)/mu_g) + 1.0) + 12.0*mu_g/(a*rho_g*u(t)))*u(t)**2

```python
sympy.pprint( SphFreeFall.subs( D, Dlow) )
```

        d                   2     ⎛                 6.0               12.0⋅μ_g ⎞
    m_p⋅──(u(t)) = - 0.5⋅π⋅a ⋅ρ_g⋅⎜0.4 + ───────────────────────── + ──────────⎟⋅u
        dt                        ⎜             ____________         a⋅ρ_g⋅u(t)⎟
                                  ⎜            ╱ a⋅ρ_g⋅u(t)                    ⎟
                                  ⎜      √2⋅  ╱  ──────────  + 1.0             ⎟
                                  ⎝         ╲╱      μ_g                        ⎠

    2
     (t) + g⋅m_p

```python

```


# Taylor Approximation and Newton's Method

## Taylor Approximation
For analytic function equivalence between value of a function at each point or value of all derivatives at a single point.

We consider approximating the function
$$h(t) = t^2 e^{3t}$$
in the neighbourhood of some value $t=t_0$.  The first-order Taylor approximation to $h(t)$ is
$$h_1(t) = h(t_0) + h'(t_0) (t-t_0),$$
where $h'(t)$ is the derivative of $h(t)$.

We define the symbolic function $h(t)$ and find its symbolic derivative $h'(t)$:

```python
import sympy as sp
sp.init_printing()
from IPython.display import display

t = sp.symbols('t');
h = sp.exp(3*t)*t**2;  # h(t)
hp = h.diff(t);  # h'(t)
hp
```

$\displaystyle 3 t^{2} e^{3 t} + 2 t e^{3 t}$

We want to evaluate $h(t_0)$ and $h'(t_0)$ so that we can use the approximation.  We don't yet have to put in a value so we just substitute the symbol `t0`.  The result will then be a symbolic expression in `t` and `t0`:

```python
t0 = sp.symbols('t0');
h0 = h.subs(t,t0);  # h(t0)
h1 = h0 + hp.subs({t:t0})*(t-t0);  # h1(t) Taylor approximation at t0
display(h1);
```

$\displaystyle t_{0}^{2} e^{3 t_{0}} + \left(t - t_{0}\right) \left(3 t_{0}^{2} e^{3 t_{0}} + 2 t_{0} e^{3 t_{0}}\right)$

For a fixed value for `t0` we can plot the symbolic function $h(t)$ and the first-order Taylor expansion $h'(t)$ at that point:

```python
h1s = h1.subs(t0, 1);  # h1(t) for value t0=1
sp.plot(h, h1s, (t,0.6,1.3));
```

The previous cell used a symbolic plotting function.  We could instead convert the symbolic expressions to functions that can be evaluated numerically using

```python
lam_h = sp.lambdify(t, h, modules=['numpy'])
lam_h1s = sp.lambdify(t, h1s, modules=['numpy'])
```

and then plot from samples of `tv`:

```python
import numpy as np
import matplotlib.pyplot as plt

tv = np.linspace(0.6, 1.3, 100);
h_vals = lam_h(tv);  h1s_vals = lam_h1s(tv)
fig, ax = plt.subplots(1,1);
ax.plot(tv, h_vals, 'r-', tv, h1s_vals, 'g-');
```

Higher-order approximations are given by the series
$$h_n(t) = h(t_0) + h'(t_0)(t-t_0) + \frac{1}{2}h''(t_0)(t-t_0)^2 + \frac{1}{6}h'''(t_0)(t-t_0)^3 + ... \frac{1}{n!}h^{(n)}(t_0)(t-t_0)^n$$

We can write this as the following sum:
$$h_n(t) = \sum_{i=0}^{n}\frac{1}{n!}h^{(i)}(t_0)(t-t_0)^i$$

An expression for the $n$th-order taylor approximation of a symbolic function $h$ about a point $t_0$ can be calculated using the sympy function `series`:

`hnt = sp.series(h,t,t0,n).removeO`

The code below plots the first- and fifth-order approximations of $h(t) = t^2 e^{3t}$

```python
%run src/labX_preamble.py  # For internal notebook functions
# %%writefileexec src/lab_taylorexp-1.py -s  # dump cell to file before execute

import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Symbolic function and derivatives
h, t = sp.symbols('h t');
h = sp.exp(3*t)*t**2;
hp = h.diff(t);
#hpp = hp.diff(t);

# Taylor expansions around point t0
t0 = 1;
h0 = h.subs(t,t0);
h1 = h0 + hp.subs({t:t0})*(t-t0);
#h2 = h0 + hp.subs(t,t0)*(t-t0) + 1/2*hpp.subs(t,t0)*(t-t0)**2;

# Direct Taylor expansion using sympy
h5s = sp.series(h, t, t0, 6).removeO();
print("Taylor 5: ")
display(h5s)

# Convert symbolic to functions that can be evaluated
lam_h = sp.lambdify(t, h, modules=['numpy']);
lam_h1 = sp.lambdify(t, h1, modules=['numpy']);
lam_h5s = sp.lambdify(t, h5s, modules=['numpy']);

# Plots
fig, ax = plt.subplots(1,1);
t_vals = np.linspace(0.5, 1.5, 100);
ax.plot(t_vals, lam_h(t_vals), 'r');
ax.plot(t_vals, lam_h1(t_vals), 'g');
#ax.plot(t_vals, lam_h5s(t_vals), 'b');

# Symbolic plotting also probably works
#sp.plot(h, h1, (t, 0, 1.5))
```

## Newton's Method for finding roots

Suppose we want to solve the equation
$$\cos(x) = x^3$$
for the variable $x$.  This is equivalent to finding the root or zero of $h(x) = \cos(x) - x^3$.

In general the function $h(x)$ might be expensive to evaluate, so plotting it and looking for the zero is not feasible.  For this demonstration, though, we can make a graph of the function over some finite domain.  The cell below defines the equation as a symbolic expression and generates the plot.  We observe that the desired solution is in the $[0.5,1]$ interval.

```python
import sympy as sp

x = sp.symbols('x');
h = sp.cos(x) - x**3;
display(h);
sp.plot(h, (x,-1.5,1.5), ylim=(-2,2));
```

One might hope to just find the solution using a symbolic solver.  However, when I uncomment the code below and run it an error is thrown saying there is no algorithm implemented to solve an equation of this type.

```python
#sp.solve(h, x);
```

Only certain cagegories of functions can be handled by the solver.  Our $h$, with polynomial and trigonometric parts, doesn't have the required structure.

The function $h(x)$ is infinitely differentiable and analytic and can be locally approximated around $x_0$ by a Taylor series expansion.

This is a polynomial in $x$ and is quite tractible to deal with.  In particular, we know how to quite easily find the zeros of low order polynomials.

This suggests an iterative strategy for finding the zero of $h(x)$.  Start with a guess or initial estimate $x_0$.  Locally approximate $h(x)$ around this point using a low-order Taylor expansion, and find the zero for this approximation.  We know that the expansion is accurate for values of $x$ near $x_0$, so if the zero is very close to $x_0$ then it's a good solution and we can terminate.  Otherwise we hope that the zero of our approximation is close to the actual zero, move $x_0$ to this new location, and repeat the process.  This is the essence of Newton's method https://en.wikipedia.org/wiki/Newton's_method.

If a first-order Taylor approximation is used then given a center $x_0$ we have
$$h(x) \approx h(x_0) + h'(x_0)(x-x_0)$$
The right-hand side of this approximation is implemented symbolically in the cell below for the given $h$.

```python
hp = sp.diff(h, x);
x0 = sp.symbols('x0');
h1 = h.subs(x,x0) + hp.subs({x:x0})*(x-x0);  # first-order Taylor at x0
display(h1);
```

$\displaystyle - x_{0}^{3} + \left(x - x_{0}\right) \left(- 3 x_{0}^{2} - \sin{\left(x_{0} \right)}\right) + \cos{\left(x_{0} \right)}$

In the code below we plot the function and its first-order Taylor expansion for the case of $x_0=0.5$.

```python
x0c = 0.5;  # current x0 value
h1s = h1.subs(x0,x0c);
sp.plot(h, h1s, (x,-1.5,1.5), ylim=(-2,2));
display(h1s)
```

A linear equation is a special case of a polynomial, and the `roots` function knows how to find zeros of these.  There is only a single zero, which is found below.

```python
zs = sp.roots(h1s);
x0n = list(zs.keys())[0];
x0n
```

$\displaystyle 1.11214163709727$

This could also just have been calculated numerically directly from the Taylor approximation:  if at $x=x_0$ we know the function value $h(x_0) = a$ and the derivative $h'(x_0) = b$ then
$$h(x) \approx h(x_0) + h'(x_0)(x-x_0) = a + b(x-x_0),$$
which has a zero at
$$x =-a/b + x_0 = -\frac{h(x_0)}{h'(x_0)} + x_0.$$

```python
lam_h = sp.lambdify(x, h, modules=['numpy']);
lam_hp = sp.lambdify(x, hp, modules=['numpy']);
x0nn = -lam_h(x0c)/lam_hp(x0c) + x0c;
x0nn
```

    1.1121416370972725

Thus we see that in order to perform the update step all we need is a method of evaluating $h(x)$ and $h'(x)$ for some given $x_0$.  The lambda functions above obtained from the symbolic expression provide this utility, but if the expressions are known then could also just be coded directly.

The code below shows a sequence of plots demonstrating Newton's method for finding the root of $h(x)$ given an initial estimate of $x_0 = 0.5$.

```python
x0c = 0.5;  # initial
for iter in range(0,4):
    h1s = h1.subs(x0,x0c)
    sp.plot(h, h1s, (x,-1.5,1.5), ylim=(-2,2));
    zs = sp.roots(h1s);
    x0c = list(zs.keys())[0];
    print('Crossing: x=%f' % x0c);
```

# Tasks

These tasks involve writing code, or modifying existing code, to meet the objectives described.

1.  Generate Taylor series expansions of order 5, 9, and 13 centered around the point $t=0$ for the function $x(t) = \sin(t)$.  Plot all of these functions on the same set of axes over the domain $t=-2\pi$ to $t=2\pi$, with a range from $-1.5$ to $1.5$.

2. Write code to recursively find the solution to $\cos(x) = x^3$ using Newton's method.  Iterate the procedure described until the change in the estimate of the zero location is small enough. Start the process from the initial point $x_0=0.5$. Easy mode: show five iterations. Challenge: stop when the magnitude of the change in the zero location drops below the threshold $1e-4$.

3. The method described uses a very rough first-order approximation to the function, and a large number of low-complexity iterations will need to be done to find the root.  You could imagine that it might be better to use a higher-order approximation, and do a smaller number of higher-complexity iterations.  Generate a single plot of the function $h(x)$ and of the second-order Taylor approximation with center point $x_0 = 0.5$, and find the roots for the approximation.

Developing a numerical method around this approach is not trivial, though, mainly because the quadratic approximation to the function could have two roots (which one do you choose?) or zero roots (what do you do then?).

4.  Use Newton's method to find the value of $\sqrt{13}$.  You should have confidence in your solution to 5 decimal places.

```python

```


# Cálculo

```python
from sympy import *
```

```python
x = Symbol('x')
```

DERIVADAS

```python
x**2 + sin(x)
```

    x**2 + sin(x)

```python
# Para arrumar a função.
init_printing(pretty_print=True)
```

```python
x**2 + sin(x)
```

$$x^{2} + \sin{\left (x \right )}$$

```python
# Para calcular a Derivada.
(x**2 + sin(x)).diff(x)
```

$$2 x + \cos{\left (x \right )}$$

```python
# Para calcular a 2ª Derivada.
(x**2 + sin(x)).diff(x,2)
```

$$- \sin{\left (x \right )} + 2$$

```python
# Para ver a expressão arrumada.
Derivative(x**2 + sin(x))
```

$$\frac{d}{d x} \left(x^{2} + \sin{\left (x \right )}\right)$$

```python
# Para calcular a Derivada.
Derivative(x**2 + sin(x)).doit()
```

$$2 x + \cos{\left (x \right )}$$

INTEGRAL INDEFINIDA

```python
# Para calcular a Integral.
integrate(x**2 + sin(x))
```

$$\frac{x^{3}}{3} - \cos{\left (x \right )}$$

```python
# Integral antes de resolver.
Integral(x**2 + sin(x))
```

$$\int \left(x^{2} + \sin{\left (x \right )}\right)\, dx$$

```python
# Para resolver a integral.
Integral(x**2 + sin(x)).doit()
```

$$\frac{x^{3}}{3} - \cos{\left (x \right )}$$

```python
# Integral antes de resolver arrumada.
Integral(x**2 + 2 + sin(x) * exp(x))
```

$$\int \left(x^{2} + e^{x} \sin{\left (x \right )} + 2\right)\, dx$$

```python
# Para resolver a Integral.
Integral(x**2 + 2 + sin(x) * exp(x)).doit()
```

$$\frac{x^{3}}{3} + 2 x + \frac{e^{x} \sin{\left (x \right )}}{2} - \frac{e^{x} \cos{\left (x \right )}}{2}$$

INTEGRAL DEFINIDA

```python
# Integral definida antes de resolver arrumada.
Integral(x**2 + 2 + sin(x) * exp(x),(x, 0, 2))
```

$$\int_{0}^{2} \left(x^{2} + e^{x} \sin{\left (x \right )} + 2\right)\, dx$$

```python
# Para resolver a Interal definida.
Integral(x**2 + 2 + sin(x) * exp(x),(x, 0, 2)).doit()
```

$$- \frac{e^{2} \cos{\left (2 \right )}}{2} + \frac{e^{2} \sin{\left (2 \right )}}{2} + \frac{43}{6}$$

```python
# Resultado numéerico.
Integral(x**2 + 2 + sin(x) * exp(x),(x, 0, 2)).doit().evalf(4)
```

$$12.06$$

```python
# Símbolo do ifinito, digitar 2 ós.
oo
```

$$\infty$$

```python
-oo
```

$$-\infty$$


# Characterization of Discrete Systems

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Linear Convolution

It was shown previously, that the convolution is an important operation in the theory of signals and linear time-invariant (LTI) systems. The linear convolution of two discrete signals $s[k]$ and $g[k]$ is defined as

\begin{equation}
\sum_{\kappa = -\infty}^{\infty} s[\kappa] \cdot g[k - \kappa] = s[k] * g[k]
\end{equation}

where $*$ is a common short-hand notation of the convolution. The general properties of the linear convolution are discussed, followed by a geometrical interpretation of the operation.

### Properties

For the discrete signals $s[k]$, $g[k]$, $h[k] \in \mathbb{C}$ the convolution shows the following properties

1. The Dirac impulse is the [identity element](https://en.wikipedia.org/wiki/Identity_element) of the convolution
    \begin{equation}
    s[k] * \delta[k] = s[k]
    \end{equation}

2. The convolution is [commutative](https://en.wikipedia.org/wiki/Commutative_property)
    \begin{equation}
    s[k] * g[k] = g[k] * s[k]
    \end{equation}

3. The convolution is [associative](https://en.wikipedia.org/wiki/Associative_property)
    \begin{equation}
    \left( s[k] * g[k] \right) * h[k] = s[k] * \left( g[k] * h[k] \right)
    \end{equation}

5. The convolution is [distributive](https://en.wikipedia.org/wiki/Distributive_property)
    \begin{equation}
    s[k] * \left( g[k] + h[k] \right) = s[k] * g[k] + s[k] * h[k]
    \end{equation}

5. Multiplication with a scalar $a \in \mathbb{C}$
    \begin{equation}
    a \cdot \left( s[k] * g[k] \right) = \left( a \cdot s[k] \right) * g[k] = s[k] * \left( a \cdot g[k] \right)
    \end{equation}

The first property is a consequence of the sifting property of the Dirac pulse, the second to fifth property can be proven by considering the definition of the convolution.

### Geometrical Interpretation

The convolution can be [interpreted in a graphical manner](https://en.wikipedia.org/wiki/Convolution#Visual_explanation). This provides valuable insights into its calculation and allows to estimate the result. The calculation of the linear convolution

\begin{equation}
y[k] = x[k] * h[k] = \sum_{\kappa = -\infty}^{\infty} x[\kappa] \cdot h[k - \kappa]
\end{equation}

can be decomposed into three steps:

1. substitute $k$ by $\kappa$ in both $x[k]$ and $h[k]$,

2. time-reverse $h[\kappa]$ (reflection at vertical axis),

3. shift $h[- \kappa]$ by $k$ to the right to yield $h[k - \kappa]$, check if it overlaps with $x[\kappa]$ and sum up the product of the overlapping samples.

**Example**

The procedure is illustrated with the signals

\begin{align}
h[k] &= \epsilon[k] \cdot e^{- \frac{k}{2}} \\
x[k] &= \text{rect}_N[k]
\end{align}

for $N=6$. Before proceeding, some helper functions are defined.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def heaviside(k):
    return np.where(k >= 0, 1.0, 0.0)

def rect(k, N):
    return np.where((0 <= k) & (k < N), 1.0, 0.0)

def plot_signals(k, x, h, xlabel, hlabel, klabel):
    plt.figure(figsize=(8,4))
    plt.stem(k, x, linefmt='b-', markerfmt='bo', label=xlabel)
    plt.stem(k, h, linefmt='r-', markerfmt='ro', label=hlabel)
    plt.xlabel(klabel)
    plt.legend()
    plt.ylim([0, 1.5])
```

Now let's compute and plot the signals $x[k]$ and $h[k]$

```python
k = np.arange(-20, 20)
h = heaviside(k) * np.exp(- k/2)
x = rect(k, 6)

plot_signals(k, x, h, '$x[k]$', '$h[k]$', '$k$')
```

The **first step** is to substitute $k$ by $\kappa$ in both $x[k]$ and $h[k]$ to yield $x[\kappa]$ and $h[\kappa]$

```python
plot_signals(k, x, h, '$x[\kappa]$', '$h[\kappa]$', '$\kappa$')
```

The **second step** is to time-reverse $h[\kappa]$ to yield $h[-\kappa]$

```python
h2 = heaviside(-k) * np.exp(k/2)

plot_signals(k, x, h2, '$x[\kappa]$', '$h[-\kappa]$', '$\kappa$')
```

The impulse response $h[-\kappa]$ is shifted by $k$ samples to the right in the **third step** to yield $h[k - \kappa]$. This is illustrated for $k = -5$

```python
h3 = heaviside(-(k+5)) * np.exp((k+5)/2)

plot_signals(k, x, h3, '$x[\kappa]$', '$h[5 -\kappa]$', '$\kappa$')
```

It now becomes evident that with respect to the overlap of $h[k -\kappa]$ and $x[\kappa]$ three cases have to be considered:

1. $k<0$: no overlap
2. $0 \leq k < N-1$: partial overlap
3. $k \geq N$: full overlap

The evaluation of the summation for the three cases is left open as an exercise.

### Finite-Length Signals

The length of a discrete signal $x[k]$ is defined as the total number of samples in between the first sample which is not zero and the last sample which is zero, plus one. A signal of finite-length or finite-length signal is a signal whose length is finite. The Dirac impulse $\delta[k]$ is for instance a finite-length signal of length one.

The convolution of two finite-length signals is of practical importance since the convolution can only be evaluated numerically for finite-length signals. Any infinite-length signal can be truncated to a finite length by multiplying it with the rectangular signal $\text{rect}_N[k]$. It is hence sufficient to consider the convolution of two rectangular signals of length $N \in \mathbb{N}$ and $M \in \mathbb{N}$

\begin{equation}
x[k] = \text{rect}_N[k] * \text{rect}_M[k]
\end{equation}

in order to derive insights on the convolution of two arbitrary finite-length signals. By following above geometrical interpretation of the linear convolution, the result for $N \leq M$ can be found as

\begin{equation}
x[k] = \begin{cases}
0 & \text{for }  k < 0 \\
k+1 & \text{for } 0 \leq k < N \\
N & \text{for } N \leq k < M \\
N+M-1-k & \text{for } M \leq k < N+M-1\\
0 & \text{for } k \geq N+M
\end{cases}
\end{equation}

The convolution of two rectangular signals results in a finite-length signal. The length of the signal is $N+M-1$. This result can be generalized to the convolution of two arbitrary finite-length signals by following above reasoning. The convolution of two finite-length signals with length $N$ and $M$, respectively results in a finite-length signal of length $N+M-1$.

For two causal signals $s[k]$ and $g[k]$ of finite length $N$ and $M$ the convolution reads

\begin{equation}
s[k] * g[k] = \sum_{\kappa = 0}^{N-1} s[\kappa] \cdot g[k - \kappa] = \sum_{\kappa = 0}^{M-1} s[k - \kappa] \cdot g[\kappa]
\end{equation}

for $0 \leq k < N+M-1$. The computation of each output sample requires at least $N$ multiplications and $N-1$ additions. The numerical complexity of the convolution for the computation of $N$ output samples is therefore [in the order of](https://en.wikipedia.org/wiki/Big_O_notation) $\mathcal{O} ( N^2 )$.

**Example**

The convolution of two rectangular signals $x[k] = \text{rect}_N[k] * \text{rect}_M[k]$ of length $N$ and $M$ is computed in the following. The resulting signal is plotted for illustration.

```python
N = 7
M = 15

x = np.convolve(np.ones(N), np.ones(M), mode='full')

plt.stem(x)
plt.xlabel('$k$')
plt.ylabel('$x[k]$')
plt.ylim([0, N+.5]);
```

**Exercise**

* Compute the convolution of two rectangular signals of equal length analytically. Check your results by modifying the numerical example.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.