# Flipping USB Connectors

*If you like this article, you might also like the second edition of [*Think Bayes*](https://greenteapress.com/wp/think-bayes/).*

[Click here to run this article on Colab](https://colab.research.google.com/github/AllenDowney/ThinkBayes2/blob/master/examples/usb.ipynb)

I am not the first person to observe that it sometimes takes several tries to plug in a USB connector (specifically the rectangular [Type A connector](https://en.wikipedia.org/wiki/USB), which is not reversible).
On Quora alone,
[people](https://www.quora.com/Why-do-you-always-have-to-flip-a-USB-twice-when-trying-to-plug-it-in-Shouldn-t-it-have-gone-in-the-first-time-when-you-had-it-the-right-way)
[have](https://www.quora.com/There-is-a-50-chance-of-plugging-in-a-USB-cable-the-right-way-on-the-first-try-Why-does-it-only-happen-20-of-the-time-when-I-do-it)
[asked](https://www.quora.com/Why-doesnt-the-USB-go-in-the-first-time-but-does-after-youve-turn-it-a-few-times)
[about](https://www.quora.com/People-that-can-plug-the-USB-in-on-their-first-try-How)
[it](https://www.quora.com/Why-cant-we-plug-in-USB-devices-correctly-the-first-time) more than once.

But I might be the first to figure out the optimal strategy for plugging in a USB connector. Specifically, I have worked out how long you should try on the first side before flipping, and how long you should try on the second side before flipping again.

Of course, my analysis is based on some modeling assumptions:

1. Initially, the probability is 0.5 that the connector is in the right orientation.

2. If it is, the time it takes to succeed follows an exponential distribution with rate parameter 1.1, which means that it takes 0.91 seconds on average.

3. Flipping the connector takes 0.1 seconds.

With that, we are ready to get started.

## Continuous Updates

The first step is to figure out the probability that the connector is in the right orientation as a function of how long you have been trying.
For that, we can use a Bayes table, which is a form of Bayes's Theorem I use in [Chapter 2 of *Think Bayes*](https://allendowney.github.io/ThinkBayes2/chap02.html#bayes-tables).

The following function takes a sequence of hypotheses, prior probabilities, and likelihoods, and returns a pandas `DataFrame` that represents a Bayes table.

```python
import pandas as pd

def bayes_table(hypos, prior, likelihood):
    """Make a table showing a Bayesian update."""
    table = pd.DataFrame(dict(prior=prior, likelihood=likelihood), index=hypos)
    table['unnorm'] = table['prior'] * table['likelihood']
    prob_data = table['unnorm'].sum()
    table['posterior'] = table['unnorm'] / prob_data
    return table
```

Now suppose that the prior probability is 0.5 that the orientation of the connector is correct, and you have been trying for 0.9 seconds.
What is the likelihood that you would have to try so long?

* If you are on the wrong side, it is 100%.

* If you are on the right side, it's given by the survival function (complementary CDF) of the exponential distribution, which is $\exp(-\lambda t)$, where $\lambda$ is the rate parameter and $t$ is time.

The following function computes this likelihood:

```python
import numpy as np

def expo_sf(t, lam):
    """Survival function of the exponential distribution."""
    return np.exp(-lam * t)
```

We can use this function to compute the likelihood of trying for 0.9 seconds or more, given $\lambda=1.1$.

```python
t = 0.9
lam = 1.1

expo_sf(t, lam)
```

    0.37157669102204566

Now let's make a Bayes table with two hypotheses -- the connector is either the right way or the wrong way -- with equal prior probabilities.

```python
hypos = ['Right way', 'Wrong way']
prior = [1/2, 1/2]
```

And here is the likelihood of the data for each hypothesis:

```python
likelihood = [expo_sf(t, lam), 1]
```

Putting it together, here's the Bayes table.

```python
bayes_table(hypos, prior, likelihood)
```

      prior
      likelihood
      unnorm
      posterior

      Right way
      0.5
      0.371577
      0.185788
      0.270912

      Wrong way
      0.5
      1.000000
      0.500000
      0.729088

After 0.9 seconds, the probability is about 73% that the orientation of the connector is wrong, so you might want to think about trying the other side.

But if it takes 0.1 seconds to flip, maybe you should keep trying a little longer. To figure out when to flip, let's do the same analysis again for general values of $\lambda$ and $t$.

## Generalization

To minimize human error, I'll use Sympy to do the algebra. Here are the symbols I'll use.

```python
from sympy import symbols, exp

t, lam, p, q, r = symbols('t lam p q r')
```

Here's the likelihood again, using the symbols.

```python
likelihood = [exp(-lam * t), 1]
likelihood
```

    [exp(-lam*t), 1]

And here's the Bayes table, using $p$ and $q$ for the prior probabilities of the hypotheses.

```python
prior = [p, q]
table = bayes_table(hypos, prior, likelihood)
table
```

      prior
      likelihood
      unnorm
      posterior

      Right way
      p
      exp(-lam*t)
      p*exp(-lam*t)
      p*exp(-lam*t)/(p*exp(-lam*t) + q)

      Wrong way
      q
      1
      q
      q/(p*exp(-lam*t) + q)

From the table I'll select the posterior probability that the orientation is correct.

```python
expr = table.loc['Right way', 'posterior']
expr.simplify()
```

$\displaystyle \frac{p}{p + q e^{lam t}}$

You might recognize this as a form of the [logistic function](https://en.wikipedia.org/wiki/Logistic_function); we can compute it like this:

```python
def logistic(p, lam, t):
    q = 1-p
    return p / (p + q * np.exp(lam * t))
```

Let's see what that looks like for a range of values of `t`, assuming that the prior probability is `p=0.5`.

```python
import matplotlib.pyplot as plt

ts = np.linspace(0, 4)
ps = logistic(p=0.5, lam=1.1, t=ts)

plt.plot(ts, ps)
plt.xlabel("How long you've been trying (s)")
plt.ylabel("Probability the orientation is right");
```

After a few seconds of fiddling, you should be reasonably convinced that the orientation is wrong.

## Strategy

Now, let's think about turning belief into action. Let me start with a conjecture: I suspect that the best strategy is to try on the first side until the probability of correct orientation drops below some threshold (to be determined), then try on the second side until the probability drops below that threshold again, and repeat until success.

To test this strategy, we will have to figure out how long to try as a function of the prior probability, `p`, and the threshold probability, `r`. Again, I'll make Sympy do the work.

Here's the equation that sets the posterior probability, which we computed in the previous section, to `r`.

```python
from sympy import Eq, solve

eqn = Eq(expr, r)
eqn
```

$\displaystyle \frac{p e^{- lam t}}{p e^{- lam t} + q} = r$

And here's the solution for `t` in terms of `p`, `q`, `r`, and `lam`.

```python
solve(eqn, t)[0]
```

$\displaystyle \frac{\log{\left(\frac{p \left(1 - r\right)}{q r} \right)}}{lam}$

And here's how we can express this solution in terms of the prior and posterior odds.

```python
def wait_time(p, lam, r):
    q = 1-p
    prior_odds = p / q
    posterior_odds = r / (1-r)
    return np.log(prior_odds / posterior_odds) / lam
```

Let's see what that looks like for a range of values of `r`, assuming that the prior probability is `p=0.5`.

```python
rs = np.linspace(0.05, 0.5)
ts = wait_time(0.5, 1.1, rs)

plt.plot(rs, ts)
plt.xlabel("Probability the orientation is right")
plt.ylabel("How long to keep trying (s)");
```

When the threshold is low, we have to wait a few seconds to reach it. As the threshold increases, the time to reach it decreases.
We'll use this function in the next section to simulate the strategy.

## Simulation

As a step toward optimization, let's run a simulation. The following function takes as parameters:

* `correct`: A Boolean indicating if the orientation is correct.
* `p`: The prior probability that the orientation is correct.
* `lam`: The rate parameter for the distribution of time until success.
* `r`: The threshold for the posterior probability.
* `flip`: The time it takes to flip the connector, in seconds.
* `trace`: A list that indicates how much time we have spent, so far, trying and flipping.

It runs the simulation and returns a sequence of waiting and flipping times. The sum of this sequence is the total time it took to connect. And we can use the length of the sequence to figure out how many times we had to flip.

```python
def simulate(correct, p, lam, r, flip, trace):
    # figure out the maximum time we should try before flipping
    wait = wait_time(p, lam, r)

    # if we're on the correct side, see if we succeed before time's up
    if correct:
        t = np.random.exponential(1/lam)
        if t < wait:
            # if so, update and return the trace
            return trace + [t]

    # if time expired, add the wait time and flip time to the trace
    # and make a recursive call to continue the simulation
    return simulate(not correct, 1-r, lam, r, flip, trace + [wait, flip])
```

Here's a test run, starting on the correct side.

```python
simulate(correct=True, p=0.5, lam=1.1, r=0.2, flip=0.1, trace=[])
```

    [0.48602719926750154]

And here's a run where we start on the wrong side.

```python
simulate(correct=False, p=0.5, lam=1.1, r=0.2, flip=0.1, trace=[])
```

    [1.2602676010180822, 0.1, 0.5625173576253926]

The following function runs the simulation many times with initial probability `p=0.5`, starting in the right orientation half the time.

It returns two arrays, containing the length of the trace and the total duration for each simulation.

```python
def run_simulations(lam, r, flip, iters=20000):
    res = []
    for i in range(iters):
        correct = i%2
        trace = simulate(correct, 0.5, lam, r, flip, [])
        res.append((len(trace), sum(trace)))

    return np.transpose(res)
```

Here's the average total duration with threshold probability `r=0.25`.

```python
lengths, totals = run_simulations(lam=1.1, r=0.25, flip=0.1)
totals.mean()
```

    2.008325460038306

With this threshold, it takes about 2 seconds to connect, on average.

## Optimization

Now let's see how the average duration varies as we sweep through a range of values for the threshold probability, `r`:

```python
rs = np.linspace(0.15, 0.4, 21)
rs
```

    array([0.15  , 0.1625, 0.175 , 0.1875, 0.2   , 0.2125, 0.225 , 0.2375,
           0.25  , 0.2625, 0.275 , 0.2875, 0.3   , 0.3125, 0.325 , 0.3375,
           0.35  , 0.3625, 0.375 , 0.3875, 0.4   ])

```python
np.random.seed(17)

res = []
for r in rs:
    lengths, totals = run_simulations(lam=1.1, r=r, flip=0.1)
    res.append((r, totals.mean()))
```

```python
from statsmodels.nonparametric.smoothers_lowess import lowess

def make_lowess(series):
    """Use LOWESS to compute a smooth line.

    series: pd.Series

    returns: pd.Series
    """
    endog = series.values
    exog = series.index.values

    smooth = lowess(endog, exog)
    index, data = np.transpose(smooth)

    return pd.Series(data, index=index)
```

```python
def plot_series_lowess(series, color):
    """Plots a series of data points and a smooth line.

    series: pd.Series
    color: string or tuple
    """
    series.plot(lw=0, marker='o', color=color, alpha=0.5)
    smooth = make_lowess(series)
    smooth.plot(label='_', color=color)
```

Here's what the results look like.

```python
rs, ts = np.transpose(res)
series = pd.Series(ts, rs)

plot_series_lowess(series, 'C1')

plt.xlabel("Threshold probability where you flip (r)")
plt.ylabel("Average total duration (seconds)");
```

The optimal value of `r` is close to `0.3`, maybe `0.29`.

With that threshold we can see how long we should try on the first side, starting with prior probability `p=0.5`.

```python
r_opt = 0.29
wait_time(p=0.5, lam=1.1, r=r_opt)
```

    0.813985497322583

With the given values of `lam` and `flip`, it turns out the optimal time to wait is about `0.8` seconds.

If we have to flip, the prior probability for the second side is `p=1-r`, so we have to wait twice as long for the posterior probability to get down to `r`.

```python
wait_time(p=1-r_opt, lam=1.1, r=r_opt)
```

    1.627970994645166

## How many flips?

Now let's run the simulations with the optimal value of `r` and see what the distributions look like for the total time and the number of flips.

```python
lengths, totals = run_simulations(lam=1.1, r=r_opt, flip=0.1)
```

Here's the distribution of total time, represented as a CDF.

```python
from empiricaldist import Cdf

Cdf.from_seq(totals).plot()

plt.xlabel('Total time to connect (seconds)')
plt.ylabel('CDF')
plt.title('Distribution of total time to connect');
```

```python
totals.mean()
```

    2.0228610633673836

The average is 2 seconds, but occasionally it takes much longer!

And here's the distribution for the total number of flips.

```python
from empiricaldist import Pmf

flips = (lengths-1) // 2
Pmf.from_seq(flips).bar()

plt.xlabel('How many times you have to flip')
plt.ylabel('PMF')
plt.title('Distribution of number of flips');
```

The probability of getting it right on the first try is only about 30%. That might seem low, because the chance of starting in the right orientation is 50%, but remember that we have a substantial chance of flipping even if we start in the right orientation (and in that case, we have to flip at least twice).

The most common outcome is that we have to flip once, about 40% of the time. And the probability of the notorious double flip is 15-20%.

Fortunately, it is rare to flip three or more times.

## Summary

With that, I think we have solved the USB connector problem.

1. For given parameters `lam` and `flip`, we can find the threshold probability, `r`, that minimizes the average time to connect.

2. Given this optimal value, we can estimate the distribution of total time and the number of times we have to flip.

Sadly, all of this fun is gradually being spoiled by the encroachment of the [USB-C](https://en.wikipedia.org/wiki/USB-C) connector, which is reversible.

*If you like this article, you might also like the second edition of [*Think Bayes*](https://greenteapress.com/wp/think-bayes/).*

Copyright 2021 Allen Downey

Code: [MIT License](https://opensource.org/licenses/MIT)

Text: [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)

```python

```


```python
## Sympy stuff

from sympy import *
x = symbols('x')
y = symbols('y')

# themes here: https://visualcomputing.github.io/colors/#/themes
```

# Areas between curves
**([Syllabus Unit 3](https://canvas.ubc.ca/courses/92840/syllabus) | [CLP2 Chapter 1.5](https://personal.math.ubc.ca/~CLP/CLP2/))**

## Introduction

### Announcements

- HW03 is now open

### Last Class

### About Me

[](https://firas.moosvi.com)

## Area under a curve

**Recall**: If $f$ is integrable and $f \geq 0$ on the interval $[a,b]$,

\begin{align}
\int_{a}^{b}f(x) dx= A,
\end{align}

this Area $A$, lies above $y=0$ and below $y=f(x)$, in the vertical strip $a\le x\le b.$

### General case: area between two curves

**Today**: We can extend this to a more general situation by replacing the $x$-axis ($y=0$) with a general lower boundary $y=g(x)$.

*Note: Setting $g(x)=0$ will recapture the original definition.*

### Leibniz-style shorthand:

$A = \int\!dA$, with $dA = (f(x)-g(x))\,dx$ over $a\leq x\leq b$.

### An intuitive description

```python
%%html

```

## Motivation

- Often when we're in school, and going through the rigours of daily life, we lose sight of the bigger picture, the important things in life.
- Things like social justice, fighting for equity, righting the wrongs of the world, and making the world a better place.
- That is why, I take every opportunity to make connections between what we're doing in class, and the real world.
- Before we get into the mechanics of computing areas between curves, let's look at this practical example.

[Source: Our World in Data](https://ourworldindata.org/income-inequality#the-gini-coefficient)

### Additional Reading

- [Applications of Area Between Curves, Section 6.5](https://math.hws.edu/~mitchell/Math131F15/tufte-latex/BetwCurves2.pdf)

## Computing Area under two curves

### Example I.1

Compute the finite area between the curves $f(x)= x+2$ and $g(x)=x-2$ from $x=0$ to $x=3$.

### Strategy

1. Given $f(x)$ and $g(x)$, integrate $f(x) - g(x)$ from $x=a$ to $x=b$ to determine the area between the curves, $A$.

    $$
    A = \int_a^b f(x) - g(x) dx
    $$

```python
f = x+2
g = x-2

a = 0 # Lower x-limit in the domain of interest
b = 3 # Upper x-limit in the domain of interest
```

```python
integrate(f)
```

$\displaystyle \frac{x^{2}}{2} + 2 x$

```python
integrate(g)
```

$\displaystyle \frac{x^{2}}{2} - 2 x$

```python
integrate(f) - integrate(g) # integrate f(x) - g(x)
```

$\displaystyle 4 x$

```python
integrate(f-g,x) # Note, this gives the same answer as above!
```

$\displaystyle 4 x$

```python
integrate(f,(x,0,3)) # integrate f(x) with x ranging from 0 to 3
```

$\displaystyle \frac{21}{2}$

```python
integrate(g,(x,0,3)) # integrate g(x) with x ranging from 0 to 3
```

$\displaystyle - \frac{3}{2}$

```python
# Final answer

A_1 = integrate(f,(x,0,3)) - integrate(g,(x,0,3))
```

The Area between $f(x) = x+2$ and $g(x) = x-2$ , between x=0 and x=3 is:

```python
A_1
```

$\displaystyle 12$

## Consideration 1: Does it matter which function is greater?

### Example I.1B

Compute the finite area between the curves $f(x)= x+2$ and $g(x)=x-2$ from $x=0$ to $x=3$.

In the previous example, what if we had gotten the order wrong, and done

$$
A = \int_a^b g(x) - f(x) dx
$$

instead?

```python
integrate(g,(x,0,3)) - integrate(f,(x,0,3))
```

$\displaystyle -12$

Interesting! It seems that if we get the order wrong, our answer will be negative.

Intuitively, this makes sense. By convention, we will require that areas in the first quadrant (top-right of an $x-y$ axes) will be **positive**.

Let's update our strategy given what we just discovered:

```{admonition} Strategy
:class: tip
1. **Given $f(x)$ and $g(x)$, determine which function is greater.**

2. **Then, assuming $f(x)$ is greater**, integrate $f(x) - g(x)$ from $x=a$ to $x=b$ to determine the area between the curves, $A$.

    $$
    A = \int_a^b f(x) - g(x) dx
    $$
```

## Consideration 2: Solving problems where the functions intersect.

What if the curves $y=f(x)$ and $y=g(x)$ cross somewhere in the given interval?

*Hint*: Remember that “area” is always a positive quantity!

### Example I.2A

Compute the finite area between where the curves $f(x)= x+2$ and $g(x)=x^2-4$ intersect.

```python
f = x + 2
g = x**2 - 4

# a and b are currently unknown! We must set f = g and solve to figure out the roots.
```

```python
a,b = solve(f-g)
display(a,b)
```

$\displaystyle -2$

$\displaystyle 3$

```python
integrate(f)
```

$\displaystyle \frac{x^{2}}{2} + 2 x$

```python
integrate(g)
```

$\displaystyle \frac{x^{3}}{3} - 4 x$

```python
integrate(f,(x,a,b))
```

$\displaystyle \frac{25}{2}$

```python
integrate(g,(x,a,b))
```

$\displaystyle - \frac{25}{3}$

```python
# Final answer

A_2a = integrate(f,(x,a,b)) - integrate(g,(x,a,b))
```

The area between $x + 2$ and $x^2 - 4$ between $x=-2$ and $x=3$:

```python
A_2a
```

$\displaystyle \frac{125}{6}$

Let's update our strategy given what we just discovered:

### Strategy

1. **Roughly sketch the curves $f(x)$ and $g(x)$.**

2. **Determine the domain of interest; usually by setting $f(x) = g(x)$ and solving for the roots. Though, other operations may be needed as well depending on the question.**

3. Given $f(x)$ and $g(x)$, determine which function is greater **within the domain of interest**.

4. Then, assuming $f(x)$ is greater, integrate $f(x) - g(x)$ from $x=a$ to $x=b$ to determine the area between the curves, $A$.

$$
A = \int_a^b f(x) - g(x) dx
$$

## Consideration 3: What if the functions cross?

### Example I.2B

Compute the finite area between the curves $f(x)= x+2$ and $g(x)=x^2-4$ between $x=-2$ and $x=4$.

Notice that we have already computed the area between $f(x)$ and $g(x)$ between $x=-2$ and $x=3$, we called that $A_{2a}$.

To find the area between $f(x)$ and $g(x)$ between $x=-2$ and $x=4$, we first notice that we need to split our integral into two pieces:

1. Integral between $x=-2$ and $x=3$ (where $f(x)$ is greater):

$$
A_{2a} = \int_{-2}^{+3} [f(x) - g(x)]dx
$$

**and (added to)**

2. Integral between $x=3$ and $x=4$ (where $g(x)$ is greater):

$$
A_{2r} = \int_{3}^{4} [g(x) - f(x)]dx
$$

The combined integral is then:

$$
A_{2b} = A_{2a} + A_{2r}
$$

```python
# Final answer

a = 3
b = 4

A_2b = A_2a + (integrate(g,(x,a,b)) - integrate(f,(x,a,b)))
```

The area between $x + 2$ and $x^2 - 4$ between $x=-2$ and $x=4$:

```python
A_2b
```

$\displaystyle \frac{71}{3}$

Let's update our strategy given what we just discovered:

### Strategy

1. Roughly sketch the curves $f(x)$ and $g(x)$.

2. Determine the domain of interest; usually by setting $f(x) = g(x)$ and solving for the roots. Though, other operations may be needed as well depending on the question.

3. Given $f(x)$ and $g(x)$, determine which function is greater within the domain(s) of interest. **If the curves cross, split the integral so each segment is considered separately.**

4. **Region 1: Assuming $f(x)$ is greater in region 1, integrate $f(x) - g(x)$ from $x=a$ to $x=b$.**

    $$
    A_1 = \int_a^b f(x) - g(x) dx
    $$

5. **Region 2: If there are one or more function crosses, there will be multiple segments to integrate. Now, $g(x)$ will be greater in region 2, so determine $A_2$:**

    $$
    A_2 = \int_c^d g(x) - f(x) dx
    $$

6. **Compute integrals of each segment and after after each cross. Add all the area segments together:**

    $$
    A_{total} = A_1 + A_2 +... A_N
    $$

## Consideration 4: Should we integrate over $x$, or over $y$?

When the upper or lower boundary curve has a piecewise definition, creative geometry-inspired thinking can offer an efficient new approach. Here is an example:

### Example I.3

Find the finite area between the curves $y^2= 4x$ and $4x-3y=4$.

First, we need to re-write $y^2= 4x$ as $f(x)$, and and $4x-3y=4$ as $g(x)$.

With a bit of algebra,

$$
f(x) = \pm\sqrt{4x}
$$
$$
g(x) = \frac{4(x-1)}{3}
$$

So let's figure out $a$ and $b$ using the same strategy we used earlier, set $f(x) = g(x)$ and solve for the roots:

```python
f = sqrt(4*x)     # Let's call the top half of the parabola +f and the bottom half -f
g = (4/3) * (x-1)

a = solve((-f)-g)[0]
b = solve((+f)-g)[0]
```

```python
print(f"a = {a:.2f}\nb = {b:.0f}")
```

    a = 0.25
    b = 4

Now, we have two options.

- We can integrate over $x$ with vertical rectangles of width $dx$.

Or,

- We can integrate over $y$ with horizontal rectangles of height $dy$.

### Option 1: Integrate over x (the long way)

We can continue doing what we've done so far and integrate over $x$ with vertical rectangles.

First, we need to setup the limits of our integral. Note that there is something a bit tricky here when we go to determine the "greater" and "lower" curves.

- Between $x=0$ and the point $x=0.25$ ($a$), where $f(x)$ and $g(x)$ first intersect, the "greater" curve is the upper half of the parabola $f(x) = +\sqrt{4x}$, and the "lower" curve is the lower half of the parabola $f(x) = -\sqrt{4x}$.

- Between $x=0.25$ ($a$) and the second point of intersection $x=4$ ($b$), the "greater" curve is the upper half of the parabola $f(x)$, and the "lower" curve is $g(x)$.

So we need to split up our integral in two parts:

\begin{align}
A_{3x} = A_{3a} + A_{3b}\\
A_{3x} = \int_0^{a} (g(x)-f(x))dx + \int_a^{b} (f(x) - g(x))dx
\end{align}

```python
A_3a = integrate(f--f,(x,0,a))
```

```python
A_3a
```

$\displaystyle 0.333333333333333$

```python
A_3b = integrate(f-g,(x,a,b))
```

```python
A_3b
```

$\displaystyle 4.875$

```python
A_3x = A_3a + A_3b
```

The Area between $f(x) = \pm \sqrt{4x}$ and $g(x) = \frac{4(x-1)}{3}$ between $x =0.25$ and $x = 4$ is:

```python
A_3x
```

$\displaystyle 5.20833333333333$

or, equivalently,

$$
\frac{125}{24}.
$$

Phew! This was a bit painful, is there a faster way?

### Option 2: Integrate over $y$ (the quick way!)

Looking at the sketch, it looks like we can also integrate over $y$, with horizontal rectangles.

The advantage of integrating over $y$, is that our integral is quite simple, because the parabola $f(x)$ is greater than $g(x)$ within our domain of interest ($x=0.25$ and $x=4$).

With a bit of algebra, we can rearrange $f(x)$ and $g(x)$ to write $f(y)$ and $g(y)$:

$$
f(y) = \frac{y^2}{4}
$$

and,

$$
g(y) = \frac{3y}{4} + 1.
$$

Since we are interested in the area between $f(x)$ and $g(x)$, the curves are bounded by the $(x,y)$ points $(a,c)$ on the left, and $(b,d)$ on the right.
We already know $a$ and $b$, what remains is to determine $c$ and $d$, the $y$-coordinates at the intersection points.

$$
A_{3y} = \int_c^{d} (g(y)-f(y))dy
$$

Because our bounds are at the points where $f(y)$ and $g(y)$ intersect, we're in luck and we can use either function to determine the corresponding $y$-values $c$ and $d$.

Let's use $g(x)$:

$$
g(x) = \frac{4(x-1)}{3}.
$$

Substituting in $a=0.25$ on the left side,
$$
c = g(0.25) = \frac{4(0.25-1)}{3} + 1
$$

we get $c = -1$.

On the right side,

$$
d = g(b)
$$

Substituting in $b=4$,
$$
d = \frac{4(4-1)}{3}
$$

we get $d = 4$.

Now that we have the limits of the $y$-integration, we can complete the integral:

$$
A_{3y} = \int_{c=-1}^{d=4} (f(y) - g(y)) dy
$$

```python
c = -1
d = 4

f_y = y**2/4
g_y = 3*y/4 + 1

A_3y = integrate(g_y-f_y,(y,c,d))
```

The Area between $f(y) = \frac{y^2}{4}$ and $g(y) = \frac{3y}{4} + 1$ between $y = -1$ and $y = 4$ is:

```python
A_3y
```

$\displaystyle \frac{125}{24}$

Great! As expected, it's the same answer as we got if we integrated over $x$ instead of $y$.
A little bit less painful, but required a bit of foresight and a good sketch of the functions!

Let's update our Strategy one last time given what we just discovered...

### Strategy

1. Roughly sketch the curves $f(x)$ and $g(x)$.

2. Determine the domain of interest; usually by setting $f(x) = g(x)$ and solving for the roots. Though, other operations may be needed as well depending on the question.

3. **Decide if it is easier to integrate over $x$ (vertical rectangles) or $y$ (horizontal rectangles). If integrating over $y$, determine $f(y)$ and $g(y)$, and the *ranges* of interest (instead of the domains, if integrating over $x$).**

4. Given $f(x)$ and $g(x)$ (**or $f(y)$ and $g(y)$**), determine which function is greater within the domain(s) (**or range(s)**) of interest. If the curves cross, split the integral so each segment is considered separately.

    **Below this, if integrating over $y$, replace $f(x)$ with $f(y)$, $dx$ with $dy$, etc...**

5. Region 1: Assuming $f(x)$ is greater in region 1, integrate $f(x) - g(x)$ from $x=a$ to $x=b$.

    $$
    A_1 = \int_a^b f(x) - g(x) dx
    $$

6. Region 2: If there are one or more function crosses, there will be multiple segments to integrate. Now, $g(x)$ will be greater in region 2, so determine $A_2$:

    $$
    A_2 = \int_c^d g(x) - f(x) dx
    $$

7. Compute integrals of each segment and after after each cross. Add all the area segments together:

    $$
    A_{total} = A_1 + A_2 +... A_N
    $$

## Summary

1. Extending definition of "Area under curve" between $f(x)$ and $y=0$, to determine area between $f(x)$ and $g(x)$.

2. Discussing income inequality and a practical application of computing area between curves.

3. Mechanics of computing areas between curves, including determining intersection points and the domain or range of integration.

4. Consideration 1: Does it matter which function is greater?

5. Consideration 2: Solving problems where the functions intersect.

6. Consideration 3: What if the functions cross?

7. Consideration 4: Should we integrate over $x$, or over $y$?

8. Development of working strategy...

Here is the final strategy that we created together to solve area between curve problems.

*Note: You will need to add or revise this strategy as you learn more skills and tricks, and become more familiar with the mechanics of doing integrals!*

### Final Strategy

1. Roughly sketch the curves $f(x)$ and $g(x)$.

2. Determine the domain of interest; usually by setting $f(x) = g(x)$ and solving for the roots. Though, other operations may be needed as well depending on the question.

3. **Decide if it is easier to integrate over $x$ (vertical rectangles) or $y$ (horizontal rectangles). If integrating over $y$, determine $f(y)$ and $g(y)$, and the *ranges* of interest (instead of the domains, if integrating over $x$).**

4. Given $f(x)$ and $g(x)$ (**or $f(y)$ and $g(y)$**), determine which function is greater within the domain(s) (**or range(s)**) of interest. If the curves cross, split the integral so each segment is considered separately.

    **Below this, if integrating over $y$, replace $f(x)$ with $f(y)$, $dx$ with $dy$, etc...**

5. Region 1: Assuming $f(x)$ is greater in region 1, integrate $f(x) - g(x)$ from $x=a$ to $x=b$.

$$
A_1 = \int_a^b f(x) - g(x) dx
$$

6. Region 2: If there are one or more function crosses, there will be multiple segments to integrate. Now, $g(x)$ will be greater in region 2, so determine $A_2$:

$$
A_2 = \int_c^d g(x) - f(x) dx
$$

7. Compute integrals of each segment and after after each cross. Add all the area segments together:

$$
A_{total} = A_1 + A_2 +... A_N
$$

## You Try!

Your post-class activity is to solve this problem.

### Example I.4

Find the area of the region bounded by the curves $y=\sin x$, $y=\cos x$, $x=0$, and $x=\pi /2$.

*Hint, the answer is not 0!*

Thanks for your time and attention today!

- Dr. Moosvi


## Gaussian Process Regression
## Part I - Multivariate Gaussian Distribution
## 2nd Machine Learning in Heliophysics
## Boulder, CO
### 21 - 25 March 2022

### Enrico Camporeale (University of Colorado, Boulder & NOAA Space Weather Prediction Center)
#### enrico.camporeale@noaa.gov
This work is licensed under a Creative Commons Attribution 4.0 International License.

Gaussian Process is a powerful technique for regression and classification
+ It is a non-parametric method
+ It has a much simpler algorithm than parametric equivalents (neural networks, etc.)
+ But it is harder to understand...

The output of GP is a fully probabilistic prediction in terms of Gaussian distributions (mean and variance)

# References
## The bible of GP

Available online (legally!)
http://www.gaussianprocess.org/gpml/chapters/

We will cover mostly Chapter 2 (Regression), Chapter 4 (Covariance Functions), Chapter 5 (Hyperparamaters)

# Gaussian distribution

There are over 100 topics all named after Gauss
https://en.wikipedia.org/wiki/List_of_things_named_after_Carl_Friedrich_Gauss

## Starting with one variable

The Gaussian distribution is arguably the most ubiquitous distribution in statistics, physics, social sciences, economy, etc.
+ Central Limit Theorem
+ Thermodynamical equilibrium (Maxwell–Boltzmann distribution)
+ Brownian motion
+ etc.

Also called  Normal distribution

$$p(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

```python
%matplotlib inline
from ipywidgets import interactive
import matplotlib.pyplot as plt
import numpy as np
def f(sigma, mu):
    plt.figure(2)
    x = np.linspace(-10, 10, num=1000)
    plt.plot(x, 1/np.sqrt(2*np.pi)/sigma * np.exp(-0.5*(x-mu)**2/sigma**2))
    plt.ylim(-0.1, 1)
    plt.show()
interactive_plot = interactive(f, sigma=(0, 3.0), mu=(-3, 3, 0.5))
output = interactive_plot.children[-1]
output.layout.height = '350px'
interactive_plot
```

    interactive(children=(FloatSlider(value=1.5, description='sigma', max=3.0), FloatSlider(value=0.0, description…

Why does the peak of the distribution change?

The distribution is normalized:

 $$\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^\infty \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)dx=1$$

$$p(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

The mean (expectation) value of a random variable $x$ normally distributed is

$\mathbb{E}(x) = \int_{-\infty}^\infty p(x) x dx = \frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^\infty \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) x dx = \frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^\infty \exp\left(-\frac{z^2}{2\sigma^2}\right) (z+\mu) dz$ =

$$\mu$$

$$p(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

The variance of a random variable $x$ is defined as

$var(x) = \mathbb{E}(x^2) - \mathbb{E}(x)^2$

When $x$ is normally distributed

$\mathbb{E}(x^2) = \frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^\infty \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) x^2 dx = \sigma^2 + \mu^2$

$var(x) = \sigma^2$

## Gaussian distribution of 2 variables

If two variables $x$ and $y$ are independent, their  joint probability is

$$p(x,y) = p(x)p(y)$$

$$p(x,y) = \frac{1}{{4\pi}\sigma_x\sigma_y}\exp\left(-\frac{(x-\mu_x)^2}{2\sigma_x^2}-\frac{(y-\mu_y)^2}{2\sigma_y^2}\right)$$

```python
%matplotlib inline
def f(sigma_x, sigma_y):
    fig = plt.figure(figsize=(10, 10))
    xx, yy = np.mgrid[-10:10:0.2, -10:10:0.2]
    f = 1/(4*np.pi)/sigma_x/sigma_y * np.exp(-0.5*(xx**2/sigma_x**2+yy**2/sigma_y**2))
    ax = plt.axes(projection='3d')
    surf = ax.plot_surface(xx, yy, f, rstride=1, cstride=1, cmap='coolwarm', edgecolor='none')
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('PDF')
    fig.colorbar(surf, shrink=0.5, aspect=5) # add color bar indicating the PDF
interactive_plot = interactive(f, sigma_x=(0, 3.0), sigma_y=(0,3.0))
output = interactive_plot.children[-1]
interactive_plot
```

    interactive(children=(FloatSlider(value=1.5, description='sigma_x', max=3.0), FloatSlider(value=1.5, descripti…

A better way of displaying 2D distributions is by using contour lines (isocontours).

What family of curves are represented by this equation ?

$\frac{(x-\mu_x)^2}{2\sigma_x^2}+\frac{(y-\mu_y)^2}{2\sigma_y^2}=const$

```python
def f(sigma_x, sigma_y):
    fig = plt.figure(figsize=(7, 7))
    xx, yy = np.mgrid[-10:10:0.2, -10:10:0.2]
    f = 1/(4*np.pi)/sigma_x/sigma_y * np.exp(-0.5*(xx**2/sigma_x**2+yy**2/sigma_y**2))
    ax = fig.gca()
    ax.set_xlim(-10, 10)
    ax.set_ylim(-10, 10)
    cfset = ax.contourf(xx, yy, f, cmap='coolwarm')
    ax.imshow(np.rot90(f), cmap='coolwarm', extent=[-10,10,-10,10]),
    cset = ax.contour(xx, yy, f, colors='k')
    ax.clabel(cset, inline=1, fontsize=10)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
interactive_plot = interactive(f, sigma_x=(0, 3.0), sigma_y=(0,3.0))
output = interactive_plot.children[-1]
#output.layout.height = '500px'

```

```python
%matplotlib inline
interactive_plot
```

    interactive(children=(FloatSlider(value=1.5, description='sigma_x', max=3.0), FloatSlider(value=1.5, descripti…

# Matrix form

$$p(x,y) = \frac{1}{{4\pi}\sigma_x\sigma_y}\exp\left(-\frac{(x-\mu_x)^2}{2\sigma_x^2}-\frac{(y-\mu_y)^2}{2\sigma_y^2}\right)$$

The 2D normal distribution can be rewritten as

$$p(x,y) = \frac{1}{4\pi\sigma_x\sigma_y}\exp\left(-\frac{1}{2}\left(\begin{bmatrix}x \\ y \end{bmatrix} -  \begin{bmatrix}\mu_x \\ \mu_y \end{bmatrix}\right)^T \begin{bmatrix} \sigma_x^2 & 0 \\ 0 & \sigma_y^2 \end{bmatrix}^{-1} \left(\begin{bmatrix}x \\ y \end{bmatrix} -  \begin{bmatrix}\mu_x \\ \mu_y \end{bmatrix}\right) \right)$$

that is

$$p(x,y) = \frac{1}{4\pi|\boldsymbol{D}|^{1/2}}\exp\left(-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^T \boldsymbol{D}^{-1}(\boldsymbol{x}-\boldsymbol{\mu}) \right)$$

where $\boldsymbol{x} = \begin{bmatrix} x \\ y \end{bmatrix}$ , $\boldsymbol{\mu} = \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix}$, $\boldsymbol{D}=\begin{bmatrix} \sigma_x^2 & 0 \\ 0 & \sigma_y^2 \end{bmatrix}$

We can now introduce a rotation of the coordinates $(x,y)$ via a rotation matrix $\boldsymbol{R}$ such that
$\boldsymbol{x}\rightarrow\boldsymbol{Rx}$

$$p(x,y) = \frac{1}{4\pi|\boldsymbol{D}|^{1/2}}\exp\left(-\frac{1}{2}(\boldsymbol{Rx}-\boldsymbol{R\mu})^T \boldsymbol{D}^{-1}(\boldsymbol{Rx}-\boldsymbol{R\mu}) \right)$$

which finally reduces to
$$p(x,y) = \frac{1}{4\pi|\boldsymbol{\Sigma}|^{1/2}}\exp\left(-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu}) \right)$$

with $\boldsymbol{\Sigma}^{-1} = \boldsymbol{R^T}\boldsymbol{D}^{-1}\boldsymbol{R}$

$\boldsymbol{R}$ is a rotation matrix, so it is unitary: $\boldsymbol{R}\boldsymbol{R}^T=\boldsymbol{I}$, hence:

$$\boldsymbol{\Sigma} = \boldsymbol{R^T}\boldsymbol{D}\boldsymbol{R}$$

(proof: $\boldsymbol{I}=\boldsymbol{\Sigma}\boldsymbol{\Sigma}^{-1} = \boldsymbol{R}^T\boldsymbol{D}\boldsymbol{R}\boldsymbol{R}^T\boldsymbol{D}^{-1}\boldsymbol{R}=\boldsymbol{I}$)

This can now be generalized to any number of variables $D$, and we have then derived the general formula for a multivariate Gaussian distribution

$$p(x,y) = \frac{1}{(2\pi\boldsymbol)^{D/2}|{\Sigma}|^{1/2}}\exp\left(-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu}) \right)$$

for which there is always an appropriate tranformation of variables (rotation) that makes the variables independent.

The general rotation matrix for an angle $\theta$ in 2D is

$R=\begin{bmatrix}\cos\theta & -\sin\theta \\ \sin\theta & \cos\theta\end{bmatrix}$

```python
def f(sigma_x, sigma_y, theta=0):
    fig = plt.figure(figsize=(7, 7))
    xx, yy = np.mgrid[-5:5:0.1, -5:5:0.1]
    theta = theta /180*np.pi
    R=np.matrix([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
    D_inv=np.matrix([[1/sigma_x**2,0],[0, 1/sigma_y**2]])
    D=np.matrix([[sigma_x**2,0],[0, sigma_y**2]])
    Sigma = np.matmul(np.matmul(np.transpose(R),D),R)
    Sigma_inv = np.matmul(np.matmul(np.transpose(R),D_inv),R)
    f = 1/(4*np.pi)/sigma_x/sigma_y * np.exp(-0.5*(xx**2*Sigma_inv[0,0]+ 2*xx*yy*Sigma_inv[0,1]+yy**2*Sigma_inv[1,1]))
    ax = fig.gca()
    ax.set_xlim(-5, 5)
    ax.set_ylim(-5, 5)
    cfset = ax.contourf(xx, yy, f, cmap='coolwarm')
    ax.imshow(np.rot90(f), cmap='coolwarm', extent=[-10,10,-10,10]),
    cset = ax.contour(xx, yy, f, colors='k')
    ax.clabel(cset, inline=1, fontsize=10)
    ax.set_xlabel('x', fontsize=16)
    ax.set_ylabel('y', fontsize=16)
    ax.text(-4,3,np.core.defchararray.add('$\Sigma^{-1}=$\n',np.array_str(Sigma_inv)), fontsize=16)
    ax.text(-4,-3.5,np.core.defchararray.add('$\Sigma=$\n',np.array_str(Sigma)), fontsize=16)
interactive_plot = interactive(f, sigma_x=(0, 3.0), sigma_y=(0,3.0),theta=(0,180))
output = interactive_plot.children[-1]
```

```python
interactive_plot
```

    interactive(children=(FloatSlider(value=1.5, description='sigma_x', max=3.0), FloatSlider(value=1.5, descripti…

Something peculiar about these matrices?

They are symmetric!

What if instead we choose to use a matrix $\Sigma^{-1}$ that is NOT symmetric?

(Note: any matrix can be decomposed in the sum of a symmetric and an anti-symmetric matrix)

Exercise: show that the anti-symmetric part disappears from the exponent in the Gaussian

Hence: without loss of generality $\Sigma^{-1}$ can be taken as symmetric.

The inverse of a symmetric matrix is symmetric: $\Sigma$ is also symmetric

$\boldsymbol{\Sigma}$ is called the  Covariance matrix

$\boldsymbol{\Sigma}^{-1}$ is called the  Precision matrix

## Covariance

If we have a set of random variables $\boldsymbol{X}=\{X_1,X_2,\ldots,X_D\}$ the covariance between two variables is defined as:

$$cov(X_i,X_j)=\mathbb{E}[(X_i-\mathbb{E}[X_i]) (X_j-\mathbb{E}[X_j])]$$

and the covariance matrix is the corresponding matrix of elements $\boldsymbol{K}_{i,j}=cov(X_i,X_j)$. Hence the diagonal entries of the covariance matrix are the variances of each element of $\mathbf{X}$.

$\mathbf{\Sigma}=\begin{bmatrix}cov(X_1,X_1) & cov(X_1,X_2) & \cdots & cov(X_1,X_D) \\ cov(X_2,X_1) & cov(X_2,X_2) & \cdots & cov(X_2,X_D)\\ \vdots & \vdots & \vdots & \vdots\\ cov(X_D,X_1) & cov(X_D,X_2) & \cdots & cov(X_D,X_D) \end{bmatrix}$

Exercise: show that if two random variables $X$ and $Y$ are independent, their covariance is equal to zero.

## Partitioned covariance and precision matrices

Assume we split our $D-$ dimensional set of random variables $\boldsymbol{X}$ in two sets $\boldsymbol{x_a}$ and $\boldsymbol{x_b}$ (each can be multi-dimensional).

Likewise, we can split the mean values in two corresponding sets  $\boldsymbol{\mu_a}$ and $\boldsymbol{\mu_b}$.
The vectors $\boldsymbol{X}$ and $\boldsymbol{\mu}$ can then be expressed as:

$\boldsymbol{X}=\begin{bmatrix}\boldsymbol{x_a}\\ \boldsymbol{x_b}\end{bmatrix} $, $\boldsymbol{\mu}=\begin{bmatrix}\boldsymbol{\mu_a}\\ \boldsymbol{\mu_b}\end{bmatrix} $.

The covariance matrix $\boldsymbol{\Sigma}$ can be partitioned as

$\boldsymbol{\Sigma}=\begin{bmatrix} \boldsymbol{\Sigma}_{aa} & \boldsymbol{\Sigma}_{ab}\\ \boldsymbol{\Sigma}_{ba} & \boldsymbol{\Sigma}_{bb}\end{bmatrix}$ Notice that $\boldsymbol{\Sigma}_{aa}$ and $\boldsymbol{\Sigma}_{bb}$ are still symmetric, while $\boldsymbol{\Sigma}_{ab}=\boldsymbol{\Sigma}_{ba}^T$

We can also introduce a similar partition for the precision matrix $\boldsymbol\Lambda=\boldsymbol\Sigma^{-1}$:

$\boldsymbol{\Lambda}=\begin{bmatrix} \boldsymbol{\Lambda}_{aa} & \boldsymbol{\Lambda}_{ab}\\ \boldsymbol{\Lambda}_{ba} & \boldsymbol{\Lambda}_{bb}\end{bmatrix}$
However, keep in mind that the partition of the inverse is not equal to the inverse of a partition!
$\boldsymbol{\Lambda}_{aa}\ne\boldsymbol{\Sigma}_{aa}^{-1}$

## When Gaussian always Gaussian

We can now reason in terms of the D-dimensional multivariate Gaussian distribution defined over the joint set $(\boldsymbol x_a,\boldsymbol x_b)$  as $p(\boldsymbol x_a,\boldsymbol x_b) = \mathcal{N}(\boldsymbol x|\boldsymbol\mu,\boldsymbol{\Sigma})$.

The Gaussian distribution has unique properties!

+ The marginal distribution $p(\boldsymbol x_a) = \int p(\boldsymbol x_a,\boldsymbol x_b) d\boldsymbol x_b$ is Gaussian
+ The conditional distribution $p(\boldsymbol x_a|\boldsymbol x_b) = \frac{p(\boldsymbol x_a,\boldsymbol x_b)}{p(\boldsymbol x_b)} = \frac{p(\boldsymbol x_a,\boldsymbol x_b)}{\int p(\boldsymbol x_a,\boldsymbol x_b)  d\boldsymbol x_a}$ is Gaussian

## Marginal distribution

$p(\boldsymbol x_a, \boldsymbol x_b)=\mathcal{N}\left(\boldsymbol x|\begin{bmatrix}\boldsymbol{\mu_a}\\ \boldsymbol{\mu_b}\end{bmatrix}  ,\begin{bmatrix} \boldsymbol{\Sigma}_{aa} & \boldsymbol{\Sigma}_{ab}\\ \boldsymbol{\Sigma}_{ba} & \boldsymbol{\Sigma}_{bb}\end{bmatrix}\right)$

The marginal distribution is obtained when we 'marginalize' (ie integrate) the distribution over a set of random variables. In the 2D graphical representation this can be understood as collapsing the distribution over one axes.

What are the mean and covariance matrix of the marginal distribution ?

$p(\boldsymbol x_a) = \int p(\boldsymbol x_a,\boldsymbol x_b) d\boldsymbol x_b = \mathcal{N}(\boldsymbol x_a| ?, ?)$

$p(\boldsymbol x_a) = \int p(\boldsymbol x_a,\boldsymbol x_b) d\boldsymbol x_b = \mathcal{N}(\boldsymbol x_a| \boldsymbol \mu_a, \boldsymbol \Sigma_{aa})$

## Conditional distribution
Whereas the result for the marginal distribution is somewhat intuitive, a less intuitive result holds for the conditional distribution, which we derive here.

The conditional distribution $p(\boldsymbol x_a| \boldsymbol x_b)$ is simply evaluated by considering the joint distribution $p(\boldsymbol x_a,\boldsymbol x_b)$ and considering $\boldsymbol x_b$ as a constant.

Using the partioning introduced above for $\boldsymbol x$, $\boldsymbol \mu$ and the precision matrix $\boldsymbol\Lambda$, we have:

$(\boldsymbol{x}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu})=(\boldsymbol{x_a}-\boldsymbol{\mu_a})^T\boldsymbol{\Lambda}_{aa}(\boldsymbol{x_a}-\boldsymbol{\mu_a})+2(\boldsymbol{x_a}-\boldsymbol{\mu_a})^T\boldsymbol{\Lambda}_{ab}(\boldsymbol{x_b}-\boldsymbol{\mu_b})+(\boldsymbol{x_b}-\boldsymbol{\mu_b})^T\boldsymbol{\Lambda}_{bb}(\boldsymbol{x_b}-\boldsymbol{\mu_b})$

Now, we expect $p(\boldsymbol x_a| \boldsymbol x_b)\sim\mathcal N(\boldsymbol x_a|\boldsymbol\mu_{a|b},\boldsymbol\Sigma_{a|b})$.
A general form for the argument of the exponent is

$(\boldsymbol{x}_a-\boldsymbol{\mu_{a|b}})^T\boldsymbol{\Sigma}^{-1}_{a|b}(\boldsymbol{x}_a-\boldsymbol{\mu}_{a|b})=\boldsymbol x_a^T \boldsymbol{\Sigma}^{-1}_{a|b} \boldsymbol x_a -2 \boldsymbol x_a^T \boldsymbol{\Sigma}^{-1}_{a|b}\boldsymbol \mu_{a|b} + \boldsymbol \mu^T_{a|b} \boldsymbol{\Sigma}^{-1}_{a|b} \boldsymbol \mu_{a|b}$ (where we have used the symmetry of $\boldsymbol\Sigma_{a|b}$).

$(\boldsymbol{x}-\boldsymbol{\mu})^T\boldsymbol{\Sigma}^{-1}(\boldsymbol{x}-\boldsymbol{\mu})=(\boldsymbol{x_a}-\boldsymbol{\mu_a})^T\boldsymbol{\Lambda}_{aa}(\boldsymbol{x_a}-\boldsymbol{\mu_a})+2(\boldsymbol{x_a}-\boldsymbol{\mu_a})^T\boldsymbol{\Lambda}_{ab}(\boldsymbol{x_b}-\boldsymbol{\mu_b})+(\boldsymbol{x_b}-\boldsymbol{\mu_b})^T\boldsymbol{\Lambda}_{bb}(\boldsymbol{x_b}-\boldsymbol{\mu_b})$

$(\boldsymbol{x}_a-\boldsymbol{\mu_{a|b}})^T\boldsymbol{\Sigma}^{-1}_{a|b}(\boldsymbol{x}_a-\boldsymbol{\mu}_{a|b})=\boldsymbol x_a^T \boldsymbol{\Sigma}^{-1}_{a|b} \boldsymbol x_a -2 \boldsymbol x_a^T \boldsymbol{\Sigma}^{-1}_{a|b}\boldsymbol \mu_{a|b} + \boldsymbol \mu^T_{a|b} \boldsymbol{\Sigma}^{-1}_{a|b} \boldsymbol \mu_{a|b}$

It is now sufficient to equate equal terms in $\boldsymbol x_a$ in the above two equations.

Terms in $\boldsymbol x_a^2\longrightarrow$: $\boldsymbol x_a^T \boldsymbol{\Sigma}^{-1}_{a|b} \boldsymbol x_a = \boldsymbol x_a^T \boldsymbol{\Lambda}_{aa} \boldsymbol x_a$, from which $\boldsymbol{\Sigma}_{a|b} = \boldsymbol{\Lambda}_{aa}^{-1}$

Terms in $\boldsymbol x_a\longrightarrow$: $2\boldsymbol x_a^T(-\boldsymbol\Lambda_{aa}\boldsymbol\mu_a+\boldsymbol\Lambda_{ab}\boldsymbol (\boldsymbol x_b-\boldsymbol \mu_b))= -2\boldsymbol x_a^T\boldsymbol\Sigma^{-1}_{a|b}\boldsymbol\mu_{a|b}$  from which $\boldsymbol\mu_{a|b}=\Sigma_{a|b}(\boldsymbol\Lambda_{aa}\boldsymbol\mu_a-\boldsymbol\Lambda_{ab}\boldsymbol (\boldsymbol x_b-\boldsymbol \mu_b))=\boldsymbol\mu_a-\boldsymbol\Lambda_{aa}^{-1}\boldsymbol\Lambda_{ab}\boldsymbol (\boldsymbol x_b-\boldsymbol \mu_b)$

So far we have:

$\boldsymbol{\Sigma}_{a|b} = \boldsymbol{\Lambda}_{aa}^{-1}$

$\boldsymbol\mu_{a|b}=\boldsymbol\mu_a-\boldsymbol\Lambda_{aa}^{-1}\boldsymbol\Lambda_{ab}\boldsymbol (\boldsymbol x_b-\boldsymbol \mu_b)$

However, we would like to express the covariance matrix and the mean of the conditional distribution $p(\boldsymbol x_a| \boldsymbol x_b)$ in terms of the partioned co-variance matrix and mean of the full distribution. We need to use the following identity that relates the inverse of a partitioned matrix, with the partition of the inverse:

$\begin{bmatrix}A & B \\ C & D\end{bmatrix}^{-1} = \begin{bmatrix}(A-BD^{-1}C)^{-1} & -(A-BD^{-1}C)^{-1}BD^{-1} \\-D^{-1}C(A-BD^{-1}C)^{-1} & D^{-1}+D^{-1}C(A-BD^{-1}C)^{-1}BD^{-1} \end{bmatrix}$

In our case

$\boldsymbol{\begin{bmatrix}\boldsymbol\Sigma_{aa} & \boldsymbol\Sigma_{ab} \\ \boldsymbol\Sigma_{ba} & \boldsymbol\Sigma_{bb}\end{bmatrix}^{-1} = \begin{bmatrix}\boldsymbol\Lambda_{aa} & \boldsymbol\Lambda_{ab} \\ \boldsymbol\Lambda_{ba} & \boldsymbol\Lambda_{bb}\end{bmatrix}}$

Hence: $\boldsymbol\Lambda_{aa} = (\boldsymbol\Sigma_{aa}- \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba})^{-1}$ and $\boldsymbol\Lambda_{ab} = - (\boldsymbol\Sigma_{aa}- \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba})^{-1}\boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}$

and finally:

\begin{equation}\boxed{\boldsymbol\mu_{a|b}=\boldsymbol\mu_a+\boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}(\boldsymbol x_b - \boldsymbol\mu_b) \\
\boldsymbol\Sigma_{a|b} = \boldsymbol\Sigma_{aa} - \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba}} \end{equation}

```python
## Example with a 2D distribution

def f(mu_a, mu_b, x_b=0, sigma_aa=1, sigma_bb=1, sigma_ab=0):
    fig = plt.figure(figsize=(7, 7))
    xx, yy = np.mgrid[-5:5:0.1, -5:5:0.1]
    y=np.linspace(-5,5,100)
    Sigma = np.matrix([[sigma_aa,sigma_ab],[sigma_ab, sigma_bb]])
    Sigma_inv = np.linalg.inv(Sigma)
    Sigma_det = np.linalg.det(Sigma)
    f = 1/(4*np.pi)/np.sqrt(Sigma_det) * np.exp(-0.5*((xx-mu_a)**2*Sigma_inv[0,0]+ 2*(xx-mu_a)*(yy-mu_b)*Sigma_inv[0,1]+(yy-mu_b)**2*Sigma_inv[1,1]))
    mu_ab = mu_a +sigma_ab/sigma_bb*(x_b-mu_b)
    Sigma_cond = sigma_aa-sigma_ab**2/sigma_bb
    ax = fig.gca()
    ax.set_xlim(-5, 5)
    ax.set_ylim(-5, 5)
    cfset = ax.contourf(xx, yy, f, cmap='coolwarm')
    cset = ax.contour(xx, yy, f, colors='k')
    ax.clabel(cset, inline=1, fontsize=10)
    ax.plot([-5, 5],[x_b,x_b], color='black')
    ax.plot(y,x_b+ 1/np.sqrt(2*np.pi)/Sigma_cond * np.exp(-0.5*(y-mu_ab)**2/Sigma_cond**2), color='yellow', linewidth=2)
    ax.set_xlabel('x_a', fontsize=16)
    ax.set_ylabel('x_b', fontsize=16)
    ax.text(-4,3,np.core.defchararray.add('$\Sigma^{-1}=$\n',np.array_str(Sigma_inv)), fontsize=16)
    ax.text(-4,-3.5,np.core.defchararray.add('$\Sigma=$\n',np.array_str(Sigma)), fontsize=16)
interactive_plot = interactive(f, sigma_aa=(0, 3.0), sigma_bb=(0,3.0), sigma_ab=(0,3.0), mu_a=(-2.0,2.0), mu_b=(-2.0,2.0),x_b=(-2.0,2.0))
output = interactive_plot.children[-1]
```

```python
interactive_plot
```

    interactive(children=(FloatSlider(value=-0.6, description='mu_a', max=2.0, min=-2.0), FloatSlider(value=0.8, d…

What are the interesting properties of $\boldsymbol\mu_{a|b}$ and $\boldsymbol\Sigma_{a|b}$ ??

$\boldsymbol\mu_{a|b}$ depends linearly on $\boldsymbol x_b$

$\boldsymbol\Sigma_{a|b}$ depends on all partitions of $\boldsymbol\Sigma$ but it is INDEPENDENT of $\boldsymbol x_b$

## End of Part 1 !!

Gaussian Process Regression Part 1.ipynb


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Power Spectral Density

The (auto-) [power spectral density](https://en.wikipedia.org/wiki/Spectral_density#Power_spectral_density) (PSD) is defined as the Fourier transformation of the [auto-correlation function](correlation_functions.ipynb) (ACF).

### Definition

For a continuous-amplitude real-valued wide-sense stationary (WSS) random signal $x[k]$ the PSD is given as

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathcal{F}_* \{ \varphi_{xx}[\kappa] \}
\end{equation}

where $\mathcal{F}_* \{ \cdot \}$ denotes the [discrete-time Fourier transformation](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform) (DTFT) and $\varphi_{xx}[\kappa]$ the ACF of $x[k]$. The ACF of a random signal of finite length $N$ can be expressed by way of a linear convolution

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{N} \cdot x_N[k] * x_N[-k]
\end{equation}

Taking the DTFT of the left- and right-hand side results in

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \, X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})\, X_N(\mathrm{e}^{-\,\mathrm{j}\,\Omega}) =
\frac{1}{N} \, | X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

The last equality results from the definition of the magnitude and the symmetry of the PSD (see below). It can be concluded from above result that the PSD quantifies the power per frequency of a random signal.

### Properties

The properties of the PSD can be deduced from the properties of the ACF and the DTFT as

1. From the symmetry of the ACF it follows that

    $$ \Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \Phi_{xx}(\mathrm{e}^{\,-\mathrm{j}\, \Omega}) $$

    and that $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) \in \mathbb{R}$.

2. The quadratic mean of a random signal is given as

    $$ E\{ x[k]^2 \} = \varphi_{xx}[0] = \frac{1}{2\pi} \int\limits_{-\pi}^{\pi} \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) \,\mathrm{d} \Omega $$

    The last relation can be found by introducing the definition of the inverse DTFT.

### Example

In this example the PSD $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j} \,\Omega})$ of a speech signal $x[k]$ is estimated by applying a discrete Fourier transformation (DFT) to its ACF. For a better interpretation of the PSD, the frequency axis $f = \frac{\Omega}{2 \pi} \cdot f_s$ has been chosen for illustration, where $f_s$ denotes the sampling frequency of the signal.

In Python the ACF is stored in a vector with indexes $0, 1, ..., 2N -1$ where the indexes correspond to the lags $\kappa = -N+1,-N+2,....,N-1$. When computing the discrete Fourier transform (DFT) of the ACF numerically by the fast Fourier transform (FFT) one has to take this shift into account. For instance, by multiplying the DFT $\Phi_{xx}[\mu]$ by $e^{j \mu \frac{2 \pi}{2N - 1} (N-1)}$ where $N$ denotes the length of the signal $N$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.io import wavfile

# read audio file
fs, x = wavfile.read('../data/speech_8k.wav')
x = np.asarray(x, dtype=float)
N = len(x)

# compute ACF
acf = 1/len(x) * np.correlate(x, x, mode='full')
# compute PSD
psd = np.fft.fft(acf)
psd = psd * np.exp(1j*np.arange(2*N-1)*2*np.pi*(N-1)/(2*N-1))
f = np.fft.fftfreq(2*N-1, d=1/fs)

# plot PSD
plt.figure(figsize = (10, 8))
plt.plot(f, np.real(psd))
plt.title('Estimated power spectral density')
plt.ylabel(r'$\hat{\Phi}_{xx}(e^{j \Omega})$')
plt.xlabel(r'$f$')
plt.axis([0, 2000, 0, 1.1*max(np.abs(psd))]);
plt.grid()
```

**Exercise**

* What does the PSD tell you about the spectral contents of a speech signal?

It can be concluded from the shown PSD that the main power of a speech signal is contained in the frequency range below 500 Hz. The speech signal exhibits furthermore a harmonic structure with a dominant fundamental frequency and a number of harmonics.

## Cross-Power Spectral Density

The cross-power spectral density is defined as the Fourier transformation of the [cross-correlation function](correlation_functions.ipynb#Cross-Correlation-Function) (CCF). For two continuous-amplitude real-valued wide-sense stationary (WSS) random signals $x[k]$ and $y[k]$ the cross-power spectral density is given as

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j} \, \Omega}) = \mathcal{F}_* \{ \varphi_{xy}[\kappa] \}
\end{equation}

where $\varphi_{xy}[\kappa]$ denotes the CCF of $x[k]$ and $y[k]$.

The symmetries of $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ can be derived from the symmetries of the CCF and the DTFT as

\begin{equation}
\underbrace {\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \Phi_{xy}^*(\mathrm{e}^{-\,\mathrm{j}\, \Omega})}_{\varphi_{xy}[\kappa] \in \mathbb{R}} =
\underbrace {\Phi_{yx}(\mathrm{e}^{\,- \mathrm{j}\, \Omega}) = \Phi_{yx}^*(\mathrm{e}^{\,\mathrm{j}\, \Omega})}_{\varphi_{yx}[-\kappa] \in \mathbb{R}}
\end{equation}

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# YOUR PROJECT TITLE

Imports and set magics:

```python
#Import packages
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy import optimize
import sympy as sm
sm.init_printing(use_unicode=True)

import ipywidgets as widgets

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2

# local modules
#import modelproject
#Denne henviser til modelproject.py - gemmer her, hvis vi vil bruge med vores model og smide noget af det derind.
```

We are solving a Microeconomics I problem from an exam. Exam 2016 winter, link is in readme and file has been uploaded as .pdf in this project. We are considering problem/exercise 2.

# Visualization

Consumer theory setup

Utility function:
$u(x_1,x_2)=x_1x_2+2x_2$

The consumption possiblity frontier is: $(x_1,x_2) \in (0, \infty) x (0,\infty)$.

Furthermore we assume: $p_1, p_2, I>0$

We start out by visualizing the consumer's utility:

```python
#Utility function:
def u(x1,x2):
    u = x1*x2+2*x2
    return u
```

```python
#Vectors:
x1_vec = np.linspace(0, 20, 50)
x2_vec = np.linspace(0, 20, 50)

#Grid:
x1_grid, x2_grid = np.meshgrid(x1_vec, x2_vec)

#Utility function in grid:
u_grid = u(x1_grid, x2_grid)

#Plot:
fig = plt.figure()
ax = fig.add_subplot(1,1,1, projection='3d')
ax.plot_surface(x1_grid, x2_grid, u_grid, cmap=plt.cm.jet, cstride=1, rstride=1)

#Labels:
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('$u$')
ax.invert_xaxis()

```

# The consumer's utility maximization problem

We have the following utility max problem:
$max_{x_1,x_2 \geq 0} \:  x_1x_2+2x_2 \: s.t. \: p_1x_1+p_2x_2=I$

```python
#Turning variables to symbolics

x1 = sm.symbols('x_1')
x2 = sm.symbols('x_2')
p1 = sm.symbols('p_1')
p2 = sm.symbols('p_2')
I = sm.symbols('I')
U = sm.symbols('U')

utility = (x1*x2 + 2*x2)
budget_constraint = sm.Eq(p1*x1 + p2*x2, I)
```

### Finding consumers demand for the goods respectively

We solve for $x_2$ in the budget constraint

```python
x2constraint = sm.solve(budget_constraint, x2)
x2constraint[0]
```

Substitute result for $x_2$

```python
utility_subx2 = utility.subs(x2, x2constraint[0])
utility_subx2
```

Finding the derivate w.r.t $x_1$ to maximize

```python
focx1 = sm.diff(utility_subx2, x1)
focx1
```

As usual this is set equal to zero and solve for $x_1$

```python
x1solv = sm.solve(sm.Eq(0,focx1),x1)
x1solv[0]
```

This resultat is substituted into the budget constraint, which then will be an expressing for $x_2$

```python
x1constraint = sm.solve(budget_constraint, x1)
utility_subx1 = utility.subs(x1, x1constraint[0])
focx2 = sm.diff(utility_subx1, x2)
x2solv = sm.solve(sm.Eq(focx2, 0), x2)
x2solv[0]
```

For simplicitiy, we proceed with this solution, which is true for $I>2p_1$, thereby we ignore the corner solutions where $I \leq 2p_1$

 ## Case with values

We set values for prices and income. $p_1=4$, $p_2=1$ and $I=16$. A change in production of the goods causes the price of good 1 to fall to $p_1=1$

```python
#Define results for x1 and x2 using the solution found above
def x1func(p1,p2,I):
    return (I/2 - p1)/p1

def x2func(p1,p2,I):
    return (I/2 + p1)/p2
```

```python
#Prices and income are implemented to determine x1, x2 and u
print('x1 =',x1func(4,1,16))
print('x2 =',x2func(4,1,16))
print('utility =',u(x1func(4,1,16),x2func(4,1,16)))
```

    x1 = 1.0
    x2 = 12.0
    utility = 36.0

```python
#Following a numerical approach we dobbelcheck the answer above
#Find optimal consumption
I = 16
p1 = 4
p2 = 1

#We define the utility function
def u_new(x_1,x_2):
    return x_1*x_2 + 2 * x_2

#We construct a function to find the optimal utility, where it is negative to use the minimizer function
def func(x_1,I,p1,p2):
    x_2 = (I-p1*x_1)/p2
    return -u_new(x_1,x_2)

#We solve the function using optimize.minimuze and our utility function
solution = optimize.minimize_scalar(
    func,method='bounded',
    bounds=(0,I/p2),args=(I,p1,p2))

x_1 = solution.x
x_2 = (I-p1*x_1)/p2
u_func = u_new(x_1,x_2)

print(f'Same results as above for p1={p1:.0f}')
print(f'x1 = {x_1:.1f}')
print(f'x2 = {x_2:.1f}')
print(f'u  = {u_func:.1f}')
print(f'I-p1*x1-p2*x2 = {I-p1*x_1-p2*x_2:.2f}')
```

    Same results as above for p1=4
    x1 = 1.0
    x2 = 12.0
    u  = 36.0
    I-p1*x1-p2*x2 = 0.00

```python
#Case with a fall in price, p1=1
print('x1 =',x1func(1,1,16))
print('x2 =',x2func(1,1,16))
print('utility =',u(x1func(1,1,16),x2func(1,1,16)))
```

    x1 = 7.0
    x2 = 9.0
    utility = 81.0

```python
#Following a numerical approach we dobbelcheck the answer above
#Find optimal consumption
I = 16
p1 = 1
p2 = 1

#We define the utility function
def u_new(x_1,x_2):
    return x_1*x_2 + 2 * x_2

#We construct a function to find the optimal utility, where it is negative to use the minimizer function
def func(x_1,I,p1,p2):
    x2 = (I-p1*x_1)/p2
    return -u_new(x_1,x_2)

#We solve the function using optimize.minimuze and our utility function
solution = optimize.minimize_scalar(
    func,method='bounded',
    bounds=(0,I/p2),args=(I,p1,p2))

x_1 = solution.x
x_2 = (I-p1*x_1)/p2
u_func = u_new(x_1,x_2)

print(f'Same results as above for p1={p1:.0f}')
print(f'x1 = {x_1:.1f}')
print(f'x2 = {x_2:.1f}')
print(f'u  = {u_func:.1f}')
print(f'I-p1*x1-p2*x2 = {I-p1*x_1-p2*x_2:.2f}')
```

    Same results as above for p1=1
    x1 = 16.0
    x2 = 0.0
    u  = 0.0
    I-p1*x1-p2*x2 = 0.00

**Results are presented graphically**

```python
# Rewriting utility
utility_Eq = sm.Eq(U,utility)

# Solving for x2 in utility function and budget constraint to plot it
print('x2(utility) =', sm.solve(utility_Eq,x2))
print('x2(budget) =', sm.solve(budget_constraint, x2))
```

    x2(utility) = [U/(x_1 + 2)]
    x2(budget) = [(I - p_1*x_1)/p_2]

```python
#We construct a function for the optimal consumption figure
def optimal_consumption_figure(x1,I,p1,p2):
    def budget(x1,I,p1,p2):
        return (I-p1*x1)/p2
    def indiff(x1,u,p1,p2):
        return u/(x1+2)

    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(x1_vec,budget(x1_vec,I,p1,p2),label='Budget Constraint')
    ax.plot(x1_vec,indiff(x1_vec,u(x1func(p1,p2,I),x2func(p1,p2,I)),p1,p2),label='Indifference Curve')
    ax.scatter(x1func(p1,p2,I),x2func(p1,p2,I),u(x1func(p1,p2,I),x2func(p1,p2,I)),color='red',label=f'Optimum ({x1func(p1,p2,I):.1f},{x2func(p1,p2,I):.1f})')
    plt.xlim(0, 20)
    plt.ylim(0, 40)

    #Title & labels
    ax.set_xlabel('$x_1$')
    ax.set_ylabel('$x_2$')
    plt.title(f'Optimal consumption bundle given $p_1$={p1:.1f}, $p_2$={p2:.1f} and $I$={I}. \n Resulting in $x_1$={x1func(p1,p2,I):.2f}, $x_2$={x2func(p1,p2,I):.2f} with utility {u(x1func(p1,p2,I),x2func(p1,p2,I)):.2f}')
    ax.legend();

```

```python
#We visualize the optimal consumption with p1=4
optimal_consumption_figure(x1_vec,16,4,1)
```

```python
#We visualize the optimal consumption with p1=1
optimal_consumption_figure(x1_vec,16,1,1)
```

```python
# We construct our model with a floatslider for p1 to anlyze the model at different price points for x1
widgets.interact(optimal_consumption_figure,
    x1=widgets.fixed(x1_vec),
    I=widgets.fixed(16),
    p1=widgets.FloatSlider(description="p_1", min=0.1, max=5, step=0.05, value=4),
    p2=widgets.fixed(1),
);
```

    interactive(children=(FloatSlider(value=4.0, description='p_1', max=5.0, min=0.1, step=0.05), Output()), _dom_…

```python
# In this model we include a floatslider to for p1, p2 and I, to analyze price changes for both goods and budget changes
widgets.interact(optimal_consumption_figure,
    x1=widgets.fixed(x1_vec),
    I=widgets.FloatSlider(description="I", min=1, max=30, step=0.5, value=16),
    p1=widgets.FloatSlider(description="p_1", min=0.1, max=8, step=0.05, value=4),
    p2=widgets.FloatSlider(description="p_2", min=0.1, max=8, step=0.05, value=1),
);
```

    interactive(children=(FloatSlider(value=16.0, description='I', max=30.0, min=1.0, step=0.5), FloatSlider(value…

# Social planner

We now implement a social planner who is the distributor of the goods that consumer A bought at $p_1=1$ and $p_2=1$. We introduce one more consumer, so that the planner has to divide the goods between consumer A and B. We have $x_1=7$ and $x_2=9$, and we assumme they have zero endowments. Consumer A still has the same utility function and consumer B has the following:
$u_B=4x_{1B}+4x_{2B}$

We introduce the following social utility function, which will maximize the utility for the consumer with the lowest utility: $S(u_A,u_B) = min(u_A,u_B)$
We wish to visualize this in a edgeworth box indicating the optimal consumtion bundle for the consumers a the given utility level.

```python
#Define symbols:
X1 = sm.symbols('X1')
X2 = sm.symbols('X2')
#Define utility functions:
uA = X1*X2 + 2*X2
uB = 4*X1 + 4*X2
```

In order to optimize the social planner's problem the first order conditions is found for both consumers and with respect to both goods.

```python
foc1uA = sm.diff(uA, X1)
foc2uA = sm.diff(uA, X2)
foc1uB = sm.diff(uB, X1)
foc2ub = sm.diff(uB, X2)
```

```python
#The marginal rate of substitution, MRS, for both
mrsA = -foc1uA/foc2uA
mrsB = -foc1uB/foc2ub
```

```python
#MRS for consumer A:
mrsA
```

```python
#MRS for consumer B:
mrsB
```

```python
#The marginal rate of substution for consumer A and B are set equal to eachother
MRSeq = sm.Eq(mrsB,mrsA)

#Solve for x2
x2_d_A = sm.solve(MRSeq,X2)
x2_d_A[0]
```

So we now know, that when $x_2=x_1+2$, then consumer A has a MRS equal to -1. The solutions here are all pareto optimal for the consumers. This is not the case for the social planner though, now we will find the utility for consumer A by substituting in $x_2=x_1+2$.

```python
#Substituting
uA_x1 = uA.subs(X2,x2_d_A[0])
uA_x1
```

```python
#Defining excess goods
e_x1 = 7 - X1
e_x2 = 9 - x2_d_A[0]

#Substitute into utility for consumber B
uB_x1 = uB.subs(X1, e_x1)
uB_x2 = uB_x1.subs(X2, e_x2)
uB_x2
```

```python
#Utility functions for the two consumers are set equal to each other. Functions expressed of x1.
uA_uB = sm.Eq(uA_x1-uB_x2,0)
uA_uB
```

```python
#Solve for optimal x1 for consumer A
x1_sol_A_ = sm.solve(uA_uB,X1)
x1_sol_A_
```

```python
#Since the goods can't be negative, we disregard the negative value and keep the positive one:
x1_sol_A = x1_sol_A_[0]
x1_sol_A
```

```python
print(f'Consumer A will get {float(x1_sol_A):.3f} x1')
```

    Consumer A will get 3.381 x1

Optimal solution for the social planner with $x_1 \approx 3,381$, this result is used to find the optimal $x_2$

```python
#Substitute optimal into x2
x2_sol_A = x2_d_A[0].subs(X1,x1_sol_A)
x2_sol_A
```

```python
print(f'Consumer A will get {float(x2_sol_A):.3f} x2')
```

    Consumer A will get 5.381 x2

These results also correspond with the function we derived earlier: $x_2=x_1+2$. Since we have $x_1 \approx 3,381$, when 2 is added we get as desired $x_2 \approx 5,381$.
Now we will proceed and solve for consumer B:

```python
#Using solution for good 1 for consumer A in consumber B's excess function:
x1_sol_B = e_x1.subs(X1,x1_sol_A)
x1_sol_B
```

```python
print(f'Consumer B will get {float(x1_sol_B):.3f} x1')
```

    Consumer B will get 3.619 x1

```python
#Using solution for consumer A in consumber B's excess function:
x2_sol_B_ = e_x2.subs(x2_d_A[0],x2_sol_A) #substitute consumer A's demand for x2 into excess function for consumer B
x2_sol_B = x2_sol_B_.subs(X1,x1_sol_A)    #substitute consumers A's demand for x1 into solution function for consumer B
x2_sol_B

```

```python
print(f'Consumer B will get {float(x2_sol_B):.3f} x2')
```

    Consumer B will get 3.619 x2

Consumer B has the same amount of good 1 and 2, which intuitively makes good sense when considering the utility function where $x_1$ and $x_2$ is perfect substitutes.

Now we proceed and find the utility for both consumers:

```python

```

```python
uA_sol = x1_sol_A*x2_sol_A + 2*x2_sol_A
uB_sol = 4*x1_sol_B + 4*x2_sol_B

print(f'Consumer A: (x1,x2)=({float(x1_sol_A):.3f},{float(x2_sol_A):.3f})')
print(f'Consumer A: (x1,x2)=({float(x1_sol_B):.3f},{float(x2_sol_B):.3f})')

print(f'utility A = {float(uA_sol):.5f}')
print(f'utility B = {float(uB_sol):.5f}')
```

    Consumer A: (x1,x2)=(3.381,5.381)
    Consumer A: (x1,x2)=(3.619,3.619)
    utility A = 28.95335
    utility B = 28.95335

MRS is equal to -1 for both, while utilites are $u_A = u_B = 28.95335$, whereas this is how the social planner would distribute the goods w.r.t. to maximizing $S(u_A,u_B) = min(u_A,u_B)$

## Plot and visualize As a Edgeworth Box

```python
X1_star = 7
X2_star = 9

#Plotting in the same diagram, by using three functions where x2 is isolated

def x2_A(x1):
    return float(uA_sol)/(x1+2)

def x2_MRS(x1):
    return x1 + 2

def x2_B(x1):
    return (float(uB_sol)-4*(x1))/4

x1 = np.linspace(0,8,100)

fig=plt.figure()
ax=fig.add_subplot(111, label="1")
ax2=fig.add_subplot(111, label="2", frame_on=False)

#Plotting Consumer A
ax.plot(x1, x2_A(x1), color="C0")
ax.scatter(float(x1_sol_A),float(x2_sol_A))
ax.plot(x1,x2_MRS(x1),color='black')
ax.set_xlabel("$x_1$ consumer A", color="C0")
ax.set_ylabel("$x_2$ consumer A", color="C0")
ax.axis([0,X1_star,0,X2_star]);
ax.tick_params(axis='x', colors="C0")
ax.tick_params(axis='y', colors="C0")
ax.legend(['Consumer A Indifference Curve','|MRS| = 1',f'Equilibrium: A:({float(x1_sol_A):.3f},{float(x2_sol_A):.3f}) B:({float(x1_sol_B):.3f},{float(x2_sol_B):.3f})'],loc=3)

#Plotting Consumer B
ax2.plot(x1, x2_B(x1), color="C1")
ax2.xaxis.tick_top()
ax2.yaxis.tick_right()
ax2.set_xlabel('$x_1$ consumer B', color="C1")
ax2.set_ylabel('$x_2$ consumer B', color="C1")
ax2.xaxis.set_label_position('top')
ax2.yaxis.set_label_position('right')
ax2.axis([X1_star,0,X2_star,0]);
ax.grid(b=True);
ax2.tick_params(axis='x', colors="C1")
ax2.tick_params(axis='y', colors="C1")
ax2.legend(['Consumer B Indifference Curve'],loc=1)

plt.show()
```

We end up with an edgewothbox giving us the equilibrium where consumer A has $3.381$ of $x_1$ and $5.381$ of $x_2$ while consumer B has $3.619$ of both $x_1$ and $x_2$

# Conclusion

We conclude that ypu can use Python to solve and visualize a consumer utility problem from MicroEconomics I. We are also able to construct a edgeworthbox by including an additional consumer to our economy, and find the social optimum for a social planner.


# Assignment 3 - Due Fri 10/22 at 11:59 PM

## Background

The data for this exercise were used in Ebonya Washington's paper: "Female Socialization: How Daughters Affect Their Legislator Fathers' Voting on Women's Issues."  published in the American Economic Review in 2008. The paper asks whether having daughters influences the voting behavior of members of the US Congress. The hypothesis is that having (more) daughters makes legislators more likely to vote liberally (in terms of political alignment, and in contrast to conservatively) on issues concerning women.

For this exercise, we will focus on votes that took place in the 108th Congress, which held session in 2003/04. As a measure of a liberal voting record, we use scores assigned by the American Association of University Women (AAUW), a liberal group that concerns itself with issues of interest to women. For the 108th Congress, the AAUW selected 9 pieces of legislation in the areas of education, equality and reproductive rights. The AAUW then assigned a score to each member of Congress. The scores range from 0 to 100 and measure the percentage of times the legislator voted in favor of the position held by the AAUW.

 The dataset `legislators.dta` contains the following characteristics for a random sample of 386 members of the 108th Congress:

 * $ngirls$ number of daughters
 * $totchi$ total number of children
 * $age$ age
 * $female$ indicator for being female
 * $repub$ indicator for being a Republican
 * $moredef$ proportion of people in the legislator's district who are in favor of "more spending on defense"
 * $aauw$ AAUW score

(For the purposes of this exercise, you can assume all members of the 108th Congress were either Democrats or Republicans and were either male or female.)

### (a) Estimate and report results for the following regression models:

Load in the data set `legislators.dta`. Remember, you will first need to call the `haven` package to do so.

Generate a variable `ngirls2` $=\text{ngirls}^2$

Generate an interaction variable `repubngirls` $= \text{repub}*\text{ngirls}$

Generate an interaction variable `repubngirls2` $=\text{repub}*\text{ngirls2}$

Estimate the following three regression models, save the output as reg1, reg2, and reg3, and show the results of each using `summary()`:

\begin{align}
aauw&=\beta_0 +\beta_1female+\beta_2repub+\beta_3ngirls+u  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1) \\
aauw&=\beta_0 +\beta_1female+\beta_2repub+\beta_3ngirls+\beta_4 ngirls2+u ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~(2)  \\
aauw&=\beta_0 + \beta_1female+\beta_2repub+\beta_3ngirls+\beta_4 ngirls2+\beta_5 repubngirls+\beta_6 repubngirls2+\beta_7 totchi + \beta_8 moredef + u\ \ \ (3)
\end{align}

*(Note: this method of generating interaction variables (multiplying them together) is appropriate when one of the interacted variables is a dummy variable, but may not be appropriate in all cases.)*

```R

```

### (b) Suggest which model is the best fit to the data. How did you determine this? (no more than 1 sentence is required)

 Text goes here

### (c) Interpret the marginal effect at the mean of the number of daughters on AAUW score in each model.
*(Hint: Calculate the total marginal effect, using the coefficients for all terms including the number of daughters. Calculating a marginal effect at the mean involves plugging in the mean number of daughters in the sample into any estimate of the total marginal effect of the number of daughters where this effect varies by the number of daughters.)*

*(Hint: Instead of typing in numbers from the regressions manually, you can call regression coefficients using summary(reg1)$coefficients[#,1] for coefficient number # starting with the intercept as number 1.)*

*(Hint: In model 3, the marginal effect will differ by particular subgroups. Interpret the different effects for each subgroup.)*

Text goes here

### (d) Test whether there is an effect of the number of daughters on AAUW scores using the second model. Be sure to describe carefully the null and alternative hypothesis.
*(Hint: You can access the residuals from a regression you have saved as reg by calling `reg$residuals`, and you can access the r-squared by calling `summary(reg)$r.squared`.)*

Text goes here

```R

```

 Text goes here

### (e) Using the third model, predict the AAUW score for male democrats who have 2 daughters and 1 son, and who have 25% of constituents who want more spending on defense, on average. Suggest a $95\%$ CI for that predicted value.

*(Hint: See part 3-A of Section Notes 8.)*

```R

```

 Text goes here

### (f) Suppose a particular male Democrat has 2 daughters and 1 son in a state where 25% of constituents want more spending on defense. Generate a $95\%$ CI for his *particular* AAUW score, continuing to use the third model.

*(Hint: See part 3-B of Section Notes 8. Note that you can use most the values you already calculated in part (e) to answer this question.)*

```R

```

 Text goes here

### (g) Suppose you think Republicans and non-Republicans may have different gender patterns in voting with respect to the AAUW score. That is, republican men may vote differently than Republican women, who may vote differently than Democratic women who may vote differently than Democratic men. Write down an estimation equation you could use to test whether Republican women, Democratic women, and Democratic men each vote differently than Republican men. Specify what your null and alternative hypotheses would be.

 Text goes here

### (h) Implement your test. Interpret each coeffcient.
*(Hint: To create dummy variables based on particular characteristics, it is easiest to first create the dummy variable and set it equal to 0 for all observations: `data$dummy<0`. Then, replace the values for that dummy with 1 for the observations that match the requirements you are looking for, as in `data[data$x1==0 & data$x2==1,]$dummy<-1`.)*

*(Hint: If you need to, you can include an interaction term in your regression using `:`. For example `lm(y~x1+x2+x1:x2,data=data)` includes an interaction between `x1` and `x2`. You will need to load the `car` package.)*

```R

```

  Text goes here

### (i) Adapt your regression to test whether Democratic women vote differently than Republican women with respect to the AAUW score. Write out the estimating equation and report your results.

```R

```

 Text goes here

## Downloading your Notebook

Download a PDF copy of your notebook by using __File > Download as HTML.__ Alternatively, go to print preview and __Save as PDF__ (but make sure your code does not get cut off at the side of the page!)


Computational Seismology

Discontinuous Galerkin Method with Physics based numerical flux - 1D Elastic Wave Equation

This notebook is based on the paper [A new discontinuous Galerkin spectral element method for elastic waves with physically motivated numerical fluxes](https://www.geophysik.uni-muenchen.de/~gabriel/kduru_waves2017.pdf)
Published in the [13th International Conference on Mathematical and Numerical Aspects of Wave Propagation](https://cceevents.umn.edu/waves-2017)

##### Authors:
* Kenneth Duru
* Ashim Rijal ([@ashimrijal](https://github.com/ashimrijal))
* Sneha Singh

---

## Basic Equations ##

The source-free elastic wave equation in a heterogeneous 1D medium is

\begin{align}
\rho(x)\partial_t v(x,t) -\partial_x \sigma(x,t) & = 0\\
\frac{1}{\mu(x)}\partial_t \sigma(x,t) -\partial_x v(x,t) & = 0
\end{align}

with $\rho(x)$ the density, $\mu(x)$ the shear modulus and $x = [0, L]$. At the boundaries $ x = 0, x = L$ we pose the general well-posed linear boundary conditions

\begin{equation}
\begin{split}
B_0(v, \sigma, Z_{s}, r_0): =\frac{Z_{s}}{2}\left({1-r_0}\right){v} -\frac{1+r_0}{2} {\sigma} = 0,  \quad \text{at} \quad x = 0, \\
 B_L(v, \sigma, Z_{s}, r_n): =\frac{Z_{s}}{2} \left({1-r_n}\right){v} + \frac{1+r_n}{2}{\sigma} = 0, \quad \text{at} \quad  x = L.
 \end{split}
\end{equation}

with the reflection coefficients $r_0$, $r_n$ being real numbers and  $|r_0|, |r_n| \le 1$.

Note that at $x = 0$,  while  $r_0 = -1$ yields a clamped wall, $r_0 = 0$  yields  an absorbing boundary, and  with $r_0 = 1$  we have a free-surface boundary condition. Similarly, at $x = L$, $r_n = -1$ yields a clamped wall, $r_n = 0$ yields an absorbing boundary, and  $r_n = 1$  gives a free-surface boundary condition.

1) Discretize the spatial domain $x$ into $K$ elements and denote the ${k}^{th}$ element $e^k = [x_{k}, x_{k+1}]$ and the element width $\Delta{x}_k = x_{k+1}-x_{k}$. Consider two adjacent elements  $e^k = [x_{k}, x_{k+1}]$ and  $e^{k+1} = [x_{k+1}, x_{k+2}]$ with an interface at $x_{k+1}$. At the interface we pose the physical conditions for a locked interface

\begin{align}
\text{force balance}:  \quad &\sigma^{-} = \sigma^{+} = \sigma, \nonumber \\
\text{no slip}: \quad & [\![ v]\!]  = 0,
\end{align}

where $[\![ v]\!] = v^{+} - v^{-}$, and $v^{-}, \sigma^{-}$ and $v^{+}, \sigma^{+}$ are the fields in $e^k = [x_{k}, x_{k+1}]$ and  $e^{k+1} = [x_{k+1}, x_{k+2}]$, respectively.

2) Within the element derive the weak form of the equation by multiplying both sides by an arbitrary test function and integrating over the element.

3) Next map the $e^k = [x_{k}, x_{k+1}]$ to a reference element $\xi = [-1, 1]$

4) Inside the transformed  element  $\xi \in [-1, 1]$, approximate the solution  and material parameters by a polynomial interpolant,  and write
\begin{equation}
v^k(\xi, t) = \sum_{j = 1}^{N+1}v_j^k(t) \mathcal{L}_j(\xi), \quad \sigma^k(\xi, t)  = \sum_{j = 1}^{N+1}\sigma_j^k(t) \mathcal{L}_j(\xi),
\end{equation}

\begin{equation}
\rho^k(\xi) = \sum_{j = 1}^{N+1}\rho_j^k \mathcal{L}_j(\xi), \quad \mu^k(\xi) = \sum_{j = 1}^{N+1}\mu_j^k \mathcal{L}_j(\xi),
\end{equation}

where $ \mathcal{L}_j$ is the $j$th interpolating polynomial of degree $N$. If we consider  nodal basis then the interpolating polynomials satisfy $ \mathcal{L}_j(\xi_i) = \delta_{ij}$.

The interpolating nodes $\xi_i$, $i = 1, 2, \dots, N+1$ are the nodes of a Gauss quadrature with

\begin{equation}
 \sum_{i = 1}^{N+1} f(\xi_i)w_i \approx \int_{-1}^{1}f(\xi) d\xi,
\end{equation}

where $w_i$ are quadrature weights.

5) At the element boundaries $\xi = \pm 1$, we generate $\widehat{v}^{k}(\pm 1, t)$ $\widehat{\sigma}^{k}(\pm 1, t)$ by solving a Riemann problem and constraining the solutions against interface and boundary conditions. Then numerical fluctuations $F^k(-1, t)$ and $G^k(1, t)$ are obtained by penalizing hat variables against the incoming characteristics only.

6) Finally, the flux fluctuations are appended to the semi-discrete PDE with special penalty weights and we have

\begin{equation}
\begin{split}
\frac{d \boldsymbol{v}^k( t)}{ d t} &= \frac{2}{\Delta{x}_k} W^{-1}({\boldsymbol{\rho}}^{k})\left(Q \boldsymbol{\sigma}^k( t) - \boldsymbol{e}_{1}F^k(-1, t)- \boldsymbol{e}_{N+1}G^k(1, t)\right),
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\frac{d \boldsymbol{\sigma}^k( t)}{ d t} &= \frac{2}{\Delta{x}_k} W^{-1}(1/{\boldsymbol{\mu}^{k}})\left(Q \boldsymbol{v}^k( t)  + \boldsymbol{e}_{1}\frac{1}{Z_{s}^{k}(-1)}F^k(-1, t)- \boldsymbol{e}_{N+1}\frac{1}{Z_{s}^{k}(1)}G^k(1, t)\right),
\end{split}
\end{equation}

where
\begin{align}
\boldsymbol{e}_{1} = [ \mathcal{L}_1(-1), \mathcal{L}_2(-1), \dots,  \mathcal{L}_{N+1}(-1) ]^T, \quad  \boldsymbol{e}_{N+1} = [ \mathcal{L}_1(1), \mathcal{L}_2(1), \dots,  \mathcal{L}_{N+1}(1) ]^T,
\end{align}
and
\begin{align}
G^k(1, t):= \frac{Z_{s}^{k}(1)}{2} \left(v^{k}(1, t)-\widehat{v}^{k}(1, t) \right) + \frac{1}{2}\left(\sigma^{k}(1, t)- \widehat{\sigma}^{k}(1, t)\right),
\end{align}
\begin{align}
F^{k}(-1, t):= \frac{Z_{s}^{k}(-1)}{2} \left(v^{k}(-1, t)-\widehat{v}^{k}(-1, t) \right) - \frac{1}{2}\left(\sigma^{k}(-1, t)- \widehat{\sigma}^{k}(-1, t)\right).
\end{align}

And the weighted elemental mass matrix $W^N(a)$ and the stiffness matrix $Q^N $ are defined by

\begin{align}
W_{ij}(a) = \sum_{m = 1}^{N+1} w_m \mathcal{L}_i(\xi_m)  {\mathcal{L}_j(\xi_m)} a(\xi_m), \quad Q_{ij} = \sum_{m = 1}^{N+1} w_m \mathcal{L}_i(\xi_m)  {\mathcal{L}_j^{\prime}(\xi_m)}.
\end{align}

7) Time extrapolation can be performed using any stable time stepping scheme like Runge-Kutta or ADER scheme.This notebook implements both Runge-Kutta and ADER schemes for solving the free source version of the elastic wave equation in a homogeneous media. To keep the problem simple, we use as spatial initial condition a Gauss function with half-width $\delta$

\begin{equation}
v(x,t=0)  = e^{-1/\delta^2 (x - x_{o})^2}, \quad \sigma(x,t=0) = 0
\end{equation}

**** Exercises****
1. Lagrange polynomial is used to interpolate the solution and the material parameters. First use polynomial degree 2 and then 6. Compare the simulation results in terms of accuracy of the solution (third and fourth figures give erros). At the end of simulation, time required to complete the simulation is also printed. Also compare the time required to complete both simulations.

2. We use quadrature rules: Gauss-Legendre-Lobatto and Gauss-Legendre. Run simulations once using Lobatto and once using Legendre rules. Compare the difference.

3. Now fix the order of polynomial to be 6, for example. Then use degree of freedom 100 and for another simulation 250. What happpens? Also compare the timre required to complete both simulations.

4. Experiment with the boundary conditions by changing the reflection coefficients $r_0$ and $r_n$.

5. You can also play around with sinusoidal initial solution instead of the Gaussian.

6. Change the time-integrator from RK to ADER. Observe if there are changes in the solution or the CFL number. Vary the polynomial order N.

```python
# Parameters initialization and plotting the simulation
# Import necessary routines
import Lagrange
import numpy as np
import timeintegrate
import quadraturerules
import specdiff
import utils
import matplotlib.pyplot as plt
import timeit                         # to check the simulation time

#plt.switch_backend("TkAgg")          # plots in external window
plt.switch_backend("nbagg")           # plots within this notebook
```

```python
# Simulation start time
start = timeit.default_timer()

# Tic
iplot = 20

# Physical domain x = [ax, bx] (km)
ax = 0.0                     # (km)
bx = 20.0                    # (km)

# Choose quadrature rules and the corresponding nodes
# We use Gauss-Legendre-Lobatto (Lobatto) or  Gauss-Legendre (Legendre) quadrature rule.

#node = 'Lobatto'
node = 'Legendre'

if node not in ('Lobatto', 'Legendre'):
    print('quadrature rule not implemented. choose node = Legendre or node = Lobatto')
    exit(-1)

# Polynomial degree N: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
N = 4                        # Lagrange polynomial degree
NP = N+1                     # quadrature nodes per element

if N < 1 or N > 12:
    print('polynomial degree not implemented. choose N>= 1 and N <= 12')
    exit(-1)

# Degrees of freedom to resolve the wavefield
deg_of_freedom = 400 # = num_element*NP

# Estimate the number of elements needed for a given polynomial degree and degrees of freedom
num_element = round(deg_of_freedom/NP) # = deg_of_freedom/NP

# Initialize the mesh
y = np.zeros(NP*num_element)

# Generate num_element dG elements in the interval [ax, bx]
x0 = np.linspace(ax,bx,num_element+1)
dx = np.diff(x0)             # element sizes

# Generate Gauss quadrature nodes (psi): [-1, 1] and weights (w)
if node == 'Legendre':
    GL_return = quadraturerules.GL(N)
    psi = GL_return['xi']
    w = GL_return['weights'];

if node == 'Lobatto':
    gll_return = quadraturerules.gll(N)
    psi = gll_return['xi']
    w = gll_return['weights']

# Use the Gauss quadrature nodes (psi) generate the mesh (y)
for i in range (1,num_element+1):
    for j in range (1,(N+2)):
        y[j+(N+1)*(i-1)-1] = dx[i-1]/2.0 * (psi[j-1] + 1.0) +x0[i-1]

# Overide with the exact degrees of freedom
deg_of_freedom = len(y);

# Generate the spectral difference operator (D) in the reference element: [-1, 1]
D = specdiff.derivative_GL(N, psi, w)

# Boundary condition reflection coefficients
r0 = 0                            # r=0:absorbing, r=1:free-surface, r=-1: clamped
rn = 0                            # r=0:absorbing, r=1:free-surface, r=-1: clamped

# Initialize the wave-fields
L = 0.5*(bx-ax)
delta = 0.01*(bx-ax)
x0 = 0.5*(bx+ax)

omega = 4.0
#u = np.sin(omega*np.pi*y/L)                                          # Sine function
u = 1/np.sqrt(2.0*np.pi*delta**2)*np.exp(-(y-x0)**2/(2.0*delta**2))   # Gaussian

u = np.transpose(u)
v = np.zeros(len(u))

U = np.zeros(len(u))
V = np.zeros(len(u))

print('points per wavelength: ', round(delta*deg_of_freedom/(2*L)))

# Material parameters
cs = 3.464                               # shear wave speed (km/s)
rho = 0*y  + 2.67                        # density (g/cm^3)
mu = 0*y + rho * cs**2                   # shear modulus (GPa)
Zs = rho*cs                              # shear impedance

# Time stepping parameters
cfl = 0.5                                # CFL number
dt = (0.25/(cs*(2*N+1)))*min(dx)         # time-step (s)
t = 0.0                                  # initial time
Tend = 2                                # final time (s)
n = 0                                    # counter

# Difference between analyticla and numerical solutions
EV = [0]                                 # initialize errors in V (velocity)
EU = [0]                                 # initialize errors in U (stress)
T = [0]                                  # later append every time steps to this

# Initialize animated plot for velocity and stress
fig1 = plt.figure(figsize=(10,10))
ax1 = fig1.add_subplot(4,1,1)
line1 = ax1.plot(y, u, 'r', y, U, 'k--')
plt.title('numerical vs exact')
plt.xlabel('x[km]')
plt.ylabel('velocity [m/s]')

ax2 = fig1.add_subplot(4,1,2)
line2 = ax2.plot(y, v, 'r', y, V, 'k--')
plt.title('numerical vs exact')
plt.xlabel('x[km]')
plt.ylabel('stress [MPa]')

# Initialize error plot (for velocity and stress)
ax3 = fig1.add_subplot(4,1,3)
line3 = ax3.plot(T, EV, 'r')
plt.title('relative error in particle velocity')
plt.xlabel('time[t]')
ax3.set_ylim([10**-5, 1])
plt.ylabel('error')

ax4 = fig1.add_subplot(4,1,4)
line4 = ax4.plot(T, EU, 'r')
plt.ylabel('error')
plt.xlabel('time[t]')
ax4.set_ylim([10**-5, 1])
plt.title('relative error in stress')

plt.tight_layout()
plt.ion()
plt.show()

A = (np.linalg.norm(1/np.sqrt(2.0*np.pi*delta**2)*0.5*Zs*(np.exp(-(y+cs*(t+1*0.5)-x0)**2/(2.0*delta**2))\
                                               - np.exp(-(y-cs*(t+1*0.5)-x0)**2/(2.0*delta**2)))))
B = (np.linalg.norm(u))

# Loop through time and evolve the wave-fields using ADER time-stepping scheme of N+1 order of accuracy
time_integrator = 'ADER'

for t in utils.drange (0.0, Tend+dt,dt):
    n = n+1

    # ADER time-integrator
    if time_integrator  in ('ADER'):
        ADER_Wave_dG_return = timeintegrate.ADER_Wave_dG(u,v,D,NP,num_element,dx,w,psi,t,r0,rn,dt,rho,mu)
        u = ADER_Wave_dG_return['Hu']
        v = ADER_Wave_dG_return['Hv']

    # Runge-Kutta time-integrator
    if time_integrator  in ('RK'):
        RK4_Wave_dG_return = timeintegrate.RK4_Wave_dG(u,v,D,NP,num_element,dx,w,psi,t,r0,rn,dt,rho,mu)
        u = RK4_Wave_dG_return['Hu']
        v = RK4_Wave_dG_return['Hv']

    # Analytical sine wave (use it when sine function is choosen above)
    #U = 0.5*(np.sin(omega*np.pi/L*(y+cs*(t+1*dt))) + np.sin(omega*np.pi/L*(y-cs*(t+1*dt))))
    #V = 0.5*Zs*(np.sin(omega*np.pi/L*(y+cs*(t+1*dt))) - np.sin(omega*np.pi/L*(y-cs*(t+1*dt))))

    # Analytical Gaussian
    U =  1/np.sqrt(2.0*np.pi*delta**2)*0.5*(np.exp(-(y+cs*(t+1*dt)-x0)**2/(2.0*delta**2))\
                                            + np.exp(-(y-cs*(t+1*dt)-x0)**2/(2.0*delta**2)))
    V =  1/np.sqrt(2.0*np.pi*delta**2)*0.5*Zs*(np.exp(-(y+cs*(t+1*dt)-x0)**2/(2.0*delta**2))\
                                               - np.exp(-(y-cs*(t+1*dt)-x0)**2/(2.0*delta**2)))
    EV.append(np.linalg.norm(V-v)/A)
    EU.append(np.linalg.norm(U-u)/B)

    T.append(t)

    # Updating plots
    if n % iplot == 0:
        for l in line1:
            l.remove()
            del l
        for l in line2:
            l.remove()
            del l
        for l in line3:
            l.remove()
            del l
        for l in line4:
            l.remove()
            del l

        # Display lines
        line1 = ax1.plot(y, u, 'r', y, U, 'k--')
        ax1.legend(iter(line1),('Numerical', 'Analytical'))
        line2 = ax2.plot(y, v, 'r', y, V, 'k--')
        ax2.legend(iter(line2),('Numerical', 'Analytical'))
        line3 = ax3.plot(T, EU, 'k--')
        ax3.set_yscale("log")#, nonposx='clip')
        line4 = ax4.plot(T, EV, 'k--')
        ax4.set_yscale("log")#, nonposx='clip')
        plt.gcf().canvas.draw()

plt.ioff()
plt.show()

# Simulation end time
stop = timeit.default_timer()
print('total simulation time = ', stop - start)                   # print the time required for simulation
print('ploynomial degree = ', N)                                  # print the polynomial degree used
print('degree of freedom = ', deg_of_freedom)                     # print the degree of freedom
print('maximum relative error in particle velocity = ', max(EU))  # max. relative error in particle velocity
print('maximum relative error in stress = ', max(EV))             # max. relative error in stress
```

```python

```


### <center> Examples of Image Decoding and Recovery using Shearlet transform </center>

```julia
\begin{equation}
    x=\sqrt{b}
\end{equation}
```

We will compute the Digital Shearlet Transform of a signal $f$ by the equation
$$\text{DST}_{j,k,m}^{2D}(f) = \left(\overline{\psi^d_{j,k}}*f_J\right)\left(2^J A_{2^j}^{-1}M_{c_j}m\right) \text{ for } j=0,1,...,J-1 $$

where $\psi^d_{j,k} = S^d_{k/2^{j/2}}(p_j*W_j)$ ( with $S^d_{k/2^{j/2}}$ the digital shearing operator already implemented, $p_j(n)$ are the Fourier coefficients of the fan filter P (associated to the shearlet generator function) and $W_j=g_{J-j}\otimes h_{J-j}$ ($h_j$ low-pass filter associated to the scaling function and $g_j$ high pass filter, mirror filter of h, associated to the wavelet function), and $A_{2^j}$ is the parabolic scaling matrix given by
$$ A_{2^j} = \left( \begin{matrix} 2^j & 0 \\ 0 & 2^{j/2} \end{matrix}\right) $$
and the Shearing transform is given by
$$ S_k = \left( \begin{matrix} 1 & k \\ 0 & 1 \end{matrix}\right)$$

Lets define the scaling and wavelet filters now in each scale by:
$$
\hat{h}_j(\xi_1)=\prod_{k=0}^{j-1}\hat{h}(2^k\xi_1)
\text{    ,    }
\hat{g}_j(\xi_1)=\hat{g}\left(\frac{2^j\xi_1}{2}\right) \hat{h}_{j-1}(\xi_1)
$$
and the directional filter transform
$p_j(\xi_1,\xi_2)=\widehat{P(2^{J-j-1}\xi_1,2^{J-j/2}\xi_2)}$
for that one need to apply padding to the directional filter, looking carefully to $p_j*W_j=p_j*g_{J-j}\otimes h_{J-j}$ that is the vertical wavelet transform of the filter $p_j$.

#### Non separable transform and Digital Shearing.

In this case it was used a nonseparable shearlet generator
$$
\hat{\psi}(\xi)=P(\xi_1/2,\xi_2/2)\widehat{\psi_1\otimes \phi_1}(\xi)
$$
Because of $\psi_{j,k,m}(\cdot) = \psi_{j,0,m}(S_{k/2^{j/2}\cdot}$ and $\psi_{j,0}^{2D}=p_j*(g_{J-j}\otimes h_{J-j/2})$,the digital shearing to mantain the domain grid will be
$$
S_{k/2^{j/2}}(x)=\left(\left( x_{\uparrow 2^{j/2}}*_1h_{j/2}\right)(S_k\cdot)*_1\overline{h_{j/2}}\right)_{\downarrow 2^{j/2}}
$$
therefore
$$
\psi_{j,k}^{2D} = S^d_{k/2^{j/2}}\psi_{j,0}^{2D}
$$
for $j\in\{0,J-1\}$ and $|k|\leq \lceil 2^{j/2}\rceil$.

```julia
# Clone the libraris
#Pkg.clone("https://github.com/arsenal9971/Shearlab.jl.git")
```

```julia
# Loard the Pkg
push!(LOAD_PATH,pwd()*"/../../src")
import Shearlab
#using Shearlab
using PyPlot
reload("Shearlab")
using Images
```

    ┌ Info: Recompiling stale cache file /home/dsweber/.julia/compiled/v1.1/Shearlab/cyzke.ji for Shearlab [bd728e11-17d4-5679-9a00-f24958f87a1c]
    └ @ Base loading.jl:1184

```julia
# Read Data
n = 512;
# The path of the image
name = "../../data_samples/kodim12.jpg";
data_nopar = Shearlab.load_image(name, n);
data_par = Shearlab.load_image(name, n,n,1);
```

```julia
# Reduce one dimension
data_nopar = data_nopar[:,:,1];
data_par = data_par[:,:,1]
Shearlab.imageplot(data_nopar);
```

```julia
sizeX_nopar = size(data_nopar,1);
sizeY_nopar = size(data_nopar,2);
sizeX_par = size(data_par,1);
sizeY_par = size(data_par,2);
```

```julia
# Set the variables for the Shearlet trasform
rows_nopar = sizeX_nopar;
cols_nopar = sizeY_nopar;
rows_par = sizeX_par;
cols_par = sizeY_par;
X_nopar = data_nopar;
X_par = data_par;
nScales = 4;
shearLevels = ceil.((1:nScales)/2)
scalingFilter = Shearlab.filt_gen("scaling_shearlet");
directionalFilter = Shearlab.filt_gen("directional_shearlet");
waveletFilter = Shearlab.mirror(scalingFilter);
scalingFilter2 = scalingFilter;
full = 0;
```

```julia
# Compute the corresponding shearlet system without gpu
@time shearletSystem_nopar= Shearlab.getshearletsystem2D(rows_nopar,cols_nopar,4, shearLevels,full,
                                directionalFilter,
                                scalingFilter,0);
```

     33.363517 seconds (6.84 M allocations: 12.524 GiB, 37.05% gc time)

```julia
#using ArrayFire
```

```julia
# Compute the corresponding shearlet system without gpu
#@time shearletSystem_par = Shearlab.getshearletsystem2D(rows_par,cols_par,nScales,shearLevels,full,
#                                directionalFilter,
#                                scalingFilter,1);
```

**Lets look at the shearlets at different scales.**

```julia
shearlet1 = Array(shearletSystem_nopar.shearlets[:,:,1]);
Shearlab.imageplot(real(shearlet1));
```

```julia
shearlet3 = Array(shearletSystem_nopar.shearlets[:,:,3]);
Shearlab.imageplot(real(shearlet3))
```

```julia
shearlet10 = Array(shearletSystem_nopar.shearlets[:,:,10]);
Shearlab.imageplot(real(shearlet10))
```

```julia
size(shearletSystem_nopar.shearlets)
```

    (512, 768, 49)

```julia
shearlet17 = Array(shearletSystem_nopar.shearlets[:,:,49]);
Shearlab.imageplot(real(shearlet17))
```

**Lets compute the shearlet coefficients for the data**

```julia
# Compute the coefficients
@time coeffs_nopar = Shearlab.SLsheardec2D(X_nopar,shearletSystem_nopar);
```

      2.122939 seconds (60.07 k allocations: 2.033 GiB, 28.44% gc time)

```julia
# Compute the coefficients in parallel
#X_par = AFArray(convert(Array{Float32},X_par));
#@time coeffs_par = Shearlab.SLsheardec2D(X_par,shearletSystem_par);
```

**Lets look at the shearlets coefficients at different scales.**

```julia
Shearlab.imageplot(real(Array(coeffs_nopar[:,:,1])))
```

```julia
Shearlab.imageplot(real(Array(coeffs_nopar[:,:,5])))
```

```julia
Shearlab.imageplot(real(Array(coeffs_nopar[:,:,10])))
```

```julia
Shearlab.imageplot(real(Array(coeffs_nopar[:,:,16])))
```

**Lets compute the shearlet reconstruction of the data**

```julia
# Make the recovery
@time Xrec_nopar=Shearlab.SLshearrec2D(coeffs_nopar,shearletSystem_nopar);
```

      2.617282 seconds (129.03 k allocations: 2.048 GiB, 42.24% gc time)

```julia
# Make the recovery in parallel
#@time Xrec_par=Shearlab.SLshearrec2D(coeffs_par,shearletSystem_par);
```

```julia
# The recovery is very good
Shearlab.imageplot(Array(Xrec_nopar));
```

```julia
# The recovery is very good
#Shearlab.imageplot(Array(Xrec_par));
```

examples/Shearlets/Examples of Image Decoding and Recovery.ipynb