# Description

Suggestion from Josh Cook (7/15/21)

The "Regularized Incomplete Beta" function I(x,a,b) tells you the probability that your population mean is <= x given a positive samples and b negative samples
8:53
for example, if you've got 100 samples and 9 are positive, the sample mean is 9%
8:53
but how probable is it that the population mean is 7% or less?

* https://mathworld.wolfram.com/RegularizedBetaFunction.html
* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.betainc.html
* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.beta.html#scipy.special.beta
* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gamma.html#scipy.special.gamma

\begin{align}
I_x(a, b) &= \frac{\Gamma(a + b)}{\Gamma(a) \Gamma(b)} \int_0^x t^{a - 1}(1 - t)^{b - 1} dt; \quad 0 \le x \le 1 \\
\Gamma(z) &= \int_0^\infty t^{z - 1}e^{-t} dt \\
\Gamma(n + 1) &= n! ; \quad \text{for natural numbers } n\\
B(a, b) &= \text{beta function} \\
  &= \int_0^1 t^{a - 1}(1 - t)^{b - 1} dt \\
  &= \frac{\Gamma(a) \Gamma(b)}{\Gamma(a + b)}
\end{align}

# Imports

```python
%load_ext blackcellmagic

```

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
```

# $\Gamma$ Function

* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gamma.html#scipy.special.gamma

\begin{align}
\Gamma(z) &= \int_0^\infty t^{z - 1}e^{-t} dt \\
\Gamma(n + 1) &= n! ; \quad \text{for natural numbers } n\\
\end{align}

```python
from scipy.special import gamma, factorial
```

```python
gamma([0, 0.5, 1, 5])
```

    array([        inf,  1.77245385,  1.        , 24.        ])

```python
z = 2.5 + 1j
```

```python
gamma(z)
```

    (0.7747621045510835+0.7076312043795929j)

```python
gamma(z + 1), z * gamma(z)
```

    ((1.229274056998117+2.5438401155000685j),
     (1.2292740569981158+2.543840115500066j))

```python
import math
gamma(0.5) ** 2 - math.pi
```

    -4.440892098500626e-16

## Plot gamma(x) for real x

```python
x = np.linspace(-3.5, 5.5, 2251)
y = gamma(x)
```

```python
plt.plot(x, y, 'b', alpha=0.6, label='gamma(x)')

k = np.arange(1, 7)
y_fact = factorial(k - 1)
plt.plot(k, y_fact, 'k*', alpha=0.6, label='(x-1)!, x=1, 2, ...')
plt.xlim(-3.5, 5.5)
plt.ylim(-10, 25)
plt.grid()
plt.xlabel('x')
plt.legend(loc='best')
```

```python
k
```

    array([1, 2, 3, 4, 5, 6])

# Beta function

* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.beta.html#scipy.special.beta

\begin{align}
B(a, b) &= \int_0^1 t^{a - 1}(1 - t)^{b - 1} dt \\
  &= \frac{\Gamma(a) \Gamma(b)}{\Gamma(a + b)}
\end{align}

Note:
* this function is symmetric
* $B(1, b) = 1/b$

https://en.wikipedia.org/wiki/Beta_distribution

In Bayesian inference, the beta distribution is the conjugate prior probability distribution for the Bernoulli, binomial, negative binomial and geometric distributions. The beta distribution is a suitable model for the random behavior of percentages and proportions.

```python
import scipy.special as sc
```

```python
from scipy.stats import beta
```

```python
sc.beta(2, 3)
```

    0.08333333333333333

```python
gamma(2) * gamma(3) / gamma(2 + 3)
```

    0.08333333333333333

```python
sc.beta(1, 5)
```

    0.2

## Plot some beta distributions

* https://www.astroml.org/book_figures/chapter3/fig_beta_distribution.html
* https://en.wikipedia.org/wiki/Beta_distribution

Pdf of the beta distribution for $0 \le x \le 1$ and shape parameters $\alpha, \beta \gt 0$

\begin{align}
f(x; \alpha, \beta) &= \frac{1}{B(\alpha, \beta)} x^{\alpha - 1}(1-x)^{\beta - 1} \\
 &= f(1-x; \beta, \alpha)
\end{align}

$B(\alpha, \beta)$ is a normalization constant (total prob = 1).

$x$ is an observed value of a random process $X$

$B(\alpha, \beta) \rightarrow $ Bernoulli distribution when $\alpha, \beta \rightarrow 0$

```python
# Define distribution parameters

alpha_values = [0.5, 1.5, 3.0, 0.5]
beta_values = [0.5, 1.5, 3.0, 1.5]
linestyles = ['-', '--', ':', '-.']
x = np.linspace(0, 1, 1002)[1:-1]

fig, ax = plt.subplots(figsize=(5, 4))
for a, b, ls in zip(alpha_values, beta_values, linestyles):
    dist = beta(a, b)
    plt.plot(x, dist.pdf(x), ls=ls,
#              c='black',
             label=r'$\alpha=%.1f,\ \beta=%.1f$' % (a, b))

plt.xlim(0, 1)
plt.ylim(0, 3)
plt.xlabel('$x$')
plt.ylabel(r'$p(x|\alpha, \beta)$')
plt.title('Beta Distribution')
plt.grid()
plt.legend()
plt.show()
```

# Incomplete Beta function

The cumulative distribution function of the beta distribution is the regularized incomplete beta function

https://en.wikipedia.org/wiki/Beta_distribution

\begin{align}
F(x; \alpha, \beta) &= \frac{B_x(\alpha, \beta)}{B(\alpha, \beta)} = I_x(\alpha, \beta) \\
  &= 1 - F(1-x; \beta, \alpha) = 1 - I_{1 - x}(\beta, \alpha)
\end{align}

* https://mathworld.wolfram.com/RegularizedBetaFunction.html
* https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.betainc.html
* Gradshteyn & Ryzhik, p950

\begin{align}
B_x(a, b) &= \int_0^x t^{a - 1}(1 - t)^{b - 1} dt; \quad \text{Incomplete beta function. Note the upper limit} \\
B(a, b) &= \int_0^1 t^{a - 1}(1 - t)^{b - 1} dt
  = \frac{\Gamma(a) \Gamma(b)}{\Gamma(a + b)} \\
\Gamma(z) &= \int_0^\infty t^{z - 1}e^{-t} dt \\
I_x(a, b) &= \frac{B_x(a, b)}{B(a, b)}; \quad \text{Regularized incomplete beta function} \quad (0 \le x \le 1)\\
    &= \frac{\Gamma(a + b)}{\Gamma(a) \Gamma(b)} \int_0^x t^{a - 1}(1 - t)^{b - 1} dt; \quad 0 \le x \le 1
\end{align}

Note

When $x = 1$

\begin{align}
  B_{x=1}(a, b) &= B(a, b) \\
    I_{x=1}(a, b) &= 1
\end{align}

```python
import scipy.special as sc
```

```python
# x = 1 -> Bx=1(a, b) = 1
sc.betainc(0.2, 3.5, 1.0)
```

    1.0

```python
sc.betainc(12, 88, [0.07, 0.1])
```

    array([0.04396123, 0.28511234])

```python
sc.betainc(1, 10, [0.01, 0.05, 0.1, 0.15, 0.25, 0.5, 0.75])
```

    array([0.09561792, 0.40126306, 0.65132156, 0.8031256 , 0.94368649,
           0.99902344, 0.99999905])

```python
sc.betainc(10, 10, [0.01, 0.05, 0.1, 0.15, 0.25, 0.5, 0.75])
```

    array([8.50910473e-16, 5.93933906e-09, 3.92988233e-06, 1.43507369e-04,
           8.90327930e-03, 5.00000000e-01, 9.91096721e-01])

## Plot cdf (regularized incomplete beta function)

```python
# Define distribution parameters

alpha_values = [1, 10, 100]
beta_values = [10, 100, 1000]
linestyles = ['-', '--', ':', '-.']
x = np.linspace(0, 1, 1002)[1:-1]

fig, ax = plt.subplots(figsize=(10, 6))
for a, b, ls in zip(alpha_values + beta_values, beta_values + alpha_values, linestyles + linestyles):
    plt.plot(x, sc.betainc(a, b, x), ls=ls,
#              c='black',
             label=r'$\alpha=%.1f,\ \beta=%.1f$' % (a, b)
            )

plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel('$x$')
plt.ylabel(r'$P(B(\alpha, \beta) < x)$')
plt.title('CDF Beta Distribution')
plt.grid()
plt.legend()
```

```python
sc.betainc(3, 97, 0.09)
```

    0.9948675347954182

```python
k = np.arange(1, 7)
y_fact = factorial(k - 1)
plt.plot(k, y_fact, 'k*', alpha=0.6, label='(x-1)!, x=1, 2, ...')
```

```python
# Define distribution parameters

def plot_betainc(alphas, betas, p=None):
    x = np.linspace(0, 1, 1002)[1:-1]
    linestyles = ["solid", "dashed", "dotted", ":", "dashdot"]
    lines = linestyles[:len(alphas)]
    fig, ax = plt.subplots(figsize=(10, 6))
    for a, b, ls in zip(alphas, betas, lines):
        plt.plot(
            x, sc.betainc(a, b, x), ls=ls, label=r"$\alpha=%.1f,\ \beta=%.1f$" % (a, b)
        )
        phat = p if p else a / b

        plt.plot(phat, sc.betainc(a, b, phat), "k*", alpha=0.6)

    plt.xlim(0, 0.3)
    plt.ylim(0, 1)
    plt.xlabel("$x$")
    plt.ylabel(r"$P(B(\alpha, \beta) < x)$")
    plt.title("Beta Distribution")
    plt.grid()
    plt.legend()
```

```python
alpha_values = [5, 5, 5, 5, 5, ]
beta_values = [45, 95, 145, 195, 295]

plot_betainc(alpha_values, beta_values, p=0.05)
```

```python
alpha_values = [1, 2, 5, 10, 20]
beta_values = [50, 100, 150, 200]

plot_betainc(alpha_values, beta_values)
```

```python
alpha_values = [5, 10, 15, 20, 25]
beta_values = [100, 100, 100, 100]

plot_betainc(alpha_values, beta_values)
```

```python
sc.betainc(1, 10, 0.1)
```

    0.6513215599

```python
sc.betainc(5, 50, 0.1)
```

    0.6385528602726217

```python
sc.betainc(10, 100, 0.1)
```

    0.6596901506172208

```python
sc.betainc(50, 500, 0.1)
```

    0.776519303109878

```python
sc.betainc(100, 1000, 0.1)
```

    0.8525208237418817

```python
sc.betainc(200, 2000, 0.1)
```

    0.9280185129747222

```python
sc.betainc(300, 3000, 0.1)
```

    0.962580883229439

## Simulate various volumes/block rates

```python
tgt_br = 0.06
pop_br = 0.05

def simulate_rep_tgt(pop_br, tgt_br):
    repairs = [5, 10, 20, 50, 100, 250, 500, 1000, 1500, 2000, 2500, 3000, 3500]

    return (
        pd.DataFrame(data={"repairs": repairs})
        .assign(
            blocks=lambda x: x.apply(lambda r: math.ceil(pop_br * r["repairs"]), axis=1),
            notblocks=lambda x: x['repairs'] - x['blocks'],
            sample_br=lambda x: x['blocks'] / x['repairs']
        )
        .assign(prob_le=lambda r: sc.betainc(r["blocks"], r["notblocks"], pop_br))
        .assign(prob_gt=lambda x: 1 - x["prob_le"])
        .assign(tgt_br=tgt_br)
        .assign(prob_le_tgt=lambda r: sc.betainc(r["blocks"], r["notblocks"], tgt_br))
        .assign(prob_gt_tgt=lambda x: 1 - x["prob_le_tgt"])
    )
```

```python
tgt_br = 0.05

tmp = simulate_rep_tgt(pop_br=0.20, tgt_br=tgt_br)
display(tmp)

fig, ax = plt.subplots(figsize=(10, 10))
simulate_rep_tgt(pop_br=0.17, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.20')

simulate_rep_tgt(pop_br=0.13, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.13')
simulate_rep_tgt(pop_br=0.12, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.12')
simulate_rep_tgt(pop_br=0.11, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.11')
simulate_rep_tgt(pop_br=0.10, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.10')
simulate_rep_tgt(pop_br=0.09, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.09')
simulate_rep_tgt(pop_br=0.08, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.08')

# simulate_rep_tgt(pop_br=0.49, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.13')
# simulate_rep_tgt(pop_br=0.31, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.12')
# simulate_rep_tgt(pop_br=0.29, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.11')
# simulate_rep_tgt(pop_br=0.27, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.10')
# simulate_rep_tgt(pop_br=0.25, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.09')
# simulate_rep_tgt(pop_br=0.23, tgt_br=tgt_br).plot('repairs', 'prob_gt_tgt', ax=ax, style='.-', label='pop_br=0.08')

ax.set_xscale('log')
plt.grid(which='both')
ax.set_title(f"Probability the BR is > {tgt_br}")
plt.show()
```

# Scipy stats

```python
from scipy import stats
```

```python
print(stats.beta.__doc__)
```

    A beta continuous random variable.

        As an instance of the `rv_continuous` class, `beta` object inherits from it
        a collection of generic methods (see below for the full list),
        and completes them with details specific for this particular distribution.

        Methods
        -------
        rvs(a, b, loc=0, scale=1, size=1, random_state=None)
            Random variates.
        pdf(x, a, b, loc=0, scale=1)
            Probability density function.
        logpdf(x, a, b, loc=0, scale=1)
            Log of the probability density function.
        cdf(x, a, b, loc=0, scale=1)
            Cumulative distribution function.
        logcdf(x, a, b, loc=0, scale=1)
            Log of the cumulative distribution function.
        sf(x, a, b, loc=0, scale=1)
            Survival function  (also defined as ``1 - cdf``, but `sf` is sometimes more accurate).
        logsf(x, a, b, loc=0, scale=1)
            Log of the survival function.
        ppf(q, a, b, loc=0, scale=1)
            Percent point function (inverse of ``cdf`` --- percentiles).
        isf(q, a, b, loc=0, scale=1)
            Inverse survival function (inverse of ``sf``).
        moment(n, a, b, loc=0, scale=1)
            Non-central moment of order n
        stats(a, b, loc=0, scale=1, moments='mv')
            Mean('m'), variance('v'), skew('s'), and/or kurtosis('k').
        entropy(a, b, loc=0, scale=1)
            (Differential) entropy of the RV.
        fit(data)
            Parameter estimates for generic data.
            See `scipy.stats.rv_continuous.fit <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_continuous.fit.html#scipy.stats.rv_continuous.fit>`__ for detailed documentation of the
            keyword arguments.
        expect(func, args=(a, b), loc=0, scale=1, lb=None, ub=None, conditional=False, **kwds)
            Expected value of a function (of one argument) with respect to the distribution.
        median(a, b, loc=0, scale=1)
            Median of the distribution.
        mean(a, b, loc=0, scale=1)
            Mean of the distribution.
        var(a, b, loc=0, scale=1)
            Variance of the distribution.
        std(a, b, loc=0, scale=1)
            Standard deviation of the distribution.
        interval(alpha, a, b, loc=0, scale=1)
            Endpoints of the range that contains alpha percent of the distribution

        Notes
        -----
        The probability density function for `beta` is:

        .. math::

            f(x, a, b) = \frac{\Gamma(a+b) x^{a-1} (1-x)^{b-1}}
                              {\Gamma(a) \Gamma(b)}

        for :math:`0 <= x <= 1`, :math:`a > 0`, :math:`b > 0`, where
        :math:`\Gamma` is the gamma function (`scipy.special.gamma`).

        `beta` takes :math:`a` and :math:`b` as shape parameters.

        The probability density above is defined in the "standardized" form. To shift
        and/or scale the distribution use the ``loc`` and ``scale`` parameters.
        Specifically, ``beta.pdf(x, a, b, loc, scale)`` is identically
        equivalent to ``beta.pdf(y, a, b) / scale`` with
        ``y = (x - loc) / scale``.

        Examples
        --------
        >>> from scipy.stats import beta
        >>> import matplotlib.pyplot as plt
        >>> fig, ax = plt.subplots(1, 1)

        Calculate a few first moments:

        >>> a, b = 2.31, 0.627
        >>> mean, var, skew, kurt = beta.stats(a, b, moments='mvsk')

        Display the probability density function (``pdf``):

        >>> x = np.linspace(beta.ppf(0.01, a, b),
        ...                 beta.ppf(0.99, a, b), 100)
        >>> ax.plot(x, beta.pdf(x, a, b),
        ...        'r-', lw=5, alpha=0.6, label='beta pdf')

        Alternatively, the distribution object can be called (as a function)
        to fix the shape, location and scale parameters. This returns a "frozen"
        RV object holding the given parameters fixed.

        Freeze the distribution and display the frozen ``pdf``:

        >>> rv = beta(a, b)
        >>> ax.plot(x, rv.pdf(x), 'k-', lw=2, label='frozen pdf')

        Check accuracy of ``cdf`` and ``ppf``:

        >>> vals = beta.ppf([0.001, 0.5, 0.999], a, b)
        >>> np.allclose([0.001, 0.5, 0.999], beta.cdf(vals, a, b))
        True

        Generate random numbers:

        >>> r = beta.rvs(a, b, size=1000)

        And compare the histogram:

        >>> ax.hist(r, density=True, histtype='stepfilled', alpha=0.2)
        >>> ax.legend(loc='best', frameon=False)
        >>> plt.show()

```python
dist = stats.beta(50, 500)
```

Calculate regularized incomplete beta function using cdf of a beta distribution and `stats.beta`

```python
dist.cdf(0.1)
```

    0.776519303109878

Calculate using the `scipy.special.betainc` function

```python
sc.betainc(50, 500, 0.1)
```

    0.776519303109878

```python

```


# POL280 Bayesian Modelling Memo & Codes

## Lecture 1: Introduction (04/13/2017)

###  Monte Carlo Simulation

```R
## Monte Carlo Simulation ##

#install.packages("plotrix")
library(plotrix)
library(grid)

## Plot Rectangle and Circle
plot(c(-1, 1), c(-1, 1), type = "n", asp = 1)
rect(-1, -1, 1, 1)
draw.circle(0, 0, 1)

## Conduct Simulation
nsamp <- 10000 # The size of sample
inside <- NA # Storage for those values inside the cirlce
for(i in 1:nsamp){
  x <- runif(2, -1, 1) ## Draw two values from uniform distribution (-1, 1)
  if (sqrt(x[1] * x[1] + x[2] * x[2]) < 1) {
    inside[i] <- 1; points(x[1], x[2], col = "orange")
  }
  if (sqrt(x[1] * x[1] + x[2] * x[2]) > 1) {
    inside[i] <- 0; points(x[1], x[2], col = "black")
  }
}

table(inside)
4 * (7848 / 10000)
```

## Lecture 2: Simple Bayesian Models and Priors (04/20/2017)

### The Monty Hall Problem

Suppose you're on a game show and you're given teh choice of three doors: Behind one door is a car; behind the otheres, goats. You pick a door, say No.1, and the host, who knows what's behind the doors, opens another door, say No.3, which has a goat. He then says to you "Do you wan to pick door No.2?" Is it to your advantage to switch your choice?

Let's pick door 1, and Monty then opens door 2 to show us a goat.

Let A represent the outcome that the car is behind door 1. Let B represent the event that we're shown a goat behind door 2:

$$P(A|B) = \frac{P(B|A) P(A)}{P(B)}$$

Let's Simulate by R!

```R
## Monty Hall Problem Simulation ##
doors <- c("1", "2", "3") # Three Doors
sim_montyhall <- function(door){
car <- sample(doors, 1) # define the door with car
pick <- doors[door] # your pick of the door
open <- sample(doors[which(doors!=car & doors!=pick)], 1) # defined the door opened
return(pick==car) # return if the car is behind your picked door.
}
sims <- replicate(10000, sim_montyhall(1) ) #repeat the process many times
mean(sims) # The probability
```

0.3372

### Bayesian Inference

#### Do Statistical Inference

In the statistical inference context, A is replaced with $\theta$ (unknown parameters), and B is replaced with $y$ (data). Therefore, the equation becomes:

\begin{align}
P(\theta|y) &= \frac{P(\theta) P(y|\theta)}{P(y)} \\
&\propto P(\theta) L(\theta|y)
\end{align}

Intuitively stated:

Posteriot Probability (of parameter level) $\propto$ Prior Probability $\times$ Likelihood Function

It's all about **distribution**.

#### Setting Prior

Given the function given above, **prior matters** the most when the **data is sparse** (small size?) Data dominates the prior when there is enough data.

The variance of prior matters. As the **variance of prior distribution increases**, the **impact of data on poterior distribution increases**.

### Bayesian Mechanics

#### Three Steps to Model Bayesan

1. Specify 1) the probability model of **how $y$ were generated**, & 2) **prior belief** of $\theta$ (probability distribution)

2. **Update $\theta$** by conditioning the probability model on the **data**.

3. Evaluate model fit and sensitivity.

#### First Step (1): Probability Models

An assumption about the probability distribution $p(y | \theta)$ that generated $y$. Defined as **PDF** (probability density functons) **if $y$ is continuous** random variables, defined as **PMF** (probability mass functions) **if $y$ is discrete** random variables.

$p(y | \theta)$ is also called **likelihood function**. It is calculated as:

$$L(\theta | y) = \Pi_{i=1}^n p(y_i | \theta)$$

Easier to with natural log of the likelihood:

$$\mathit{l}(\theta | y) = \mbox{log}L(\theta | y)$$

Because the original likelihood often become very small, so we want to deal with the manageable numbers by taking log of it.

* Use of MLE($\hat{\theta}$) by frequentists cannot answer questions such as: $p (\theta > 0)$; $p(\theta \in (a, b))$ or relative likelihood of competing models.

#### First Step (2): Specifying Priors

**Beta-Binomial Distribution** = Beta posterior

\begin{align}
\mbox{Prior } p(\theta) &= \frac{\Gamma(\alpha + \beta)}{\Gamma (\alpha) + \Gamma (\beta)} \theta^{\alpha-1} (1-\theta)^{\beta-1} \\
\mbox{Likelihood } &=
\begin{pmatrix}
n \\ y
\end{pmatrix}
\theta^y (1-\theta)^{n-y}
\end{align}

Then:

\begin{align}
\mbox{Posterior } &\propto \mbox{Prior} \times \mbox{Likelihood} \\
\pi(\theta | y) &\propto \left\{ \frac{\Gamma(\alpha + \beta)}{\Gamma (\alpha) + \Gamma (\beta)} \theta^{\alpha-1} (1-\theta)^{\beta-1} \right\} \times \left\{
\begin{pmatrix}
n \\ y
\end{pmatrix}
\theta^y (1-\theta)^{n-y} \right\} \\
&\propto \theta^{\alpha - 1 + y} + (1 - \theta)^{\beta - 1 + n - y}
\end{align}

The Solution is:

\begin{align}
\pi (\theta | y) &\approx \mbox{Beta}( \alpha + y, \beta + n - y) \\
E(\theta | y) &\approx \frac{\alpha + y}{\alpha + \beta + n}
\end{align}

Try in R! (Data is not provided)

```R
library(dplyr)
setwd("c:/")
load("Dropbox/Files/baseball.Rda")
#
career_filtered <- career %>% filter(AB >= 500)
m <- MASS::fitdistr(career_filtered$average, dbeta,
start = list(shape1 = 1, shape2 = 10))
alpha0 <- m$estimate[1]
beta0 <- m$estimate[2]
#
career_eb <- career %>%
mutate(eb_estimate = (H + alpha0) / (AB + alpha0 + beta0))
```

**Gamma Distribution** = Gamma Posterior

\begin{align}
\mbox{Gamma } &= \frac{\beta^{\alpha}}{\Gamma (\alpha)} \theta^{\alpha-1} e^{- \beta \theta} \\
\mbox{Poisson PMF } &= p(y | \theta) = \frac{e^{-\theta} \theta^{y_i}}{y_i !} \\
\mbox{Poisson Likelihood } &= \mathit{L}(\theta | y) = \hat{\Pi}_{i=1}^n \frac{e^{-\theta} \theta^{y_i}}{y_i !} \\
&= \frac{e^{-\theta n} \theta^{\sum_{i=1}{n} y_i} }{y_1 ! y_2 ! \dots y_n !} \\
\pi(\theta | y) &\propto \frac{\beta^{\alpha}}{\Gamma (\alpha)} \theta^{\alpha-1} e^{- \beta \theta} \times \frac{e^{-\theta n} \theta^{\sum_{i=1}{n} y_i} }{y_1 ! y_2 ! \dots y_n !} \\
&\propto \theta^{\alpha - 1 + \Sigma y} e^{- \theta (\beta + n)} \\
&\propto \mbox{Gamma }(\alpha + \Sigma y, \beta + n)
\end{align}


# Sinais periódicos

Neste notebook avaliaremos os sinais periódicos e quais são as condições necessárias para periodicidade.

Esta propriedade dos sinais está ligada ao ***deslocamento no tempo***, uma transformação da variável independente.

Um sinal periódico, contínuo, é aquele para o qual a seguinte propriedade é válida:

\begin{equation}
x(t) = x(t \pm mT_p),
\end{equation}
ou seja, o valor do sinal no instante $t$ [s] é o mesmo para o instante $t \pm mT_p$ [s]. Dessa forma, o sinal se repete a cada
período $T_p$.

$T_p$ é o chamado período fundamental do sinal periódico. Neste caso, $x(t) = x(t \pm T_p) = x(t \pm 2T_p) = ... = x(t \pm kT_p)$.

Para os sinais discretos a definição é análoga:

\begin{equation}
x[n] = x[n \pm m N_p],
\end{equation}
com $N_p$ sendo um número de amostras inteiro.

Um sinal que não é periódico é chamado de aperiódico.

Vamos ver alguns exemplos de sinais periódicos contínuos e discretos.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
from scipy import signal # some signals
import IPython.display as ipd # to play signals
```

```python
# Configurações gerais
fs = 44100
t = np.linspace(0, 1, fs) # vetor temporal
freq = 1000 # Frequencia fundamental
```

```python
# seno ou cosseno
xt = np.sin(2*np.pi*freq*t)
# Figura
plt.figure()
plt.title('Seno')
plt.plot(t, xt, '-b', linewidth = 2, label = 'seno - 1000 [Hz]')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/1000))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

## Um seno com 2 frequências
Se tivermos um sinal

\begin{equation}
x(t) = x(t) = \mathrm{sin}(2  \pi \ m_1 \ f t) + \mathrm{sin}(2  \pi \ m_2 \ f t),
\end{equation}

ele será um sinal periódico desde que $\frac{m_2}{m_1}$ seja um número racional. Do contrário, o sinal será quase-periódico. Ele parecerá periódico, mas se você olhar os detalhes, vai notar que o sinal nunca se repete.

```python
# seno ou cosseno - 2 frequencias
m = 3 #1.4*np.sqrt(2)
xt = np.sin(2*np.pi*freq*t) +  np.sin(2*np.pi*m*freq*t)
# Figura
plt.figure()
plt.title('Seno')
plt.plot(t, xt, '-b', linewidth = 2, label = 'seno - 2 freq')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
# dente de serra
xt = signal.sawtooth(2 * np.pi * freq * t)
# Figura
plt.figure()
plt.title('Dente de serra')
plt.plot(t, xt, '-b', linewidth = 2, label = 'sawtooth')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
# onda quadrada
xt = signal.square(2 * np.pi * freq * t)
# Figura
plt.figure()
plt.title('Onda quadrada')
plt.plot(t, xt, '-b', linewidth = 2, label = 'square')
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, 3/freq))
plt.tight_layout()
plt.show()
# play
ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python
N = 9
n = np.arange(N)
xn = [2, 1, -1, 2, 1, -1, 2, 1, -1]

# Figura
plt.figure()
plt.title('Sinal discreto periódico')
plt.stem(n, xn, '-b', basefmt=" ", use_line_collection=  True)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Amostra [-]')
plt.ylabel('Amplitude [-]')
plt.ylim((-2.2, 2.2))
plt.tight_layout()
plt.show()
```

## Sinais discretos periódicos

A periodicidade em sinais discretos tem um limite prático. Pra pensar nisso, podemos imaginar um sinal contínuo $x(t) = \mathrm{cos}(\omega t)$. À medida que a frequência, $f$, do sinal aumenta, sua taxa de oscilação também aumenta. Mas, o que aconteceria no caso de um sinal do tipo

\begin{equation}
x[n] = \mathrm{cos}(\omega n) \ ?
\end{equation}

```python
N = 50
n = np.arange(N)
w = 0
xn = np.cos(w*n)

# Figura
plt.figure()
plt.title('Cosseno discreto')
plt.stem(n, xn, '-b', label = r'\omega = {:.3} [rad/s]'.format(float(w)), basefmt=" ", use_line_collection=  True)
plt.legend(loc = 'best')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Amostra [-]')
plt.ylabel('Amplitude [-]')
plt.ylim((-1.2, 1.2))
plt.tight_layout()
plt.show()
```

Aula 5 - sinais periodicos/sinais periodicos.ipynb


```python
%display typeset
```

# Usando Modelos Populacionais Logísticos para Epidemias
Nesta aula vamos conhecer o modelo logístico de Richards, e estuda sua aplicabilidade no estudo de epidemias. Este notebook se baseia [neste artigo](https://userweb.ucs.louisiana.edu/~xxw6637/papers/JTB2012.pdf)

O modelo de Richards se baseia em um modelo proposto por Verhulst em 1838.

$$\frac{dN(t)}{dt}=r N(t)\left[1-\frac{N(t)}{K}\right]$$

Richards generalizou este modelo em 1959, adicionando o parâmetro $a$ permitindo o desvio da dependência estrita da densidade.

```python
var('C r1 K a t')
C = function('C')(t)
dcdt = diff(C,t)==r1*C(1-(C/K)^a)
dcdt
```

```python
desolve(dcdt, C, ivar=t)
```

O modelo de Richards admite uma solução explícita:

$$C(t)=K\left[1+a e^{-a r1(t-t_c)}\right]^{-1/a},$$

onde $t_c$ é o ponto no tempo em que a segunda derivada de $C(t)$ se torna 0, ou seja, quando $C(t)=K(1+a)^{-1/a}$. Seja $r=a r1$, então a equação acima se torna:
$$C(t)=K\left[1+a e^{-r(t-t_c)}\right]^{-1/a}.$$

## Relação com o modelo SIR
Seja o modelo SIR,
$$
\frac{dS}{dt} = -\beta S I/N \label{dsdt}
$$
$$\frac{dI}{dt} = \beta S I/N -\gamma I$$
$$\frac{dR}{dt} = \gamma I$$

Onde podemos considerar que $N=S+I$. A partir das equações acima temos que

$$\frac{d(S+I)}{ds}=\frac{\gamma (S+I)}{\beta S}$$

cuja solução é:

$$S(t)+I(t) = cS(t)^{\gamma/\beta} \label{eq10}$$

onde

$$c:=[S(0)+I(0)]S(0)^{\gamma/\beta}$$

combinando (\ref{dsdt}) e (\ref{eq10}) obtemos:

$$\frac{dS}{dt}=-\beta S[1-(S/L)^\alpha] \label{eq12}$$

onde $\alpha:= 1-\gamma/\beta$ e $L:=c^{1/(1-\gamma/\beta)}$.

Como a equação (\ref{eq12}) tem a mesma forma que a equação (\ref{dsdt}), pode ser resolvida como:

$$S(t)=L[1+\alpha e^{b(t-t_j)}]^{-1/\alpha}$$

onde $b=\alpha\beta$ e $t_j$ o tempo finito em que a segunda derivada de $S(t)$ é $0$.

Como estamos interessados no numero acumulado de de casos, podemos definir:

\begin{align}
J(t) :=& I(t)+R(t)\nonumber\\
=& N-S(t)\nonumber\\
=& N-L[1+\alpha e^{b(t-t_j)}]^{-1/\alpha}
\end{align}

Sendo $N=S(0)+I(0)$ então $N \approx L$. Como $I(0)/S(0)\rightarrow 0$,

$$J(t)\approx L-L[1+\alpha e^{b(t-t_j)}]^{-1/\alpha}.$$

Esta última equação nos dá uma forma eficiente de mapear uma curva de casos acumulados ao modelo SIR sem utilizar equações diferenciais.

Na prática podemos igualar $L$ ao tamanho final da epidemia, e $t_j$ ao ponto de inflexão da curva. Temos ainda que $\alpha=1-1/{\cal R}_0$, e $b=\beta -\gamma$ a taxa de geração de infeções.

### Instante do pico, $t_i$
O pico da epidemia ocorre quando $\dot{I}(t_i)=0$ o que implica que
$$\frac{\beta S(t_i)}{S(t_i)+I(t_i)}=\gamma$$

Considerando as equações (\ref{eq10}) e $\alpha:= 1-\gamma/\beta$ e $L:=c^{1/(1-\gamma/\beta)}$, podemos obter

$$S(t_i)=L (\beta/\gamma)^{-1/\alpha}$$

combinando as duas equações acima temos

$$I(t_i)=L{\cal R}_0^{-{\cal R}_0/({\cal R}_0-1)}({\cal R}_0-1)$$

com

$${\cal R}_0=e^{b(t_i-t_j)}$$.

## Ajustando o modelo de Richards a Dados
Agora vamos ver como o modelo de Richards se presta para representar uma epidemia com múltiplas ondas. Vamos utilizar os dados da Epidemia da COVID-19 em 2020 e 21

```python
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
ukd = pd.read_csv('UK_covid.csv')
ukd['date'] = pd.to_datetime(ukd.date)
ukd.set_index('date', inplace=True)
ukd
```

      total_cases
      new_cases
      location
      total_deaths
      new_deaths
      total_vaccinations
      new_vaccinations
      stringency_index
      population

      date

      2020-01-31
      2
      2
      United Kingdom
      NaN
      NaN
      NaN
      NaN
      8.33
      68207114

      2020-02-01
      2
      0
      United Kingdom
      NaN
      NaN
      NaN
      NaN
      8.33
      68207114

      2020-02-02
      2
      0
      United Kingdom
      NaN
      NaN
      NaN
      NaN
      11.11
      68207114

      2020-02-03
      8
      6
      United Kingdom
      NaN
      NaN
      NaN
      NaN
      11.11
      68207114

      2020-02-04
      8
      0
      United Kingdom
      NaN
      NaN
      NaN
      NaN
      11.11
      68207114

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      2021-08-24
      6586181
      30762
      United Kingdom
      132174.0
      174.0
      89865264.0
      186086.0
      43.98
      68207114

      2021-08-25
      6621799
      35618
      United Kingdom
      132323.0
      149.0
      90095045.0
      229781.0
      43.98
      68207114

      2021-08-26
      6659916
      38117
      United Kingdom
      132465.0
      142.0
      90295121.0
      200076.0
      43.98
      68207114

      2021-08-27
      6697770
      37854
      United Kingdom
      132566.0
      101.0
      90466529.0
      171408.0
      43.98
      68207114

      2021-08-28
      6729912
      32142
      United Kingdom
      132699.0
      133.0
      NaN
      NaN
      NaN
      68207114

576 rows × 9 columns

```python
fig , [a1,a2] = plt.subplots(2,1, figsize=[15,6], sharex=True)
ukd.total_cases.plot(ax=a1);
ukd.new_cases.plot(ax=a2);
```

```python
def rich(L,a,b,t,tj):
    j=L-L*(1+a*np.exp(b*(t-tj)))**(-1/a)
    return j
df = ukd.reset_index()
@interact
def plot_richards(w1=range_slider(1,180,step_size=1,default=(1,160)),
                  tp1=(86,(10,150)),L1=(.3e6,(.1e6,1.5e6)),a1=(0.1,(0.1,1)),b1=(0.2,(0.001,.2,.01)),
                  w2=range_slider(180,350,step_size=1,default=(210,300)),
                  tp2=(277,(230,350)),L2=(1e6,(.9e6,3.5e6)),a2=(0.1,(0.1,1)),b2=(0.2,(0.001,.2,.01)),
                  w3=range_slider(220,400,step_size=1,default=(300,400)),
                  tp3=(329,(220,400)),L3=(2e6,(1e6,4.5e6)),a3=(0.1,(0.1,1)),b3=(0.2,(0.001,.2,.01)),
                 ):
    fig,ax = plt.subplots(1,1, figsize=(15,8))
    df.total_cases.plot(ax=ax, label='Total cases',style='+')
    plt.plot(range(w1[0],w1[1]), [rich(L1,a1,b1,t,tp1) for t in range(w1[0],w1[1])], label='onda 1')
    plt.plot(range(w2[0],w2[1]), [L1+rich(L2,a2,b2,t,tp2) for t in range(w2[0],w2[1])], label='onda 2')
    plt.plot(range(w3[0],w3[1]), [L2+rich(L3,a3,b3,t,tp3) for t in range(w3[0],w3[1])], label='onda 3')
    plt.grid()
    plt.legend()
```

    Interactive function <function plot_richards at 0x7f046fa235e0> with 15 widgets
      w1: TransformIntRangeSlider(…

## Encontrando os Parâmetros por otimização

```python
import sherpa
```

```python
!pip install gpyopt
```

    Requirement already satisfied: gpyopt in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (1.2.6)
    Requirement already satisfied: numpy>=1.7 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from gpyopt) (1.19.5)
    Requirement already satisfied: scipy>=0.16 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from gpyopt) (1.5.4)
    Requirement already satisfied: GPy>=1.8 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from gpyopt) (1.10.0)
    Requirement already satisfied: cython>=0.29 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from GPy>=1.8->gpyopt) (0.29.21)
    Requirement already satisfied: paramz>=0.9.0 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from GPy>=1.8->gpyopt) (0.9.5)
    Requirement already satisfied: six in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from GPy>=1.8->gpyopt) (1.15.0)
    Requirement already satisfied: decorator>=4.0.10 in /home/fccoelho/Downloads/SageMath/local/lib/python3.9/site-packages (from paramz>=0.9.0->GPy>=1.8->gpyopt) (4.4.2)
    WARNING: You are using pip version 21.0.1; however, version 21.2.4 is available.
    You should consider upgrading via the '/home/fccoelho/Downloads/SageMath/local/bin/python3 -m pip install --upgrade pip' command.

```python
parameters = [
    sherpa.Discrete(name='L1', range=[.1e6,1e6]),
    sherpa.Discrete(name='tp1', range=[10,150]),
#     sherpa.Discrete(name='s1', range=[1,180]), #inicio da onda
#     sherpa.Discrete(name='d1', range=[10,180]), # duração da onda
    sherpa.Continuous(name='a1', range=[0,1]),
    sherpa.Continuous(name='b1', range=[0,1]),
]
```

```python
# algorithm = sherpa.algorithms.RandomSearch(max_num_trials=2000)
algorithm = sherpa.algorithms.SuccessiveHalving(max_finished_configs=500)
study = sherpa.Study(parameters=parameters,
                     algorithm=algorithm,
                     lower_is_better=True,
                     disable_dashboard=True)
```

```python
trial = study.get_suggestion()
trial.parameters
```

    {'L1': 257465,
     'tp1': 116,
     'a1': 0.3479896982752564,
     'b1': 0.06494139156730783,
     'resource': 1,
     'rung': 0,
     'load_from': '',
     'save_to': '1'}

```python
s=1
e=160
for trial in study:
    L = trial.parameters['L1']
    tp = trial.parameters['tp1']
    a = trial.parameters['a1']
    b = trial.parameters['b1']
    dado = df.loc[s:e].total_cases.values
    sig = [rich(L,a,b,t,tp) for t in range(s,e)]

    loss = sum((dado[s:e]-sig)**2)/(e-s)
    study.add_observation(trial=trial,
                          objective=loss,
                          )
    study.finalize(trial)
```

```python
res = study.get_best_result()
res
```

    {'Trial-ID': 5801,
     'Iteration': 1,
     'L1': 280608,
     'a1': 0.298767528283268,
     'b1': 0.056357823970169973,
     'load_from': '',
     'resource': 1,
     'rung': 0,
     'save_to': '5801',
     'tp1': 91,
     'Objective': 152888426.20476854}

```python
sig = [rich(res['L1'],res['a1'],res['b1'],t,res['tp1']) for t in range(s,e+1)]
```

```python
plt.plot(range(s,e+1),dado,label='dados')
plt.plot(range(s,e+1),sig, label='richards')
plt.grid()
plt.legend();
```

```python

```

Planilhas Sage/Suplemento 3 - o Modelo de Richards.ipynb


# Performing the Linear Noise Approximation (LNA) and conducting Approximate Bayesian Computation (ABC)

This notebook will guide the user through performing the linear noise approximation (LNA) on any system the user wishes, and through using the LNA when conducting ABC for parameter estimation.

## Toy Example: Stochastic three gene interaction network

Consider the Three Gene Example described in (Karlebach & Shamir, 2008) where Gene 1, 2 and 3 are represented by $g_1$, $g_2$ and $g_3$ respectively and $\textbf{g} = (g_1, g_2, g_3)^T$. The kinetic parameters of the system are represented through $k_*$. In this system, the levels of these genes vary and depend on each other, in a way that is determined by the equations below:

$$
\begin{align*}
\emptyset & \overset{\hphantom{k_{1}} \, k_{1,s} \, g_3 \hphantom{k}}{\longrightarrow}   g_1  \\
g_1 & \overset{\hphantom{k_{1,s}} \, k_{1,d} \,  \hphantom{k_{1,s}}}{\longrightarrow} \emptyset \\
g_1 & \overset{ \hphantom{k_{1}} \,k_{2,s} \, g_1 \hphantom{k_{1,s}}}{\longrightarrow} g_1 + g_2 \\
g_2 & \overset{\hphantom{k_{1,s}}\, k_{2,d} \, \hphantom{k_{1,s}}}{\longrightarrow} \emptyset \\
g_1 + g_2 & \overset{k_{3,s} \, k_{3,1} \, k_{3,2} }{\longrightarrow} g_1 + g_2 + g_3 \\
g_3 & \overset{ \hphantom{k_{1,s}} \, k_{3,d} \, \hphantom{k_{1,s}}}{\longrightarrow}  \emptyset
\end{align*}
$$

These equations are visualised in the below diagram:

The interactions between the genes can be seen: Gene 1 activates Gene 2 and Gene 3, Gene 2 activates Gene 3 and finally Gene 3 represses Gene 1 \[With the degradation of all the genes and the activation of Gene 1 from a constant source being supressed from the illustration\]. The reaction rates of this model are:

 $\textbf{a}= \begin{bmatrix}
      \frac{ k_{1,s}}{1+k_{1,3} \, \textbf{g}_3}, & k_{1,d} \, \textbf{g}_1, & \frac{k_{2,s} \, k_{2,1}\, \textbf{g}_1}{1+k_{2,1} \, \textbf{g}_1}, & k_{2,d} \, \textbf{g}_2, & \frac{k_{3,s} \, k_{3,1} \, \textbf{g}_1 \,  k_{3,2} \, \textbf{g}_2}{(1+k_{3,1} \, \textbf{g}_1)(1+k_{3,2} \, \textbf{g}_2)}, & k_{3,d} \, \textbf{g}_3
     \end{bmatrix} ^T$.

The Stoichiometry matrix of the system is:

$$\mathcal{S} =
 \begin{pmatrix}
  1 & -1 & 0 & 0 & 0 & 0 \\
  0 & 0 & 1 & -1 & 0 & 0 \\
  0 & 0 & 0 & 0 & 1 & -1
 \end{pmatrix} $$

In this example, the parameters $[k_{1,s}, k_{2,s}, k_{3,s}, k_{1,d}, k_{1,d}, k_{2,d}, k_{3,d}, k_{1,3}, k_{2,1}, k_{3,1}, k_{3,1}]$ have been chosen to take values $[2.0, 1.0, 15.0, 1.0, 1.0, 1.0, 100.0, 1.0, 1.0, 1.0]$ arbitrarily.

The user requires $\mathcal{S}$ and $\textbf{a}$ when constructing a stochastic model of a system. So, we construct $\mathcal{S}$ and $\textbf{a}$, where the reaction rates (reaction propensity vector), $\textbf{a}$, must be stated in a function.

### References

- Karlebach, G., & Shamir, R. (2008). Modelling and analysis of gene regulatory networks. Nature Reviews Molecular Cell Biology, 9(10), 770–780. https://doi.org/10.1038/nrm2503

# To run this notebook locally:
* Make sure that [Jupyter](https://jupyter.org/) is installed, along with [Julia](https://www.julialang.org/) and its [Jupyter Kernel](https://github.com/JuliaLang/IJulia.jl).
* Clone or download [GpABC.jl](https://github.com/tanhevg/GpABC.jl) to your machine.
* Run `jupyter` from the `examples` directory of the local copy of `GpABC.jl`:
```bash
$ cd GpABC.jl/examples
$ jupyter notebook lna-example.ipynb
```
* The first line of the cell below activates the local project environment and downloads all the dependencies.

This notebook was tested under Julia 1.4

```julia
import Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()
using GpABC, OrdinaryDiffEq, Distances, Distributions, LinearAlgebra, StochasticDiffEq, Plots
pyplot()
```

     Activating environment at `~/gaussian_processes/GpABC.jl/examples/Project.toml`
    ┌ Info: Precompiling StochasticDiffEq [789caeaf-c7a9-5a7d-9973-96adeb23e2a0]
    └ @ Base loading.jl:1260

    Plots.PyPlotBackend()

```julia
#
# Stochastic Model
#
S = [1.0 -1.0  0.0  0.0  0.0  0.0;  # stoichometry matrix
     0.0  0.0  1.0 -1.0  0.0  0.0;
     0.0  0.0  0.0  0.0  1.0 -1.0]

params = [2.0, 1.0, 15.0, 1.0, 1.0, 1.0, 100.0, 1.0, 1.0, 1.0]  #parameters

reaction_rate_function = function(x,params)  #reaction rates of model (stated in a function)
    f = [params[1]/(1+params[7]*x[3]),
        params[4]*x[1],
        params[2]*params[8]*x[1]/(1+params[8]*x[1]),
        params[5]*x[2],
        params[3]*params[9]*x[1]*params[10]*x[2]/(1+params[9]*x[1])/(1+params[10]*x[2]),
        params[6]*x[3]]
    return f
end
```

    #3 (generic function with 1 method)

An Ordinary Differential Equation (ODE) model is given by the relationship $\dot{\textbf{g}} = \mathcal{S} \, \textbf{a} $.

So we define the ODE, `GeneReg`, by: $\mathcal{S} \, \textbf{a}$. (Giving the same example used in [`abc-examples.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb)).

The initial conditions, timspan, solver and saveat are arbitrary and of the same form needed to pass to an [`OrdinaryDiffEq.jl`](https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl) DE solver.

```julia
#
# ODE
#
GeneReg = function(params::AbstractArray{Float64,1},
    Tspan::Tuple{Float64,Float64}, x0::AbstractArray{Float64,1},
    solver::OrdinaryDiffEq.OrdinaryDiffEqAlgorithm, saveat::Float64)

  if size(params,1) != 10
    throw(ArgumentError("GeneReg needs 10 parameters, $(size(params,1)) were provided"))
  end

  function ODE_3GeneReg(dx, x, par, t)
    dx[1:3] = S*reaction_rate_function(x, par)
  end

  prob = ODEProblem(ODE_3GeneReg, x0 ,Tspan, params)
  Obs = solve(prob, solver, saveat=saveat)

  return Obs
end

#
# ODE settings
#
Tspan = (0., 10.) #timspan
x0 = [3.0, 2.0, 1.0] #initial conditions
solver = RK4()
saveat = 0.1

#
# Solve and plot the ODE
#
ODE_solution = GeneReg(params, Tspan, x0, solver, saveat)
plot(ODE_solution)
```

## Stochastic Example

Before calculating the LNA, let's view the stochastic simulation of `GeneReg`:

```julia
#
# SDE
#
StochGeneReg = function(params::AbstractArray{Float64,1},
    Tspan::Tuple{Float64,Float64}, x0::AbstractArray{Float64,1},
    solver::StochasticDiffEq.StochasticDiffEqAlgorithm, dt::Float64)

  if size(params,1) != 10
    throw(ArgumentError("GeneReg needs 10 parameters, $(size(params,1)) were provided"))
  end

  function sigma_3GeneReg(dx, x, par, t)
    dx[1] = 0.3
    dx[2] = 0.3
    dx[3] = 0.3
  end

  function ODE_3GeneReg(dx, x, par, t)
    dx[1:3] = S*reaction_rate_function(x, par)
  end

  prob = SDEProblem(ODE_3GeneReg, sigma_3GeneReg, x0 ,Tspan, params)
  Obs = solve(prob, solver, dt=dt)

  return Obs
end

#
# SDE settings
#
solver = EM()
dt = 0.1

#
# Solve and plot the SDE
#
SDE_solution = StochGeneReg(params, Tspan, x0, solver, saveat)
plot(SDE_solution)
```

## Computing the LNA

Now, if the user wishes to compute the LNA of the above stochastic problem, the process begins by setting up an [`LNAInput`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.LNAInput) structure:

```julia
volume = 100.0
input = LNAInput(params, S, reaction_rate_function, volume);
```

where `volume` is a parameter. The `volume` can be viewed as the volume that the reactions are taking place in, or molecule number. For example, if `volume` becomes very large the LNA will be very close to the deterministic equations of the system.

***
The [LNA](https://tanhevg.github.io/GpABC.jl/latest/overview-lna/) is the solution to the two ODEs, where $\Omega$ is the volume:

$$
\begin{align}
\frac{d\phi}{dt}& =\mathcal{S}\textbf{a}(\phi) \label{mean} \\
\frac{d\Sigma}{dt}&=\mathcal{A} \, \Sigma + \Sigma \, \mathcal{A}^T + \frac{1}{\sqrt{\Omega}} \, \mathcal{S} \, \text{diag}(\textbf{a}(\phi)) \, \mathcal{S}^T \label{covar}
\end{align}
$$
where, $\mathcal{A} = \mathcal{S} \cdot \mathcal{J}$ ($\mathcal{J}$ is the Jacobian), $\phi$(t) are the mean trajectories and $\Sigma$(t) is the covariance of the trajectories.

If the user wishes to see the results from the two ODEs, the user must provide the initial condtions of the ODEs in the form of (initial conditions of the species, initial covariance) and then call [`compute_LNA`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.compute_LNA).

This returns a [`LNA`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.LNA) structure that holds the solutions to the above ODEs. The means, $\phi$ can be called by `lna.traj_means` and the covariance $\Sigma$ can be called by `lna.traj_covars`. The time points the system of ODEs were solved for can be called by `lna.time_points`.

### References

- Komorowski, M., Finkenstädt, B., Harper, C.V., and Rand, D.A. (2009). Bayesian inference of biochemical kinetic parameters using the linear noise approximation. BMC Bioinformatics, 10:343. https://doi.org/10.1186/1471-2105-10-343

- Schnoerr, D., Sanguinetti, G., and Grima, R. (2017). Approximation and inference methods for stochastic biochemical kinetics—a tutorial review. Journal of Physics A: Mathematical and Theoretical, 50(9), 093001. https://doi.org/10.1088/1751-8121/aa54d9

```julia
x0 = ([3.0, 2.0, 1.0], Matrix(1.0I, 3,3)) #initial conditions
                                           #...in the form of
                                           #...(initial conditions of the species, initial covariance)
solver = RK4()
saveat = 0.1

lna = compute_LNA(input, x0, Tspan, saveat)
t = lna.time_points

#
# Calculate the variance of the trajectories for visualisation
#
variance = zeros(size(S)[1], length(t))
for (idx, elt) in enumerate(lna.traj_covars)
    variance[:, idx] = diag(elt)
end

#
# Plot the trajectories and confidence intervals
#
plot(t, lna.traj_means[1,:], ribbon=1.96*sqrt.(variance[1,:]), c=:blue, leg=false)
plot!(t, lna.traj_means[2,:], ribbon=1.96*sqrt.(variance[2,:]), c=:red, leg=false)
plot!(t, lna.traj_means[3,:], ribbon=1.96*sqrt.(variance[3,:]), c=:green, leg=false)
```

The three means are plotted with the shaded areas being the 95% confidence intervals.

## Sampling from the LNA

The LNA gives the mean of the trajectories and the covariance between them; hence a single trajectory can be sampled from a Multivariate Normal distribution, i.e. $\textbf{x}(t) \sim \mathcal{N}(\phi(t), \Sigma(t))$. The function [`sample_LNA_trajectories`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.sample_LNA_trajectories-Tuple{LNA,Int64}) automatically does this.

The user can also sample more than one trajectory, which are then averaged, by changing `n_samples`. The larger the number of samples that are averaged over, the less noisy the outputs are. The effect of varying `n_samples` is shown below:

```julia
n_samples = 1

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

```julia
n_samples = 10

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

```julia
n_samples = 100

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

## ABC using the LNA

The same examples that can be located in [`abc-examples.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb) are shown, but using the LNA. The only difference is in the `simulator_function`.

Let's try and estimate the first three parameters of the model and compare simulation based ABC and emulation based ABC.

### Simulation-Based Rejection ABC

So, lets define the `simulator_function`. We need to create a function to be passed to ABC algortihms that takes a vector of the parameters we wish to estimate as an argument and outputs model results (as stated in the [documentation](https://tanhevg.github.io/GpABC.jl/latest/ref-abc/#abc_ref-1)).

So we define a function to create a `LNAInput` object and then call the function [`get_LNA_trajectories`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.get_LNA_trajectories) on this object. The output of `get_LNA_trajectories` conforms to the desired output of the `simulator_function`.

```julia
function simulator_function(var_params)
    input = LNAInput(vcat(var_params, params[n_var_params+1:end]), S, reaction_rate_function, volume)
    return get_LNA_trajectories(input, n_samples, x0, Tspan, saveat)
end
```

    simulator_function (generic function with 1 method)

Let's take the reference data to be `lna_trajectories` for the purpose of this example, calculated above using the following:
```
@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)
```
This provides some fake data where we already know the nomimal parameters.

```julia
#
# SimulatedABCRejection arguments
#
priors = [Uniform(0., 5.), Uniform(0., 5.), Uniform(10., 20.)] #priors for the unknown parameters
n_var_params = length(priors) #no. of parameters to vary
n_particles = 500
threshold = 2.0
progress_every = 100
reference_data = lna_trajectories

sim_result = SimulatedABCRejection(reference_data,
                        simulator_function,
                        priors,
                        threshold,
                        n_particles;
                        write_progress = false
                        )

plot(sim_result)
```

Parameter 1, 2 and 3's nominal values were 2.0, 1.0 and 15.0; which haven't been uniquely returned following simulated-based ABC. However, a positive correlation between Parameter 2 and 3 has been recovered.

The above becomes our "gold-standard" that we should try to reproduce when conducting emulation-based rejection ABC.

### Emulation-Based Rejection ABC

The only new argument to sepcify is `n_design_points`; the number of design points (explained in [`abc-example.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb)), when calling `EmulatedABCRejection`.

```julia
n_design_points = 200

emu_result = EmulatedABCRejection(reference_data,
                        simulator_function,
                        priors,
                        threshold,
                        n_particles,
                        n_design_points;
                        write_progress = false)

plot(emu_result)
```

The same information about Parameters 1, 2 and 3 can be visually deduced.

To show that the above still holds for [Simulation-Based](https://tanhevg.github.io/GpABC.jl/latest/overview-abc/#Simulation-based-ABC-SMC-1) and [Emulation Based](https://tanhevg.github.io/GpABC.jl/latest/overview-abc/#Emulation-based-ABC-SMC-1) ABC-SMC using the LNA, the same example as above is conducted below:

### Simulation-Based ABC-SMC

```julia
threshold_schedule = [4.0, 3.0, 2.0]

sim_abcsmc_res = SimulatedABCSMC(reference_data,
                        simulator_function,
                        priors,
                        threshold_schedule,
                        n_particles;
                        write_progress = false
                        )

population_colors=["#FF2F4E", "#A20018", "#800013"]
plot(sim_abcsmc_res, population_colors=population_colors)
```

### Emulation-Based ABC-SMC

```julia
emu_abcsmc_res = EmulatedABCSMC(reference_data,
                        simulator_function,
                        priors,
                        threshold_schedule,
                        n_particles,
                        n_design_points;
                        write_progress = false
                        )

plot(emu_abcsmc_res, population_colors=population_colors)
```

Again, the results are similar for emulation and simulation.


| Name | Description | Date
| :- |-------------: | :-:
|Reza Hashemi| Random function generator.  | On 15th of July 2019

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
def random_binning(N, d,return_string=False):
    """
    Function to distribute a sum total of N into d bins
    Returns a list
    """
    import numpy as np

    count=N
    sum_count=0
    lst=[]
    for i in range(d-1):
        x=np.random.randint(0,count+1)
        if return_string:
            lst.append(str(x))
        else:
            lst.append(x)
        count=count-x
        sum_count+=x

    if return_string:
        lst.append(str(N-sum_count))
    else:
        lst.append(N-sum_count)
    return lst
```

### Function to generate single term from a list of exponents. E.g. if we pass 1,2,0,3 it will generate: $x_1.x_2^2.x_4^3$

```python
def gen_single_term(lst):
    """
    Function to generate single term from a list of exponents.
    """
    term=''
    for i in range(1,len(lst)+1):
        if lst[i-1]==0:
            pass
        elif lst[i-1]==1:
            term+='x'+str(i)+'.'
        else:
            term+='x'+str(i)+'^'+str(lst[i-1])+'.'

    return term[:-1]
```

```python
gen_single_term([1,1,2,0,1])
```

    'x1.x2.x3^2.x5'

```python
def count_symbol(lst):
    syms = set(lst)
    syms_dict={}
    term_lst=[]
    for s in syms:
        syms_dict[s]=lst.count(s)

    for k,v in syms_dict.items():
        if v!=1:
            term_lst.append(str(v)+'.'+k)
        else:
            term_lst.append(k)

    return term_lst
```

```python
def gen_multinomial(n_features=5,max_power=5,max_terms=10,fixed_terms=None,coefficients=True,
                   prob_negative_sign=0.3):
    """
    Generates multinomial expression.
    n_features: Number of independent variables
    max_power: Maximum exponent each terms can be raised to. A random power will be chosen up to this maximum.
    max_terms: Maximum number of terms. A random number of terms will be chosen up to this maximum.
    fixed_terms: Attempt will be made to generate only this many terms. Sometimes does not work.
    coefficients (boolean): Adds (or does not) random integer coefficients in front of the terms.
    prob_negative_sign: Probability of putting a negative term.
                        Each term's sign wil be chosen randomly based on this probability
    """
    import numpy as np
    eqn=''
    eqn_terms=[]

    if fixed_terms!=None:
        n_terms=fixed_terms
    else:
        n_terms=np.random.randint(2,max_terms+1)

    for i in range(n_terms):
        power=np.random.randint(1,max_power+1)
        #power=max_power
        power_lst=random_binning(power,n_features)
        term=gen_single_term(power_lst)
        if coefficients:
            coeff=np.random.randint(1,11)
            if coeff!=1:
                coeff=str(coeff)
                term=coeff+'.'+term
        eqn_terms.append(term)

    eqn_terms=count_symbol(eqn_terms)

    for e in eqn_terms:
        eqn+=e
        sign=np.random.choice(['+','-'],p=[prob_negative_sign,1-prob_negative_sign])
        eqn= eqn+' '+sign+' '

    return eqn[:-3]
```

```python
gen_multinomial(coefficients=True)
```

    '6.x1^2.x2.x4 - 4.x1^2.x4^3 - 7.x1^2.x2^2 - 3.x1 - 7.x1^3.x3.x5 - 5.x1^5 - 6.x5'

```python
def make_equation_system(n_samples=10,n_features=5,max_power=5,max_terms=10,
                               fixed_terms=None,coefficients=False,prob_negative_sign=0.3):
    """
    Generates multiple samples of multinomials for constructing a system of equations
    """
    regressions=[]
    for i in range(n_samples):
        regressions.append(gen_multinomial(n_features=n_features,max_power=max_power,
                                           max_terms=max_terms,fixed_terms=fixed_terms,
                                           coefficients=coefficients,prob_negative_sign=prob_negative_sign))

    return regressions
```

```python
make_equation_system(n_samples=10,n_features=4,max_power=3,max_terms=8,fixed_terms=4,
                     coefficients=True,prob_negative_sign=0.5)
```

    ['10.x2 - x1 - 7.x1 - 6.x1',
     '2.x2.x4^2 - 2.x1 + 8.x1.x2.x4 - 10.x4',
     '8.x1^2 - 4.x1.x2 + 6.x1^3 + 4.x1',
     '2.x1^3 + 3.x1.x2^2 - 9.x1^2.x4 - 9.x1',
     '7.x1.x3 - x1^3 + x1^2.x3 + 3.x2^2.x3',
     '10.x2^2.x4 - 8.x2 + 5.x1.x3 + x2^2.x4',
     '2.x1^3 - x1.x4 - 5.x1^2 + 2.x3',
     '10.x1 + 4.x1.x2 - 9.x1^2.x4 + 3.x1^2.x4',
     'x2 - x1^3 + x2.x3^2 - 10.x1^2',
     '3.x1.x2.x4 - 3.x1^3 + 3.x1^2 - 3.x1']

### Evaluate a polynomial string

```python
from sympy import *
```

```python
def symbolize(s):
    """
    Converts a a string (equation) to a SymPy symbol object
    """
    from sympy import sympify
    s1=s.replace('.','*')
    s2=s1.replace('^','**')
    s3=sympify(s2)

    return(s3)
```

```python
def eval_multinomial(s,vals=None,symbolic_eval=False):
    """
    Evaluates polynomial at vals.
    vals can be simple list, dictionary, or tuple of values.
    vals can also contain symbols instead of real values provided those symbols have been declared before using SymPy
    """
    from sympy import Symbol,sympify,symbols

    sym_s=symbolize(s)
    sym_set=sym_s.atoms(Symbol)
    sym_lst=[]
    for s in sym_set:
        sym_lst.append(str(s))
    sym_lst.sort()
    if symbolic_eval==False and len(sym_set)!=len(vals):
        print("Length of the input values did not match number of variables and symbolic evaluation is not selected")
        return None
    else:
        if type(vals)==list:
            sub=list(zip(sym_lst,vals))
        elif type(vals)==dict:
            l=list(vals.keys())
            l.sort()
            lst=[]
            for i in l:
                lst.append(vals[i])
            sub=list(zip(sym_lst,lst))
        elif type(vals)==tuple:
            sub=list(zip(sym_lst,list(vals)))
        result=sym_s.subs(sub)

    return result
```

```python
s1=gen_multinomial(fixed_terms=5,coefficients=True)
```

```python
s1
```

    '2.x4 - 6.x2^2.x3 - 10.x1 - 6.x3^2.x4^2 - 8.x1^3.x2.x5'

```python
eval_multinomial(s1,{'x1':2,'x3':1,'x4':0,'x2':3,'x5':1},symbolic_eval=True)
```

    -266

```python
vals=(2,3,1)
eval_multinomial(s1,vals)
```

    Length of the input values did not match number of variables and symbolic evaluation is not selected

```python
from sympy import symbols
x,y=symbols('x y')
vals=[2,x,2,2,y]
eval_multinomial(s1,vals,symbolic_eval=True)
```

    -12*x**2 - 64*x*y - 112

### Pretty printing and LaTeX format output

```python
def pretty_multinomial(n_features=5,max_power=5,max_terms=10,fixed_terms=None,coefficients=False):
    from sympy import init_printing
    init_printing()
    s=gen_multinomial(n_features=n_features,max_power=max_power,max_terms=max_terms,
                      fixed_terms=fixed_terms,coefficients=coefficients)
    sym_s=symbolize(s)
    return(sym_s)
```

```python
pretty_multinomial(coefficients=True,fixed_terms=4)
```

```python
def latex_multinomial(n_features=5,max_power=5,max_terms=10,fixed_terms=None,coefficients=False):
    from sympy import latex
    s=gen_multinomial(n_features=n_features,max_power=max_power,max_terms=max_terms,
                      fixed_terms=fixed_terms,coefficients=coefficients)
    sym_s=symbolize(s)
    return(latex(sym_s))
```

```python
latex_multinomial(coefficients=True,fixed_terms=4)
```

    '5 x_{1}^{4} - 10 x_{1}^{2} x_{2} x_{5} + x_{1}^{2} x_{3} - 10 x_{2}^{3}'

```python
lst=[]
for i in range(6):
    lst.append(pretty_multinomial(coefficients=True))
lst
```

```python
pretty_multinomial(n_features=2,max_power=4,coefficients=True)
```

```python

```

Random Function Generator/Random function generator.ipynb


# FRED Examples

```python
# Import the pandas library as pd
import pandas as pd

# Import the plotting library matplotlib.pyplot as plt
import matplotlib.pyplot as plt

plt.style.use('classic')
%matplotlib inline
```

## Example: M2 Money Stock

The monetary aggregate M2 has FRED series ID `M2` and is available here: https://fred.stlouisfed.org/series/M2. To find the data in plaintext, under "Observation:", click "(+more)" and then in the drop-down menu, click "View All". You will be taken to a page that has the data. The URL is for the data is: https://fred.stlouisfed.org/data/M2.txt.

Notice that the data URL has the form: `https://fred.stlouisfed.org/data/SERIESID.txt`, where `SERIESID` in this case is `M2`.

Notice that there are 19 lines of text *before* the line starting with "DATE". That means you need to tell Pandas to skip the first 19 rows of the text when importing.

```python
# Download M2 data
m2 = pd.read_csv('https://fred.stlouisfed.org/data/M2.txt',skiprows=19,sep='\s+',index_col=0,parse_dates = True)

# First 5 rows of m2
m2.head()
```

      VALUE

      DATE

      1980-11-03
      1591.4

      1980-11-10
      1592.9

      1980-11-17
      1596.3

      1980-11-24
      1597.2

      1980-12-01
      1596.1

Note the the imported data are a DataFrame with a column called `'VALUE'`. It will be convenient later if we go ahead and extract the column as a Pandas Series and set `m2` equal to that.

```python
# set m2 equal to 'VALUE' column of m2
m2 = m2['VALUE']

# First 5 rows of m2
m2.head()
```

    DATE
    1980-11-03    1591.4
    1980-11-10    1592.9
    1980-11-17    1596.3
    1980-11-24    1597.2
    1980-12-01    1596.1
    Name: VALUE, dtype: float64

## Plot most recent year of M2 data

You can see from FRED that the data are at a weekly observation frequency. So we plot the final 52 elements of `m2`.

```python
# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot
ax.plot(m2[-53:],'-',lw=3)

# Add title and y-axis label
ax.set_title('M2 Money Stock')
ax.set_ylabel('Billions of Dollars')

# Auto-rotate the x-axis labels
fig.autofmt_xdate()

ax.grid()

# Save the figure as a png file. bbox_inches = 'tight' reduces whitespace around the figure
fig.savefig('m2.png',dpi = 120,bbox_inches='tight')
```

Evidently the money supply in the US has expanded considerably in the past year.

## Plot most recent year of M2 growth

Recall that the growth rate of a quantity $X_t$ is:
\begin{align}
\text{growth rate of }X_t & = \frac{X_t - X_{t-1}}{X_{t-1}}
\end{align}

You can use the `.shift()` method for Pandas objects to create the lagged ($t-1$) value of $X$.

```python
# Create legged M2
m2_lag = m2.shift(1)

m2_lag.head()
```

    DATE
    1980-11-03       NaN
    1980-11-10    1591.4
    1980-11-17    1592.9
    1980-11-24    1596.3
    1980-12-01    1597.2
    Name: VALUE, dtype: float64

The data have been shifted forward one time increment and so there is a missing value for the first date.

```python
# Create growth rate of m2
m2_growth = m2/m2_lag - 1

# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot. Multiply growth by 100 to get units in percent
ax.plot(100*m2_growth[-53:],'-',lw=3)

# Add title and y-axis label
ax.set_title('Weekly Growth of M2 Money Stock')
ax.set_ylabel('Percent')

# Auto-rotate the x-axis labels
fig.autofmt_xdate()

ax.grid()

# Save the figure as a png file. bbox_inches = 'tight' reduces whitespace around the figure
fig.savefig('m2_growth.png',dpi = 120,bbox_inches='tight')
```

Maybe you'd like a bar chart instead?

```python
# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot. Multiply growth by 100 to get units in percent.
ax.bar(m2_growth[-53:].index,100*m2_growth[-53:],width=2.5)
fig.autofmt_xdate()

# Add title and y-axis label
ax.set_title('Weekly Growth of M2 Money Stock')
ax.set_ylabel('Percent')

# Auto-rotate the x-axis labels
fig.autofmt_xdate()

ax.grid()

# Save the figure as a png file. bbox_inches = 'tight' reduces whitespace around the figure
fig.savefig('m2_growth_bar.png',dpi = 120,bbox_inches='tight')
```

## Compute growth rate of M2 over previous year

```python
m2_growth_one_year = 100*(m2.iloc[-1] / m2.iloc[-53]- 1)

print('M2 growth over previous year:',round(m2_growth_one_year,5),'%')
```

    M2 growth over previous year: 27.17106 %

# Example: Consumer Price Index

The consumer price index is a measure of the price level. "Consumer Price Index for All Urban Consumers: All Items in U.S. City Average" has FRED series ID `CPIAUCSL` and is available here: https://fred.stlouisfed.org/series/CPIAUCSL.

Notice that here there are now 54 lines of text *before* the line starting with "DATE". That means you now need to tell Pandas to skip the first 54 rows of the text when importing.

```python
# Download CPI data
cpi = pd.read_csv('https://fred.stlouisfed.org/data/CPIAUCSL.txt',skiprows=54,sep='\s+',index_col=0,parse_dates = True)

# set cpi equal to 'VALUE' column of cpi
cpi = cpi['VALUE']

# First 5 rows of CPI
cpi.head()
```

    DATE
    1947-01-01    21.48
    1947-02-01    21.62
    1947-03-01    22.00
    1947-04-01    22.00
    1947-05-01    21.95
    Name: VALUE, dtype: float64

## Plot most recent 2 years of CPI data

You can see from FRED that the data are at a monthly observation frequency. So we plot the final 24 elements of `cpi`.

```python
# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot
ax.plot(cpi[-24:],'-',lw=3)

# Add title and y-axis label
ax.set_title('Consumer Price Index')
ax.set_ylabel('Index (1982-84=100)')

# Auto-rotate the x-axis labels
fig.autofmt_xdate()

ax.grid()

# Save the figure as a png file. bbox_inches = 'tight' reduces whitespace around the figure
fig.savefig('cpi.png',dpi = 120,bbox_inches='tight')
```

Look at the big drop in consumer prices in February, March, and April that was followed by fairly rapid recovery.

## Monthly CPI inflation over previous 2 years

We can obtain a measure of inflation as the growth rate of the CPI.

```python
# Compute CPI inflation
cpi_inflation = cpi/cpi.shift(1)-1

# Create figure and axis
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot
ax.plot(100*12*cpi_inflation[-24:],'-',lw=3)

# Add title and y-axis label
ax.set_title('CPI Inflation')
ax.set_ylabel('Percent (Annualized)')

# Auto-rotate the x-axis labels
fig.autofmt_xdate()

ax.grid()

# Save the figure as a png file. bbox_inches = 'tight' reduces whitespace around the figure
fig.savefig('cpi_inflation.png',dpi = 120,bbox_inches='tight')
```

Notice that the data are multiplied by 100 and 12 so that the units are annualized percent change. For example, in April 2020, the CPI declined by 9.5% on an annual basis. That means that the CPI actually only fell by 0.8%, but if that same percent change were to persist for a full year, then the CPI would end up falling by 9.5%. It is a convention to annualize inflation and interest data to standardize the units.

## Average CPI inflation over different subsamples

Suppose you want to compare the average rate of CPI inflation over the past year with that over the past 2 years, 5 years, and 50 years. Then use the formula:

\begin{align}
\text{CPI}_T & = (1 + \pi)^T \text{CPI}_0
\end{align}

to compute

\begin{align}
\pi & = \left(\frac{\text{CPI}_T}{\text{CPI}_0}\right)^{1/T} - 1
\end{align}

```python
# Compute and print average inflation rates
print('CPI inflation over previous year:     ',round(100*(cpi.iloc[-1]/cpi.iloc[-1-1*12]-1),5),'%')
print('CPI inflation over previous 2 years:  ',round(100*((cpi.iloc[-1]/cpi.iloc[-1-2*12])**(1/2)-1),5),'%')
print('CPI inflation over previous 10 years: ',round(100*((cpi.iloc[-1]/cpi.iloc[-1-10*12])**(1/10)-1),5),'%')
print('CPI inflation over previous 50 years: ',round(100*((cpi.iloc[-1]/cpi.iloc[-1-50*12])**(1/50)-1),5),'%')
print('CPI inflation over previous all years:',round(100*((cpi.iloc[-1]/cpi.iloc[0])**(1/(len(cpi)-1)*12)-1),5),'%')
```

    CPI inflation over previous year:      1.28887 %
    CPI inflation over previous 2 years:   1.78924 %
    CPI inflation over previous 10 years:  1.73197 %
    CPI inflation over previous 50 years:  3.8391 %
    CPI inflation over previous all years: 3.44054 %

See how much lower the average inflation rate over the last year has been compared to the larger subsamples?


# PIT on VCT data revisited

# Purpose
PIT on VCT has previously been conducted in:
* [04.03_PIT_VCT_linear_VMM.ipynb](04.03_PIT_VCT_linear_VMM.ipynb)
* [04.03_PIT_VCT_nonlinear_VMM.ipynb](04.03_PIT_VCT_nonlinear_VMM.ipynbb)
...but that was a while a ago, so this is a retake on this one...

# Methodology
* Load VCT data
* Convert it to prime system
* Regress fx, fy, mz
* simulate with these coefficients

# Setup

```python
# %load imports.py
## Local packages:

%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
np.set_printoptions(linewidth=150)

import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl

from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.regression import show_pred, show_pred_captive
from src.visualization.plot import track_plot

## Load models:
# (Uncomment these for faster loading):
import src.models.vmm_linear as vmm
from src.models.vmm import ModelSimulator
```

```python
from src.data.wpcc import ship_parameters, df_parameters, ps, ship_parameters_prime, ps_ship
```

## Load VCT data

```python
df_VCT_all = pd.read_csv('../data/external/vct.csv', index_col=0)
df_VCT = df_VCT_all.groupby(by=['model_name']).get_group('V2_5_MDL_modelScale')
```

```python
df_VCT.describe()
```

# Subtract the resistance

```python
df_resistance = df_VCT.groupby(by='test type').get_group('resistance')

X = df_resistance[['u','fx']].copy()
X['u**2'] = X['u']**2
y = X.pop('fx')

model_resistance = sm.OLS(y,X)
results_resistance = model_resistance.fit()

X_pred = pd.DataFrame()
X_pred['u'] = np.linspace(X['u'].min(), X['u'].max(), 20)
X_pred['u**2'] = X_pred['u']**2
X_pred['fx'] = results_resistance.predict(X_pred)

fig,ax=plt.subplots()
df_resistance.plot(x='u', y='fx', style='.', ax=ax)
X_pred.plot(x='u', y='fx', style='--', ax=ax);

```

```python
df_VCT_0_resistance = df_VCT.copy()
df_VCT_0_resistance['u**2'] = df_VCT_0_resistance['u']**2
df_VCT_0_resistance['fx']-= results_resistance.predict(df_VCT_0_resistance[['u','u**2']])
```

## VCT to prime system

```python
interesting = [
    'u',
    'v',
    'r',
    'delta',
    'fx',
    'fy',
    'mz',
    'thrust',
]
#df_VCT_prime = ps_ship.prime(df_VCT[interesting], U=df_VCT['V'])
df_VCT_prime = ps_ship.prime(df_VCT_0_resistance[interesting], U=df_VCT_0_resistance['V'])
```

## Regression

### N

```python
Math(vlatex(vmm.N_qs_eq))
```

```python
N_ = sp.symbols('N_')
diff_eq_N = regression.DiffEqToMatrix(ode=vmm.N_qs_eq.subs(N_qs,N_),
                                      label=N_, base_features=[delta,u,v,r])

X = diff_eq_N.calculate_features(data=df_VCT_prime)
y = diff_eq_N.calculate_label(y=df_VCT_prime['mz'])

model_N = sm.OLS(y,X)
results_N = model_N.fit()

show_pred_captive(X=X,y=y,results=results_N, label=r'$N$')
```

## Y

```python
Math(vlatex(vmm.Y_qs_eq))
```

```python
Y_ = sp.symbols('Y_')
diff_eq_Y = regression.DiffEqToMatrix(ode=vmm.Y_qs_eq.subs(Y_qs,Y_),
                                      label=Y_, base_features=[delta,u,v,r])

X = diff_eq_Y.calculate_features(data=df_VCT_prime)
y = diff_eq_Y.calculate_label(y=df_VCT_prime['fy'])

model_Y = sm.OLS(y,X)
results_Y = model_Y.fit()

show_pred_captive(X=X,y=y,results=results_Y, label=r'$Y$')
```

## X

```python
Math(vlatex(vmm.X_qs_eq))
```

```python
X_ = sp.symbols('X_')
ode=vmm.X_qs_eq.subs([(X_qs,X_),
                      (thrust,0),  # Removing thrust from the equation
                     ])

diff_eq_X = regression.DiffEqToMatrix(ode=ode,
                                      label=X_, base_features=[delta,u,v,r])

X = diff_eq_X.calculate_features(data=df_VCT_prime)
y = diff_eq_X.calculate_label(y=df_VCT_prime['fx'])

model_X = sm.OLS(y,X)
results_X = model_X.fit()

show_pred_captive(X=X,y=y,results=results_X, label=r'$X$')
```

```python
results_summary_X = regression.results_summary_to_dataframe(results_X)
results_summary_Y = regression.results_summary_to_dataframe(results_Y)
results_summary_N = regression.results_summary_to_dataframe(results_N)
```

```python
df_parameters_all = df_parameters.copy()
for other in [results_summary_X, results_summary_Y, results_summary_N]:
    df_parameters_all = df_parameters_all.combine_first(other)

df_parameters_all.rename(columns={'coeff':'regressed'}, inplace=True)
df_parameters_all.drop(columns=['brix_lambda'], inplace=True)

df_parameters_all['regressed'] = df_parameters_all['regressed'].combine_first(df_parameters_all['prime'])  # prefer regressed
#df_parameters_all['regressed'].fillna(0,inplace=True)
```

## Create a simulator with regressed parameters

```python
model = ModelSimulator(simulator=vmm.simulator,
                       parameters=df_parameters_all['regressed'],
                       ship_parameters=ship_parameters,
                       control_keys=['delta'],
                       primed_parameters=True,
                       prime_system=ps,)
model.save('../models/model_VCT_linear.pkl')
```

# Simulation

A simulation with the regressed hydrodynamic coefficients can now be conducted. The figures below compare results from these simulation and the original data.

## Load test

```python
def load_test(id):

    df, units, meta_data = mdl.load(id=id, dir_path='../data/processed/kalman_cut/')
    df.index = df.index.total_seconds()
    df = df.iloc[0:-100].copy()
    df.index-=df.index[0]
    df.sort_index(inplace=True)
    df['thrust'] = df['Prop/PS/Thrust'] + df['Prop/SB/Thrust']

    return df

tests = {
    22774 : load_test(id=22774).loc[80:],
    22773 : load_test(id=22773),
    22770 : load_test(id=22770).iloc[100:],
}

```

```python
results = {}
for id, df_ in tests.items():

    result_regression = model.simulate(df_=df_,
                                       name='regressed', method='RK45')

    results[id] = result_regression

```

```python
for id, result_regression in results.items():

    result_regression.track_plot(compare=True);

    fig = result_regression.plot(compare=True, ax=ax);
    fig.set_size_inches(15,10)
    plt.tight_layout()
```


## 2.6 Convergence of Average Consensus

In Sec.2.2--2.5, we introduced the network topology, the combination matrix, and the consensus average algorithm. In this section, we will discuss the convergence of average consensus, and how network toplogy affect the convergence rate.

### 2.6.1 Riview of average consensus algorithm

Average consensus is a decentralized algorithm to let each agent reach the global average of each local variables. Given a network of $n$ computing agents, the combination matrix $W$, and the local variables $x_1^0, \cdots, x_n^0$ (each variable has dimension $d$), the average consensus algorithm is

$$x_i^{k} = \sum_{j=1}^n w_{ij} x_j^{k-1} = w_{ii} x_i^{k-1} + \sum_{j\in \mathcal{N}(i)} w_{ij} x_j^{k-1}, \quad \forall\ i=1,\cdots, n.$$

Average consensus is a recursive algorithm. Each recursion only incurs neighborhood communication. Each recursion of average consensus is conducted in parallel. By repeating a series of partial averaging, it is guaranteed by theory that $x_i^{k} \to \bar{x}^0 = \frac{1}{n}\sum_{i=1}^n x_i^0$.

**Notation.** We introduce some notations to facilitate the analysis.

- We let ${X} = [x_1^T;\cdots;x_n^T]\in \mathbb{R}^{n\times d}$ to collect each local variable $x_i$ into a matrix, and $X^{(k)}$ be the $k$-th iterate of variable $X$;

- Given matrices $W\in \mathbb{R}^{n\times n}$ and $X\in \mathbb{R}^{n\times d}$, it holds that

\begin{align}\label{submulti}
\|W X\|_F \le \|W\|_2 \|X\|_F.
\end{align}

To verify it, by letting $X_j$ be the $j$-th column of $X$, we have

$$\|WX\|_F^2 = \sum_{j=1}^d \|X_j\|_2^2 \le \sum_{j=1}^d \|W\|_2^2 \|X_j\|_2^2=\|W\|_2^2\|X\|_F^2.$$

- Let $\lambda_i(A)$ be the $i$-th eigenvalue of the matrix $A$, and $\rho(A)$ be the spectrum radius of the matrix $A$.

### 2.6.2 Convergence analysis

**Reformulation of aveage consensus.** With notation $X^{(k)}$ and the combination matrix $W$, we can rewrite the avearge consensus algorithm as

$$X^{(k+1)} = W X^{(k)}$$

**Assumption on $W$.** Recall the assumption on the combination matrix $W$:

> **Assumption 1 (Doubly stochastic)** We assume $W$ is irreducible and non-periodic, and it satisfies $W \mathbf{1} = \mathbf{1}$ and $\mathbf{1}^T W = \mathbf{1}^T.$

**Properties of $W$.** If $W$ satisfies Assumption 1, it has the following properties:

> **Proposition 1** If matrix $W \in \mathbb{R}^{n\times n}$ satisfes Assumption 1, it has the following properties:
>
> - The spectral radius of $W$ equals 1, i.e., $\rho(W) = \max\{|\lambda_1(A)|, |\lambda_2(A)|, \ldots,  |\lambda_n(A)|\} = 1$. Furthermore, there is one and only one eigenvalue with magnitude being $1$.
>
>
> - The spectral radius of $W-\frac{1}{n}\mathbf{1} \mathbf{1}^T$ is strictly less than $1$, i.e., $\rho(W-\frac{1}{n}\mathbf{1} \mathbf{1}^T) < 1$.

**Convergence of average consensus.** Give $X^{(0)} = [(x_1^{(0)})^T;\cdots;(x_n^{(0)})^T] \in \mathbb{R}^{n\times d}$, the goal of average consensus is to let $X^{(k)}$ converge to $\frac{1}{n}\mathbf{1}\mathbf{1}^TX^{(0)}$. As a result, we have

\begin{align*}
\|X^{(k+1)} - \frac{1}{n}\mathbf{1}\mathbf{1}^T X^{(0)}\|_F &= \|W^{k+1}X^{(0)} - \frac{1}{n}\mathbf{1}\mathbf{1}^TX^{(0)}\| \nonumber \\
&\le \|W^{k+1} - \frac{1}{n}\mathbf{1}\mathbf{1}^T\|\|X^{(0)}\|_F \nonumber \\
&= \|W - \frac{1}{n}\mathbf{1}\mathbf{1}^T\|^{k+1} \|X^{(0)}\|_F \nonumber \\
&= \rho^{k+1} \|X^{(0)}\|_F
\end{align*}

where $\rho = \rho(W - \frac{1}{n}\mathbf{1}\mathbf{1}^T) < 1$ because $W$ is doubly stochastic. The above derivation leads to the following theorem:

> **Theorem 1.** Under Assumption 1, $x_i^{(k)}$ generated by average consensus will converge to $\bar{x}^{(0)} = \frac{1}{n}\sum_{i=1}^n x_i^{(0)}$ at linear rate, i.e.,
>
> $$\|x_i^{(k)} - \bar{x}^{(0)}\| \le \rho^k \|X^{(0)}\|_F$$
>
> where $\rho = \rho(W - \frac{1}{n}\mathbf{1}\mathbf{1}^T) < 1$.


```python
from itertools import chain, combinations
def get_set_of_subsets(omega):
    return set([frozenset(s) for s in chain.from_iterable(combinations(omega, r) for r in range(len(omega) + 1))])

```

```python

omega = {"♦", "♥", "♠", "♣"}
SS1 = get_set_of_subsets(omega)
SS1

{frozenset(),
frozenset({'♠'}),
frozenset({'♠', '♣', '♥'}),
frozenset({'♠', '♥', '♦'}),
frozenset({'♣'}),
frozenset({'♠', '♣'}),
frozenset({'♦'}),
frozenset({'♠', '♦'}),
frozenset({'♠', '♣', '♦'}),
frozenset({'♥'}),
frozenset({'♥', '♦'}),
frozenset({'♠', '♥'}),
frozenset({'♣', '♥'}),
frozenset({'♣', '♦'}),
frozenset({'♣', '♥', '♦'}),
frozenset({'♠', '♣', '♥', '♦'})}

```

    {frozenset(),
     frozenset({'♠'}),
     frozenset({'♦'}),
     frozenset({'♥'}),
     frozenset({'♥', '♦'}),
     frozenset({'♠', '♥'}),
     frozenset({'♣'}),
     frozenset({'♣', '♥'}),
     frozenset({'♣', '♥', '♦'}),
     frozenset({'♠', '♣'}),
     frozenset({'♠', '♣', '♥'}),
     frozenset({'♣', '♦'}),
     frozenset({'♠', '♦'}),
     frozenset({'♠', '♣', '♦'}),
     frozenset({'♠', '♥', '♦'}),
     frozenset({'♠', '♣', '♥', '♦'})}

딕셔너리, 정의역

```python
x = np.arange(-6, 6, 0.01)
x
```

    array([-6.  , -5.99, -5.98, ...,  5.97,  5.98,  5.99])

```python

def relu_func(x):
    return (x>0)*x

plt.plot(x, relu_func(x), linestyle='--', label="ReLU")

```

```python
def leaky_relu(x):
    return np.maximum(0.1*x,x)

xx = np.linspace(-2, 2, 100)
plt.plot(xx, relu(xx))
plt.plot(0, 0, marker='o', ms=10)
plt.title("ReLU")
plt.xlabel("$x$")
plt.ylabel("$ReLU(x)$")
plt.show()
```

```python
import numpy as np
import matplotlib.pylab as plt

x = np.arange(-6, 6, 0.01)
```

```python
def relu_func(x): # ReLU(Rectified Linear Unit, 정류된 선형 유닛) 함수
    return (x>0)*x
    # return np.maximum(0,x) # same

#그래프 출력
plt.plot(x, relu_func(x), linestyle='--', label="ReLU")
```

```python
def leakyrelu_func(x): # Leaky ReLU(Rectified Linear Unit, 정류된 선형 유닛) 함수
    return (x>=0)*x + (x<0)*0.1*x # 알파값(보통 0.01) 조정가능
    # return np.maximum(0.01*x,x) # same

#그래프 출력
plt.plot(x, leakyrelu_func(x), linestyle='--', label="Leaky ReLU")

```

```python
def f3(x):
    return (1 - x[0]) ** 2 + (1 - x[1]) ** 2 + 100 * ((x[1] - x[0] ** 2) ** 2 + (x[2] - x[1] ** 2) ** 2)
def f3g(x):
    return np.array([-400 * x[0] * (-x[0] ** 2 + x[1]) + 2 * x[0] - 2,
            -200 * x[0] ** 2 - 400 * x[1] * (-x[1] ** 2 + x[2]) + 202 * x[1] - 2,
            -200 * x[1] ** 2 + 200 * x[2]])
x0 = (0, 0, 0)
result = sp.optimize.minimize(f3, x0, jac=f3g)
print(result)
```

```python
import sympy

# Juypter 노트북에서 수학식의 LaTeX 표현을 위해 필요함
sympy.init_printing(use_latex='mathjax')

x1, x2, x3 = sympy.symbols('x1 x2 x3')
f = (1 - x1) ** 2 + (1 - x2) ** 2 + 100 * ((x2 - x1 ** 2) ** 2) ** 2 + ((x3 - x2 ** 2) ** 2)
f
```

$\displaystyle \left(1 - x_{1}\right)^{2} + \left(1 - x_{2}\right)^{2} + 100 \left(- x_{1}^{2} + x_{2}\right)^{4} + \left(- x_{2}^{2} + x_{3}\right)^{2}$

```python
sympy.simplify(sympy.diff(f, x1))
```

$\displaystyle 800 x_{1} \left(x_{1}^{2} - x_{2}\right)^{3} + 2 x_{1} - 2$

```python
sympy.simplify(sympy.diff(f, x2))
```

$\displaystyle 4 x_{2} \left(x_{2}^{2} - x_{3}\right) + 2 x_{2} - 400 \left(x_{1}^{2} - x_{2}\right)^{3} - 2$

```python
sympy.simplify(sympy.diff(f, x3))
```

$\displaystyle - 2 x_{2}^{2} + 2 x_{3}$

```python
sympy.simplify(sympy.diff(f, x1, x1))
```

$\displaystyle 4800 x_{1}^{2} \left(x_{1}^{2} - x_{2}\right)^{2} + 800 \left(x_{1}^{2} - x_{2}\right)^{3} + 2$

```python
sympy.simplify(sympy.diff(f, x1, x2))
```

$\displaystyle - 2400 x_{1} \left(x_{1}^{2} - x_{2}\right)^{2}$

```python
sympy.simplify(sympy.diff(f, x1, x3))
```

$\displaystyle 0$

```python
sympy.simplify(sympy.diff(f, x2, x1))
```

$\displaystyle - 2400 x_{1} \left(x_{1}^{2} - x_{2}\right)^{2}$

```python
sympy.simplify(sympy.diff(f, x2, x2))
```

$\displaystyle 12 x_{2}^{2} - 4 x_{3} + 1200 \left(x_{1}^{2} - x_{2}\right)^{2} + 2$

```python
sympy.simplify(sympy.diff(f, x2, x3))
```

$\displaystyle - 4 x_{2}$

```python
sympy.simplify(sympy.diff(f, x3, x1))
```

$\displaystyle 0$

```python
sympy.simplify(sympy.diff(f, x3, x2))
```

$\displaystyle - 4 x_{2}$

```python
sympy.simplify(sympy.diff(f, x3, x3))
```

$\displaystyle 2$

```python
x = sympy.symbols('x')
f = - sympy.exp(-x)
f

```

$\displaystyle - e^{- x}$

```python
sp.integrate.quad(f, 0, np.inf)
```

```python
import sympy

# Juypter 노트북에서 수학식의 LaTeX 표현을 위해 필요함
sympy.init_printing(use_latex='mathjax')

x1, x2, x3 = sympy.symbols('x1 x2 x3')
f = (1 - x1[0]) ** 2 + (1 - x2[1]) ** 2 + (100 * (((x2[1] - (x1[0] ** 2)) ** 2) + (x3[2] - (x2[1] ** 2) ** 2)))
f

```

```python
def f3(x):
    return (1 - x[0]) ** 2 + (1 - x[1]) ** 2 + (100 * (((x[1] - (x[0] ** 2)) ** 2) + (x[2] - (x[1] ** 2) ** 2)))
```

```python
x0 = (0, 0, 0)
result = sp.optimize.minimize(f3, x0)
print(result)
```

          fun: -2.747970332905848e+35
     hess_inv: array([[-2.30020321e-05, -2.56201954e-05,  1.41650738e-03],
           [-2.56202010e-05, -2.81597115e-05,  1.55052625e-03],
           [ 1.41650733e-03,  1.55052594e-03,  1.25254861e+03]])
          jac: array([0., 0., 0.])
      message: 'Optimization terminated successfully.'
         nfev: 355
          nit: 3
         njev: 71
       status: 0
      success: True
            x: array([-4.04676690e+08, -4.14667004e+08, -3.24614504e+12])

```python
# 400𝑥1(𝑥21−𝑥2)+2𝑥1−2

def flp(x):
    return np.array([-400*x[0]*(-x[0]**2 + x[1])])
```

```python
x0 = (0, 0, 0)
result = sp.optimize.minimize(f3, x0, jac = flp)
print(result)
```

          fun: 2.0
     hess_inv: array([[1, 0, 0],
           [0, 1, 0],
           [0, 0, 1]])
          jac: array([-2., -2., -2.])
      message: 'Desired error not necessarily achieved due to precision loss.'
         nfev: 107
          nit: 0
         njev: 95
       status: 2
      success: False
            x: array([0., 0., 0.])

```python
def f3(x):
    return (1 - x[0]) ** 2 + (1 - x[1]) ** 2 + 100 * ((x[1] - x[0] ** 2) ** 2 + (x[2] - x[1] ** 2) ** 2)
def f3g(x):
    return np.array([-400 * x[0] * (-x[0] ** 2 + x[1]) + 2 * x[0] - 2,
            -200 * x[0] ** 2 - 400 * x[1] * (-x[1] ** 2 + x[2]) + 202 * x[1] - 2,
            -200 * x[1] ** 2 + 200 * x[2]])
x0 = (0, 0, 0)
result = sp.optimize.minimize(f3, x0, jac=f3g)
print(result)
```

          fun: 1.9237776594950597e-14
     hess_inv: array([[0.09997913, 0.19848124, 0.3967629 ],
           [0.19848124, 0.39887784, 0.79749982],
           [0.3967629 , 0.79749982, 1.5994276 ]])
          jac: array([ 5.46870419e-06, -3.63809107e-06,  4.44125590e-07])
      message: 'Optimization terminated successfully.'
         nfev: 37
          nit: 28
         njev: 37
       status: 0
      success: True
            x: array([1.        , 0.99999999, 0.99999998])

-  두배 오르거나 1/2 내리거나
-  2년 후의 주식가격
- 표본공간

```python
f = {
    1: 'uu',
    2: 'ud',
    3: 'du',
    4: 'dd'
}
```

```python
f[1]
```

    'uu'

```python

```