# Survial Models

## Time to event data

Often, we are interested in determining how some variable or intervention affects the time to occurence of a certain outcome. For example, a researcher may be interested in how a particular drug affects how long a person will survive with a particular disease. Here we want to examine the waiting time for the occurence of such events.

In survival analysis, the likelihood is somewhat more involved in its formulation. To be able to properly discuss the likelihood specification in survival models, however, we must first discuss several key quantities in survival analysis. As discussed, our main outcome of interest is time, which has support $0\leq t< \infty$. We denote our time as with a random variable $T$.

## Important quantities in Survival Analysis

If the random variable $T$ is the time to the event we are studying, we can make probability statements about the lifetime of individuals. For example, we can calculate the probability of surviving past a particular time $t$ (i.e. $P(T>t)$). This probability, known as the survival function is the primary object of interest in survival analysis. The survival function is denoted by

$$S(t) = P(T > t) = 1 - F(t),$$

where $F$ is the [CDF](https://en.wikipedia.org/wiki/Cumulative_distribution_function) of $T$.  It is mathematically convenient to express the survival function in terms of the [hazard rate](https://en.wikipedia.org/wiki/Survival_analysis#Hazard_function_and_cumulative_hazard_function), $h(t)$.  The hazard rate is the instantaneous probability that the event occurs at time $t$ given that it has not yet occured.  That is,

$$\begin{align*}
h(t)
    & = \lim_{\Delta t \to 0} \frac{P(t < T < t + \Delta t\ |\ T > t)}{\Delta t} \\
    & = \frac{f(t)}{S(t)}
\end{align*}$$

Furthermore, we have that

$$S(t) = \exp\left(-\int_0^t h(s)\ ds\right).$$

This representation of the survival function shows that the cumulative hazard function

$$H(t) = \int_0^t h(s)\ ds$$

is an important quantity in survival analysis, since we may consicesly write $S(t) = \exp(-H(t)).$

An important, but subtle, point in survival analysis is [censoring](https://en.wikipedia.org/wiki/Survival_analysis#Censoring). In many survival studies, many data points are `right censored` or rather survival times are usually only known for a portion for those under study where as the remainder are only known to exceed certain values. Specifically, an observation is said to be right censored at $c$ if the exact value of the observation is not known but only that it is greater than or equal to $c$.

### Data notation

Suppose that there are $n$ subjects under study and that associated with the $i^{th}$ individual is a survival time $t_i*$ and a fixed censoring time $c_i$. The exact survival time $t_i$ will be observed only if $t_i* \leq c_i$. Hence our data is of the form $(y_i,\delta_i)$ where
$$
\begin{equation}
t_i = min(t_i*, c_i)
\end{equation}
$$
and
$$
\begin{equation}
\delta_i= I(t_i*\leq c_i)
\end{equation}
$$

In our dataset, $\delta_i$ is given by `df.event` and

```python
%matplotlib inline
import numpy as np
import pandas as pd
import pymc3 as pm
import arviz as az
from pymc3 import Gamma, Poisson, Normal, Model, sample, forestplot, NUTS, Metropolis, find_MAP, starting, traceplot
import seaborn as sns
import matplotlib.pyplot as plt
from statsmodels import datasets
from theano import tensor as tt
from io import StringIO
import warnings
warnings.simplefilter('ignore')

sns.set_style("ticks")

SEEDS = 2234323, 172354266
np.random.seed(SEEDS)
```

```python
blue, _, red = sns.color_palette()[:3]
```

### Mastectomy Data

In this tutorial, we will use an oncological dataset.
In this dataset, each row represents observations from a woman diagnosed with breast cancer that underwent a mastectomy. The column `time` ($t_i$) represents the time (in months) post-surgery that the woman was observed. The column `event` indicates whether or not the woman died during the observation period (i.e. $\delta_i$). The column `metastized` represents whether the cancer had metastized prior to surgery.

We will analyze the relationship between survival time post-mastectomy and whether or not the cancer had metastized.

```python
df = datasets.get_rdataset('mastectomy', 'HSAUR', cache=True).data
df.event = df.event.astype(np.int64)
df.metastized = (df.metastized == 'yes').astype(np.int64)
n_patients = df.shape[0]
patients = np.arange(n_patients)
```

```python
fig, ax = plt.subplots(figsize=(14, 10))

blue, _, green = sns.color_palette()[:3]

ax.hlines(patients[df.metastized.values == 0], 0, df[df.metastized.values == 0].time,
          color=blue, label='Not metastized')

ax.hlines(patients[df.metastized.values == 1], 0, df[df.metastized.values == 1].time,
          color=green, label='Metastized')

ax.scatter(df[df.event.values == 1].time, patients[df.event.values == 1],
           color='k', zorder=10, label='Death')

ax.set_xlim(left=0)
ax.set_xlabel('Months since mastectomy')
ax.set_yticks([])
ax.set_ylabel('Subject')

ax.set_ylim(-0.25, n_patients + 0.25)

ax.legend(loc='center right');
```

As an initial look into survival, we plot the non-parametric Kaplan Meier estimates of the survival via the package `lifelines`

```python
!conda install -y lifelines
```

    Collecting package metadata (current_repodata.json): done
    Solving environment: done

    ## Package Plan ##

      environment location: /Users/twiecki/miniconda3

      added / updated specs:
        - lifelines

    The following packages will be downloaded:

        package                    |            build
        ---------------------------|-----------------
        libopenblas-0.3.10         |openmp_h63d9170_4         8.2 MB  conda-forge
        llvm-openmp-10.0.1         |       h28b9765_0         265 KB  conda-forge
        matplotlib-base-3.1.3      |   py37h11da6c2_0         6.6 MB  conda-forge
        numpy-1.19.1               |   py37h7e69742_0         5.1 MB  conda-forge
        scipy-1.3.3                |   py37h82752d6_0        18.3 MB  conda-forge
        ------------------------------------------------------------
                                               Total:        38.5 MB

    The following NEW packages will be INSTALLED:

      autograd           conda-forge/noarch::autograd-1.3-py_0
      autograd-gamma     conda-forge/noarch::autograd-gamma-0.4.1-py_0
      cycler             conda-forge/noarch::cycler-0.10.0-py_2
      freetype           conda-forge/osx-64::freetype-2.10.2-h8da9a1a_0
      future             conda-forge/osx-64::future-0.18.2-py37hc8dfbb8_1
      kiwisolver         conda-forge/osx-64::kiwisolver-1.2.0-py37ha1cc60f_0
      libblas            conda-forge/osx-64::libblas-3.8.0-17_openblas
      libcblas           conda-forge/osx-64::libcblas-3.8.0-17_openblas
      libgfortran        conda-forge/osx-64::libgfortran-4.0.0-2
      liblapack          conda-forge/osx-64::liblapack-3.8.0-17_openblas
      libopenblas        conda-forge/osx-64::libopenblas-0.3.10-openmp_h63d9170_4
      lifelines          conda-forge/noarch::lifelines-0.25.0-py_0
      llvm-openmp        conda-forge/osx-64::llvm-openmp-10.0.1-h28b9765_0
      matplotlib-base    conda-forge/osx-64::matplotlib-base-3.1.3-py37h11da6c2_0
      numpy              conda-forge/osx-64::numpy-1.19.1-py37h7e69742_0
      pandas             conda-forge/osx-64::pandas-1.0.5-py37h94625e5_0
      patsy              conda-forge/noarch::patsy-0.5.1-py_0
      pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
      pytz               conda-forge/noarch::pytz-2020.1-pyh9f0ad1d_0
      scipy              conda-forge/osx-64::scipy-1.3.3-py37h82752d6_0

    Downloading and Extracting Packages
    numpy-1.19.1         | 5.1 MB    | ##################################### | 100%
    llvm-openmp-10.0.1   | 265 KB    | ##################################### | 100%
    matplotlib-base-3.1. | 6.6 MB    | ##################################### | 100%
    libopenblas-0.3.10   | 8.2 MB    | ##################################### | 100%
    scipy-1.3.3          | 18.3 MB   | ##################################### | 100%
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done

```python
from lifelines import KaplanMeierFitter
kmf = KaplanMeierFitter()
```

```python
kmf.fit(df.time, event_observed=df.event)
```

    <lifelines.KaplanMeierFitter:"KM_estimate", fitted with 44 total observations, 18 right-censored observations>

```python
time = df.time
event = df.event
ix0 = df.metastized==0
ix1 = df.metastized==1

kmf.fit(time[ix0], event[ix0], label='no Metastasis')
ax = kmf.plot(color=green)

kmf.fit(time[ix1], event[ix1], label='Metasasis')
kmf.plot(ax=ax, ls='dashed', color=blue, figsize=(14,10))
ax.set_xlabel('Time (months)')
ax.set_ylabel('Survival');
```

# Model Specification

We wish to relate a treatment effect in some way to the time to event outcome. There are several ways to do this, but two main paradigms are pervasive in the literature:

1. Proportional Hazards
2. Accelerated Failure Time

In this lecture we will focus on the first of these models.

There are a number of probability models one could use for lifetime data but the Weibull distribution is a very flexible model that is popular with many researchers. Its flexibility comes from the fact that it has a hazard rate which is either monotone increasing, decreasing, or constant. Additionally, it is the only parametric regression model which has both a proportional hazards representation and an accelerated failure-time representation. Again, however, we will only be focusing on the Proportional hazards specification.

## Proportional Hazards Model

The main objective at this point is to specify a model that relates the group effect to the time outcome. Recall above that the hazard quantifies the instantaneous risk of disease at a given time point.

It turns out that the easiest survival parameter to model is the hazard rate: which tells us how quickly individuals of a certain age are experiencing the event of interest. The major approach to modeling the effects of covariates on survival is to model the conditional hazard rate as a function of the covariates. Two general classes of models have been used to relate covariate effects to survival:

1. the family of multiplicative hazard models, and
2. the family of additive hazard rate models.

We will focus on the former of these here.

For the family of multiplicative hazard rate models the conditional hazard rate of an individual with covariate vector $X$ as a product of a baseline hazard rate $h_0(t)$ and a non-negative function of the covariates,
$c(\beta^T X)$, that is, $h (t | z) =h_0(t)c(\beta^T X)$.

   -$h_0(x)$ may have a specified parametric form or it may be left as an arbitrary non-negative function.

   - Any non-negative function can be used for the link function $c()$. Most applications use the Cox (1972) model with $c(\beta^T X) = \exp(\beta^T X)$ which is chosen for its simplicity
   - Key Feature: when all the covariates are fixed at time 0, the hazard rates of two individuals with distinct
   values of X are proportional.

Using the exponential link, we have
$$
{\displaystyle h (t|X_{i})=h _{0}(t)\exp(\beta^T X_{i} )}
$$

That is to say, the model is formulated in such a fashion that the group effect has a multiplicative effect on the hazard. Typically, semi-parametric models which don't have explicit representations of $h_0(t)$ are preferred. However, for the purposes of this tutorial, we will assume a particular form for this baseline hazard.

The Weibull proportional hazards model models the hazard as

$$
{\displaystyle h (t|X_{i})=\alpha\lambda t^{\alpha-1}\exp(\beta^TX_{i})}
$$

## Likelihood Specification

Due to right censoring, we need to adjust our likelihood to account for observations where we know that an individual survived up to a particular point. If a person is known to have survived at least to time $t$, then their likelihood contribution is simply the survival function evaluated at the censoring time. We typically express this with an indicator variable $(\delta_i)$ for each observation denoting whether or not the individual was right censored or observed.

First, recall that the Weibull proportional hazards model models the hazard as a function of the covariates. The probability density function is related to the hazard by:

$$
\begin{equation}
f(t) = h(t)S(t)
\end{equation}
$$

The likelihood specification for an individual is therefore given by:
$$
\begin{eqnarray}
L_i &=& [f(t|X)]^{\delta_i}[S(t|X)]^{1-\delta_i}\\
&=& [h(t|X)S(t|X)]^{\delta_i}[S(t|X)]^{1-\delta_i}\\
&=& [\alpha\lambda t^{\alpha-1}\exp(\beta^T X)exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}\\
&=&[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}\\
\end{eqnarray}
$$

### Specification in PyMC3

We've formulated the mathematical form of our likelihood - we now need to implement it. PyMC3 makes this really easy to do with the `DensityDist` class. All that is needed to specify a likelihood is a python function that passes the log of the likelihood.

Continuing from before:

$$
\begin{eqnarray}
log L_i &=& \log[[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}]\\
&=& \delta_i \log[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]+(1-\delta_i)(-\lambda t^{\alpha}\exp(\beta^T X))\\
&=&\delta_i(\log\alpha+\log\lambda+(\alpha-1)\log t + \beta^T X) - \lambda t^{\alpha}\exp(\beta^T X)
\end{eqnarray}
$$

```python
## program log likelihood

def logp(event):
    return (event*(tt.log(α) + tt.log(λ)
                   + tt.log(tt.exp(β*np.array(metast)))
                   + (α-1)*tt.log(np.array(time)))
            - (λ*tt.exp(β*np.array(metast))
                   * np.array(time)**α))

    survival = pm.DensityDist('survival', logp,
                              observed={'event': event})
```

### Prior Distribution specification:
Recall that in a Bayesian framework, estimated parameters are considered random variables - so we need to place a prior distribution on them. Looking at the parameters to estimate, we have

1. Weibull baseline hazard parameters: $\alpha$,$\lambda>0$
    - A natural prior for these parameters would be the Gamma distribution.

```python
from pymc3 import Gamma

params = (5, 1), (1, 3), (5, 5), (0.5, 0.5), (10, 1)

fig, axes = plt.subplots(1, len(params), figsize=(14, 4), sharey=True)
for ax, (alpha, beta) in zip(axes, params):
    sns.distplot(Gamma.dist(alpha, beta).random(size=1000), ax=ax, kde=False)
    ax.set_xlim(0, 20)
    ax.set_title(r'$\alpha={0}, \beta={1}$'.format(alpha, beta));
```

 - Alternatively, one might specify a Half Cauchy distribution:

```python
sns.distplot(pm.HalfCauchy.dist(1).random(size=1000), kde=False)
```

2. Group effect: $-\infty<\beta<\infty$

For the effect of metastasis, the most natural prior is the Normal distribution.

```python
sns.distplot(pm.Normal.dist(0,100).random(size=3000), kde=False)
```

### Implementation in PyMC3

Putting all the pieces together, we can fit the model using the `pymc` framework as so:

```python
metast = df.metastized.values
```

```python
n_iterations = 2000
n_tune = 1000
```

```python
with Model() as weibull_model:
    α = pm.HalfCauchy("α", 1)
    # Weibull shape parameter
    λ = pm.HalfCauchy("λ", 1)
    # Effects of metastisis
    β = pm.Normal('β', 0, 100)
    # Weibull scales
    linear = pm.Deterministic('linear', tt.exp(β*metast))

    # Weibull survival likelihood, accounting for censoring
    def logp(event):
        return event*(tt.log(α) + tt.log(λ) + tt.log(linear)
                      + (α-1)*tt.log(np.array(time))) - (λ*linear * np.array(time)**α)

    survival = pm.DensityDist('survival', logp,
                              observed={'event': event})
    weibull_trace = pm.sample(n_iterations, tune=n_tune, cores=2)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [β, λ, α]

<div>
    <style>
        /* Turns off some styling */
        progress {
            /* gets rid of default border in Firefox and Opera. */
            border: none;
            /* Needs to be in here for Safari polyfill so background images work as expected. */
            background-size: auto;
        }
        .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
            background: #F44336;
        }
    </style>
  <progress value='6000' class='' max='6000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [6000/6000 00:16<00:00 Sampling 2 chains, 0 divergences]
</div>

    Sampling 2 chains for 1_000 tune and 2_000 draw iterations (2_000 + 4_000 draws total) took 16 seconds.
    The acceptance probability does not match the target. It is 0.887443419244335, but should be close to 0.8. Try to increase the number of tuning steps.

We can view the posterior samples and assess convergence of the Markov Chain:

```python
az.plot_trace(weibull_trace, var_names=['α', 'λ', 'β']);
```

Since we have fit a proportional hazards model, the interpretation of the group effect is straightforward (at least with respect to the hazard). We need only to exponentiate the $\beta$ posterior samples for the effect size and corresponding credible intervals.

```python
hr_samps = np.exp(weibull_trace['β'])
print('Hazard ratio (metastisized vs not metastisized) at 5 years: {0:.2f} {1}'.format(hr_samps.mean(),
                                                            np.percentile(hr_samps, [2.5, 97.5]).round(2)))
```

    Hazard ratio (metastisized vs not metastisized) at 5 years: 2.28 [0.84 5.23]

> Interpretation: This means that the hazard for the group with metastasis is bigger
than the one for the group with no metastasis. This indicates metastasis has a negative effect on survival.

### Posterior Survival quantites

Given that we have posterior samples for each of the parameters, it is straightforward to obtain posterior survival curves, hazard rates, etc, with corresponding credible intervals.

```python
## Plot survival curves
n_intervals = 120
interval_bounds = np.arange(0, n_intervals + 1, 1)
```

```python
#baseline_lam = np.exp(weibull_trace['β'][n_burn:,0])
baseline_lam = np.repeat(1., n_iterations*2)
metast_lam = np.exp(weibull_trace['β'])
λ_trace = weibull_trace['λ']
α_trace = weibull_trace['α']
```

```python
def plot_with_hpd(x, lam, f, ax, linestyle='-', color=None, label=None, alpha=0.05, debug=False, **kwargs):

    m = np.median(f(lam, **kwargs), axis=0)

    percentiles = 100 * np.array([alpha / 2., 1. - alpha / 2.])
    hpd = np.percentile(f(lam, **kwargs), percentiles, axis=0)

    if debug:
        import pdb;pdb.set_trace()
    ax.fill_between(x, hpd[0], hpd[1], color=color, alpha=0.15)
    ax.step(x, m, color=color, linestyle=linestyle, label=label);
```

```python
def cum_weibull_hazard(lam, α=α_trace ,λ=λ_trace, t=n_intervals):
    return (λ_trace.reshape(-1,1)*lam.reshape(-1,1) * np.arange(t)) ** α.reshape(-1,1)

def weibull_survival(lam,α=α_trace, λ=λ_trace, t=n_intervals):
    return np.exp(-cum_weibull_hazard(lam, α, λ, t))

def cum_hazard(hazard, n=n_intervals):
    return (np.arange(n) * hazard.reshape(-1,1)).cumsum(axis=-1)

def survival(hazard):
    return np.exp(-cum_hazard(hazard))
```

```python
fig, (hazard_ax, surv_ax) = plt.subplots(ncols=2, sharex=True, sharey=False, figsize=(16, 6))

plot_with_hpd(interval_bounds[:-1], baseline_lam, cum_weibull_hazard,
              hazard_ax, color=green, linestyle=':', label='No Metastasis')
plot_with_hpd(interval_bounds[:-1], metast_lam, cum_weibull_hazard,
              hazard_ax, color=blue, linestyle='--', label='Metastisized')

hazard_ax.set_xlim(0, n_intervals);
hazard_ax.set_xlabel('Followup time');

hazard_ax.set_ylabel(r'Cumulative hazard $\Lambda(t)$');

hazard_ax.legend(loc=2);

plot_with_hpd(interval_bounds[:-1], baseline_lam, weibull_survival,
              surv_ax, color=green, linestyle=':')
plot_with_hpd(interval_bounds[:-1], metast_lam, weibull_survival,
              surv_ax, color=blue, linestyle='--')

surv_ax.set_xlim(0, n_intervals);
surv_ax.set_xlabel('Followup time');

surv_ax.set_ylabel('Survival function $S(t)$');

fig.suptitle('Bayesian survival model');
```

Looking at our KM estimate from earlier, we see that the posterior survival curves follow closely to the non-parametric K-M estimates.

```python
## Hazard ratio comparing 2 groups at a given time point
def weibull_hazard(lam, t, α=α_trace, λ=λ_trace):
    return (α.reshape(-1,1) * λ.reshape(-1,1) * lam.reshape(-1,1) * t**(α.reshape(-1,1) - 1))
```

We can of course, obtain our hazard ratio from the posterior samples:

```python
hazard_ratio = weibull_hazard(metast_lam, 60)/weibull_hazard(baseline_lam, 60)
print('Hazard ratio (metastisized vs notmetastisized) at 5 years: {0} {1}'.format(hazard_ratio.mean(),                                                     np.percentile(hazard_ratio, [2.5, 97.5])))
```

    Hazard ratio (metastisized vs notmetastisized) at 5 years: 2.2798788499285894 [0.83868425 5.23182762]

```python
### Other quantities of interest:

baseline_survival = weibull_survival(baseline_lam)
metast_survival = weibull_survival(metast_lam)
```

```python
#Time where surivial is .5
np.where(baseline_survival.mean(axis=0)<=0.5)[0][0]
np.where(metast_survival.mean(axis=0)<=0.5)[0][0]
```

    38

```python
baseline_survival_5 = baseline_survival[:, 60]
metast_survival_5 = metast_survival[:, 60]
print('Baseline 5-year median survival: {0:.2f} {1}'.format(np.median(baseline_survival),
                                                        np.percentile(baseline_survival, [2.5, 97.5]).round(2)))

print('Metastasis group median 5-year survival: {0:.2f} {1}'.format(np.median(metast_survival_5),
                                                        np.percentile(metast_survival_5, [2.5, 97.5]).round(2)))
```

    Baseline 5-year median survival: 0.60 [0.14 0.98]
    Metastasis group median 5-year survival: 0.34 [0.11 0.81]

## Exercise: Heart transplant survival

Try your hand at building a survival model using PyMC3. Here is a small dataset based on 69 patients receiving heart transplants. Taken from "The Statistical Analysis of Failure Time Data" by Kalbfleisch and Prentice, Appendix I, pages 230-232 from stalib data depository. http://lib.stat.cmu.edu/datasets/stanford

Data columns:
- Age at transplant in years
- Survival Status 1=dead 0=alive
- Survival Time after transplant in days

Estimate the relationship between age at transplant and survival probability.

```python
data_str = """
41 1 5
40 1 16
54 1 16
29 1 17
55 1 28
52 1 30
40 1 39
35 0 39
56 1 43
36 1 45
42 1 51
50 1 53
42 1 58
52 1 61
61 1 66
45 1 68
49 1 68
53 1 72
47 1 72
64 1 77
51 1 78
53 1 80
54 1 81
56 1 90
53 1 96
48 1 100
28 0 109
46 1 110
23 0 131
47 1 153
43 1 165
26 0 180
52 1 186
47 1 188
51 1 207
51 1 219
47 0 265
48 1 285
19 1 285
49 1 308
42 1 334
44 0 340
47 1 342
54 0 370
48 0 397
52 0 445
46 0 482
48 0 515
52 0 545
48 1 583
26 0 596
47 0 630
47 0 670
50 1 675
58 1 733
32 0 841
44 1 852
41 0 915
38 0 941
45 1 979
48 1 995
43 1 1032
36 0 1141
45 0 1321
53 1 1386
48 0 1407
40 0 1571
48 0 1586
33 0 1799
"""

transplant_data = pd.read_table(StringIO(data_str), sep='\s+', names=['transplant_age', 'death', 'time'])
transplant_data.head()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>transplant_age</th>
      <th>death</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>41</td>
      <td>1</td>
      <td>5</td>
    </tr>
    <tr>
      <th>1</th>
      <td>40</td>
      <td>1</td>
      <td>16</td>
    </tr>
    <tr>
      <th>2</th>
      <td>54</td>
      <td>1</td>
      <td>16</td>
    </tr>
    <tr>
      <th>3</th>
      <td>29</td>
      <td>1</td>
      <td>17</td>
    </tr>
    <tr>
      <th>4</th>
      <td>55</td>
      <td>1</td>
      <td>28</td>
    </tr>
  </tbody>
</table>
</div>

```python

```

---
# References

Ibrahim, Joseph G., Ming‐Hui Chen, and Debajyoti Sinha. Bayesian survival analysis. John Wiley & Sons, Ltd, 2005.

Klein, John P., Moeschberger, Melvin L. Survival Analysis: Techniques for Censored and Truncated Data Second Edition. Springer, 2003.


```python
import sys
sys.path.append("..")
from gblend.geometry import *
from gblend.levels import *
from gblend.variables import x,y,z,rxy

import graphics.d3 as g3d
from sympy import pi
```

```python
t1 = RevolutionSurface3D(1/4-z**2-(rxy-2)**2)
t1.displace((x**2)/5+(y**2)/5)
t2 = RevolutionSurface3D(1/4-z**2-(rxy-2)**2)
t1.translate(-1,0,0)
t2.translate(1,0,0)

obj = t1 | t2
res = 120 # resolution for extracting mesh
sz = 5

ps, ts, ns, vals = geometry_level(obj,[-sz,sz,-sz,sz,-sz,sz], [res,res,res])
mesh = g3d.mesh3d(ps, ts)
seat_fig = g3d.get_figure([mesh])
seat_fig
```

```python
b1 = Ball3D(k=1/3)
b1.scale(1,1,.5)
b1.displace((x**2+y**2)/2)

b2 = Ball3D(k=1/3)
b2.scale(.6,.6,.6)
b2.translate(0,0,.5)

t1 = Torus3D(.3,2,k=3/4)
t1.rotate(1,0,0,pi/2)

obj = (b1-b2) | t1
res = 150 # resolution for extracting mesh
sz = 4

ps, ts, ns, vals = geometry_level(obj,[-sz,sz,-sz,sz,-sz,sz], [res,res,res])
mesh = g3d.mesh3d(ps, ts)
craddle_fig = g3d.get_figure([mesh])
craddle_fig
```

```python
#seat_fig.write_image("seat.png")
#craddle_fig.write_image("craddle.png")
```

```python
import math
```

```python
import sys
sys.path.append("..")
import gblend
import gblend.geometry
import gblend.levels
import gblend.variables
help(gblend.variables)
```

    Help on module gblend.variables in gblend:

    NAME
        gblend.variables - Some useful variables to define functions in 3-dimensional space

    DATA
        k = k
        rxy = rxy
        x = x
        y = y
        z = z

    FILE
        /Users/valeroc/Dropbox/MyProjects/geometry-blender/gblend/variables.py

```python

```

```python

```


# Rješavanje jednadžbi za dobivanje prijenosne funkcije

U ovom poglavlju biti će opisano rješavanje linearnih jednadžbi supstitucijom, rješavanje sustava linearnih jednadžbi te supstitucije varijabli.

Da bi smo dobili prijenosnu funkciju sustava potrebno je rješiti sustav jednadžbi koje definiraju sustav, u elektrotehnici su to npr. jednadžbe dobivene metodom konturnih struji, napona čvorova ili nekom metodom koja opisuje strujno-naponske zavisnosti unutar sustava.

Rješavanju jednadžbi potrebno je pristupiti analitički, te standardnim postupcima supstitucije doći do rješenja.

Python funkcije koje se koriste za ove operacije dio su ```Sympy``` knjižnice.
```python
* Eq
* symbols
* Function
* sympy.solvers solve
```

Za početak potrebno je obrisati sve varijable iz memorije.

```python
import os
clear = lambda: os.system('cls')
clear()
```

    0

Potom importati sve funkcije koje koristimo.

```python
from sympy.interactive import printing
printing.init_printing(use_latex=True)

from sympy import Eq, symbols, Function
from sympy.solvers import solve
```

### Primjer: Linearni električni sustav (Izvor: N.Stojkovic, N.Mijat - Teorija mreža i linija (Zbirka zadataka), zadatak 3.05.)

Za početak je potrebno definirati sva imena matematičkih varijabli koje koristimo kao Sympy simbole.

```python
Uul, Uiz = symbols('Uul Uiz')
Z1, Z2 = symbols('Z1 Z2')
I1, I2 = symbols('I1 I2')
r = symbols('r')
Tf = symbols('Tf')
s = symbols('s')
L1, L2 = symbols('L1 L2')
```

1.) Postavimo jednadžbe koje opisuju sustav.

```python
eq = Eq(Uul, I2*Z1 + Uiz + I2*Z2)
display(eq)
```

```python
i2 = (1/r) * Uul
print('I2 =')
display(i2)
```

Za supstituiranje I2 u gornjoj jednadžbi s izrazom za I2 potrebno je koristiti funkciju ```.subs()```

Funkcija ```.subs()``` kao argument ima varijablu koju supstituiramo te izraz koji unosi ta supstitucija.
Simbol (Matematicka varijabla) i python varijabla ne smiju imati isto ime, no ukoliko nam je simbol ```I2```, a python varijabla ```i2``` tada možemo supstituirati simbol s izrazom za ```I2``` koji je spremljen kao python varijabla ```i2```.
Iz tog razloga je gornja jednadžba zapisana kao ```i2=...``` a ne ```I2=...```.

```.subs(VARIJABLA, IZRAZ)```, u ovom slučaju:

```python
eq = eq.subs(I2, i2)
display(eq)
```

Isto smo mogli napraviti s:

```python
eq = eq.subs(I2, (1/r) * Uul)
```

jer je u memoriji računala (1/r) * Uul spremljeno kao i2.

Sada kada smo dobili izraz iz kojeg mozemo izlučiti ```Uul``` te ```Uiz```, samim time dobiti prijenosnu funkciju ```Tf = Uiz/Uul``` potrebno je rješiti jednadžbu za varijablu ```Uul``` uz pomoć funkcije ```solve()```, to će nam dati izraz u obliku ```Uul = ...```

```python
Uul = solve(eq, Uul)
display(Uul)
```

Vidimo da je zapis u obliku matrice, s matricom ne možemo raditi daljnje potrebne operacije stoga je potrebno izvući izraz iz matrice i spremiti ga u "običnu" varijablu.
S obzirom da smo jednadžbu rješili samo za jednu varijablu, rezultat je izražen kao samo jedno polje te je dovoljno napraviti sljedeće:

```python
Uul = Uul[0]
display(Uul)
```

Sada kada imamo definiran izraz za ```Uul``` možemo jednostavno definirati ```Tf``` kao ```Uiz/Uul``` te dobiti prijenosnu funkciju.

```python
Tf = Uiz/Uul
display(Tf)
```

```Z1``` i ```Z2``` možemo supstituirati kao ```sL1``` i ```sL2``` putem funkcije ```.subs()``` kao u ranijem koraku.

```python
Tf = Tf.subs(Z1, s*L1)
Tf = Tf.subs(Z2, s*L2)
display(Tf)
```

Potom vrijednosti ```L1```, ```L2``` i ```r``` možemo unositi istim postupkom supstitucije, npr. ```L1=2```, ```L2=3```, ```r=1/2```

```python
Tf = Tf.subs(L1, 2)
Tf = Tf.subs(L2, 3)
Tf = Tf.subs(r, 1/2)
display(Tf)
```

Isto smo mogli napraviti na način:
```python
l1 = 2
l2 = 3
R = 1/2
Tf = Tf.subs(L1, l1)
Tf = Tf.subs(L2, l2)
Tf = Tf.subs(r, R)
display(Tf)
```

### Primjer: Linearni električni sustav (Izvor: N.Stojkovic, N.Mijat - Teorija mreža i linija (Zbirka zadataka), zadatak 3.12.)

```python
import os
clear = lambda: os.system('cls')
clear()
```

```python
from sympy.interactive import printing
printing.init_printing(use_latex=True)

from sympy import Eq, symbols, Function
from sympy.solvers import solve
import sympy as sp
```

```python
U1, U2 = symbols('U1 U2')
Uiz, Uul = symbols('Uiz Uul')
I1, I2 = symbols('I1 I2')
Tf = symbols('Tf')
s = symbols('s')
R1, R2, L, C = symbols('R1 R2 L C')
```

1) Definiramo jednadžbe

```python
eq1 = Eq(U1 * ((1/(s*L))+s*C) - Uul*(1/(s*L)), 0)
eq2 = Eq(U2 * ((1/R1)+(1/R2)) - Uiz*(1/R1), 0)

display(eq1)
display(eq2)
```

Po definiciji operacijskog pojačala ```Uiz=A(U1-U2)```, gdje A teži ka beskonačno dolazimo do izraza ```U1=U2```
U idućem koraku rješimo gornje jednadžbe za varijable ```U1``` i ```U2```.

```python
u1 = solve(eq1, U1)
u2 = solve(eq2, U2)
u1 = u1[0]
u2 = u2[0]
display(u1)
display(u2)
```

Vratimo se na definiciju ```U1 = U2```

```python
eq3 = Eq(U1, U2)
display(eq3)
```

Te uvrstimo supstituciju s gore dobivenim izrazima za ```U1``` i ```U2```

```python
eq3 = eq3.subs(U1, u1)
eq3 = eq3.subs(U2, u2)
display(eq3)
```

Potom rješimo jednadžbu za varijablu ```Uiz``` te podijelimo s ```Uul```.

```python
tf = solve(eq3, Uiz)
tf = tf[0]
display(tf)
```

```python
tf = tf/Uul
display(tf)
```

Uvrstimo zadane vrijednosti parametara te supstituiramo.

```python
r1 = 9
r2 = 1
l = 2
c = 1
```

```python
tf = tf.subs(R1, r1)
tf = tf.subs(R2, r2)
tf = tf.subs(L, l)
tf = tf.subs(C, c)
display(tf)
```


```python
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats
import random

%matplotlib inline
```

**Note:**  This Jupyter notebook is a slightly shortened version of "HT&GT.ipynb" found at this GitHub repository __[here](https://github.com/craw-daddy/Introductory-DS)__.

## Statistical Hypothesis Testing

Hypothesis testing typically involves comparing two datasets to look for a statistical relationship between the two datasets.  In other cases, testing will involve a dataset versus an "idealized" dataset.

### Examples of hypothesis testing
1.  Is the mean height of Scandinavian men the same as the mean height of other European (non-Scandinavian) men?
2.  Are the number of males/females fined for speeding in New York "significantly different" than male/female arrests for speeding in Massachusetts in 2017?  (This is essentially asking if the number of fines is independent of gender and location.)
3.  A certain machine for manufacturing cardboard boxes is supposed to manufacture ones that are 1/5" thick.  A quality control technician in the company checks a sample of the output to test if the thickness is smaller than this desired target (meaning the boxes will be too weak).
4.  When making tea, is there is a difference in taste whether the tea is added to the cup first, or the milk is added to the cup first?  (A somewhat famous example described by Ron Fisher who helped lay the foundations of statistical hypothesis testing.  Dr. Muriel Bristol, a colleague of Fisher, claimed to be able to tell which was added first.)

### Null hypothesis and alternative hypothesis

$H_O$:  The null hypothesis, assumed to be true.

$H_A$:  The alternative hypothesis, accepted if the samples/observations support this hypothesis "with sufficient evidence".

For some of the examples given above, $H_O$ and $H_A$ might be stated as follows:
1. $H_O$: $\mu_S = \mu_E$ where $\mu_S$ is the mean height of Scandinavian men and $\mu_E$ is the mean height of other European (non-Scandinavian) men.

   $H_A$: $\mu_S \not= \mu_E$

2.  $H_O$:  The frequency of speeding fines is independent of gender and location.

    $H_A$:  The frequency of speeding fines is not independent of gender and/or location.

3.  $H_O$:  $\mu = 0.20$ where $\mu$ is the mean thickness of the boxes in inches.

    $H_A$:  $\mu < 0.20$

###  Common assumptions about samples/observations
1.  Samples come from a normal distribution, or at least one that is symmteric.  Alternatively, the number of samples should be high enough (at least $30$) so that the "Law of Large Numbers", i.e. the Central Limit Theorem applies.
2.  Samples are independent from one another.
3.  For multiple datasets, it is typically assumed they have a common variance.  (That itself could constitute another hypothesis test.)

(Other assumptions might be used if data is categorical in nature.)

### Test statistic

The test statistic summarizes the dataset(s) into one value that can be used to (try to) distinguish the null hypothesis from the alternative hypothesis.

Common test statistic distributions:  Student's t distribution, normal distribution (using Law of Large Numbers with known variance), $\chi^2$ distribution

### Significance level
The significance level, $\alpha$, is the probability threshold below which the null hypothesis will be rejected.  What this means is the following:  Assume the null hypothesis is true, and let $T$ denote the test statistic that will be used in the test.  The significance level partitions the possible values of $T$ into regions where the null hypothesis will be rejected (the _critical region_), and those where it will not be rejected.  (E.g. for a normal distribution, the critical region will be the tail(s) of the distribution.)  The probability of the critical region is equal to $\alpha$.  Typical values for $\alpha$ are $0.05$ and $0.01$.

### Procedure of the test
1.  Having chosen the test statistic $T$, and the significance level $\alpha$, we compute the observed value of $T$ using the samples/observations in our dataset.  Call this value $t_{obs}$.
2.  Check if $t_{obs}$ lies in the critical region.  If so, we reject the null hypothesis in favor of the alternative hypothesis.  If not, we cannot reject the null hypothesis.
3.  Equivalently, the value of the test statistic corresponds to a $p$-value, which is the probability of that value $t_{obs}$ occurring, assuming that the null hypothesis is true.  If the $p$-value is smaller than or equal to the significance level, we reject the null hypothesis in favor of the alternative hypothesis.

### Errors in hypothesis testing
It is possible that we make an error in hypothesis testing, e.g. rejecting the null hypothesis incorrectly or failing
to reject the null hypothesis when it is not the truth (but the samples we have do not support rejecting it).

A common analogy used in hypothesis testing is to a criminal trial.  In most countries, a defendant on trial is presumed to be innocent (the "null hypothesis").  The evidence presented during the course of a trial is analogous to the samples taken in a statistical test.  If the evidence is sufficiently persuasive, the jury can find the defendant guilty (the "alternative hypothesis").  If the evidence is not persuasive, the defendant is found "not guilty".  It's possible, of course, that the jury's decision is wrong.

	Truth

        (Jury) Decision
        _Not Guilty_
_(Null Hypothesis)_
        _Guilty_
_(Alternative Hypothesis)_

        Not Guilty
(Accept Null Hypothesis)
        Ok
        Error
(**Type II Error**)

        Guilty
(Accept Alternative Hypothesis)
        Error
(**Type I Error**)
        Ok

In hypothesis testing, as in jury trials, we want to minimize the "conviction of an innocent person", or the incorrect rejection of the null hypothesis (the "Type I Error").  There is an asymmetry in that lowering the chances of a Type I error magnifies the chances of a Type II error occurring.

**Note: The significance level $\alpha$ is the (highest) probability of a Type I Error occurring.**

```python
#  Cardboard box samples
samples=[0.1803, 0.2160, 0.1622, 0.2277, 0.2253, 0.1809, 0.1765, 0.1861, 0.1814, 0.1698,
         0.1853, 0.2086, 0.1839, 0.1783, 0.1814, 0.1565, 0.2127, 0.1811, 0.1718, 0.2089,
         0.2067, 0.1614, 0.1690, 0.1812, 0.2172, 0.1555, 0.1623, 0.1887, 0.2069, 0.1676,
         0.1845, 0.1859, 0.1917, 0.2170, 0.1943, 0.1813, 0.2017, 0.2097, 0.1737, 0.2076]

print(len(samples))
print(np.mean(samples))
```

    40
    0.188465

Since we only have the samples in hand, with no before-hand knowledge about the variance
of the distribution, we perform a one-sample, one-sided t-test.

In this case we compute $(\bar{X}-\mu)/(s/\sqrt{n})$, where $\bar{X}$ is the sample mean, $s$ is the sample standard deviation, $n$ is the number of samples, and $\mu$ is the (assumed) mean of the distribution (from the null hypothesis).  We compare this to the Student's $t$-distribution with $n-1$ degrees of freedom.

```python
alpha = 0.05

#  Perform the t-test.
(statistic, p_value) = stats.ttest_1samp(samples, 0.20)
```

```python
#  Note that, by default, Python performs a two-sided t-test.
#  To get the one-sided test we want, we reject the null hypothessis if and only if
#  the test statistic is negative (based on the alternative hypothesis),
#  and we have p_value/2 < alpha.

print(statistic)
print(p_value/2 < alpha)
```

    -3.72303812077
    True

So in this case we reject the null hypothesis in favor of the alternative hypothesis ($\mu < 0.20$).

Alternatively, we can find the "critical value", where if the test statistic is less than this value, we reject the null hypothesis.  Since this is a one-sided test, we want the value of the test statistic where 5% of the cdf is below this value.

```python
t_var = stats.t(len(samples)-1)
t_critical = t_var.ppf(0.05)  #  "Invert" the cdf to find the critical value.

print(t_critical)
print(statistic < t_critical)
```

    -1.6848751195
    True

Since the value of the test statistic is less than this critical value, we reject the null hypothesis.

## Bayes' Theorem  (Introduction)

Bayes' Theorem (or Bayes' Rule) is used to describe the probability of an event, given other knowledge related to that event. It is a way to update our (probabilistic) beliefs as new information is revealed.

As a simple example, consider the case of a mother who has two children.
Assuming that it is equally likely that each child is a girl or a boy, consider these questions:

 - What is the probability that both of the mother's children are girls?
 - If I tell you that at least one of the children is a girl, what is the probability that both children are girls?
 - If I tell you that the eldest child is a girl, what is the probability that both children are girls?

These answers are, respectively, $\frac{1}{4}$, $\frac{1}{3}$, and $\frac{1}{2}$, so additional information about the mother's children changes our assessment of these probabilities.  (See [answers](#mother) at the end of this lesson if you are unfamiliar with this solution.)

### Bayesian inference
Bayesian inference is the use of Bayes' Theorem to update the probability of a hypothesis as more evidence becomes available.  Also used to infer values of parameters of probability distributions from observations.

### Applications of Bayes' Theorem and Bayesian Inference
1. Medical testing
2. Spam (email) detection
3. Weather prediction
4. Estimating parameters for probability distributions
5. Cracking the Enigma Code
6. The famous ``Monty Hall'' problem  (Do you win the car or the goat?)

#### An example
Suppose after a horrible week of upset stomachs, little sleep, pain, and weak knees, you go to the doctor.  After running a variety of tests, the doctor tells you that you have a life-threatening disease?  Should you be worried?

You need more information in order to decide if you should be (very) concerned or not, so asking the doctor, she tells you the following pieces of information about this disease and the testing method:
 - For people who have the disease, the test will correctly detect this 99% of the time.  (The _sensitivity_ is 99%.)
 - For people without the disease, the test will correctly conclude this 99% of the time.  (The _specificity_ is 99%.)
 - It is believed (estimated) that 0.1% of people have this disease.

Letting D denote the event that you have the disease, and P denoting the event that you test positive for having the disease, we are trying to determine this quantity:

$Pr(D \mid P)$ = the probability that you have the disease, given that you test positive for it

Before getting to the statement of Bayes' Theorem, let's solve this problem "by hand".

 - In population of 1000 people, we expect there will be one person with this disease, i.e. $1000 \times 0.001 = 1$.

 - For this one person with the disease, the test will be correct with probability $1\times 0.99 \approx 1$, so the test will correctly identify this person.

 - Out of the 999 people without the disease, the test will incorrectly test positive for (about) $999\times 0.01 \approx 10$ of them.

Thus, there are eleven people that test positive, but only one of them has the disease, so the chances that you have
the disease, given that you test positive is about $1/11 \approx 9\%$.

**Conclusion:**  If it was me, I would be worried enough to get tested again (by another doctor, who uses a different lab to process the test).

## Bayes' Theorem
Let A and B denote two events, where $Pr(B) > 0$.  Bayes' Theorem allows us to express $Pr(A\mid B)$ (the probability that A happens, given that B has happened) in terms of other probabilities.

$$Pr(A \mid B) = \frac{Pr(B \mid A)\cdot Pr(A)}{Pr(B)}$$

###  Some terminology and assumptions
$Pr(A \mid B)$, the conditional probability that the event $A$ happens given the evidence that $B$ has occurred, is typically called the _posterior probability_.  The quantity $P(A)$ is usually called the _prior probability_, and $Pr(B\mid A)$ is often called the _likelihood_.

Implicit in the use of Bayes' Theorem are the assumptions that the posterior probability and the likelihood are known quantities (or we have very good estimates).  If we gain additional knowledge that the prior probability or the likelihood has somehow changed, we would use them to update $Pr(A\mid B)$ accordingly.

#### Our example
We want to compute $Pr(D\mid P)$.
To apply Bayes' Theorem we need our other probabilities, $Pr(P \mid D)$, $Pr(D)$, and $Pr(P)$.

```python
# The likelihood  Pr(P | D)
p_Pos_given_Disease = 0.99

#  The prior probability  Pr(D)
p_Disease = 0.001
```

Finding $Pr(P)$ is the slightly tricky one here.  This uses the so-called "Law of Total Probability".

```python
p_Pos = 0.001*0.99 + 0.999*0.01
```

Then we apply Bayes' Theorem.

```python
p_Disease_given_Pos = p_Pos_given_Disease * p_Disease / p_Pos

print(p_Disease_given_Pos)
```

    0.09016393442622951

### The second test
Suppose having tested positive for the disease, you go for a second test, which is positive again.

What's your estimation now of the probability you have the disease?  (See [end](#second-test-answer) for answer.)

### Bayesian inference

Bayes' Theorem can be used to estimate parameters of probability distributions, given an assumption about the underlying distribution.

For example, coin flipping can be represented by a Bernoulli random variable with a (possibly unknown) parameter $P$, which equals the probability of obtaining a "heads" in one flip.  Our goal might be to estimate the parameter $P$ based on observed evidence of coin flips.

Recall that repeated Bernoulli trials corresponds to the binomial distribution, giving the probability of $k$ "successes" in $n$ trials, where success happens with probability $P$.  The binomial probability mass function has this expression:
$$Pr(k;n,P) = {n \choose k}P^k(1-P)^{n-k}.$$

As stated, we want to use Bayes' Theorem to help us estimate $P$.  Bayes' Theorem tells us that

$$Pr(P=p\mid heads) \propto Pr(heads \mid P=p)\cdot Pr(P=p)$$

where "$\propto$" means "proportional to".  We know that $Pr(heads \mid P=p) = p$, so if we know (or could "guess") a possible distribution for the random variable $P$, we could update our knowledge about $P$ given new evidence of an observed coin flip.

In this case the "right" probability distribution to try for $P$ is the Beta distribution, denoted $\mathbb{B}(\alpha,\beta)$, which is a distribution on the interval $[0,1]$ with the probability density function

$$p(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)}.$$

Here $B(\alpha,\beta)$ denotes the Beta function, which is the normalizing constant that makes this a probability distribution.

Our application of Bayes' Theorem becomes:

$$
 \begin{align}
 Pr(P=p\mid heads) & \propto\,\, Pr(heads \mid P=p)\cdot Pr(P=p) \\
                   & \propto\,\, p\cdot p^{\alpha-1}(1-p)^{\beta-1} \\
                   & \propto\,\, p^{\alpha}(1-p)^{\beta-1}.
 \end{align}
$$

It isn't too difficult to show that
$$Pr(P=p \mid heads) = \mathbb{B}(\alpha+1,\beta),$$
and similarly that
$$Pr(P=p \mid tails) = \mathbb{B}(\alpha, \beta+1).$$

In other words, if $P$ (the prior) is distributed according to the Beta distribution $\mathbb{B}(\alpha,\beta)$, then the _new_ distribution of $P$ (the posterior) is a Beta distribution that depends upon observing a "heads" or "tails" of the new coin flip, either $\mathbb{B}(\alpha+1,\beta)$ or $\mathbb{B}(\alpha,\beta+1)$, respectively.

**Terminology:** With the assumption that the prior of $P$ is the Beta distribution, and the posterior for $P$ is another Beta distribution (with different parameters) we say that the Beta distribution is a "conjugate prior" for Bernoulli trials.

From the properties of the Beta distribution, we know that $\mathbb{E}(\mathbb{B}(\alpha,\beta))=\frac{\alpha}{\alpha+\beta}$.
Therefore, we have that our prior probability is $P=Pr(heads) = \frac{\alpha}{\alpha+\beta}$.  Observations make us readjust this belief about $P$ for every coin flip we observe.

If we observe a "heads", we want to adjust $P$ upwards, and observing "tails" makes us adjust $P$ downwards.  We do this by changing the parameters of the Beta distribution as noted above.

As the number of observations increases, our estimate for P should get better.

```python
P = 0.8

alpha = 1
beta = 1
l = []
dists = []

for k in range(101):
    estP = alpha/(alpha+beta)
    if k % 10 == 0:
        l.append([alpha, beta, round(estP,4)])
        dists.append(stats.beta(a=alpha,b=beta))
    r = random.random()
    if r <= P:
        alpha = alpha+1
    else:
        beta = beta+1

print(l)
print(round(alpha/(alpha+beta),4))
```

    [[1, 1, 0.5], [10, 2, 0.8333], [18, 4, 0.8182], [26, 6, 0.8125], [36, 6, 0.8571], [45, 7, 0.8654], [53, 9, 0.8548], [61, 11, 0.8472], [70, 12, 0.8537], [77, 15, 0.837], [84, 18, 0.8235]]
    0.8155

```python
plt.figure()
for k, d in enumerate(dists[0::len(dists)//5]):
    ax=plt.subplot(2,3,k+1)
    xPoints = np.arange(0,1,1/200)
    ax.plot(xPoints, d.pdf(xPoints))
    ax.set_title("Beta(a={a}, b={b})".format(**d.kwds))
plt.tight_layout()
```

####  Estimation the probability of "heads"
Suppose we are examining another coin.  What's the estimate for p, the probability of a "heads" for this coin, when we are given this list of 150 coin tosses?  (See answer [below](#coin-example-answer).)

```python
ht = ['H','T','H','H','T','T','T','H','H','T','T','T','H','T','T',
      'T','T','H','H','H','T','H','T','T','T','T','H','H','H','T',
      'T','T','H','H','T','H','T','T','H','H','H','H','T','T','H',
      'H','H','T','T','H','H','H','T','T','T','T','T','H','T','H',
      'H','T','H','T','T','T','T','T','H','T','T','T','H','T','T',
      'H','T','T','H','T','T','H','T','T','T','T','H','T','T','H',
      'T','H','H','T','H','T','T','T','H','T','T','T','T','H','H',
      'T','T','T','H','H','T','T','T','H','H','T','H','T','H','T',
      'T','T','H','T','H','T','T','T','T','T','H','T','T','T','T',
      'H','H','T','T','T','H','H','T','H','T','T','H','T','T','H']
```

### Answers to qustions asked

#### A mother's children (solution)

There are four possibilities for the mother's two children, GG, GB, BG, and BB (given in birth order).  Assuming that B and G are equally likely, then these four possibilities are also equally likely.  Hence, with no information, we see the probability that the mother has two girls is $1/4$.

If we are told that at least one child is a girl, this eliminates the BB option, leaving three equally likely options, so the probability that she has two girls is $1/3$.

Finally, if we are told that the eldest child is a girl, this leaves only the two choices $GG$ and $GB$, each equally likely, so the probability she has two girls is $1/2$.

[[Back to Introduction to Bayes' Thm](#intro)]

#### The second test (solution)
Suppose having tested positive for the disease, you go for a second test, which is positive again.

What's your estimation now of the probability you have the disease?

The first test (effectively) alters the prior probability $Pr(D)$, from 0.1% to (about) 9%.  This isn't quite accurate, but is close enough for a good estimate.

```python
# New prior
p_Disease = 0.09

# New denominator, since the prior has been updated (given the positive test)
p_Pos = 0.09*0.99+0.91*0.01

p_Disease_given_Pos = p_Pos_given_Disease * p_Disease / p_Pos
print(p_Disease_given_Pos)
```

    0.9073319755600815

So now you're about 91% positive that you have the disease.

[[Back to "The second test"](#second-test)]

#### Estimating the probability of "heads" (solution)

```python
ht = ['H','T','H','H','T','T','T','H','H','T','T','T','H','T','T',
      'T','T','H','H','H','T','H','T','T','T','T','H','H','H','T',
      'T','T','H','H','T','H','T','T','H','H','H','H','T','T','H',
      'H','H','T','T','H','H','H','T','T','T','T','T','H','T','H',
      'H','T','H','T','T','T','T','T','H','T','T','T','H','T','T',
      'H','T','T','H','T','T','H','T','T','T','T','H','T','T','H',
      'T','H','H','T','H','T','T','T','H','T','T','T','T','H','H',
      'T','T','T','H','H','T','T','T','H','H','T','H','T','H','T',
      'T','T','H','T','H','T','T','T','T','T','H','T','T','T','T',
      'H','H','T','T','T','H','H','T','H','T','T','H','T','T','H']
```

As before, we use the Beta distribution as a prior, and update this distribution for each "heads" or "tails" we see.  In this case, we start with $\alpha = \beta = 1$ and increase the appropriate variable for "heads" or "tails".

```python
alpha = 1 + sum(1 for x in ht if x == "H")
beta = 1 + sum(1 for x in ht if x == "T")

print(alpha, beta)
print("Estimate for p: ", alpha/(alpha+beta))
```

    60 92
    Estimate for p:  0.39473684210526316


```python
import matplotlib.pyplot as plt
import numpy as np
import sympy
```

# Math Magic: Final Week Review

# Review: A bit of modern math philosophy

### What is the most elementary, simple, basic, primitive kind of math operation?

### What do you call it when you do a bunch of ^that kind of operation in a row?

### What do you call it when you do a bunch of ^that kind of operation in a row?

### Believe it or not, you can keep going down that road. If you do a bunch of ^that in a row, it is called `tetration`!

### A `hyperoperation`
- The name for any `normal` operation (like the ones above).
- This recursive repeated arithmetic operation can be described by:

# Anyway, that's not even review. Back to the review!

# 1. What is the sine of theta? $sin(\theta) = ?$
# 2. What is the cosine of theta? $cos(\theta) = ?$
# 3. What is the tangent of theta? $tan(\theta) = ?$

# 4. What is the arcsine (inverse sine) of y / r ? $arcsin(\frac{y}{r}) = ?$
# 5. What is the arccosine (inverse cosine) of x / r ? $arccos(\frac{x}{r}) = ?$
# 6. What is the arctangent (inverse tangent) of y / x? $arctan(\frac{y}{x}) = ?$

# 1. What the heck is this demonic drawing above?
# 2. How many $\pi$ radians is 0 degrees?
# 3. How many $\pi$ radians is 90 degrees?
# 4. How many $\pi$ radians is 270 degrees?

# 5. Calculate in python the conversion between any *degrees* angle to *radians*

```python
# YOUR CODE HERE! Try converting degrees to radians

degrees = 42.42  # convert me to radians!

degrees * (2*np.pi/360)

# HINT: use np.pi, it is a highly accurate pi
```

    0.7403686686959946

# Cowcooloose review

```
                                       /;    ;\
                                   __  \____//
                                  /{_\_/   `'\____
                                  \___ ---(=)--(=)--}
       _____________________________/          :--'
   ,-,'`@@@@@@@@       @@@@@@         \_    `__\
  ;:(  @@@@@@@@@        @@@             \___(o'o)
  :: )  @@@@          @@@@@@        ,'@@(  `===='
  :: : @@@@@:          @@@@         `@@@:
  :: \  @@@@@:       @@@@@@@)    (  '@@@'
  ;; /\      /`,    @@@@@@@@@\\   :@@@@@)
  ::/  )    {_----------------:  :~`, ;
 ;;'`; :   )                  :  / `; ;
;;;; : :   ;                  :  ;  ; :
`'`' / :  :                   :  :  : :
    )_ \__;      ";"          :_ ;  \_\       `,','
    :__\  \    * `,'*         \  \  :  \   *  8`;'*  *
        `^'     \ :/           `^'  `-^-'   \v/ :  \/   -Bill Ames-
```

### 1. What do we mean by the phrase "rate of change"?
### 2. What do we call the "rate of change" of position?
### 3. What do we call the "rate of change" of velocity/speed?
### 4. What 1-word definition do we use for the phrase "rate of change"?

### 5. What does this thing represent?
## $\frac{df(x)}{dx} = f'(x)$

# Recall that...
## $x(t) = position$
## $\frac{dx(t)}{dt}  = x'(t) = velocity$
## $\frac{dx^2(t)}{dt}  = x''(t) = acceleration$

### 6. The function $f(x) = e^x$ is special for what reason?

# GUESS WHAT

# I THINK

#  WE WILL LEAVE IT AT THAT

# Thank you for a very fun semester!
- I hope you learned something useful!

```python

```


```python
import open3d as o3d
import numpy as np
import os
import sys

# monkey patches visualization and provides helpers to load geometries
sys.path.append('..')
import open3d_tutorial as o3dtut
# change to True if you want to interact with the visualization windows
o3dtut.interactive = not "CI" in os.environ
```

# Mesh deformation
If we want to deform a triangle mesh according a small number of constraints, we can use mesh deformation algorithms. Open3D implements the as-rigid-as-possible method by [\[SorkineAndAlexa2007\]](../reference.html#SorkineAndAlexa2007) that optimizes the following energy functional

\begin{equation}
\sum_i \sum_{j \in \mathcal{N}(i)} w_{ij} || (\mathbf{p}'_i - \mathbf{p}'_j) - \mathbf{R}_i (\mathbf{p}_i - \mathbf{p}_j)||^2 \,,
\end{equation}

where $\mathbf{R}_i$ are rotation matrices that we want to optimize for, and $\mathbf{p}'_i$ and $\mathbf{p}_i$ are the vertex positions after and before the optimization, respectively. $\mathcal{N}(i)$ is the set of neighbours of vertex $i$. The weights $w_{ij}$ are cot weights.

Open3D implementes this method in `deform_as_rigid_as_possible`. The first argument to this method is a set of `constraint_ids` that refer to the vertices in the triangle mesh. The second argument `constrint_pos` defines at which position those vertices should be after the optimization. The optimization process is an iterative scheme. Hence, we also can define the number of iterations via `max_iter`.

```python
mesh = o3dtut.get_armadillo_mesh()

vertices = np.asarray(mesh.vertices)
static_ids = [idx for idx in np.where(vertices[:, 1] < -30)[0]]
static_pos = []
for id in static_ids:
    static_pos.append(vertices[id])
handle_ids = [2490]
handle_pos = [vertices[2490] + np.array((-40, -40, -40))]
constraint_ids = o3d.utility.IntVector(static_ids + handle_ids)
constraint_pos = o3d.utility.Vector3dVector(static_pos + handle_pos)

with o3d.utility.VerbosityContextManager(o3d.utility.VerbosityLevel.Debug) as cm:
    mesh_prime = mesh.deform_as_rigid_as_possible(
            constraint_ids, constraint_pos, max_iter=50)
```

```python
print('Original Mesh')
R = mesh.get_rotation_matrix_from_xyz((0,np.pi,0))
o3d.visualization.draw_geometries([mesh.rotate(R, center=mesh.get_center())])
print('Deformed Mesh')
mesh_prime.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh_prime.rotate(R, center=mesh_prime.get_center())])
```

## Smoothed ARAP
Open3D also implements a smoothed version of the ARAP objective defined as

\begin{equation}
\sum_i \sum_{j \in \mathcal{N}(i)} w_{ij} || (\mathbf{p}'_i - \mathbf{p}'_j) - \mathbf{R}_i (\mathbf{p}_i - \mathbf{p}_j)||^2 + \alpha A ||\mathbf{R}_i - \mathbf{R}_j||^2\,,
\end{equation}

that penalizes a deviation of neighbouring rotation matrices. $\alpha$ is a trade-off parameter for the regularization term and $A$ is the surface area.

The smoothed objective can be used in `deform_as_rigid_as_possible` by using the argument `energy` with the parameter `Smoothed`.


# R36 DERIVATION

**We know the following from our geometry:**
```
1. R0u = R06 * R6g * Rgu
2. R03 = R03 * R36
3. R6g = I
4. R0u = rotz(yaw) * roty(pitch) * rotx(roll)
5. Rgu = rotz(pi) * roty(-pi/2)
```

**For any matrix recall the following properties:**
```
1. R = I * R = R * I
2. (A * B) * C = A * (B * C)
3. if A = B then A * C = B * C and vice versa
4. if A = B then C * A = C * C and vice versa
5. R * R.inverse() = R.inverse() * R = I
```

**Note that for rotation matrices**
```
R.inverse() = R.T
Therefore: R.T * R = R * R.T = I
```

**From the discussion above, you can do the following:**
- _Subsitution_
- - `R0u = (R03 * R36) * I * Rgu`
- _Multiply both sides by `Rgu.inverse() = Rgu.T` at the right_
- - `R0u * Rgu.T = R03 * R36 `
- _Multiply both sides by `R03.inverse() = R03.T` at the left_
- - `R03.T * R0u * Rgu.T = R36 `

**We conclude then that:**
```
R36 = R03.T * R0u * Rgu.T
R36 = R03.T * R0g
where:
R0u = rotz(yaw) * roty(pitch) * rotx(roll)
Rgu = rotz(pi) * roty(-pi/2)
```
`

```python
from sympy import symbols, cos, sin, pi, simplify, trigsimp, expand_trig, pprint, sqrt, atan2
from sympy.matrices import Matrix

def rotx(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [1., 0., 0.],
    [0., cq,-sq],
    [0., sq, cq]
  ])

  return r

def roty(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [ cq, 0., sq],
    [ 0., 1., 0.],
    [-sq, 0., cq]
  ])

  return r

def rotz(q):

  sq, cq = sin(q), cos(q)

  r = Matrix([
    [cq,-sq, 0.],
    [sq, cq, 0.],
    [0., 0., 1.]
  ])

  return r
```

```python
q1, q2, q3, q4, q5, q6= symbols('q1:7')

R03 = Matrix([
[sin(q2 + q3)*cos(q1), cos(q1)*cos(q2 + q3), -sin(q1)],
[sin(q1)*sin(q2 + q3), sin(q1)*cos(q2 + q3),  cos(q1)],
[        cos(q2 + q3),        -sin(q2 + q3),        0]])

R03T = R03.T
pprint(R03T)
```

    ⎡sin(q₂ + q₃)⋅cos(q₁)  sin(q₁)⋅sin(q₂ + q₃)  cos(q₂ + q₃) ⎤
    ⎢                                                         ⎥
    ⎢cos(q₁)⋅cos(q₂ + q₃)  sin(q₁)⋅cos(q₂ + q₃)  -sin(q₂ + q₃)⎥
    ⎢                                                         ⎥
    ⎣      -sin(q₁)              cos(q₁)               0      ⎦

```python
R36 = Matrix([[-sin(q4)*sin(q6) + cos(q4)*cos(q5)*cos(q6), -sin(q4)*cos(q6) - sin(q6)*cos(q4)*cos(q5), -sin(q5)*cos(q4)],
  [                           sin(q5)*cos(q6),                           -sin(q5)*sin(q6),          cos(q5)],
  [-sin(q4)*cos(q5)*cos(q6) - sin(q6)*cos(q4),  sin(q4)*sin(q6)*cos(q5) - cos(q4)*cos(q6),  sin(q4)*sin(q5)]])

```

```python
alpha, beta, gamma = symbols('alpha beta gamma', real = True)
R0u = rotz(alpha) * roty(beta) * rotx(gamma)
pprint(R0u)
```

    ⎡1.0⋅cos(α)⋅cos(β)  -1.0⋅sin(α)⋅cos(γ) + sin(β)⋅sin(γ)⋅cos(α)  1.0⋅sin(α)⋅sin(
    ⎢
    ⎢1.0⋅sin(α)⋅cos(β)  sin(α)⋅sin(β)⋅sin(γ) + 1.0⋅cos(α)⋅cos(γ)   sin(α)⋅sin(β)⋅c
    ⎢
    ⎣   -1.0⋅sin(β)                 1.0⋅sin(γ)⋅cos(β)                         1.0⋅

    γ) + sin(β)⋅cos(α)⋅cos(γ)⎤
                             ⎥
    os(γ) - 1.0⋅sin(γ)⋅cos(α)⎥
                             ⎥
    cos(β)⋅cos(γ)            ⎦

```python
RugT = (rotz(pi) * roty(-pi/2)).T
pprint(RugT)
```

    ⎡0   0    1.0⎤
    ⎢            ⎥
    ⎢0  -1.0   0 ⎥
    ⎢            ⎥
    ⎣1   0     0 ⎦

```python
R36 = R03T * R0u * RugT
pprint(R36)
```

    ⎡(1.0⋅sin(α)⋅sin(γ) + sin(β)⋅cos(α)⋅cos(γ))⋅sin(q₂ + q₃)⋅cos(q₁) + (sin(α)⋅sin
    ⎢
    ⎢(1.0⋅sin(α)⋅sin(γ) + sin(β)⋅cos(α)⋅cos(γ))⋅cos(q₁)⋅cos(q₂ + q₃) + (sin(α)⋅sin
    ⎢
    ⎣                             -(1.0⋅sin(α)⋅sin(γ) + sin(β)⋅cos(α)⋅cos(γ))⋅sin(

    (β)⋅cos(γ) - 1.0⋅sin(γ)⋅cos(α))⋅sin(q₁)⋅sin(q₂ + q₃) + 1.0⋅cos(β)⋅cos(γ)⋅cos(q

    (β)⋅cos(γ) - 1.0⋅sin(γ)⋅cos(α))⋅sin(q₁)⋅cos(q₂ + q₃) - 1.0⋅sin(q₂ + q₃)⋅cos(β)

    q₁) + (sin(α)⋅sin(β)⋅cos(γ) - 1.0⋅sin(γ)⋅cos(α))⋅cos(q₁)

    ₂ + q₃)  -1.0⋅(-1.0⋅sin(α)⋅cos(γ) + sin(β)⋅sin(γ)⋅cos(α))⋅sin(q₂ + q₃)⋅cos(q₁)

    ⋅cos(γ)  -1.0⋅(-1.0⋅sin(α)⋅cos(γ) + sin(β)⋅sin(γ)⋅cos(α))⋅cos(q₁)⋅cos(q₂ + q₃)

                                           1.0⋅(-1.0⋅sin(α)⋅cos(γ) + sin(β)⋅sin(γ)

     - 1.0⋅(sin(α)⋅sin(β)⋅sin(γ) + 1.0⋅cos(α)⋅cos(γ))⋅sin(q₁)⋅sin(q₂ + q₃) - 1.0⋅s

     - 1.0⋅(sin(α)⋅sin(β)⋅sin(γ) + 1.0⋅cos(α)⋅cos(γ))⋅sin(q₁)⋅cos(q₂ + q₃) + 1.0⋅s

    ⋅cos(α))⋅sin(q₁) - 1.0⋅(sin(α)⋅sin(β)⋅sin(γ) + 1.0⋅cos(α)⋅cos(γ))⋅cos(q₁)

    in(γ)⋅cos(β)⋅cos(q₂ + q₃)  1.0⋅sin(α)⋅sin(q₁)⋅sin(q₂ + q₃)⋅cos(β) - 1.0⋅sin(β)

    in(γ)⋅sin(q₂ + q₃)⋅cos(β)  1.0⋅sin(α)⋅sin(q₁)⋅cos(β)⋅cos(q₂ + q₃) + 1.0⋅sin(β)

                                                         1.0⋅sin(α)⋅cos(β)⋅cos(q₁)

    ⋅cos(q₂ + q₃) + 1.0⋅sin(q₂ + q₃)⋅cos(α)⋅cos(β)⋅cos(q₁)⎤
                                                          ⎥
    ⋅sin(q₂ + q₃) + 1.0⋅cos(α)⋅cos(β)⋅cos(q₁)⋅cos(q₂ + q₃)⎥
                                                          ⎥
     - 1.0⋅sin(q₁)⋅cos(α)⋅cos(β)                          ⎦

```python
q1, q2, q3, q4, q5, q6= symbols('q1:7')

R03 = Matrix([
[sin(q2 + q3)*cos(q1), cos(q1)*cos(q2 + q3), -sin(q1)],
[sin(q1)*sin(q2 + q3), sin(q1)*cos(q2 + q3),  cos(q1)],
[        cos(q2 + q3),        -sin(q2 + q3),        0]])

R03T = R03.T

alpha, beta, gamma = symbols('alpha beta gamma')
R0u = rotz(alpha) * roty(beta) * rotx(gamma)

RugT = (rotz(pi) * roty(-pi/2)).T
R36 = R03T * R0u * RugT

roll, pitch, yaw = 0.366, -0.078, 2.561

variables = {
  q1: 1.01249,
  q2: -0.2758,
  q3: -0.11568,
  alpha: yaw,
  beta: pitch,
  gamma: roll
}

R0g = R0u * Rug
R36eval = R36.evalf(subs = variables)

pprint(R36eval)

print(get_spherical_ik(R36eval))
```

    ⎡ 0.724598074610823   -0.686235503719083   0.0635489079820028⎤
    ⎢                                                            ⎥
    ⎢ 0.684428504026748    0.727345057020894   0.0502671950976013⎥
    ⎢                                                            ⎥
    ⎣-0.0807171180481332  0.00707117123884714  0.996711967115533 ⎦
    (0.756897144385541, 0.0808050247662215, 0.00709437915922838)

```python
print(simplify(R03I) == R03.T)
print(Rug == Rug.T)
```

    True
    True

```python
R03 = Matrix([
[sin(q2 + q3)*cos(q1), cos(q1)*cos(q2 + q3), -sin(q1)],
[sin(q1)*sin(q2 + q3), sin(q1)*cos(q2 + q3),  cos(q1)],
[        cos(q2 + q3),        -sin(q2 + q3),        0]])

roll, pitch, yaw = 0.366, -0.078, 2.561
R0u = rotz(yaw)* roty(pitch) * rotx(roll)
Rug = (rotz(pi) * roty(-pi/2)).T

R0u.evalf(subs = {alpha: yaw, beta: pitch, gamma: roll})
R36eval2 = R36.evalf(subs = {q1: 1.01249, q2: -0.2758, q3: -0.11568})
pprint(R36eval2)
```

    ⎡ 0.724598074610823   -0.686235503719083   0.0635489079820028⎤
    ⎢                                                            ⎥
    ⎢ 0.684428504026748    0.727345057020894   0.0502671950976012⎥
    ⎢                                                            ⎥
    ⎣-0.0807171180481332  0.00707117123884723  0.996711967115533 ⎦

```python

def ik(R):
  r12, r13 = R[0,1], R[0,2]
  r21, r22, r23 = R[1,0], R[1,1], R[1,2]
  r32, r33 = R[2,1], R[2,2]
  q5 = atan2(sqrt(r13**2 + r33**2), r23)
  q4 = atan2(r33, -r13)
  q6 = atan2(-r22, r21)
  return q4.evalf(), q5.evalf(), q6.evalf()

print(ik(R36eval))

print(R36eval)
```

    (1.63446868902482, 1.52050793847535, -0.815787839019348)
    Matrix([[0.724598074610823, -0.686235503719083, 0.0635489079820028], [0.684428504026748, 0.727345057020894, 0.0502671950976013], [-0.0807171180481332, 0.00707117123884714, 0.996711967115533]])

```python
R0g = rotz(yaw) * roty(pitch) * rotx(roll) * Rug
print(R0g)
```

    Matrix([[0.257143295038827, 0.488872082559650, -0.833595473062544], [0.259329420712765, 0.796053601157403, 0.546851822377060], [0.930927267496960, -0.356795110642117, 0.0779209320563015]])

```python
R03T_eval =R03T.evalf(subs = { q1: 1.01249, q2: -0.2758, q3: -0.11568})
pprint(R03T_eval)
pprint(R03T)
```

    ⎡-0.202129925357168  -0.323618808899201  0.924345368248129⎤
    ⎢                                                         ⎥
    ⎢0.489672387232293   0.783986806638987   0.381556863649747⎥
    ⎢                                                         ⎥
    ⎣-0.848153551226034  0.529750463466212           0        ⎦
    ⎡sin(q₂ + q₃)⋅cos(q₁)  sin(q₁)⋅sin(q₂ + q₃)  cos(q₂ + q₃) ⎤
    ⎢                                                         ⎥
    ⎢cos(q₁)⋅cos(q₂ + q₃)  sin(q₁)⋅cos(q₂ + q₃)  -sin(q₂ + q₃)⎥
    ⎢                                                         ⎥
    ⎣      -sin(q₁)              cos(q₁)               0      ⎦

```python

```


# YOUR PROJECT TITLE

> **Note the following:**
> 1. This is *not* meant to be an example of an actual **model analysis project**, just an example of how to structure such a project.
> 1. Remember the general advice on structuring and commenting your code from [lecture 5](https://numeconcopenhagen.netlify.com/lectures/Workflow_and_debugging).
> 1. Remember this [guide](https://www.markdownguide.org/basic-syntax/) on markdown and (a bit of) latex.
> 1. Turn on automatic numbering by clicking on the small icon on top of the table of contents in the left sidebar.
> 1. The `modelproject.py` file includes a function which could be used multiple times in this notebook.

Imports and set magics:

```python
import numpy as np
from scipy import optimize
import sympy as sm

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2

# local modules
import modelproject
```

# OverLapping Generation (OLG)Model

Consider an economy where individuals live for two periods, and the population grows at a constant rate $n>0$. Identical competitive firms maximize their profits employing a Cobb-Douglas technology that combines labor, $L_{t}$, and capital, $K_{t}$, so that $Y_{t}=A K_{t}^{\alpha} L_{t}^{1-\alpha}$, with $\alpha \in(0,1)$. Assume full capital depreciation (i.e., $\delta=1$ ). Under these assumptions, profit maximization leads to:
$$
\begin{aligned}
r_{t} &=\alpha A k_{t}^{\alpha-1}, \\
w_{t} &=(1-\alpha) A k_{t}^{\alpha},
\end{aligned}
$$
where $r_{t}$ is the (net) rental rate of capital, $w_{t}$ is the wage rate, and $k_{t}$ denotes capital in per-worker units.
Utility for young individuals born in period $t$ is
$$
U_{t}=\ln c_{1 t}+\frac{1}{1+\rho} \ln c_{2 t+1},
$$
with $\rho>-1 . c_{1 t}$ denotes consumption when young, $c_{2 t+1}$ consumption when old. Young agents spend their entire time endowment, which is normalized to one, working. Suppose the government runs an unfunded (pay-as-you-go) social security system, according to which the young pay a contribution $d_{t}$ that

amounts to a fraction $\tau \in(0,1)$ of their wages. Thus, the contributions are paid out in the same period to the current old. The latter do not work, and sustain their consumption through their savings and the social security benefits. Thus, the budget constraints in each period of life read as:
$$
\begin{aligned}
c_{1 t}+s_{t} &=(1-\tau) w_{t} \\
c_{2 t+1} &=s_{t} R_{t+1}+(1+n) d_{t+1}
\end{aligned}
$$
where $R_{t+1}=r_{t+1}$ under $\delta=1$.
a Set up and solve the individual's problem of optimal intertemporal allocation of resources. Derive the Euler equation. Show that individual saving behavior is characterized by
$$
s_{t}=\frac{1}{2+\rho}(1-\tau) w_{t}-\tau \frac{1+\rho}{2+\rho} \frac{1+n}{1+r_{t+1}} w_{t+1}
$$
b Show that the capital accumulation equation that gives $k_{t+1}$, as a function of $k_{t}$, is given by
$$
k_{t+1}=\frac{1}{1+\frac{1+\rho}{2+\rho} \frac{(1-\alpha)}{\alpha} \tau}\left[\frac{(1-\alpha)(1-\tau)}{(1+n)(2+\rho)} A k_{t}^{\alpha}\right]
$$

Show also that, in the steady state, the amount of capital-per-worker is
$$
\bar{k}=\left[\frac{1}{1+\frac{1+\rho}{2+\rho} \frac{(1-\alpha)}{\alpha} \tau} \frac{(1-\alpha)(1-\tau) A}{(1+n)(2+\rho)}\right]^{\frac{1}{1-\alpha}} .
$$
c Suppose that, at time $T$, before saving decisions are made, the government decides to switch to a fully funded social security system according to which the young pay a contribution $d_{T}$ that amounts to a fraction $\tau \in$ $(0,1)$ of their wages. These contributions are then paid out in the next period, together with the accrued interest rate. The budget constraints in each period of life now read as:
$$
\begin{aligned}
c_{1 t}+s_{t} &=(1-\tau) w_{t}, \\

c_{2 t+1} &=\left(s_{t}+\tau w_{t}\right) R_{t+1}, \quad \text { for } t \geq T
\end{aligned}
$$
Show that the new steady-state capital-per-worker, which is denoted by $\bar{k}^{\prime}$, is such that
$$
\bar{k}^{\prime}=\left[\frac{(1-\alpha) A}{(1+n)(2+\rho)}\right]^{\frac{1}{1-\alpha}} .
$$

d In the absence of any compensation from the government, the old generation at time $T$ is worse off, after the social security system is changed. Explain why. How could the government intervene to compensate them, without imposing any burden on the current generation of workers?

# OLG model different pension schemes

## Model description

### Unfunded social security system

**Time:** Discrete and indexed by $t\in\{0,1,\dots\}$.

**Demographics:** Population grows at a constant rate $n>0$ and a life consists of
two periods; *young* and *old*.

**Households:** Utility for young individuals born in period $t$ is
$$
\begin{aligned}
 & U_{t}=\ln c_{1 t}+\frac{1}{1+\rho} \ln c_{2 t+1} \\
 & \text{s.t.}\\
 & S_{t}=s_{t}(1-\tau_{w})w_{t}\\
 & C_{1t}=(1-s_{t})(1-\tau_{w})w_{t}\\
 & C_{2 t+1}=s_{t} r_{t+1}+(1+n) d_{t+1}
\end{aligned}
$$
with $\rho>-1 . c_{1 t}$ denotes consumption when young, $c_{2 t+1}$ consumption when old, where the agents do not work.  Young agents spend their entire time endowment, which is normalized to one, working. Suppose the government runs an unfunded (pay-as-you-go) social security system, according to which the young pay a contribution $d_{t}$ that amounts to a fraction $\tau \in(0,1)$ of their wages.

**Firms:** Firms rent capital $K_{t-1}$ at the rental rate $r_{t}^{K}$,
and hires labor $E_{t}$ at the wage rate $w_{t}$. Firms have access
to the production function

$$
\begin{aligned}
Y_{t}=F(K_{t-1},E_{t})=A K_{t-1}^{\alpha} E_{t}^{1-\alpha},\,\,\,,\alpha\in(0,1)
\end{aligned}
$$

Profits are

$$
\begin{aligned}
\Pi_{t}=Y_{t}-w_{t}E_{t}-r_{t}^{K}K_{t-1}
\end{aligned}
$$

**Government:** The Government is not directly included in the model, but runs the social security system by paying the contribution $d_t=\tau w_t$ of the young generation's wage to the old generation.

**Capital:** Depreciates with a rate of $\delta \in [0,1]$.

### Fully funded social security system

The young pay a contribution $d_{T}$ that amounts to a fraction $\tau \in$ $(0,1)$ of their wages. hese contributions are then paid out in the next period, together with the accrued interest rate. The budget constraints are now:

$$
\begin{aligned}
c_{1 t}+s_{t} &=(1-\tau) w_{t}, \\
c_{2 t+1} &=\left(s_{t}+\tau w_{t}\right) r_{t+1}, \quad \text { for } t \geq T
\end{aligned}
$$

```python

```

## Analytical solution

If your model allows for an analytical solution, you should provide here.

You may use Sympy for this. Then you can characterize the solution as a function of a parameter of the model.

To characterize the solution, first derive a steady state equation as a function of a parameter using Sympy.solve and then turn it into a python function by Sympy.lambdify. See the lecture notes for details.

## Numerical solution

You can always solve a model numerically.

Define first the set of parameters you need.

Then choose one of the optimization algorithms that we have gone through in the lectures based on what you think is most fitting for your model.

Are there any problems with convergence? Does the model converge for all starting values? Make a lot of testing to figure these things out.

# Further analysis

Make detailed vizualizations of how your model changes with parameter values.

Try to make an extension of the model.

# Conclusion

Add concise conclusion.


# Logistic Regression

- Logistic regression is a classification algorithm, don’t confuse with the name regression.
- Logistic Regression is basically Supervised ML algorithm that is used for __binary classification__
- In classification problem the values we predict take on only a small number of discrete values. For example, "yes or no", "cat or dog", "cat or dog or bird", etc.
- In logistic regression we estimate the probability of a certain discrete value.
- Our main goal is to __predict the probability of the discrete class labels__

Some examples of classification problems:
1. Spam detection
2. Detection of fraudulent transanctions
3. Malignant/Benign tumor

__Assumption of Logistic Regression:__

* Input space can be __separated into two nice ‘regions’, one for each class, by a defined boundary.__

* Your data must be **linearly** separable in __n dimensions__

### Hypothesis for classification problem

In classification problem we want the probability of the outcomes. So we would like f(x) to be between 0 and 1.

\begin{equation}
0
\leq
f_\theta(x)
\leq
1
\end{equation}

* Mathematically, in Linear Regression __hypothesis function $f_{\theta}(x)$ can be >1 or <0__

* For Logistic Regression we need __hypothesis function which can satisfy the math rule:__ $0 \leq f_{\theta}(x)\leq 1$

### Intuitive Example to understand math behind LoR

__Example / Problem statement: __

* We are provided a sample of 100 customers.

* We need to predict the probability whether a customer will buy (y) a particular magazine or not based on their
  age

* As you can see, we’ve a categorical outcome variable, we’ll use logistic regression.

#### Step 1:

To start with logistic regression, I’ll first write the simple linear regression equation with dependent variable

$$ \large f_{\theta}(x) = mx + C ------ (1) $$

#### Step 2:

So, final equation form based on our example will be

$$ \large f(readers\;will\;buy\; magazine\; or\; not) = m * x_{age} + C ------(2)$$

* Now consider Y as function of probability. Why? because we need the chance (probability of the whether buyer will buy the magazine or not)

* This function is established using two things:
  * Probability of Success(p) means reader buys magazine
  * Probability of Failure(1-p) means reader doesn't buy magazine.

* Probability p should meet following criteria:
    1. It must always be positive (since p >= 0)
    2. It must always be less than equals to 1 (since p <= 1)

#### Step 3:

__Since probability must always be positive, we’ll put the linear equation in exponential form.__

For any value of slope and dependent variable, exponent of this equation will never be negative.

$$ \large f_{\theta}(x) = exp(C + mx_{Age}) = e^{(C + mx_{Age})}    ------- (3) $$

Here,

* P  =  Probability of readers' buy magazine or not
* exp OR e =  exponential symbol OR Euler's number.

#### Step 4:

To make the probability less than 1, we must divide p by a number greater than p. This can simply be done by:

$$ \large f_{\theta}(x)  =  \frac{exp(C + mx_{Age})}{exp(C + mx_{Age}) + 1}   =  \frac{ e^{(C + mx_{Age})}}{e^{(C + mx_{Age})} + 1 } = p  ----- (4)$$

#### Step 5:

Now by using equation 1, 2, 3 and 4 we can redefine the probability as:

 $$ p = \frac {e^{g}}{ 1 + e^{g}} ----- (5)$$

 where $ g_{\theta}(x) = mx + C = Decision\_boundary $

 * Here we have just __substitute the C + mx{Age} with Y__

 * Here p is the probability of success. This __Equation (5) is the Logit Function__

```

```

If p is the probability of success, 1-p will be the probability of failure which can be written as:

$$q = 1 - p = 1 - \frac {e^{g}}{ 1 + e^{g}}    --- (6)$$

* Here q is the probability of failure

#### Step 6:

Now let's __divide the equation 5 and 6__ so that we get following equation

$$ \frac {p}{1-p} = e^{g}    --- (7)$$

#### Step 7:

After __taking log on both side__, So that we can get the value of Y

$$ \large log(\frac{p}{1-p}) = g_\theta(x)    --- (8) $$

Now, let's __put the value of Y in equation 8 from equation 2__

$$ \large log(\frac{p}{1-p}) = C + mx_{Age}   --- (9) $$

#### Notes:

* The __equation 9 is used in Logistic Regression and It is called Logit(p). __

* Here __(p/1-p) is called the Odd Ratio (OR) or Odds(p)__

* Whenever the __log of Odd Ratio is found to be positive, then the probability of success is always more than
  50%.__

* You can see __probability never goes below 0 and above 1. __

### Why we take odds instead of probabilities?

Let $P(X)$ denote the probability of an event X occurring. In that case, the __odds ratio (OR(X))__ is defined as

$$ \large \frac{P(X)}{1-P(X)}$$

* It is essentially the ratio of the probability of the event happening, vs. it not happening.

* It is clear that __probability and odds__ convey the __exact same information.__

* __But as $P(X)$ goes from 0 to 1, OR(X) goes from 0 to infinity__

* The advantage is that the __odds defined on 0 to infinity__ which map to __log-odds on -infinity to +infinity__

#### Sigmoid function

We will now see how we arrive at the sigmoid function, starting form the log_odds function:

$log(\frac{y}{1-y}) = g_\theta(x) = \theta^TX$

$y = (1-y)*e^{\theta^TX}$

$y = e^x - y*e^{\theta^TX}$

$y + ye^x = e^{\theta^TX}$

$y*(1 + e^{\theta^TX}) = e^{\theta^TX}$

$y = \frac{e^{\theta^TX}}{1+e^{\theta^TX}}$

$y = \frac{1}{\frac{1}{e^{\theta^TX}} + 1}$

$y = \frac{1}{1 + e^{-\theta^TX}}$

Finally, the Sigmoid function, as derived above, is given below:

$$\sigma(x)=\frac{1}{1+e^{-x}}$$

Sigmoid function is used to restrict the f(x) between 0 and 1.

**Remark:** The derivative of logistic function can be written $\text{d}\sigma = \sigma (1-\sigma)$.

```python
'''Explore the signature of sigmoid function'''
import numpy as np
import matplotlib.pyplot as plt
import math
%matplotlib inline

sigmoid = lambda x: 1/(1 + np.exp(-x))
x = np.linspace(-10,10,100)

plt.plot(x, sigmoid(x), 'o')

# Draw the grid line in background.
plt.grid()

# Title & Subtitle
plt.title('Sigmoid function')

# write the Sigmoid formula
plt.text(4, 0.8, r'$\sigma(x)=\frac{1}{1+e^{-x}}$', fontsize=15)

plt.show()
```

### Why do we need sigmoid function?

* This function will give us the __output value between [0,1]__

* When you __perform derivative of $e^{x}$ it gives you back $e^{x}$ and this function is the only known
  differentiable function with this mathematical property. That makes it more convenient for calculation.__

## Cost functions for Logistic Regression

#### Can we directly use cost function of linear regression which is Sum of sequared Errors?

* The Answer is __No, we can't__

* Linear regression uses mean squared error as its cost function. If this is used for logistic regression, then it
  will be generate a non-convex function of our input intercept values or parameter (theta) values

* Gradient Descent will converge into global minimum for function that is convex.

## Cost function for a value of f(x):

$$ \large \text{Cost function} (f_{\theta}(x))  = \begin{cases}
  -\log(f_{\theta}(x)) & \text{if $y = 1$} \\
  -\log(1\; -\; f_{\theta}(x) ) & \text{if $y = 0$}
\end{cases}$$

$$ \text{Cost function for entire data set with m observation} = J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \text{Cost function} (f_{\theta}(x^{i},y^{i})) $$

### Combine both cases of Cost function in one equation

$$ \large \text{Cost function} (f_{\theta}(x),y)  =
  -y\log(f_{\theta}(x))-(1-y)\log(1\; -\; f_{\theta}(x) )
$$

```python
'''Explore the signature of sigmoid function'''
import numpy as np
import matplotlib.pyplot as plt
import math
%matplotlib inline

sigmoid = lambda x: 1/(1 + np.exp(-x))
x = np.linspace(-10,10,100)

plt.plot(x, [-math.log(x)-math.log(1-x) for x in sigmoid(x)], 'o')

# Draw the grid line in background.
plt.grid()

# Title & Subtitle
plt.title('Loss function for Logistic Regression')

# write the Loss formula
plt.text(-8, 9.5, r'$ -y\log(f_{\theta}(x))-(1-y)\log(1\; -\; f_{\theta}(x) )$', fontsize=15)

plt.show()
```

$$\large J(\theta) = - \frac{1}{m}\sum_{i=1}^{m}[y^{(i)}\log(f_{\theta}(x^{(i)}) + (1-y^{(i)})\log(1-f_{\theta}(x^{(i)}))]$$

__Now we need to minimize this cost function $J(\theta)$__

### Gradient Descent

* In order to minimize the cost function we need to derive its __partial derivative with respect to $\theta$__

$$\large J(\theta) = - \frac{1}{m}\sum_{i=1}^{m}[y^{(i)}\log(\frac{1}{1+e^{-\theta x^{(i)}}}) + (1-y^{(i)})\log(1-\frac{1}{1+e^{-\theta x^{(i)}}})]$$

__Partial derivate that we use for updating our cost function or minimize our error function is given below__

$$\large \frac{\partial J}{\partial \theta}  = \frac{1}{m}\sum_{i=1}^{m}[x^{(i)}_{j}f_{\theta}(x^{(i)}) - y^{(i)}x^{(i)}_{j}]$$

#### Final Gradient descent equation is as below

__Repeat this until we get $min_{\theta}J(\theta)$:__

        {

$$ \large \theta_{j} := \theta_{j} -\alpha \sum_{i=0}^{m}[x^{(i)}_{j}f_{\theta}(x^{(i)}) - y^{(i)}x^{(i)}_{j}] $$

        }

__Here $\alpha$ is Learning Rate __

### Comparing Classification & Regression

|Properties| Supervised Classification (Logistic Regression) | Regression (Linear Regression)|
|:---:|:---: |:---:|
|  Output Type | Discrete Value (Class labels) | Continuous (number) |
|  What are you trying to achieve? | Decision Boundary or How to choose value for parameters theta | Best line of fit |
|  Evaluation | Accuracy | Sum of Sequared Error or $r^{2} Error$|

### Intution of logistic regression

Its about finding the decision boundary.

#### Scenario 1

```python
import matplotlib.pyplot as plt
x1 = np.random.randn(100)
y1 = np.random.randn(100)
x2 = np.random.randn(100)
y2 = np.random.randn(100) + 6
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```

#### Scenario 2

```python
x1 = np.random.randn(100)
y1 = np.random.randn(100) + 3
x2 = np.random.randn(100) + 3
y2 = np.random.randn(100)
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```

#### Scenario 3

```python
x1 = np.random.randn(100)
y1 = np.random.randn(100)
x3 = np.random.randn(25) + 3
y3 = np.random.randn(25)
x4 = np.random.randn(25) - 3
y4 = np.random.randn(25)
x5 = np.random.randn(25)
y5 = np.random.randn(25) + 3
x6 = np.random.randn(25)
y6 = np.random.randn(25) - 3
x3 = np.append(x3 , x5)
y3 = np.append(y3, y5)
x4 = np.append(x4, x6)
y4 = np.append(y4, y6)
x2 = np.append(x3, x4)
y2 = np.append(y3, y4)
#x2 = np.append(x3, x6)
#y2 = np.append(y2 , y6)
plt.plot(x1, y1, "+", x2, y2, "x")
plt.axis('equal')
plt.show()
```


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Power Spectral Densitity

For a wide-sense stationary (WSS) real-valued random process $x[k]$, the [power spectral density](../random_signals/power_spectral_densities.ipynb#Power-Spectral-Density) (PSD) $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given as the discrete-time Fourier transformation (DTFT) of the auto-correlation function (ACF) $\varphi_{xx}[\kappa]$

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\kappa = -\infty}^{\infty} \varphi_{xx}[\kappa] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,\kappa}
\end{equation}

Under the assumption of a real-valued LTI system with impulse response $h[k] \in \mathbb{R}$, the PSD $\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the output signal of an LTI system $y[k] = \mathcal{H} \{ x[k] \}$ is derived by taking the DTFT of the [ACF of the output signal](../random_signals_LTI_systems/correlation_functions.ipynb#Auto-Correlation-Function) $\varphi_{yy}[\kappa]$

\begin{align}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) &= \sum_{\kappa = -\infty}^{\infty} \underbrace{h[\kappa] * h[-\kappa]}_{\varphi_{hh}[\kappa]} * \varphi_{xx}[\kappa] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,\kappa} \\
&= H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot H(\mathrm{e}^{\,-\mathrm{j}\,\Omega}) \cdot \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = | H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2 \cdot \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{align}

The PSD of the output signal $\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of an LTI system is given by the  PSD of the input signal $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ multiplied with the squared magnitude $| H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2$ of the transfer function of the system.

### Example - Pink Noise

It can be concluded from above findings, that filtering can be applied to a white noise random signal $x[k]$ with $\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0$ in order to create a random signal $y[k] = x[k] * h[k]$ with a desired PSD

\begin{equation}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = N_0 \cdot | H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

where $N_0$ denotes the power per frequency of the white noise. Such a random signal is commonly termed as [*colored noise*](https://en.wikipedia.org/wiki/Colors_of_noise). Different application specific types of colored noise exist. One of these is [*pink noise*](https://en.wikipedia.org/wiki/Pink_noise) whose PSD is inversely proportional to the frequency. The approximation of a pink noise signal by filtering is illustrated by the following example. The PSDs $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ are estimated from $x[k]$ and $y[k]$ using the [Welch technique](../spectral_estimation_random_signals/welch_method.ipynb).

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

fs = 44100
N = 5*fs

# generate uniformly distributed white noise
np.random.seed(1)
x = np.random.uniform(size=N) - .5
# filter white noise to yield pink noise
# see http://www.firstpr.com.au/dsp/pink-noise/#Filtering
a = np.poly([0.99572754, 0.94790649, 0.53567505])  # denominator coefficients
b = np.poly([0.98443604, 0.83392334, 0.07568359])  # numerator coefficients
y = 1/3 * sig.lfilter(b, a, x)
# estimate PSDs using Welch's technique
f, Pxx = sig.csd(x, x, nperseg = 256)
f, Pyy = sig.csd(y, y, nperseg = 256)

# PSDs
Om = f * 2 * np.pi
plt.plot(Om, 20*np.log10(np.abs(.5*Pxx)), label=r'$| \Phi_{xx}(e^{j \Omega}) |$ in dB')
plt.plot(Om, 20*np.log10(np.abs(.5*Pyy)), label=r'$| \Phi_{yy}(e^{j \Omega}) |$ in dB')
plt.title('Power Spectral Density')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -60, -10])
plt.grid()
```

Let's listen to white and pink noise

```python
from scipy.io import wavfile

wavfile.write('uniform_white_noise.wav', fs, np.int16(x*32768))
wavfile.write('uniform_pink_noise.wav', fs, np.int16(y*32768))
```

**White noise**
<audio src="./uniform_white_noise.wav" controls>Your browser does not support the audio element.</audio>[./uniform_white_noise.wav](./uniform_white_noise.wav)

**Pink noise**
<audio src="./uniform_pink_noise.wav" controls>Your browser does not support the audio element.</audio>[./uniform_pink_noise.wav](./uniform_white_noise.wav)

## Cross-Power Spectral Densities

The cross-power spectral densities $\Phi_{yx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ between the in- and output of an LTI system are given by taking the DTFT of the [cross-correlation functions](../random_signals_LTI_systems/correlation_functions.ipynb#Cross-Correlation-Function) (CCF) $\varphi_{yx}[\kappa]$ and $\varphi_{xy}[\kappa]$. Hence,

\begin{equation}
\Phi_{yx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\kappa = -\infty}^{\infty} h[\kappa] * \varphi_{xx}[\kappa] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,\kappa} = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot H(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

and

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\kappa = -\infty}^{\infty} h[-\kappa] * \varphi_{xx}[\kappa] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,\kappa} = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot H(\mathrm{e}^{\,-\mathrm{j}\,\Omega})
\end{equation}

## System Identification by Spectral Division

Using the result above for the cross-power spectral density $\Phi_{yx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ between out- and input, and the relation of the [CCF of finite-length signals to the convolution](../random_signals/correlation_functions.ipynb#Definition) yields

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{yx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})} = \frac{\frac{1}{K} Y(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot X(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}{\frac{1}{K} X(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot X(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}
= \frac{Y(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{X(\mathrm{e}^{\,\mathrm{j}\,\Omega})}
\end{equation}

holding for $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \neq 0$ and $X(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \neq 0$. Hence, the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of an unknown system can be derived by dividing the spectrum of the output signal $Y(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ through the spectrum of the input signal $X(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. This is equal to the [definition of the transfer function](https://en.wikipedia.org/wiki/Transfer_function). However, care has to be taken that the spectrum of the input signal does not contain zeros.

Above relation can be realized by the discrete Fourier transformation (DFT) by taking into account that a multiplication of two spectra $X[\mu] \cdot Y[\mu]$ results in the cyclic/periodic convolution $x[k] \circledast y[k]$. Since we aim at a linear convolution, zero-padding of the in- and output signal has to be applied.

### Example

We consider the estimation of the impulse response $h[k] = \mathcal{F}_*^{-1} \{ H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \}$ of an unknown system using the spectral division method. Normal distributed white noise with variance $\sigma_n^2 = 1$ is used as wide-sense ergodic input signal $x[k]$. In order to show the effect of sensor noise, normally distributed white noise $n[k]$ with the variance $\sigma_n^2 = 0.01$ is added to the output signal $y[k] = x[k] * h[k] + n[k]$.

```python
N = 1000  # number of samples for input signal

# generate input signal
np.random.seed(1)
x = np.random.normal(size=N, scale=1)  # normally distributed (zero-mean, unit-variance) white noise
# impulse response of the system
h = np.concatenate((np.zeros(20), np.ones(10), np.zeros(20)))
# output signal by convolution
y = np.convolve(h, x, mode='full')
# add noise to the output signal
y = y + np.random.normal(size=y.shape, scale=.1)

# zero-padding of input signal
x = np.concatenate((x, np.zeros(len(h)-1)))
# estimate transfer function
H = np.fft.rfft(y)/np.fft.rfft(x)
# compute inpulse response
he = np.fft.irfft(H)
he = he[0:len(h)]

# plot impulse response
plt.figure()
plt.stem(he, label='estimated')
plt.plot(h, 'g-', label='true')
plt.title('Estimated impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$\hat{h}[k]$')
plt.legend();
```

**Exercise**

* Change the length `N` of the input signal. What happens?
* Change the variance $\sigma_n^2$ of the additive noise. What happens?

Solution: Increasing the length `N` of the input signal lowers the uncertainty in estimating the impulse response. The higher the variance of the additive white noise, the higher the uncertainties in the estimated impulse response.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.