# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Superposition of Random Signals

The superposition of two random signals

\begin{equation}
y[k] = x[k] + n[k]
\end{equation}

is a frequently applied operation in statistical signal processing. For instance, to model a measurement procedure or communication channel as superposition of the desired signal with noise. We assume that the statistical properties of the real-valued signals $x[k]$ and $n[k]$ are known. We are interested in the statistical properties of $y[k]$, as well as the joint statistical properties between the signals and their superposition $y[k]$. It is assumed for the following that $x[k]$ and $n[k]$ are drawn from wide-sense stationary (WSS) real-valued random processes.

### Cumulative Distribution and Probability Density Function

The cumulative distribution function (CDF) $P_y(\theta)$ of $y[k]$ is given by rewriting it in terms of the joint probability density function (PDF) $p_{xn}(\theta_x, \theta_n)$

\begin{equation}
P_y(\theta) = \Pr \{ y[k] \leq \theta \} = \Pr \{ (x[k] + n[k]) \leq \theta \} =
\int\limits_{-\infty}^{\infty} \int\limits_{-\infty}^{\theta - \theta_n} p_{xn}(\theta_x, \theta_n) \; \mathrm{d}\theta_x\,\mathrm{d}\theta_n
\end{equation}

Its PDF is computed by introducing above result into the [definition](distributions.ipynb#Univariate-Probability-Density-Function) of the PDF

\begin{equation}
p_y(\theta) = \frac{\mathrm{d} P_y(\theta)}{\mathrm{d}\theta} = \int\limits_{-\infty}^{\infty} p_{xn}(\theta - \theta_n, \theta_n) \; \mathrm{d}\theta_n
\end{equation}

since the inner integral on the right hand side of $P_y(\theta)$ can be interpreted as the inverse operation to the derivation with respect to $\theta$.

An important special case is that $x[k]$ and $n[k]$ are uncorrelated. Under this assumption, the joint PDF $p_{xn}(\theta_x, \theta_n)$ can be written as $p_{xn}(\theta_x, \theta_n) = p_x(\theta_x) \cdot p_n(\theta_n)$. Introducing this into above result yields

\begin{align}
p_y(\theta) &= \int\limits_{-\infty}^{\infty} p_x(\theta - \theta_n) \cdot p_n(\theta_n) \; \mathrm{d}\theta_n \\
&= p_x(\theta) * p_n(\theta)
\end{align}

Hence, the PDF of the superposition of two uncorrelated signals is given by the convolution of the PDFs of both signals.

#### Example

The following example estimates the PDF of a superposition of two uncorrelated signals drawn from random processes generating samples according to the [uniformly distributed](important_distributions.ipynb#Uniform-Distribution) white noise model with $a=0$ and $b=1$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

N = 10000  # length of random signals

# generate random signals
np.random.seed(2)
x = np.random.uniform(size=N)
n = np.random.uniform(size=N)
y = x + n

# plot estimated pdf
plt.figure(figsize = (10, 6))
plt.hist(y, 50, normed=True)
plt.title('Estimated PDF')
plt.xlabel(r'$\theta$')
plt.ylabel(r'$\hat{p}_y(\theta)$');
```

**Exercise**

* Check the result of the numerical simulation by calculating the theoretical PDF of $y[k]$

The PDF of both signals $x[k]$ and $n[k]$ is given as $p_x(\theta) = p_n(\theta) = \text{rect}(\theta - \frac{1}{2})$, according to the assumptions stated above. The PDF of $y[k] = x[k] + n[k]$ is consequently given by $p_y(\theta) = \text{rect}(\theta - \frac{1}{2}) * \text{rect}(\theta - \frac{1}{2})$. The convolution of two rectangular signals results in a triangular signal. Hence

\begin{equation}
p_y(\theta) =
\begin{cases}
1 - |\theta -1| & \text{for } 0 < \theta < 2 \\
0 & \text{otherwise}
\end{cases}
\end{equation}

### Linear Mean

The linear mean $\mu_y$ of the superposition is derived by introducing $y[k] = x[k] + n[k]$ into the [definition of the linear mean](ensemble_averages.ipynb#Linear-mean) and exploiting the [linearity of the expectation operator](ensemble_averages.ipynb#Properties) as

\begin{equation}
\mu_y[k] = E \{ x[k] + n[k] \} = \mu_x[k] + \mu_n[k]
\end{equation}

The linear mean of the superposition of two random signals is the superposition of its linear means.

### Auto-Correlation Function and Power Spectral Density

The ACF is computed in the same manner as the linear mean by inserting the superposition into the [definition](correlation_functions.ipynb#Auto-Correlation-Function) of the ACF and rearranging terms

\begin{align}
\varphi_{yy}[\kappa] &= E\{ y[k] \cdot y[k-\kappa] \} \\
&= E\{ (x[k] + n[k]) \cdot (x[k-\kappa] + n[k-\kappa]) \} \\
&= \varphi_{xx}[\kappa] + \varphi_{xn}[\kappa] + \varphi_{nx}[\kappa] + \varphi_{nn}[\kappa]
\end{align}

The ACF of the superposition of two random signals is given as the superposition of all auto- and cross-correlation functions (CCFs) of the two random signals. The power spectral density (PSD) is derived by performing a discrete-time Fourier transform (DTFT) of the ACF

\begin{equation}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

This can be simplified further by exploiting the symmetry property of the CCFs $\varphi_{xn}[\kappa] = \varphi_{nx}[-\kappa]$ and the DTFT for real-valued signals as

\begin{equation}
\Phi_{yy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + 2\,\Re \{ \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \} + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

where $\Re \{ \cdot \}$ denotes the real part of its argument.

### Cross-Correlation Function and Cross Power Spectral Density

The CCF $\varphi_{ny}[\kappa]$ between the random signal $n[k]$ and the superposition $y[k]$ is derived again by introducing the superposition into the [definition of the CCF](correlation_functions.ipynb#Cross-Correlation-Function)

\begin{equation}
\varphi_{ny}[\kappa] = E\{ n[k] \cdot (x[k-\kappa] + n[k-\kappa]) \} = \varphi_{nx}[\kappa] + \varphi_{nn}[\kappa]
\end{equation}

It is given as the superposition of the CCF between the two random signals and the ACF of $n[k]$. The cross PSD is derived by applying the DTFT to $\varphi_{ny}[\kappa]$

\begin{equation}
\Phi_{ny}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{nx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

The CCF $\varphi_{xy}[\kappa]$ and cross PSD $\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can be derived by exchanging the signals $n[k]$ and $x[k]$

\begin{equation}
\varphi_{xy}[\kappa] = E\{ x[k] \cdot (x[k-\kappa] + n[k-\kappa]) \} = \varphi_{xx}[\kappa] + \varphi_{xn}[\kappa]
\end{equation}

and

\begin{equation}
\Phi_{xy}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{xn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

### Additive White Gaussian Noise

In order to model the effect of distortions, it is often assumed that a random signal $x[k]$ is distorted by additive normal distributed white noise $n[k]$ resulting in the observed signal $y[k] = x[k] + n[k]$. It is furthermore assumed that the noise $n[k]$ is uncorrelated to the signal $x[k]$. This model is known as [additive white Gaussian noise](https://en.wikipedia.org/wiki/Additive_white_Gaussian_noise) (AWGN) model.

For zero-mean random processes and from the properties of the AWGN model, it follows that $\varphi_{xn}[\kappa] = \varphi_{nx}[\kappa] = 0$ and $\varphi_{nn}[\kappa] = N_0 \cdot \delta[\kappa]$. Introducing this into above results for additive random signals yields the following relations for the AWGN model

\begin{align}
\varphi_{yy}[\kappa] &= \varphi_{xx}[\kappa] + N_0 \cdot \delta[\kappa] \\
\varphi_{ny}[\kappa] &= N_0 \cdot \delta[\kappa] \\
\varphi_{xy}[\kappa] &= \varphi_{xx}[\kappa]
\end{align}

The PSDs are given as the DTFTs of the ACF/CCFs. The AWGN model is frequently assumed in communications as well as in the measurement of physical quantities to cope for background, sensor and amplifier noise.

#### Example

For the following numerical example, the disturbance of a harmonic signal $x[k] = \cos(\Omega_0 k)$ by unit variance AWGN is considered.

```python
N = 1024  # length of signals
K = 20  # maximum lag for ACF/CCF

# generate signals
x = np.cos(20*2*np.pi/N*np.arange(N))
np.random.seed(2)
n = np.random.normal(size=N)
# superposition of signals
y = x + n

# compute and truncate ACF of superposition
acf = 1/N * np.correlate(y, y, mode='full')
acf = acf[(len(x)-1)-(K-1):(len(x)-1)+K]
# compute and truncate CCF of superposition and noise
ccf = 1/N * np.correlate(n, y, mode='full')
ccf = ccf[(len(x)-1)-(K-1):(len(x)-1)+K]

# plot results
kappa = np.arange(-(K-1), K)
plt.figure(figsize=(10, 4))

plt.subplot(121)
plt.stem(kappa, acf)
plt.title('ACF of superposition')
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\varphi_{yy}[\kappa]$')
plt.axis([-K, K, -.5, 1.1*np.max(acf)])

plt.subplot(122)
plt.stem(kappa, ccf)
plt.title('CCF between noise and superposition')
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\varphi_{ny}[\kappa]$')
plt.axis([-K, K, -.2, 1.1]);
```

**Exercise**

* Derive the theoretic result for $\varphi_{xx}[\kappa]$
* Based on this, can you explain the results of the numerical simulation?

The ACF of the deterministic signal $x[k] = \cos(\Omega_0 k)$ is computed via temporal averaging. Since $x[k]$ is periodic, the ACF $\varphi_{xx}[\kappa]$ is also periodic. It is hence sufficient to perform the temporal averaging over one period $N = \frac{2 \pi}{\Omega_0}$ with $N \in \mathbb{Z}$

\begin{align}
\varphi_{xx}[\kappa] &= \frac{1}{N} \sum_{k=0}^{N-1} x[k] \cdot x[k-\kappa] \\
&= \frac{1}{N} \sum_{k=0}^{N-1} \cos(\Omega_0 k) \cdot \cos(\Omega_0 (k-\kappa))
\end{align}

Applying the [product-to-sum identity](https://en.wikipedia.org/wiki/List_of_trigonometric_identities#Product-to-sum_and_sum-to-product_identities) of the cosine function and rearranging terms yields

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{2} \cos(\Omega_0 \kappa) + \frac{1}{2N} \sum_{k=0}^{N-1} \cos(2 \Omega_0 k - \Omega_0 \kappa)
\end{equation}

The remaining sum over the phase shifted cosine $\cos(2 \Omega_0 k - \Omega_0 \kappa)$ is zero since the summation is carried out over two full periods. The ACF of a cosine signal is then given as

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{2} \cos(\Omega_0 \kappa)
\end{equation}

Introducing this into the relations for the AWGN model, the ACF of the superposition $\varphi_{yy}[\kappa]= \frac{1}{2} \cos(\Omega_0 \kappa) + \delta[\kappa]$. Taking the statistical uncertainty due to a finite number of samples into account, this analytic result coincides with the result of the numerical evaluation shown above.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


Problem: Heavy hitter

Reference:
- Privacy at Scale: Local Differential Privacy in Practice

```python
%load_ext autoreload
%autoreload 2
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

Implementation of Random Response Protocol $\pi$ (for user with value $v$) as
\begin{equation}
  \forall_{y\in D} Pr[\pi(v)=y]=
  \begin{cases}
    \frac{e^\epsilon}{e^\epsilon+|D|-1}, & \text{if $y=v$}.\\
    \frac{1}{e^\epsilon+|D|-1}, & \text{if $y\neq v$}.\\
  \end{cases}
\end{equation}

Intuition: We sample the correct answer (response $y$ equals truth value $v$) with higher probability. Therefore, if $\epsilon$ is higher, probability of returning the correct answer is higher which means the privacy is lower. When $\epsilon = 0$, we have the highest privacy protection.

```python
from heavyhitter import heavyhitter as hh
import numpy as np
class ROUser(hh.User):
  def set_epsilon(self, epsilon=0):
    self.eps = epsilon
    pk = [np.exp(self.eps) if np.array_equal(self.val, np.asarray([x])) else 1
                                                 for x in self.rv.xk]
    pk /= np.exp(self.eps)+len(self.rv.xk)-1
    self.response_rv = stats.rv_discrete(name='response', values=(self.rv.xk, pk))

  def response(self, query):
    assert hasattr(self, 'response_rv'), "response requires setting privacy budget epsilon"
    return self.response_rv.rvs(size=1) == query

class ROAggregator(hh.Aggregator):
  def aggregate(self, responses):
    return np.mean(responses)
```

```python
from scipy import stats
import random

# create the random variable
domain_size = 5
domain = np.arange(domain_size)
prob = np.asarray([random.random() for _ in range(5)])
prob /= sum(prob)
rv = stats.rv_discrete(name='rv', values=(domain, prob))

# plot prob
plt.figure()
fig, ax = plt.subplots(1, 1)
ax.plot(domain, rv.pmf(domain), 'ro', ms=12, mec='r')
ax.vlines(domain, 0, rv.pmf(domain), colors='r', lw=4)
ax.set_xticks(domain)
plt.show()

# sanity check
from collections import Counter
c = Counter([sample for sample in rv.rvs(size=1000)])
print "Generating 1000 samples\n", list(c.items())
```

Frequency query with local Differential Privacy. The untrusted aggregator is trying to estimate the underlying true distribtion so it needs to query the users to report their values.

There are two sources of randomness
- Each user is a sample of the true distribution.
- To protect their privacy, each user reports a random response.

Note that the randomness of the response in the equation depends on the domain size but not the number of users.

```python
users = [ROUser(index=i, rv=rv) for i in range(1000)]
aggregator = ROAggregator(index=0)
aggregator.subscribe(users)
```

```python
# Frequency query for elements
for eps in [0.0, 5.0]:
  for user in users:
    user.set_epsilon(epsilon=eps)
  print "==== %s users with epsilon=%s ===="%(len(users), users[0].eps)
  for x, p in zip(rv.xk, rv.pk):
    frequency = aggregator.aggregate(aggregator.query(np.asarray([x])))
    print "ele:%s, truth:%0.4f, estimate:%0.4f, diff:%0.4f"%(x,p,frequency, abs(p-frequency))
```

    ==== 1000 users with epsilon=0.0 ====
    ele:0, truth:0.2380, estimate:0.1990, diff:0.0390
    ele:1, truth:0.0092, estimate:0.1940, diff:0.1848
    ele:2, truth:0.1772, estimate:0.1950, diff:0.0178
    ele:3, truth:0.3103, estimate:0.2040, diff:0.1063
    ele:4, truth:0.2653, estimate:0.2130, diff:0.0523
    ==== 1000 users with epsilon=5.0 ====
    ele:0, truth:0.2380, estimate:0.2320, diff:0.0060
    ele:1, truth:0.0092, estimate:0.0130, diff:0.0038
    ele:2, truth:0.1772, estimate:0.1810, diff:0.0038
    ele:3, truth:0.3103, estimate:0.2720, diff:0.0383
    ele:4, truth:0.2653, estimate:0.3000, diff:0.0347


# $R_0$ Analysis

One of the key quantities of interest for any infectious pathogen is the _reproductive number_, denoted $R_0$.

This is the expected number of new cases caused by each case.

Most estimates place this number for the 2019-nCov (novel coronavirus) as between 2 and 3. For example, Zhang et al ([link](https://www.ncbi.nlm.nih.gov/pubmed/32097725)), who analyzed the outbreak on the Diamond Princess cruise ship, say:

> The Maximum-Likelihood (ML) value of R0 was 2.28 for COVID-19 outbreak at the early stage on the ship. The median with 95% confidence interval (CI) of R0 values was 2.28 (2.06-2.52).

In this notebook, we try to reproduce these findings within Canada. That is, we run a hypothesis test:

$$ H_0: R_0 \in [2.06, 2.52] $$

$$ H_A: R_0 < 2.06 \text{ or } R_0 > 2.52 $$

### Setup

```julia
# Load dependencies
using Pkg
pkg"activate .."
using JSON, Statistics, Plots, Dates, LsqFit, TimeSeries, DataFrames
using StatsModels, GLM, HypothesisTests, StatsFuns
```

     Activating environment at `~/classes/326/ECON-326-Project/Project.toml`

```julia
# Load data
data = JSON.parsefile("../data/covid_data.json");
```

### Data Filtering

There is a potential heteroskedasticity problem, in that if we run our regression against the entire timespan (from Jan. 22 to April 5), we're ignoring all the changes in the public health environment which have taken place.

For example, we would expect that the observed $\hat{R}_0$ in Canada is lower in recent weeks, as Canada has aggressively enforced social distancing.

So, we'll **remove all data** dating to after the WHO pandemic declaration on **Mar. 11, 2020**.

```julia
# Unpack data
pandemic_day = Date("2020-03-11")
canada = data["Canada"]

confirmed = [point["confirmed"] for point in canada if Date(point["date"]) <= pandemic_day]
recovered = [point["recovered"] for point in canada if  Date(point["date"]) <= pandemic_day]
deaths = [point["deaths"] for point in canada if Date(point["date"]) <= pandemic_day]
dates = Date.([point["date"] for point in canada if Date(point["date"]) <= pandemic_day]);
```

### Regression Model

There is another difficulty, which is that there is not a uniform incubation period for the virus.

However, the median incubation period is roughly 5 days (see [here](https://annals.org/aim/fullarticle/2762808/incubation-period-coronavirus-disease-2019-covid-19-from-publicly-reported), _Annals of Internal Medicine_), so we will use that as an approximation.

Our guess for the true spread process is then:

$$ x_{t + 5} = R_0 x_t + \epsilon $$

If Canada was a closed system (i.e., there was no immigration), and the incubation period was exactly and uniformly 5 days, with each case causing exactly $R_0$ others, then $\epsilon = 0$.

Note that this implies an exponential spread:

$$\begin{align}
x_{t + 5} - x_t = (R_0 - 1)x_t  \\
\frac{d x}{d t} = \frac{1}{5} \big( (R_0 - 1) \big) x
\end{align}$$

Which is a first-order ODE with solution:

$$ x(t) = \alpha e^{\frac{1}{5} \big( (R_0 - 1) \big)t} $$

with $\alpha$ an integration constant (the num. of cases at $t = 0$.)

### Linear Regression and Assumptions

So, the exact regression we are running is simply:

$$ y_i = \beta_1 x_i + \beta_0 $$

Where the data is grouped $(x_i, y_i) = (x_t, x_{t+5})$.

Before we begin, recall the assumptions:

1. $0 < \mathbb{E}X^4 < \infty$ and $0 < \mathbb{E}Y^4 < \infty$: This is true. We plotted the distributions earlier, and nothing was truly obscene. Also, Canada has a finite population, and the data are small relative to that.

2. $\mathbb{E}[u_i | X_i] = 0$: This one is shaky. As $X_i$ increases, for example, $Y_i$ becomes less predictable (the 5-day incubation time was a rough estimate, and one that's liable to break down as more and more people get the virus.) But there's no easy way around it.

3. $(X_i, Y_i)$ are i.i.d. draws from their joint distribution: Certainly there wasn't any preferential sampling done (and the future 5 days away is unknowable when each $X_i$ was "sampled.") But perhaps the cutoff rule we used (there are only time periods of interest; before Mar. 11, and after) is too coarse.

   The key question is if there is too much serial dependence (the data point $(X_t, X_{t+5})$ is correlated with $(X_{t+5}, X_{t+10})$). But short of using sophisticated statistical techniques and multiple independent data sets, we can't see an obvious answer.

### Model Fitting

```julia
df = DataFrame(X=confirmed[1:end-5], Y=confirmed[6:end]);
```

```julia
ols = lm(@formula(Y ~ X), df)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

    Y ~ 1 + X

    Coefficients:
    ───────────────────────────────────────────────────────────────────────────
                 Estimate  Std. Error   t value  Pr(>|t|)  Lower 95%  Upper 95%
    ───────────────────────────────────────────────────────────────────────────
    (Intercept)  -3.69961   0.840939   -4.39939    <1e-4    -5.39553   -2.0037
    X             2.22754   0.0584838  38.0881     <1e-34    2.1096     2.34548
    ───────────────────────────────────────────────────────────────────────────

So, our fitted model is:

$$ x_{t+5} = 2.22754x_t -3.69961 $$

### Goodness of Fit

Before doing anything quantitative, we can get a visual picture.

```julia
predicted_data = predict(ols);
plot(dates[6:end], confirmed[6:end], lw = 3, label = "Actual Case Count", title = "Canada", legend = :topleft)
plot!(dates[6:end], predicted_data, lw = 3, label = "Predicted Case Count", xticks = (dates[6:end])[1:8:end])
```

We can examine the $R^2$ value for our regression:

```julia
r2(ols) # R^2
```

    0.9712125257563606

By construction, the residuals are uncorrelated with $X$.

We can see this below.

```julia
plot(df.X, residuals(ols), lw = 3, legend = false, title = "Regression Residuals Against X", xlabel = "X_i", ylabel = "u_i")
```

However, it's possible that the residuals are correlated _serially_. That is:

$$ \text{Cov}[u_i, u_{i-1}] \neq 0 $$

Ordinarily, the indexing of the data doesn't matter.

However, this is a time series.

Julia provides a statistical test for this:

```julia
DurbinWatsonTest(Float64.(df.X), residuals(ols)) # check for serial correlation
```

    Durbin-Watson autocorrelation test
    ----------------------------------
    Population details:
        parameter of interest:   sample autocorrelation parameter
        value under h_0:         "0"
        point estimate:          0.7920867976746211

    Test summary:
        outcome with 95% confidence: reject h_0
        two-sided p-value:           <1e-10

    Details:
        number of observations:     45
        DW statistic:               0.4158264046507579

And we can confirm visually that there is some serial correlation:

```julia
plot(residuals(ols), lw = 3, legend = false, title = "Regression Residuals Against Time", xlabel = "t", ylabel = "u_i")
```

So, although we don't notice much $\text{Cov}(u_i, X_i)$, we do see significant $\text{Cov}(u_i, i)$.

### Hypothesis Testing

We found

$$ \hat{R}_0 = \hat{\beta}_1 = 2.22754 $$

To test our composite hypothesis, we can compute the $t$ statistic for each interval endpoint (2.06 and 2.52.)

Testing

$$ H_A: R_0 < 2.06 $$

```julia
t_left = (2.22754 - 2.06)/0.0584838
```

    2.8647249323744317

```julia
p_left = normcdf(t_left)
```

    0.9979131419179907

Testing

$$ H_A : R_0 > 2.52 $$

```julia
t_right = (2.22754 - 2.52)/0.0584838
```

    -5.000701048837459

```julia
p_right = 1 - normcdf(t_right)
```

    0.9999997143888665

Therefore, we fail to reject both prongs of the null hypothesis, which is the Zhang et al results for the $R_0$.

(That is, under the null hypothesis, something like 99.7% of tests would produce the results we saw.)

# Pandemic Response Analysis

Another interesting question is how various countries compare to one another in their responses.

Specifically, we're interested in whether Canada has responded "better" than the US since the WHO pandemic declaration.

So:

$$ H_A: \text{"Canada responded better than the US, since the WHO declared a pandemic.}" $$

$$ H_0: \text{"Canada didn't respond better than the US, since....}"$$

We'll make this precise shortly.

Note that this is a **one-sided alternative hypothesis**, run with **two populations** (Canada and the US.)

### Quantifying "Success"

A natural way to capture success is to compare the empirical $\hat{R}_0$ before and after the WHO pandemic declaration. That is:

$$ \hat{\Delta} =  \hat{R}^{\text{before Mar. 11}}_0 - \hat{R}^{\text{after Mar. 11}}_0 $$

The test is therefore:

$$ H_0: \Delta_{\text{Canada}} - \Delta_{\text{US}} \leq 0 $$

$$ H_A: \Delta_{\text{Canada}} - \Delta_{\text{US}} > 0$$

### Preparing Data

```julia
canada = data["Canada"]
us = data["US"]

canada_confirmed_before = [point["confirmed"] for point in canada if Date(point["date"]) <= pandemic_day]
canada_confirmed_after = [point["confirmed"] for point in canada if Date(point["date"]) > pandemic_day]
us_confirmed_before = [point["confirmed"] for point in us if Date(point["date"]) <= pandemic_day]
us_confirmed_after = [point["confirmed"] for point in us if Date(point["date"]) > pandemic_day];
```

### Run Regressions

```julia
df_can_before = DataFrame(X=canada_confirmed_before[1:end-5], Y=canada_confirmed_before[6:end]);
df_can_after = DataFrame(X=canada_confirmed_after[1:end-5], Y=canada_confirmed_after[6:end]);
df_us_before = DataFrame(X=us_confirmed_before[1:end-5], Y=us_confirmed_before[6:end]);
df_us_after = DataFrame(X=us_confirmed_after[1:end-5], Y=us_confirmed_after[6:end]);
```

```julia
ols_can_before = lm(@formula(Y ~ X), df_can_before)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

    Y ~ 1 + X

    Coefficients:
    ───────────────────────────────────────────────────────────────────────────
                 Estimate  Std. Error   t value  Pr(>|t|)  Lower 95%  Upper 95%
    ───────────────────────────────────────────────────────────────────────────
    (Intercept)  -3.69961   0.840939   -4.39939    <1e-4    -5.39553   -2.0037
    X             2.22754   0.0584838  38.0881     <1e-34    2.1096     2.34548
    ───────────────────────────────────────────────────────────────────────────

```julia
ols_can_after = lm(@formula(Y ~ X), df_can_after)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

    Y ~ 1 + X

    Coefficients:
    ───────────────────────────────────────────────────────────────────────────────
                   Estimate   Std. Error   t value  Pr(>|t|)  Lower 95%   Upper 95%
    ───────────────────────────────────────────────────────────────────────────────
    (Intercept)  1143.12     208.145       5.49191    <1e-4   707.462    1578.77
    X               1.71873    0.0505082  34.0288     <1e-17    1.61302     1.82445
    ───────────────────────────────────────────────────────────────────────────────

```julia
ols_us_before = lm(@formula(Y ~ X), df_us_before)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

    Y ~ 1 + X

    Coefficients:
    ───────────────────────────────────────────────────────────────────────────
                 Estimate  Std. Error   t value  Pr(>|t|)  Lower 95%  Upper 95%
    ───────────────────────────────────────────────────────────────────────────
    (Intercept)  -41.3603     8.59528  -4.81198    <1e-4   -58.6943   -24.0263
    X              4.5445     0.13208  34.4072     <1e-32    4.27813    4.81086
    ───────────────────────────────────────────────────────────────────────────

```julia
ols_us_after = lm(@formula(Y ~ X), df_us_after)
```

    StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Array{Float64,1}},GLM.DensePredChol{Float64,LinearAlgebra.Cholesky{Float64,Array{Float64,2}}}},Array{Float64,2}}

    Y ~ 1 + X

    Coefficients:
    ───────────────────────────────────────────────────────────────────────────────────
                    Estimate   Std. Error   t value  Pr(>|t|)    Lower 95%    Upper 95%
    ───────────────────────────────────────────────────────────────────────────────────
    (Intercept)  28012.4      4842.54       5.78466    <1e-4   17876.9      38148.0
    X                1.71707     0.053714  31.9668     <1e-17      1.60464      1.82949
    ───────────────────────────────────────────────────────────────────────────────────

### Calculate Deltas

```julia
Δ_canada = 2.22754 - 1.71873
```

    0.5088099999999998

```julia
Δ_us = 4.5445 - 1.71707
```

    2.82743

**Note:** There's no clear formula we could find for the standard error of a difference of OLS estimates (let alone a difference of differences of OLS estimates.) But a plausible lower bound would be to add the SE of each term (this corresponds to the assumptions that the various countries before/after the declaration are independent.)

This is a weakness in our study.

```julia
SE = 0.053714 + 0.13208 + 0.0505082 + 0.0584838
```

    0.294786

### Test Hypotheses

To reiterate, we are testing

$$ H_0: \Delta_{\text{Canada}} - \Delta_{\text{US}} \leq 0 $$

$$ H_A: \Delta_{\text{Canada}} - \Delta_{\text{US}} > 0 $$

```julia
t = (Δ_canada - Δ_us)/0.294786
```

    -7.865434586445761

```julia
1 - normcdf(t)
```

    0.9999999999999981

Since our $p$-value here is 99.999999999999%, we fail to reject the null.

Put another way, our $t$ statistic is extremely far from the rejection region of (say) 1.96 or higher, so we fail to reject.


# Assigment 2 - Aakash ME16B001
### Problem 4

```python
from sympy import *
from sympy.solvers import solve
from sympy import Symbol
import matplotlib.pyplot as plt
import numpy as np

x=Symbol('x')
a1=Symbol('a1')
a2=Symbol('a2')
a3=Symbol('a3')
a4=Symbol('a4')
a5=Symbol('a5')
eqn_gal2 =[]
eqn_gal3 =[]
eqn_gal4 =[]

u2 = a1 + a2*x + a3*x*x
z2=u2.subs(a1,solve(u2.subs(x,1)-2 , a1)[0])
z2=z2.subs(a2,solve((x*diff(u2,x)+0.5).subs(x,2),a2)[0])
# finding residual
r_gal2=diff(x*(diff(z2,x)),x)-(2/(x*x))
eqn_gal2.append(integrate(r_gal2*x*x, (x, 1, 2)))
z_gal2 = solve(eqn_gal2, a3)
print("Equation for Quadratic",eqn_gal2)
print(z_gal2)

u3 = a1 + a2*x + a3*x*x + a4*x*x*x
z3=u3.subs(a1,solve(u3.subs(x,1)-2 , a1)[0])
z3=z3.subs(a2,solve((x*diff(u3,x)+0.5).subs(x,2),a2)[0])
# finding residual
r_gal3=diff(x*(diff(z3,x)),x)-(2/(x*x))
# galerkian integral wrt weighting functions
eqn_gal3.append(integrate(r_gal3*x*x, (x, 1, 2)))
eqn_gal3.append(integrate(r_gal3*x*x*x, (x, 1, 2)))
z_gal3 = solve(eqn_gal3, [a3,a4])
print("Equation for Cubic",eqn_gal3)
print(z_gal3)

u4 = a1 + a2*x + a3*x*x + a4*x*x*x + a5*x*x*x*x
z4=u4.subs(a1,solve(u4.subs(x,1)-2 , a1)[0])
z4=z4.subs(a2,solve((x*diff(u4,x)+0.5).subs(x,2),a2)[0])
# finding residual
r_gal4=diff(x*(diff(z4,x)),x)-(2/(x*x))
# galerkian integral wrt weighting functions
eqn_gal4.append(integrate(r_gal4*x*x, (x, 1, 2)))
eqn_gal4.append(integrate(r_gal4*x*x*x, (x, 1, 2)))
eqn_gal4.append(integrate(r_gal4*x*x*x*x, (x, 1, 2)))
z_gal4 = solve(eqn_gal4, [a3,a4,a5])
print("Equation for Biquadratic",eqn_gal4)
print(z_gal4)

# plotting the curve
u_gal2=[]
u_gal3=[]
u_gal4=[]
u_exact=[]
y = []
i=1
while i<=2:
    y.append(i)
    u_gal2.append((z2.subs(a3,z_gal2.get(a3))).subs(x,i))
    u_gal3.append(((z3.subs(a3,z_gal3.get(a3))).subs(a4,z_gal3.get(a4))).subs(x,i))
    u_gal4.append((((z4.subs(a3,z_gal4.get(a3))).subs(a4,z_gal4.get(a4))).subs(a5,z_gal4.get(a5))).subs(x,i))
    u_exact.append((2/i)+0.5*np.log(i))
    i+=0.1

plt.plot(y, u_gal2, label="Quadratic")
plt.plot(y, u_gal3, label="Cubic")
plt.plot(y, u_gal4, label="Biquadratic")
plt.plot(y, u_exact, label="Exact")
plt.legend()
plt.xlabel('x', fontsize=16)
plt.ylabel('u', fontsize=16)
plt.suptitle('Solutions with different Trial functions')
plt.show()

print("It can be observed from the graph that Biquadratic trial function \nhas the least error and almost coincides with the exact soluton")

```

### Problem 2

```python
from sympy import *
from sympy.solvers import solve
from sympy import Symbol
import matplotlib.pyplot as plt
import numpy as np

x=Symbol('x')
a1=Symbol('a1')
a2=Symbol('a2')
a3=Symbol('a3')
wf1=Symbol('wf1')
wf2=Symbol('wf2')
eqn=[]

u=2+(x-1)*a1+(x*x-1)*a2+a3*(x**3-1)
up=diff(u,x)

n=[x-1,x*x-1,x**3-1]
print(n)
for i in range(3):
    print(n[i])
    w=n[i]*(x**2)*(5-2*u)-integrate(diff(n[i],x)*(x**2)*up,x)+integrate((x-1)*n[i],x)
    wf1=w.subs(x,1)
    wf2=w.subs(x,2)
    eqn.append(wf2-wf1)
print(w)
print(eqn)
sol=solve(eqn,[a1, a2, a3])
print(sol)

u_ext=[]
u_gal=[]
y = []
i=1
while i<=2:
    y.append(i)
    u_ext.append(((0.1)*(29.9096-(5*x)-(4.9096/x))+log(x)).subs(x,i))
    u_gal.append((((u.subs(a1,sol.get(a1))).subs(a2,sol.get(a2))).subs(a3,sol.get(a3))).subs(x,i))
    i+=0.01
plt.plot(y, u_ext, label="Exact")
plt.plot(y, u_gal, label="Galerkin")
plt.legend()
plt.xlabel('x', fontsize=16)
plt.ylabel('u', fontsize=16)
plt.suptitle('Exact soluton vs Galerkin weak form')
plt.show()
```

```python

```


# Lecture 4 - SciPy

What we have seen so far
- How to setup a python environment and jupyter notebooks
- Basic python language features
- Introduction to NumPy
- Plotting using matplotlib

Scipy is a collection of packages that provide useful mathematical functions commonly used for scientific computing.

List of subpackages
- cluster : Clustering algorithms
- constants : Physical and mathematical constants
- fftpack : Fast Fourier Transform routines
- integrate : Integration and ordinary differential equation solvers
- interpolate : Interpolation and smoothing splines
- io : Input and Output
- linalg : Linear algebra
- ndimage : N-dimensional image processing
- odr : Orthogonal distance regression
- optimize : Optimization and root-finding routines
- signal : Signal processing
- sparse : Sparse matrices and associated routines
- spatial : Spatial data structures and algorithms
- special : Special functions
- stats : Statistical distributions and functions

We cannot cover all of them in detail but we will go through some of the packages and their capabilities today

- interpolate
- optimize
- stats
- integrate

We will also briefly look at some other useful packages
- networkx
- sympy

```python
import numpy as np
import matplotlib.pyplot as plt
```

## Interpolation : `scipy.interpolate`

```python
import scipy.interpolate as interp
```

```python
x = np.linspace(-1,2,5);
y = x**2
plt.plot(x,y,'ro')
```

```python
f = interp.interp1d(x,y,kind="linear")
```

```python
type(f)
```

    scipy.interpolate.interpolate.interp1d

```python
x_fine = np.linspace(-1,2,100)
plt.plot(x_fine,f(x_fine))
plt.plot(x,y,'ro')
```

```python
plt.plot(x_fine,interp.interp1d(x,y,kind="zero")(x_fine))
plt.plot(x_fine,interp.interp1d(x,y,kind="linear")(x_fine))
plt.plot(x_fine,interp.interp1d(x,y,kind="cubic")(x_fine))
plt.plot(x,y,'ro')
```

```python
interp.interp1d?
```

```python
interp.interp2d?
```

## Optimization : `scipy.optimize`

Contains functions to find minima, roots and fit parameters

```python
from scipy import optimize
```

```python
def f(x):
    return x**2 + np.sin(2*x)
```

```python
x = np.linspace(-5,5,100)
plt.plot(x,f(x));
```

```python
results = optimize.minimize(f, -4)
results
```

          fun: -0.5920740012779424
     hess_inv: array([[0.18432423]])
          jac: array([1.49011612e-07])
      message: 'Optimization terminated successfully.'
         nfev: 33
          nit: 6
         njev: 11
       status: 0
      success: True
            x: array([-0.51493324])

```python
x_opt = results.x
```

```python
plt.plot(x,f(x));
plt.plot(x_opt,f(x_opt),'ro');
```

```python
optimize.minimize?
```

```python
def f(x):
    return x[0]*x[0] + x[1]*x[1] + 5*(np.sin(2*x[0]) + np.sin(2*x[1]) )
```

```python
x=np.linspace(-5,5,100)
y=np.linspace(-5,5,100)
X,Y = np.meshgrid(x,y)
```

```python
plt.imshow(f((X,Y)))
```

```python
optimize.minimize(f,x0=[2,2])
```

          fun: 0.07912876341589659
     hess_inv: array([[ 0.52488677, -0.47511323],
           [-0.47511323,  0.52488677]])
          jac: array([1.1920929e-07, 1.1920929e-07])
      message: 'Optimization terminated successfully.'
         nfev: 24
          nit: 4
         njev: 6
       status: 0
      success: True
            x: array([2.13554766, 2.13554766])

You can use the function `basinhopping` to find the global minima

```python
optimize.basinhopping(f,[1,4])
```

```python
optimize.basinhopping?
```

## Curve Fitting

```python
x = np.linspace(-2,2,30)
y = x+np.sin(5.2*x)+0.3*np.random.randn(30)
plt.plot(x,y,'ro')
```

```python
def f(x,a,b,c):
    return a*x + b*np.sin(c*x)
```

```python
((a,b,c),cov) = optimize.curve_fit(f,x,y,(0,0,4))
a,b,c
```

    (1.0112575930259058, 1.0024366873075972, 5.1915165536399055)

```python
cov
```

    array([[ 0.001428  , -0.00016983,  0.00077266],
           [-0.00016983,  0.00355949, -0.00031009],
           [ 0.00077266, -0.00031009,  0.00268351]])

```python
x_fine = np.linspace(-2,2,200)
plt.plot(x_fine,f(x_fine,a,b,c))
plt.plot(x,y,'ro')
```

### Root Finding

```python
def f(x):
    return (x+2)*(x-1)*(x-5)
```

```python
optimize.root(f,0)
```

        fjac: array([[-1.]])
         fun: array([7.99360578e-15])
     message: 'The solution converged.'
        nfev: 8
         qtf: array([-1.11436793e-08])
           r: array([12.00000791])
      status: 1
     success: True
           x: array([1.])

## Statistics : `scipy.stats`

```python
from scipy import stats
```

Find the maximum likelihood estimate for parameters

```python
samples = 3*np.random.randn(1000)+2
plt.hist(samples);
```

```python
stats.norm.fit(samples)
```

    (1.9458130358594379, 2.9667022271670134)

```python
np.mean(samples),np.median(samples)
```

    (1.9458130358594379, 1.996030772212961)

```python
stats.scoreatpercentile(samples,20)
```

    -0.5886852239653766

```python
a = np.random.randn(30)
b = np.random.randn(30) + 0.1
```

```python
stats.ttest_ind(a,b)
```

    Ttest_indResult(statistic=-1.1920408671206213, pvalue=0.23809979134147563)

You can also perform kernel density estimation

```python
x = np.concatenate(( 2*np.random.randn(1000)+5,  0.6*np.random.randn(1000)-1) )
```

```python
plt.hist(x);
```

```python
pdf = stats.kde.gaussian_kde(x)
```

```python
counts,bins,_ = plt.hist(x)
x_fine=np.linspace(-2,10,100)
plt.plot(x_fine,np.sum(counts)*pdf(x_fine))
```

```python
bins
```

## Numerical Integration : `scipy.integrate`

```python
import scipy.integrate as integ
```

You can compute integral using the `quad` funtion

```python
def f(x):
    return x**2 + 5*x + np.sin(x)
```

```python
integ.quad(f,-1,1)
```

```python
integ.quad?
```

You can also solve ODEs of the form
$$ \frac{dy}{dt} = f(y,t) $$

```python
def f(y,t):
    return (y[1], -y[1]-9*y[0])
```

```python
t = np.linspace(0,10,100)
Y = integ.odeint(f,[1,1],t)
```

```python
plt.plot(t,Y[:,1])
```

# Other useful packages

## `networkx`
Useful Package to handle graphs.

Install by running `conda install networkx`

```python
import networkx as nx
```

```python
G = nx.Graph()
G.add_nodes_from([1,2,3,4])
G.add_edge(1,2)
G.add_edge(2,3)
G.add_edge(3,1)
G.add_edge(3,4)
```

```python
nx.draw(G)
```

```python
G = nx.complete_graph(10)
nx.draw(G)
```

## `sympy`

Package for performing symbolic computation and manipulation.

Install it in your environment by running `conda install sympy`

```python
from sympy import *
```

```python
x,y = symbols("x y")
```

```python
expr = x+y**2
```

```python
x*expr
```

```python
expand(x*expr)
```

```python
factor(x**2 -2*x*y + y**2)
```

```python
latex(expr)
```

```python
init_printing()
```

```python
simplify( (x-y)**2 + (x+y)**2)
```

```python
x**2/(y**3+y)
```

```python
(x**2/(y**3+y)).subs(y,1/(1+x))
```

```python
(x**2/(y**3+y)).evalf(subs={'x':2, 'y':4})
```

```python
Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
I = Integral(exp(-x**2 - y**2), (x, -oo, oo), (y, -oo, oo))
```

```python
I.doit()
```

```python
(sin(x)/(1+cos(x)))
```

```python
(sin(x)/(1+cos(x))).series(x,0,10)
```

```python

```

## Exercises
The following exercises requires the combined usage of the packages we learnt today.

1. Generate 10 random polynomials of order 5
    - Numerically and analytically integrate them from 0 to 1 and compare the answers.
    - Compute one minima for each polynomial and show that the analytically computed derivative is 0 at the minima
    - Randomly sample the polynomials in the range from 0 to 1, and see if you can recover the original coefficents by trying to fit a 5th order polynomial to the samples.
2. Read and learn about [Erdos-Renyi Random Graphs](https://en.wikipedia.org/wiki/Erd%C5%91s%E2%80%93R%C3%A9nyi_model). See if you can numerically verify some of the properties mentioned in the wiki, such as for what parameter values is the graph most likely connected.

```python

```


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

23. Foco: $F(-7,3)$; diretriz: $x+3=0$

Arrumando a equação da diretriz

$d: x = -3 $

Fazendo um esboço é possível perceber que a parábola é paralela ao eixo $x$, logo sua equação é dada por $(y-k)^2 = 2p(x-h)$

Sabendo que a distância da diretriz até o foco é $p$, podemos calcular sua distância para achar $\frac{p}{2}$ usando o ponto$P(-3,3)$ da diretriz

$p = \sqrt{(-7-(-3))^2 + (3-3)^2}$

$p = \sqrt{-4^2 + 0}$

$p = \pm \sqrt{16}$

$p = -4$

$\frac{p}{2} = -2$

Somando $\frac{p}{2}$ no eixo $x$ da diretriz, obtemos as coordenadas do vértice

$V(-5,3)$

Substituindo agora os pontos dos vértice e o valor de $p$ na fórmula, temos que

$(y-3)^2 = 2 \cdot -4 \cdot (x+5)$

$(y-3)^2 = -8(x+5)$

$y^2 - 6y + 9 = -8x - 40$

$y^2 - 6y + 8x + 49 = 0$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y-3)**2, -8*(x+5)), (x,-20,20), (y,-20,20),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/23.ipynb


# Simple Electrical Circuit

This notebook is a simple tutorial on how to use the julia package `BondGraphs.jl` to simulate a simple electrical circuit.

This tutorial has been adapted from https://bondgraphtools.readthedocs.io/en/latest/tutorials/RC.html

```julia
# Since BondGraphs is not yet in the package manager, we will need to include it directly from Github
# NOTE: You will need Julia >= 1.7
using Pkg; Pkg.add(url="https://github.com/jedforrest/BondGraphs.jl")
using BondGraphs
```

Our first example will be a simple electric circuit of a capacitor, resistor, and current supply in parallel. We will first model this circuit without the current supply.

## Model Construction
We first create a `BondGraph` object which will hold all our components

```julia
model = BondGraph("RC Circuit")
```

    BondGraph RC Circuit (0 Nodes, 0 Bonds)

We then create all the components of our model. Each component contains a constitutive equation which builds our model

```julia
C = Component(:C)
constitutive_relations(C)
```

\begin{align}
0 =& \frac{q\left( t \right)}{C} - E_1\left( t \right) \\
\frac{dq(t)}{dt} =& F_1\left( t \right)
\end{align}

```julia
R = Component(:R)
constitutive_relations(R)
```

\begin{align}
0 =&  - R F_1\left( t \right) + E_1\left( t \right)
\end{align}

We also create an `EqualEffort` node which represents Kirchoff's Voltage Law for shared effort (voltage) between components

```julia
kvl = EqualEffort()
```

    𝟎

Components and nodes are added to the model, and connected together as a graph network

```julia
add_node!(model, [C, R, kvl])
connect!(model, R, kvl)
connect!(model, C, kvl)
model
```

    BondGraph RC Circuit (3 Nodes, 2 Bonds)

Our bond graph is fundamentally a graph object, and can be manipulated using Julia's graph functionality

```julia
using Graphs
adjacency_matrix(model)
```

    3×3 SparseArrays.SparseMatrixCSC{Int64, Int64} with 2 stored entries:
     ⋅  ⋅  1
     ⋅  ⋅  1
     ⋅  ⋅  ⋅

We can view our model structure by plotting the bond graph as a graph network. A plot recipe is available for `Plots.jl`

```julia
using Plots
plot(model, fontsize=12)
```

## Simulating our Model
With a bond graph we can automatically generate a series of differential equations which combine all the constitutive relations from the components, with efforts and flows shared according to the graph structure.

```julia
constitutive_relations(model)
```

\begin{align}
\frac{dC_{+}q(t)}{dt} =& \frac{ - C_{+q}\left( t \right)}{C_{+}C R_{+}R}
\end{align}

We set values for parameters in the graph model itself. Each component comes with default values.

When substituted into our equations, we get the following relation for the capacitor charge `C.q(t)`

```julia
C.C = 1
R.R = 2
constitutive_relations(model; sub_defaults=true)
```

\begin{align}
\frac{dC_{+}q(t)}{dt} =& \frac{ - C_{+q}\left( t \right)}{2}
\end{align}

We can solve this the bond graph directly using the in-built `simulate` function.

```julia
tspan = (0., 10.)
u0 = [1] # initial value for C.q(t)
sol = simulate(model, tspan; u0)
plot(sol)
```

Under the hood, our `simulate` function is converting our bond graph into an `ODESystem` from `ModelingToolkit.jl`.

We can chose instead to create an `ODESystem` directly and handle it with whatever tools we like.

```julia
using ModelingToolkit
sys = ODESystem(model)
display("text/plain", sys)
```

    Model RC Circuit with 1 equations
    States (1):
      C₊q(t) [defaults to 0.0]
    Parameters (2):
      C₊C [defaults to 1]
      R₊R [defaults to 2]
    Incidence matrix:1×2 SparseArrays.SparseMatrixCSC{Num, Int64} with 1 stored entry:
     ×  ⋅

## Expanding the circuit with a current supply
We will expand our model by adding an external current (flow) supply in parallel, represented by the component `Sf` (Source of Flow)

```julia
Is = Component(:Sf, "Is")
add_node!(model, Is)
connect!(model, Is, kvl)
plot(model, fontsize = 10)
```

The simplest external input is a constant. We set the parameter function of the current supply component to be `fs(t) = 2`. Note the additional '2' in the constitutive relation below

```julia
Is.fs = t -> 2 # Note that this is still a function of time
constitutive_relations(model)
```

\begin{align}
\frac{dC_{+}q(t)}{dt} =& \frac{ - C_{+q}\left( t \right) + 2.0 C_{+}C R_{+}R}{C_{+}C R_{+}R}
\end{align}

```julia
sol = simulate(model, tspan; u0)
plot(sol)
```

The forcing function can be more complex, such as `fs(t) = sin(2t)`

```julia
Is.fs = t -> sin(2t)
constitutive_relations(model; sub_defaults=true)
```

\begin{align}
\frac{dC_{+}q(t)}{dt} =& \frac{ - C_{+q}\left( t \right) + 2 \sin\left( 2 t \right)}{2}
\end{align}

```julia
sol = simulate(model, tspan; u0)
plot(sol)
```

The input can be any arbitrary julia function of t, so long as it returns a sensible output. Note that for this to work you must register the custom function with `@register_symbolic`, so that the library knows not to simplify this function further. Note the addition of `f(t)` in the equation.

```julia
f(t) = t % 2 <= 1 ? 0 : 1 # repeating square wave
@register_symbolic f(t)

Is.fs = t -> f(t)
constitutive_relations(model; sub_defaults=true)
```

\begin{align}
\frac{dC_{+}q(t)}{dt} =& \frac{ - C_{+q}\left( t \right) + 2 f\left( t \right)}{2}
\end{align}

```julia
sol = simulate(model, tspan; u0)
plot(sol)
```

Once a function `f(t)` is registered in in the model, it can be repeatedly modified and called without having to reconstruct the model

```julia
p = plot();
for i in 1:4
    f(t) = cos(i * t)
    sol = simulate(model, (0., 5.); u0)
    plot!(p, sol, label = "f(t) = cos($(i)t)", lw=2)
end
plot(p)
```


# Example: compare GS and SOR

Let's look at

$$
\begin{equation}
\begin{pmatrix}
  4  &  3 & 0 \\
  3  &  4 & -1 \\
  0  & -1 & 4
\end{pmatrix}
%
\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix}
%
= \begin{pmatrix} 24 \\ 30 \\ -24 \end{pmatrix} \nonumber
\end{equation}
$$

with both GS and SOR. In  both cases we'll start with $\vec{x}^{(0)} = [1, 1, 1]^T$.

The exact solution is $[3, 4, -5]^T$.

## Gauss Seidel

Recall:

for $i = 1, \dots, n$:
$$\qquad x^{(k+1)}_i = \frac{1}{a_{ii}}(b_i - \sum_{j=1}^{i-1} a_{ij} x_j^{(k+1)} - \sum_{j=i+1}^{n} a_{ij} x_j^{(k)}) \:.$$

Writing this out for our case gives

\begin{align}
x_1^{(k+1)} &= \frac{1}{4}(24 - 3 x_2^{(k)} - 0 x_3^{(k)})
= 6 - \frac{3}{4}x_2^{(k)} \nonumber \\
%
x_2^{(k+1)} &= \frac{1}{4}(30 - 3 x_1^{(k+1)} - (-1) x_3^{(k)})
= \frac{30}{4} - \frac{3}{4}x_1^{(k+1)} + \frac{1}{4}x_3^{(k)}\nonumber \\
%
x_3^{(k+1)} &= \frac{1}{4}(-24 - 0 x_1^{(k+1)} - (-1) x_2^{(k+1)})
= -6 + \frac{1}{4}x_2^{(k+1)}\nonumber
\end{align}

```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def GS(A, b, x_0, K, ref):

    # Define the size of our system
    n = len(x_0)

    # We've got an initial guess, x_0, and an iteration count to hit, K
    # To track progress through the iterations, we'll store all of our
    # estimates. You wouldn't typically do this, but we will for illustration.
    x = np.zeros((K+1, n))
    x[0] = x_0
    # We'd also like to track our error (since we know the real solution)
    error = np.zeros((K+1, n))
    error[0] = np.abs(ref - x[0])

    for k in range(0,K):

        for i in range(0, n):

            # Initialize the summation terms to zero on each iteration
            sum_1 = 0.0
            sum_2 = 0.0

            # the k+1 term from 0 to i-1
            for j in range(i):

                sum_1 = sum_1 + A[i][j] * x[k+1][j]

            # the k term from i+1 to n-1
            for j in range(i+1, n):

                sum_2 = sum_2 + A[i][j] * x[k][j]

            x[k+1][i] = 1.0/A[i][i] * (b[i] - sum_1 - sum_2)

        # compute the error
        error[k+1] = np.abs(ref - x[k+1])

    return x, error

x_0 = np.ones(3)
A = [[ 4,  3, 0], [3,  4, -1] ,[0, -1, 4]]
b = [24, 30, -24]
ref = [3, 4, -5]

x_GS, error_GS = GS(A, b, x_0, 4, ref)
print x_GS, "\n"
print error_GS

```

    [[ 1.          1.          1.        ]
     [ 5.25        3.8125     -5.046875  ]
     [ 3.140625    3.8828125  -5.02929688]
     [ 3.08789062  3.92675781 -5.01831055]
     [ 3.05493164  3.95422363 -5.01144409]]

    [[ 2.          3.          6.        ]
     [ 2.25        0.1875      0.046875  ]
     [ 0.140625    0.1171875   0.02929688]
     [ 0.08789062  0.07324219  0.01831055]
     [ 0.05493164  0.04577637  0.01144409]]

## Successive Over Relaxation

Recall:

for $i = 1, \dots, n$:
$$\qquad x^{(k+1)}_i = (1-\omega)x_i^{(k)} + \frac{\omega}{a_{ii}}(b_i - \sum_{j=1}^{i-1} a_{ij} x_j^{(k+1)} - \sum_{j=i+1}^{n} a_{ij} x_j^{(k)}) \:.$$

Let $\omega = 1.25 = 5/4$

Writing this out for our case gives

$$\begin{align}
x_1^{(k+1)} &= (1-\frac{5}{4})x_1^{(k)} + \frac{\frac{5}{4}}{4}(24 - 3 x_2^{(k)} - 0 x_3^{(k)})
= \frac{15}{2} - \frac{1}{4}x_1^{(k)} - \frac{15}{16}x_2^{(k)} \nonumber \\
%
x_2^{(k+1)} &= (1-\frac{5}{4})x_2^{(k)} + \frac{\frac{5}{4}}{4}(30 - 3 x_1^{(k+1)} - (-1) x_3^{(k)})
= \frac{150}{16} - \frac{15}{16}x_1^{(k+1)} - \frac{1}{4}x_2^{(k)} + \frac{5}{16}x_3^{(k)}\nonumber \\
%
x_3^{(k+1)} &= (1-\frac{5}{4})x_3^{(k)} + \frac{\frac{5}{4}}{4}(-24 - 0 x_1^{(k+1)} - (-1) x_2^{(k+1)})
= -\frac{15}{2} + \frac{5}{16}x_2^{(k+1)} - \frac{1}{4}x_3^{(k)}\nonumber
\end{align}$$

```
def SOR(A, b, x_0, w, K, ref):

    # Define the size of our system
    n = len(x_0)

    # We've got an initial guess, x_0, and an iteration count to hit, K
    # To track progress through the iterations, we'll store all of our
    # estimates. You wouldn't typically do this, but we will for illustration.
    x = np.zeros((K+1, n))
    x[0] = x_0
    # We'd also like to track our error (since we know the real solution)
    error = np.zeros((K+1, n))
    error[0] = np.abs(ref - x[0])

    for k in range(0,K):

        for i in range(0, n):

            # Initialize the summation terms to zero on each iteration
            sum_1 = 0.0
            sum_2 = 0.0

            # the k+1 term from 0 to i-1
            for j in range(i):

                sum_1 = sum_1 + A[i][j] * x[k+1][j]

            # the k term from i+1 to n-1
            for j in range(i+1, n):

                sum_2 = sum_2 + A[i][j] * x[k][j]

            x[k+1][i] = (1 - w) * x[k][i] + w/A[i][i] * (b[i] - sum_1 - sum_2)

        # compute the error
        error[k+1] = np.abs(ref - x[k+1])

    return x, error

x_SOR, error_SOR = SOR(A, b, x_0, 1.25, 4, ref)
print x_SOR, "\n"
print error_SOR
```

    [[ 1.          1.          1.        ]
     [ 6.3125      3.51953125 -6.65014648]
     [ 2.62231445  3.95852661 -4.60042381]
     [ 3.13330269  4.01026464 -5.09668635]
     [ 2.95705123  4.00748383 -4.97348972]]

    [[ 2.          3.          6.        ]
     [ 3.3125      0.48046875  1.65014648]
     [ 0.37768555  0.04147339  0.39957619]
     [ 0.13330269  0.01026464  0.09668635]
     [ 0.04294877  0.00748383  0.02651028]]

Let's compare with a plot or two

```
idx = [1, 2, 3]
plt.plot(idx, x_GS[4], 'b*')
plt.plot(idx, x_SOR[4], 'g+')
plt.plot(idx, ref, 'ko')
plt.axis([0, 4, -6, 6])
plt.legend(("GS", "SOR", "ref"), loc="lower left")
```

```
plt.axis([0.5, 4.5, 1e-2, 1e1])
for k in range(1,5):
    #print error_GS[k][0]
    plt.semilogy(k, error_GS[k][0], 'g*')
plt.title("GS error at point 1")
```

```
plt.axis([0.5, 4.5, 1e-2, 1e1])
for k in range(1,5):
    #print error_SOR[k][0]
    plt.semilogy(k, error_SOR[k][0], 'g*')
plt.title("SOR error at point 1")
```

```

```


# Complementarity and Optimal Stopping

- Prof. Jesse Perla, (jesse.perla@ubc.ca)
- Arnav Sood (arnav.sood@ubc.ca)

The goal of this notebook is to illustrate and compare different Julia approaches to solving [optimal stopping problems](https://en.wikipedia.org/wiki/Optimal_stopping).

In particular, we will be examining reformulations of these problems as [linear complementarity problems](https://en.wikipedia.org/wiki/Linear_complementarity_problem).

### Introduction to Optimal Stopping

----------

Optimal stopping problems concern a decision of when to exit some process and "accept" the current realization of some state variable. Many option problems (e.g., when to exercise a stock option at a current price, when to hire a worker at a current quality, etc.) take this form.

An familiar example for economists is the [McCall job-search model](https://en.wikipedia.org/wiki/Secretary_problem), whose central question ("when, in terms of the offer received, should I stop looking for a job?") is essentially the same thing.

### Vanilla (Bellman) Formulation (Continuous-Time, 1D)

-----------

The standard approach (as described by, e.g. [Stokey and Lucas](http://www.princeton.edu/~moll/HACTproject/option_simple.pdf)) is to write down a recursive problem.

Here's the value function:

$$
\begin{align}v(x) &= \max_{\tau \geq 0} \mathbb{E}_0 \big\{ e^{-\rho t} u(x_t)d_t + e^{-\rho \tau}S(x_\tau) \big\} \\
dx_t &= \mu(x_t) + \sigma(x_t)dW_t
\end{align}$$

In this setup:

- $\rho$ is the discount rate
- $x_t$ is the current state
- $W_t$ is some random shock, like a white noise process
- $\mu(x_t)$ is the **state-dependent** drift of the state
- $\sigma(x_t)$ is the **state-dependent** loading on the shock
- $u(x_t)$ is the instantaneous flow utility under the optimal policy
- $S(x_\tau)$ is the **state-dependent** value of stopping (i.e., think of this as "exiting" a search with the current realization of some state variable)

The traditional next step is to write a Bellman equation (or, more precisely in the continuous-time case, a Hamilton-Jacobi-Bellman equation):

$$ \rho v(x) = u(x) + \mu(x) v'(x) + \frac{\sigma^2(x)}{2} v''(x) $$

We can then solve this problem (after choosing a discrete grid of points) by iterating to a fixed point.

Let's set up some primitives.

```julia
using Parameters, LinearAlgebra, Distributions

# generator for stopping problem with default values
StoppingProblem = @with_kw (ρ = 0.05, # discount rate
                            μ = x -> -0.01 * x, # drift
                            σ = x -> 0.05 * x, # volatility
                            u = x -> x^(1/2), # flow utility
                            S = x -> 20. - 0.01 * x, # stopping value
                            W = Normal(0.0, 0.01), # shock
                            x_min = 0.0,
                            x_max = 10.0,
                            M = 300) # number of grid points
```

    #72 (generic function with 2 methods)

To use this generator:

```julia
default_problem = StoppingProblem();
@show default_problem.M; # should be 300
```

    default_problem.M = 300

```julia
new_problem = StoppingProblem(M = 400, ρ = 0.02);
@show new_problem.M, new_problem.ρ, new_problem.μ(1); # should be 400, 0.02, -0.1
```

    (new_problem.M, new_problem.ρ, new_problem.μ(1)) = (400, 0.02, -0.01)

So, parameters we explicitly change use the new values, and everything else uses the defaults.

```julia

```

### Stuff from Before

-------

```julia
# Diffusion operators with regular grids
function diffusionoperators(x::AbstractRange) # ξ = 0.0 corresponds to the unrescaled case from PTW
    Δ = step(x)
    P = length(x)

    dl_1 = zeros(P-1)
    d_1 = -ones(P)
    du_1 = ones(P-1)
    d_1[end] = d_1[end] + du_1[end]
    L_1_plus = Tridiagonal(dl_1, d_1, du_1)/Δ

    dl_m1 = -ones(P-1)
    d_m1 = ones(P)
    du_m1 = zeros(P-1)
    d_m1[1] = d_m1[1] + dl_m1[1]
    L_1_minus = Tridiagonal(dl_m1, d_m1, du_m1)/Δ

    dl_2 = ones(P-1)
    d_2 = -2 * ones(P)
    d_2[1] = -1
    d_2[end] = -1
    du_2 = ones(P-1)
    L_2 = Tridiagonal(dl_2, d_2, du_2)/(Δ^2)

    return (L_1_minus, L_1_plus, L_2)
end
```

    diffusionoperators (generic function with 1 method)

```julia
function LCP_objects(sp)
    # setup work
    @unpack ρ, μ, σ, S, W, x_min, x_max, M, W, u = sp
    grid = range(x_min, x_max, length = M) # instantiate grid
    L_1_minus, L_1_plus, L_2 = diffusionoperators(grid)
    A = Diagonal(μ.(grid)) * L_1_minus + 1/2 * Diagonal(σ.(grid))^2 * L_2
    B = ρ*I - A
    S_vec = S.(grid)
    q = -u.(grid) + B*S_vec
    return (A = A, B = B, S = S_vec, q = q)
end
```

    LCP_objects (generic function with 1 method)

```julia
A, B, S, q = LCP_objects(default_problem);
```

```julia
using PATHSolver
```

```julia
f = z -> B*z + q
n = default_problem.M
lb = zeros(n)
ub = 1000*ones(n) # a reasonable guess?
options(output = :no, time_limit = 600) # 10 minute budget
@time exit_code, sol_z, sol_f = solveLCP(f, lb, ub)
```

      0.283578 seconds (784.73 k allocations: 41.943 MiB, 7.80% gc time)
    Reading options file path.opt
    Read of options file complete.

    300 row/cols, 897 non-zeros, 1.00% dense.

    Path 4.7.03 (Thu Jan 24 15:44:12 2013)
    Written by Todd Munson, Steven Dirkse, and Michael Ferris

     ** EXIT - solution found.

    (:Solved, [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0  …  34.803, 34.8324, 34.8589, 34.8825, 34.9031, 34.9206, 34.9349, 34.9458, 34.9533, 34.9573], [1.0, 0.817101, 0.74133, 0.683184, 0.634161, 0.590969, 0.551919, 0.516007, 0.482579, 0.451182  …  2.30926e-14, -5.68878e-13, 7.16316e-13, -4.14779e-13, -8.81073e-13, -9.34364e-13, 6.72351e-13, -2.40252e-13, 1.69909e-12, -8.32667e-13])

```julia
using Plots
v = sol_z + S
plot(v, title = "Value Function for Vanilla Model", legend = false, xlabel = "x", ylabel = "v(x)")
```

```julia
using NLsolve
```

```julia
function f!(F, z)
    F .= B*z + q # in place assignment
end

@time r = mcpsolve(f!, lb, ub,
                zeros(n), # initial condition
                reformulation = :smooth, # uses a so-called "Fischer function" to smooth out the problem
                autodiff = :forward,
                ftol = 1e-12);
```

      4.466888 seconds (12.39 M allocations: 670.624 MiB, 9.50% gc time)

```julia
v = r.zero + S
plot(v, title = "Value Function for Vanilla Model", legend = false, xlabel = "x", ylabel = "v(x)")
```

## TBD

------

- upwind scheme
- 1D LCP (NLsolve, PATHSolver)
- Should really get some JuMP in there...
- Multiple dimensions
- Discretization
- probably simulate the process

```julia

```


# Seção 1.3 - Carga e Corrente

**1.1 Quantos coulombs são representados pelas seguintes quantidades de elétrons?**

(a) 6,482 * 10^17

(b) 1,24 * 10^18

(c) 2,46 * 10^19

(d) 1,628 * 10^20

```python
print("1.1")
carga_eletron = 1.602*10**(-19) #coulombs
n_eletrons = 1/carga_eletron
a = 6.482*10**17
b = 1.24*10**18
c = 2.46*10**19
d = 1.628*10**20
carga_a = -a / n_eletrons
carga_b = -b / n_eletrons
carga_c = -c / n_eletrons
carga_d = -d / n_eletrons
print("Carga (a):", carga_a, "C")
print("Carga (b):", carga_b, "C")
print("Carga (c):", carga_c, "C")
print("Carga (d):", carga_d, "C")
```

    1.1
    Carga (a): -0.10384164 C
    Carga (b): -0.198648 C
    Carga (c): -3.9409199999999998 C
    Carga (d): -26.08056 C

**1.2 Determine a corrente que flui por um elemento se o fluxo de carga for dado por:**

(a) q(t) = (3t + 8) mC

(b) q(t) = (8t^2 + 4t - 2) C

(c) q(t) = (3e^-t - 5e^-2t) nC

(d) q(t) = 10sen(120*pi*t) pC

(e) q(t) = 20e^-4tcos(50t) uC

```python
print("1.2")
import numpy as np
from sympy import *

t = symbols('t')
q = 3*t + 8
i = diff(q,t)
print("Resposta (a):", i, "mA")
q = 8*t**2 + 4*t - 2
i = diff(q,t)
print("Resposta (b):", i, "A")
q = 3*exp(-t) - 5*exp(-2*t)
i = diff(q,t)
print("Resposta (c):", i, "nA")
q = 10*sin(120*np.pi*t)
i = diff(q,t)
print("Resposta (d):", i, "pA")
q = 20*exp(-4*t)*cos(50*t)
i = diff(q,t)
print("Resposta (e):", i, "uA")
```

    1.2
    Resposta (a): 3 mA
    Resposta (b): 16*t + 4 A
    Resposta (c): -3*exp(-t) + 10*exp(-2*t) nA
    Resposta (d): 3769.91118430775*cos(376.991118430775*t) pA
    Resposta (e): -1000*exp(-4*t)*sin(50*t) - 80*exp(-4*t)*cos(50*t) uA

**1.3 Determine a carga q(t) que flui por um dispositivo se a corrente for:**

(a) i(t) = 3 A, q(0) = 1 C

(b) i(t) = (2t + 5) mA, q(0) = 0

(c) i(t) = 20cos(10t + pi/6) uA, q(0) = uC

(d) i(t) = 10e^(-30t)sen(40t) A, q(0) = 0

```python
print("1.3")
i = 3
q = integrate(i,t)
q0 = 1
print("Resposta (a):", q + q0, "C")
i = 2*t + 5
q = integrate(i,t)
q0 = 0
print("Resposta (b):", q + q0, "mC")
i = 20*cos(10*t + np.pi/6)
q = integrate(i,t)
q0 = 2
print("Resposta (c):", q + q0, "uC")
i = 10*exp(-30*t)*sin(40*t)
q = integrate(i,t)
q0 = 0
print("Resposta (d):", q + q0, "C")
```

    1.3
    Resposta (a): 3*t + 1 C
    Resposta (b): t**2 + 5*t mC
    Resposta (c): 2*sin(10*t + 0.523598775598299) + 2 uC
    Resposta (d): -3*exp(-30*t)*sin(40*t)/25 - 4*exp(-30*t)*cos(40*t)/25 C

**1.4 Uma corrente de 7,4 A passa por um condutor. Calcule
qual a carga que passa através de qualquer seção transversal desse condutor em 20 s.**

```python
print("1.4")

from sympy import *
i = 7.4
t = symbols('t')
tf = 20
q = integrate(i,(t,0,tf))
print("Carga total é:", q, "C")
```

    1.4
    Carga total é: 148.000000000000 C

**1.5 Determine a carga total transferida ao longo do intervalo 0 <= t <= 10 s quando i(t) = 1/2t A.**

```python
print("1.5")
from sympy import *
t = symbols('t')
i = (1/2)*t
ti = 0 #tempo inicial
tf = 10 #tempo final
q = integrate(i,(t,ti,tf))
print("Carga total é:", q, "C")
```

    1.5
    Carga total é: 25.0000000000000 C

**1.6 A carga que entra em determinado elemento é mostrada na Figura 1.23. Determine a corrente em:**

(a) t = 1ms

(b) t = 6ms

(c) t = 10ms

```python
print("1.6")
from sympy import *
t = symbols('t')
qa = (30/2)*t #intervalo de 0ms a 2ms
i = diff(qa,t)
print("Resposta (a):", i.subs(t,1), "mA")
qb = 30 #intervalo de 2ms a 8ms
i = diff(qb,t)
print("Resposta (b):", i.subs(t,6), "mA")
qc = (-30/4)*t #intervalo de 8ms a 12ms
i = diff(qc,t)
print("Resposta (c):", i.subs(t,10), "mA")
```

    1.6
    Resposta (a): 15.0000000000000 mA
    Resposta (b): 0 mA
    Resposta (c): -7.50000000000000 mA

**1.7 A carga que flui por um fio é representada na Figura 1.24. Represente a corrente correspondente.**

```python
print("1.7")
import matplotlib.pyplot as plt
from sympy import *
from scipy import signal
import numpy as np
import matplotlib.pyplot as plt

t = np.linspace(0, 8, 500)
triangle = 50*signal.sawtooth(np.pi * 0.5 * t, 0.5)
plt.plot(t, triangle)
plt.xlabel('tempo(s)')
plt.ylabel('carga(C)')
plt.show()

i = np.diff(triangle)
t = np.linspace(0,8,499)
plt.plot(t,i)
plt.xlabel('tempo(s)')
plt.ylabel('corrente(A)')
plt.show()
```

**1.8 A corrente que flui por um ponto em um dispositivo é mostrada na Figura 1.25. Calcule a carga total através do ponto.**

```python
print("1.8")
from sympy import *

t = symbols('t')
i1 = 10*t
q1 = integrate(i1,(t,0,1)) #carga do intervalo 0ms a 1ms
i2 = 10
q2 = integrate(i2,(t,1,2)) #carga do intervalo 1ms a 2msb
print("Carga total:", q1+q2, "mC")
```

    1.8
    Carga total: 15 mC

**1.9 A corrente através de um elemento é ilustrada na Figura 1.26. Determine a carga total que passa pelo elemento em:**

(a) t = 1s

(b) t = 3s

(c) t = 5s

```python
print("1.9")
from sympy import *

t = symbols('t')
i1 = 10
q1 = integrate(i1,(t,0,1))
print("Resposta (a):", q1, "C")
i2 = 10 - 5*t
q2 = integrate(i2,(t,0,1))
i3 = 5
q3 = integrate(i3,(t,0,1))
print("Resposta (b):", q1+q2+q3, "C")
q3 = integrate(i3,(t,0,2))
i4 = 5 - 5*t
q4 = integrate(i4,(t,0,1))
print("Resposta (c):", q1+q2+q3+q4, "C")
```

    1.9
    Resposta (a): 10 C
    Resposta (b): 45/2 C
    Resposta (c): 30 C

# Seções 1.4 e 1.5 Tensão, potência e energia


# Working with Sampling Distributions
Most statistical analysis involves working with distributions - usually of sample data.

## Sampling and Sampling Distributions
As we discussed earlier, when working with statistics, we usually base our calculations on a sample and not the full population of data. This means we need to allow for some variation between the sample statistics and the true parameters of the full population.

In the previous example, we knew the probability that a security search would be triggered was 25%, so it's pretty easy to calculate that the expected value for a random variable indicating the number of searches per 100 passengers is 25. What if we hadn't known the probability of a search? How could we estimate the expected mean number of searches for a given number of passengers based purely on sample data collected by observing passengers go through security?

### Creating a Proportion Distribution from a Sample
We know that the each passenger will either be searched or not searched, and we can assign the values ***0*** (for not searched) and ***1*** (for searched) to these outcomes. We can conduct a Bernoulli trial in which we sample 16 passengers and calculate the fraction (or *proportion*) of passengers that were searched (which we'll call ***p***), and the remaining proportion of passengers (which are the ones who weren't searched, and can be calculated as ***1-p***).

Let's say we record the following values for our 16-person sample:

    0,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0

In this sample, there were 3 searches out of 16 passengers; which as a proportion is 3/16 or 0.1875. This is our proportion (or **p**); but because we know that this is based on a sample, we call it **p̂** (or p-hat). The remaining proportion of passengers is 1-p; in this case 1 - 0.1875, which is 0.8125.

The data itself is *qualitative* (categorical) - we're indicating "no search" or "search"; but because we're using numeric values (0 and 1), we can treat these values as numeric and create a binomial distribution from them - it's the simplest form of a binomial distribution - a Bernoulli distribution with two values.

Because we're treating the results as a numberic distribution, we can also calculate statistics like *mean* and *standard deviation*:

To calculate these, you can use the following formulae:

\begin{equation}\mu_{\hat{p}} = \hat{p}\end{equation}

\begin{equation}\sigma_{\hat{p}} = \sqrt{\hat{p}(1-\hat{p})}\end{equation}

The mean is just the value of **p̂**, so in the case of the passenger search sample it is 0.1875.

The standard deviation is calculated as:

\begin{equation}\sigma_{\hat{p}} = \sqrt{0.1875 \times 0.8125} \approx 0.39\end{equation}

We can use Python to plot the sample distribution and calculate the mean and standard deviation of our sample like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

searches = np.array([0,1,0,0,1,0,0,0,0,0,0,0,1,0,0,0])

# Set up the graph
plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
print('Mean: ' + str(np.mean(searches)))
print('StDev: ' + str(np.std(searches)))
```

When talking about probability, the *mean* is also known as the *expected value*; so based on our single sample of 16 passengers, should we expect the proportion of searched passengers to be 0.1875 (18.75%)?

Well, using a single sample like this can be misleading because the number of searches can vary with each sample. Another person observing 100 passengers may get a (very) different result from you. One way to address this problem is to take multiple samples and combine the resulting means to form a *sampling* distribution. This will help us ensure that the distribution and statistics of our sample data is closer to the true values; even if we can't measure the full population.

### Creating a Sampling Distribution of a Sample Proportion
So, let's collect mulitple 16-passenger samples - here are the resulting sample proportions for 12 samples:

| Sample | Result |
|--------|--------|
| p̂1| 0.1875 |
| p̂2| 0.2500 |
| p̂3| 0.3125 |
| p̂4| 0.1875 |
| p̂5| 0.1250 |
| p̂6| 0.3750 |
| p̂7| 0.2500 |
| p̂8| 0.1875 |
| p̂9| 0.3125 |
| p̂10| 0.2500 |
| p̂11| 0.2500 |
| p̂12| 0.3125 |

We can plot these as a sampling distribution like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

searches = np.array([0.1875,0.25,0.3125,0.1875,0.125,0.375,0.25,0.1875,0.3125,0.25,0.25,0.3125])

# Set up the graph
plt.xlabel('Search Results')
plt.ylabel('Frequency')
plt.hist(searches)
plt.show()
```

#### The Central Limit Theorem
You saw previously with the binomial probability distribution, with a large enough sample size (the *n* value indicating the number of binomial experiments), the distribution of values for a random variable started to form an approximately *normal* curve. This is the effect of the *central limit theorem*, and it applies to any distribution of sample data if the size of the sample is large enough. For our airport passenger data, if we collect a large enough number of samples, each based on a large enough number of passenger observations, the sampling distribution will be approximately normal. The larger the sample size, the closer to a perfect *normal* distribution the data will be, and the less variance around the mean there will be.

Run the cell below to see a simulated distribution created by 10,000 random 100-passenger samples:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

# Plot the distribution as a histogram
means = df['p-hat']
means.plot.hist(title='Simulated Sampling Distribution')
plt.show()
print ('Mean: ' + str(means.mean()))
print ('Std: ' + str(means.std()))
```

### Mean and Standard Error of a Sampling Distribution of Proportion
The sampling distribution is created from the means of multiple samples, and its mean is therefore the mean of all the sample means. For a distribution of proportion means, this is considered to be the same as **p** (the population mean). In the case of our passenger search samples, this is 0.25.

Because the sampling distribution is based on means, and not totals, its standard deviation is referred to as its *standard error*, and its formula is:

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{p(1-p)}{n}}\end{equation}

In this formula, *n* is the size of each sample; and we divide by this to correct for the error introduced by the average values used in the sampling distribution. In this case, our samples were based on observing 16-passengers, so:

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{16}} \approx 0.11\end{equation}

In our simulation of 100-passenger samples, the mean remains 0.25. The standard error is:

\begin{equation}\sigma_{\hat{p}} = \sqrt{\frac{0.25 \times 0.75}{100}} \approx 0.043\end{equation}

Note that the effect of the central limit theorem is that as you increase the number and/or size of samples, the mean remains constant but the amount of variance around it is reduced.

Being able to calculate the mean (or *expected value*) and standard error is useful, because we can apply these to what we know about an approximately normal distribution to estimate probabilities for particular values. For example, we know that in a normal distribution, around 95.4% of the values are within two standard deviations of the mean. If we apply that to our sampling distribution of ten thousand 100-passenger samples, we can determine that the proportion of searched passengers in 95.4% of the samples was between 0.164 (16.4%) and 0.336 (36.6%).

How do we know this?

We know that the mean is ***0.25*** and the standard error (which is the same thing as the standard deviation for our sampling distribution) is ***0.043***. We also know that because this is a *normal* distribution, ***95.4%*** of the data lies within two standard deviations (so 2 x 0.043) of the mean, so the value for 95.4% of our samples is 0.25 ± (*plus or minus*) 0.086.

The *plus or minus* value is known as the *margin of error*, and the range of values within it is known as a *confidence interval* - we'll look at these in more detail later. For now, run the following cell to see a visualization of this interval:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

n, p, s = 100, 0.25, 10000
df = pd.DataFrame(np.random.binomial(n,p,s)/n, columns=['p-hat'])

# Plot the distribution as a histogram
means = df['p-hat']
m = means.mean()
sd = means.std()
moe1 = m - (sd * 2)
moe2 = m + (sd * 2)

means.plot.hist(title='Simulated Sampling Distribution')

plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(moe1, color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(moe2, color='magenta', linestyle='dashed', linewidth=2)
plt.show()
```

### Creating a Sampling Distribution of Sample Means
In the previous example, we created a sampling distribution of proportions; which is a suitable way to handle discrete values, like the number of passengers searched or not searched. When you need to work with continuous data, you use slightly different formulae to work with the sampling distribution.

For example, suppose we want to examine the weight of the hand luggage carried by each passenger. It's impractical to weigh every bag that is carried through security, but we could weigh one or more samples, for say, 5 passengers at a time, on twelve occassions. We might end up with some data like this:

| Sample | Weights |
|--------|---------|
| 1      | [4.020992,2.143457,2.260409,2.339641,4.699211] |
| 2      | [3.38532,4.438345,3.170228,3.499913,4.489557] |
| 3      | [3.338228,1.825221,3.53633,3.507952,2.698669] |
| 4      | [2.992756,3.292431,3.38148,3.479455,3.051273] |
| 5      | [2.969977,3.869029,4.149342,2.785682,3.03557] |
| 6      | [3.138055,2.535442,3.530052,3.029846,2.881217] |
| 7      | [1.596558,1.486385,3.122378,3.684084,3.501813] |
| 8      | [2.997384,3.818661,3.118434,3.455269,3.026508] |
| 9      | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 10     | [2.532509,3.064274,3.32908,2.981303,3.915995] |
| 11     | [4.078268,2.283018,3.606384,4.555053,3.344701] |
| 12     | [2.532509,3.064274,3.32908,2.981303,3.915995] |

Just as we did before, we could take the mean of each of these samples and combine them to form a sampling distribution of the sample means (which we'll call **X**, and which will contain a mean for each sample, which we'll label x̄n):

| Sample | Mean Weight |
|--------|---------|
| x̄1 | 3.092742  |
| x̄2 | 3.7966726 |
| x̄3 | 2.98128   |
| x̄4 | 3.239479  |
| x̄5 | 3.36192   |
| x̄6 | 3.0229224 |
| x̄7 | 2.6782436 |
| x̄8 | 3.2832512 |
| x̄9 | 3.5734848 |
| x̄10 | 3.1646322 |
| x̄11 | 3.5734848 |
| x̄12 | 3.1646322 |

We can plot the distribution for the sampling distribution like this:

```python
%matplotlib inline
from matplotlib import pyplot as plt
import numpy as np

meanweights = np.array([3.092742,
                        3.7966726,
                        2.98128,
                        3.239479,
                        3.36192,
                        3.0229224,
                        2.6782436,
                        3.2832512,
                        3.5734848,
                        3.1646322,
                        3.5734848,
                        3.1646322])

# Set up the graph
plt.xlabel('Mean Weights')
plt.ylabel('Frequency')
plt.hist(meanweights, bins=6)
plt.show()

print('Mean: ' + str(meanweights.mean()))
print('Std: ' + str(meanweights.std()))
```

Just as before, as we increase the sample size, the central limit theorem ensures that our sampling distribution starts to approximate a normal distribution. Our current distribution is based on the means generated from twelve samples, each containing 5 weight observations. Run the following code to see a distribution created from a simulation of 10,000 samples each containing weights for 500 passengers:

>This may take a few minutes to run. The code is not the most efficient way to generate a sample distribution, but it reflects the principle that our sampling distribution is made up of the means from multiple samples. In reality, you could simulate the sampling by just creating a single sample from the ***random.normal*** function with a larger ***n*** value.

```python

```

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))

# data will hold all of the sample data
data = np.array([])

# sampling will hold the means of the samples
sampling = np.array([])

# Perform 10,000 samples
for s in samples:
    # In each sample, get 500 data points from a normal distribution
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

# Create a dataframe with the sampling of means
df = pd.DataFrame(sampling, columns=['mean'])

# Plot the distribution as a histogram
means = df['mean']
means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.show()

# Print the Mean and StdDev for the full sample and for the sampling distribution
print('Sample Mean: ' + str(data.mean()))
print('Sample StdDev: ' + str(data.std()))
print ('Sampling Mean: ' + str(means.mean()))
print ('Sampling StdErr: ' + str(means.std()))
```

### Mean and Variance of the Sampling Distribution

The following variables are printed beneath the histogram:

- **Sample Mean**: This is the mean for the complete set of sample data - all 10,000 x 500 bag weights.
- **Sample StdDev**: This is the standard deviation for the complete set of sample data - all 10,000 x 500 bag weights.
- **Sampling Mean**: This is the mean for the sampling distribution - the means of the means!
- **Sampling StdErr**: This is the standard deviation (or *standard error*) for the sampling distribution

If we assume that **X** is a random variable representing every possible bag weight, then its mean (indicated as **μx**) is the population mean (**μ**). The mean of the **X** sampling distribution (which is indicated as **μx̄**) is considered to have the same value. Or, as an equation:

\begin{equation}\mu_{x} = \mu_{\bar{x}}\end{equation}

In this case, the full population mean is unknown (unless we weigh every bag in the world!), but we do have the mean of the full set of sample observations we collected (**x̄**), and if we check the values generated by Python for the sample mean and the sampling mean, they're more or less the same: around 3.2.

To find the standard deviation of the sample mean, which is technically the *standard error*, we can use this formula:

\begin{equation}\sigma_{\bar{x}}  = \frac{\sigma}{\sqrt{n}}\end{equation}

In this formula, ***σ*** is the population standard deviation and ***n*** is the size of each sample.

Since our the population standard deviation is unknown, we can use the full sample standard deviation instead:

\begin{equation}SE_{\bar{x}} \approx \frac{s}{\sqrt{n}}\end{equation}

In this case, the standard deviation of our set of sample data is around 1.2, and we have used 500 variables in each sample to calculate our sample means, so:

\begin{equation}SE_{\bar{x}} \approx \frac{1.2}{\sqrt{500}} = \frac{1.2}{22.36} \approx 0.053\end{equation}

## Confidence Intervals
A confidence interval is a range of values around a sample statistic within which we are confident that the true parameter lies. For example, our bag weight sampling distribution is based on samples of the weights of bags carried by passengers through our airport security line. We know that the mean weight (the *expected value* for the weight of a bag) in our sampling distribution is 3.2, and we assume this is also the population mean for all bags; but how confident can we be that the true mean weight of all carry-on bags is close to the value?

Let's start to put some precision onto these terms. We could state the question another way. What's the range of weights within which are confident that the mean weight of a carry-on bag will be 95% of the time? To calculate this, we need to determine the range of values within which the population mean weight is likely to be in 95% of samples. This is known as a *confidence interval*; and it's based on the Z-scores inherent in a normal distribution.

Confidence intervals are expressed as a sample statistic ± (*plus or minus*) a margin of error. To calculate the margin of error, you need to determine the confidence level you want to find (for example, 95%), and determine the Z score that marks the threshold above or below which the values that are *not* within the chosen interval reside. For example, to calculate a 95% confidence interval, you need the critical Z scores that exclude 5% of the values under the curve; with 2.5% of them being lower than the values in the confidence interval range, and 2.5% being higher. In a normal distribution, 95% of the area under the curve is between a Z score of ± 1.96. The following table shows the critical Z values for some other popular confidence interval ranges:

| Confidence  | Z Score |
|-------------|---------|
| 90%         | 1.645   |
| 95%         | 1.96    |
| 99%         | 2.576   |

To calculate a confidence interval around a sample statistic, we simply calculate the *standard error* for that statistic as described previously, and multiply this by the approriate Z score for the confidence interval we want.

To calculate the 95% confidence interval margin of error for our bag weights, we multiply our standard error of 0.053 by the Z score for a 95% confidence level, which is 1.96:

\begin{equation}MoE = 0.053 \times 1.96 = 0.10388 \end{equation}

So we can say that we're confident that the population mean weight is in the range of the sample mean ± 0.10388 with 95% confidence. Thanks to the central limit theorem, if we used an even bigger sample size, the confidence interval would become smaller as the amount of variance in the distribution is reduced. If the number of samples were infinite, the standard error would be 0 and the confidence interval would become a certain value that reflects the true mean weight for all carry-on bags:

\begin{equation}\lim_{n \to \infty} \frac{\sigma}{\sqrt{n}} = 0\end{equation}

In Python, you can use the *scipy.stats.**norm.interval*** function to calculate a confidence interval for a normal distribution. Run the following code to recreate the sampling distribution for bag searches with the same parameters, and display the 95% confidence interval for the mean (again, this may take some time to run):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from scipy import stats

mu, sigma, n = 3.2, 1.2, 500
samples = list(range(0, 10000))

# data will hold all of the sample data
data = np.array([])

# sampling will hold the means of the samples
sampling = np.array([])

# Perform 10,000 samples
for s in samples:
    # In each sample, get 500 data points from a normal distribution
    sample = np.random.normal(mu, sigma, n)
    data = np.append(data,sample)
    sampling = np.append(sampling,sample.mean())

# Create a dataframe with the sampling of means
df = pd.DataFrame(sampling, columns=['mean'])

# Get the Mean, StdDev, and 95% CI of the means
means = df['mean']
m = means.mean()
sd = means.std()
ci = stats.norm.interval(0.95, m, sd)

# Plot the distribution, mean, and CI
means.plot.hist(title='Simulated Sampling Distribution', bins=100)
plt.axvline(m, color='red', linestyle='dashed', linewidth=2)
plt.axvline(ci[0], color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(ci[1], color='magenta', linestyle='dashed', linewidth=2)
plt.show()

# Print the Mean, StdDev and 95% CI
print ('Sampling Mean: ' + str(m))
print ('Sampling StdErr: ' + str(sd))
print ('95% Confidence Interval: ' + str(ci))
```


## Principal Component Analysis
- Here we are going to revisit PCA and introduce some new concepts along the way

###  Uses
- Feature extraction
- Used as input into other models
- Visualization of high dimensions in lower spaces
- Examples:
    - Can use to visualize differences in MNIST $(28\times 28)$ numbers
    - Consider 1080p or 720p images -> millions of pixels
    - Genome $3\times 10^9$base pairs
        - Lots of data
        - Need to be able to work with high dimensional data
- Question How do we go from miliions of sequences to 2 or 3 dimensions?
    - Example: DNA sequences from Ancestry or 23AndMe

### How PCA works
- PCA is a linear transformation:
$$
Z = Q^Tx
$$
where $Z$ is the output matrix, $Q^T$ is the affine transformation and $x$ is the input
- Matrix $Q$ allows rotation and scaling of $X$
- Variance is information
    - Directions of high variability indicate most amount of information
- PCA results in orthogonal transformation of the input space
     - Linear combinatio of inputs
     - Arrange columns such that largest variaces in collumns appear first
     - Signal to Noise selection
         - Large vairations in the data are signal
         - Small variations in the data are noise
         - Select only columns such that $\sum Var_i\approx 95\%,98\%,99\%,...$ of variance

### Derivation
- We'll derive principal components in 2 steps
    1. Calculate the empiricale covariance matrix
    2. Decompose the empiricale covariance matrix through eigen decomposition

#### Calculating the empirical covariance matrix
- Suppose we have the observation matrix $X_{N\times M}$ then
$$
\Sigma_X =\frac{1}{N}(X-\mu_X)^T(X-\mu_X)\quad;\quad \mu_X = \frac{\sum_{i=1}^N X_{ij}}{N}
$$
$\Sigma_X$ is the empricale covariance matrix and $\mu_X$ is empircale mean

#### Eigendecomposition and selection of Principal components
- By definition the eigendecomposition of the empricale covariance matrix is one such that:
$$
\Sigma_X V = V \Lambda \quad;\quad
\Lambda =
\begin{bmatrix}
\lambda_1& 0 & \cdots & 0 \\
0& \lambda_2 & \cdots & 0 \\
\vdots&\vdots&\ddots&\vdots\\
0& 0 & \ddots & \lambda_N \\
\end{bmatrix}
$$
- By definition $V^T$ is orthogonal to $V$ which implies $V^T=V^{-1}$, which means that by multiplying equation by $V^T$ we have:
$$
\Sigma_X = V\Lambda V^{-1} = V\Lambda V^T \Leftrightarrow V^T\Sigma_XV = \Lambda
$$
- Then for some linear transformation $Z = XQ$ we have $\mu_Z = \mu_XQ$ which implies:
$$
\begin{align}
\Sigma_Z &= \frac{1}{N}(Z-\mu_Z)^T(Z-\mu_Z) = \frac{1}{N}(XQ-\mu_Z)^T(XQ-\mu_Z) \\
& = \frac{1}{N}(XQ-\mu_XQ)^T(XQ-\mu_XQ) = \frac{1}{N}Q^T(X-\mu_X)^T(X-\mu_X)Q\\
&=Q^T\Sigma_XQ
\end{align}
$$
- It follows that if $Q^TQ=I$ then
$$
\Sigma_Z = \Lambda
$$
and $\Sigma_Z$ is our principal component matrix

#### Reconstruction error
- In unsupervised machine learning our objective function is the __reconstruction error__
- It is the function against which we "optimize" our values
- The reconstrcution error for PCA is defined as:
$$
J = \sum_n(x_n-x_{reconstructed,n})^2
$$
- The reconstuction is the loss from using our approximation
- Selecting the top $k$ eigenvalues and associated principal vectors we re-write the reconstruction error through the Frobenius norm:
$$
J = |X- Z_kQ_k^{-1}| = |X- XQ_kQ_k^T|_F^2 \quad\mbox{since } Q_k^{-1}=Q_k^T \mbox{ and } Z_k = XQ_k
$$
where $X\in \mathbb{R}^{N\times D}$ and $Q\in \mathbb{R}^{D\times K}$
- Selecting the eignevectors $Q_i$ associated with the $k$ top principal component yields the principal compnent algorithm, that is
$$
J_min = \min_k|X-XQ_kQ_k^T|_F^2\qquad s.t. k = \lambda_1>\lambda_2>...>\lambda_k
$$

```python
# ---- PCA: Example ----
import keras as k
import numpy as np
from matplotlib.mlab import PCA

```

    c:\users\munol\appdata\local\programs\python\python35\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters
    Using TensorFlow backend.

### Limitations of PCA
- Only allows linear projections
    - Co-variance matrix is of size dxd. If $d=10^5$, then $|\Sigma|= 10^{10}$
    - __Solution__: singular value decomposition (SVD)
- PCA restricts to orthogonal vectors in feature space that minimize reconstruction error
     - __Solution__: independent component analysis (ICA) seeks directions that are statistically independent
- Assumes points are multivariate Gaussian
     - __Solution:__ Kernel PCA that transforms input data to other spaces

### Interpretation of PCA
- Two common interpretations of PCA are through
    1. Maximum variance direction - find the vectors of the PC such that the projection of the data onto this direction maximizes variances (out of all possible dimensional projections)
    2. Minimum reconstruction error - find the PC vectors such that the projection of the data onto these vectors minimizes the reconstruction error

- In both cases PCA provides an understanding of our data in some transformed __latent space__
- Frequently these latent spaces don't have an interpretable understanding
- Intuitively the idea of linear decomposition of our data is attractive

```python
#### Eigenfaces Example ####
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_lfw_people
from sklearn.metrics import classification_report
from sklearn.decomposition import PCA
from sklearn.neural_network import MLPClassifier

# Load data
lfw_dataset = fetch_lfw_people(min_faces_per_person=100)

_, h, w = lfw_dataset.images.shape
X = lfw_dataset.data
y = lfw_dataset.target
target_names = lfw_dataset.target_names

# split into a training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

# Compute a PCA
n_components = 100
pca = PCA(n_components=n_components, whiten=True).fit(X_train)

# Calculate the eignefaces
eigenfaces = pca.components_.reshape((n_components, h, w))

# apply PCA transformation
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)
```

```python
# Visualization
def plot_gallery(images, titles, h, w, n_row=3, n_col=4):
    """Helper function to plot a gallery of portraits"""
    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))
    plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
    for i in range(n_row * n_col):
        plt.subplot(n_row, n_col, i + 1)
        plt.imshow(images[i].reshape((h, w)), cmap=plt.cm.gray)
        # plt.title(titles[i], size=12)
        plt.xticks(())
        plt.yticks(())

def titles(y_test, target_names):
    for i in range(target_names.shape[0]):
        true_name = target_names[y_test[i]].split(' ')[-1]
        yield 'Names:{name}'.format(name = true_name)
```

```python
eigenface_titles = ["eigenface %d" % i for i in range(0,11)]
plot_gallery(eigenfaces, eigenface_titles, h, w)

plt.show()
```

- We can take a linear composition of these faces to generate, with 100 components we recreate the images
- As an exercise try increasing the number of PCA components

```python
reconstructed_names = target_names[y_train[0:4]]
X_projected_all = pca.inverse_transform(X_train_pca)
X_projected = X_projected_all[0:4]
loss = ((X_train - X_projected_all) ** 2).mean()
plot_gallery(X_projected, reconstructed_names, h, w, n_row=1, n_col=4)
print("The total training loss is {}".format(loss))
```

```python
# True values
plot_gallery(X_train[0:12], reconstructed_names, h, w, n_row=1, n_col=4)
```

Materials/L6 - Unsupervised Methods in DL/PCA.ipynb


# Section 2.2 $\qquad$ Solving Linear Systems

### Example 1

Solve the linear system whose augmented matrix has the echelon form:
$$
  \left[
     \begin{array}{ccc|c}
       1 & 2 & 0 & 3 \\
       0 & 1 & 1 & 2 \\
       0 & 0 & 1 & -1 \\
     \end{array}
   \right]
$$

```python
from numpy import *
from numpy.linalg import solve

A = array([[1, 2, 0], [0, 1, 1], [0, 0, 1]]);
b = array([3, 2, -1]);

solve(A, b)
```

    array([-3.,  3., -1.])

To solve a linear system $A\mathbf{x}=\mathbf{b}$,
- form the augmented matrix

- transform to a matrix $[C~|~\mathbf{d}]$ in

- find solution of the linear system w.r.t. $[C~|~\mathbf{d}]$ using

**Remark:**

If $A$ is $n\times n$, and the linear system $A\mathbf{x}=\mathbf{b}$ has a unique solution, then

### Example 2

Solve the linear system
\begin{eqnarray*}
  x+2y+3z &=& 9 \\
  2x-y+z &=& 8 \\
  3x-z &=& 3
\end{eqnarray*}

```python
from numpy import *
from numpy.linalg import solve

A = array([[1, 2, 3], [2, -1, 1], [3, 0, -1]]);
b = array([9, 8, 3]);

solve(A,b)
```

    array([ 2., -1.,  3.])

In general, $A$ is $m\times n$ and the augmented matrix $[C~|~\mathbf{d}]$ might have the following form
\begin{equation*}
  \left[
    \begin{array}{ccccc|c}
      1 & c_{12} & \cdots &  & c_{1n} &  d_1 \\
      0 & 1 & c_{23} & \cdots &   c_{2n} & d_2 \\
      \vdots & \vdots & \ddots & \ddots &  &  \\
      0 & 0  &  & 1 & c_{k-1,n} & d_{k-1} \\
      0 & 0 & \cdots & 0 &  c_{k,n}& d_k \\
      0 & 0 & \cdots &  &  0& d_{k+1} \\
      \vdots & \vdots &  &  &\vdots  &  \\
      0 & 0 & \cdots &  &  0& d_{m} \\
    \end{array}
  \right]
\end{equation*}
- If $d_{k+1} = 1$,

- If $d_{k+1} = 0$, $c_{k,n} = 1$

- If $c_{k,n} = 0$

### Example 3

Let
\begin{equation*}
  [C~|~\mathbf{d}] = \left[
        \begin{array}{ccccc|c}
          1 & 2 & 3 & 4 & 5 & 6 \\
          0 & 1 & 2 & 3 & -1 & 7 \\
          0 & 0 & 1 & 2 & 3 & 7 \\
          0 & 0 & 0 & 1 & 2 & 9 \\
        \end{array}
      \right].
\end{equation*}

### Example 4

Let
\begin{equation*}
  [C~|~\mathbf{d}] = \left[
        \begin{array}{cccc|c}
          1 & 2 & 3 & 4 & 5 \\
          0 & 1 & 2 & 3 & 6 \\
          0 & 0 & 0 & 0 & 1 \\
        \end{array}
      \right].
\end{equation*}

### Example 5

Solve the linear system
\begin{eqnarray*}
% \nonumber to remove numbering (before each equation)
  x+2y+3z &=& 6 \\
  2x-3y+2z &=& 14 \\
  3x+y-z &=& -2
\end{eqnarray*}

```python
from numpy import *
from numpy.linalg import solve

A = array([[1, 2, 3], [2, -3, 2], [3, 1, -1]]);
b = array([6, 14, -2]);

solve(A,b)
```

    array([ 1., -2.,  3.])

## Homogeneous System

### Example 6

Solve the linear system
\begin{eqnarray*}
% \nonumber to remove numbering (before each equation)
  x+y+z+w &=& 0 \\
  x+w &=& 0 \\
  x+2y+z &=& 0
\end{eqnarray*}

```python
from sympy import *

x, y, z, w = symbols('x y z w');

eq1 = x + y + z + w;
eq2 = x + w;
eq3 = x + 2*y + z;

solve([eq1, eq2, eq3], (x, y, z, w))
```

    {x: -w, y: w, z: -w}

**Remark:**

## Relationship between nonhomogeneous and homogeneous systems

Let $A\mathbf{x}=\mathbf{b}$, $\mathbf{b}\neq\mathbf{0}$ be a consistent system. The associated homogeneous system is

-

-