# Supervised Learning: Regression

[Regression](https://en.wikipedia.org/wiki/Regression_analysis) analysis is a statistical process for estimating the relationships between two or more variables. The relationship is modeled as

$$y \sim x$$
or
$$y = f(x)\text{.}$$

Both model descriptions indicate  that the variable $y$ is a function  of $x$. Therefore the variable $y$ is denoted as **response variable** or **dependent variable**, whereas the variable $x$ is denoted as **predictor variable** or **independent variable**.

The function $f(x;\beta)$ is called a **regression function** and $\beta$ are its free parameters.

Source: [xkcd](https://xkcd.com/605/)

**Import libraries**

```python
%matplotlib inline
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
```

## The machine learning model development process

## Data preparation

In this section we work with the *trees* data set, a data set that ships with the [R-package `datasets`](https://stat.ethz.ch/R-manual/R-devel/library/datasets/html/trees.html) (Ryan et al. 1976). The *trees* data set provides measurements of the girth, height and volume of timber in 31 felled black cherry trees, also known as [*Prunus serotina*](https://en.wikipedia.org/wiki/Prunus_serotina). The height of the trees is given in feet (ft) and the volume is given in in cubic feet (cft). The girth is the diameter of the tree (in inches) measured at 4 ft 6 in (approx. 1.37 m) above the ground.

We load the data set using pandas `read_csv` and take a quick look at the data set by applying the `head` function.

```python
trees_raw = pd.read_csv("../datasets/trees.csv", index_col=0)
print(trees_raw.shape)
trees_raw.head()
```

For a better understanding we convert the variables in meters, respectively, cubic meters, by applying these equations

$$ 1\; \text{in} = 0.0254\;\text{m,}$$

$$ 1\; \text{ft} = 0.3048\;\text{m,}$$

and

$$ 1\; \text{cft} = 0.0283168\;\text{m}^3\text{.}$$

We write a function to convert the features `Girth`,`Height` and `Volume` from inches, feet and cubic feet into meters and cubic meters and apply it on the data set.

```python
def convert2meter(s, input_unit="in"):
    '''
    Function to convert inches, feet and cubic feet to meters and cubic meters
    '''
    if input_unit == "in":
        return s*0.0254
    elif input_unit == "ft":
        return s*0.3048
    elif input_unit == "cft":
        return s*0.0283168
    else:
        print("Error: Input unit is unknown.")
```

```python
## Apply function
measurement_unit = {"Girth":"in",
                    "Height":"ft",
                    "Volume":"cft"}

trees = trees_raw.copy()
for feature in ["Girth", "Height", "Volume"]:
    trees[feature] = trees_raw[feature].apply(lambda x: convert2meter(s=x,
                                 input_unit=measurement_unit.get(feature)))
```

```python
trees.head()
```

It is always a good idea to visualize the data we want a work with. However, instead of a scatter plot, which is fine for the comparison of two variables, we are going to plot a scatter plot matrix to account for multiple variables. The `seaborn` library provides the handy `pairplot` function for plotting scatter plot matrices.

```python
sns.pairplot(trees);
```

A quick visual inspection indicates that there is an approximately linear relationship between girth and to somehow a less pronounced linear relationship between height and volume. Any relation between height and girth is less obvious.

> The goal in this example is to build a linear regression model with `Volume` being the dependent variable and `Height` and `Girth` being the independent (explanatory) variables.

## Train-(Validation)-Test-Split

We write a function to split our data set into two partitions: a train set and a test set.

```python
def train_test_split(df, ratio=0.3, rs=42):
    np.random.seed(rs)
    idx = np.random.choice([True, False], size=df.shape[0],
                           replace=True, p=[1-ratio, ratio])
    train = df.loc[idx,:]
    test = df.loc[~idx,:]
    print("Train set: {}".format(train.shape))
    print("Test set: {}".format(test.shape))
    return train, test
```

```python
train, test = train_test_split(trees, ratio=0.4, rs=42)
```

```python
fig, ax = plt.subplots(figsize=(9,4))
train.plot.scatter(x="Girth", y="Volume", ax=ax,
                   alpha=0.5, s=60, color="r", label="Training set")
test.plot.scatter(x="Girth", y="Volume", ax=ax,
                  alpha=0.5, s=60, color="g", label="Test set");
```

## Learning Algorithm

### Hypothesis and Model Parameters

#### Simple Linear Regression

In [**linear regression**](https://en.wikipedia.org/wiki/Simple_linear_regression) models a relationship between the response variable $y$ and and the predictor variable $x$ is given in form of a **linear** equation

$$y = \beta_0 + \beta_1 x + \epsilon$$

where $\beta_0$ and $\beta_1$ are constants and $\epsilon$ is a **random error term** added to the linear model equation, thus resulting in a probabilistic model. The number $\beta_0$ is called **intercept**, sometimes referred to as *bias, shift* or *offset*, and defines the point of intersection of the line  and the $y$-axis $(x=0)$. The number $\beta_1$ is called **regression coefficient**. It is a measure of the slope of the **regression line**. Thus, $\beta_1$ indicates how much the $y$-value changes when the $x$-value increases by 1 unit. The error term $\epsilon_i$, sometimes referred to as *residuals*,  is assumed to consist of independent normal distributed values, $e_i \sim N(0, \sigma^2)$.

The error $\epsilon_i$ for each particular pair of values ($x_i,y_i$) is computed by the difference of the observed value $y_i$ and the predicted value given by $\hat y_i$.

$$\epsilon_i = y_i - \hat y_i$$

Depending on the data $\epsilon_i$ is a negative number if $y_i$ plots below the regression line or it is a positive number if $y_i$ plots above the regression line.

#### Multiple Linear Regression

A [multiple linear regression](https://en.wikipedia.org/wiki/Regression_analysis) model is a generalization of the simple linear regression model.

By extending the equation $y = \beta_0 + \beta_1 x + \epsilon$ to a set of $n$ observations and $d$ explanatory variables, $x_1,\cdots, x_d$ the regression model can be written as

$$
\begin{align}
y_i & = \beta_0 + \sum_{j=1}^dx_{ij}\beta_j + \epsilon_i \\
& = \beta_0 + \beta_1x_{1i} + \beta_2x_{2i} +\cdots + \beta_dx_{di} + \epsilon_i \text{.}
\quad i = 1,2,\dots n\text{, } x\in \mathbb R^d\text{.}
\end{align}
$$

#### Matrix Notation

A regression model based upon $n$ observations (measurements) consists of of $n$ response variables, $y_1, y_2,...y_n$. For the ease of notation we write the response variables as a one-dimensional column vector of the size $y_{n \times 1}$.

$$
\begin{align}
\mathbf y_{n \times 1}=
       \begin{bmatrix}
        y_{1} \\
        y_{2} \\
        \vdots \\
        y_{n} \\
        \end{bmatrix}
\end{align}
$$

Moreover, for each particular observation $x_i$ $(x_1, x_2,..., x_n)$ we represent the associated explanatory variables $d$ as a column vector as well.

$$
\begin{align}
\mathbf x_{i}=
       \begin{bmatrix}
        x_{i1} \\
        x_{i2} \\
        \vdots \\
        x_{id} \\
        \end{bmatrix}
\end{align}
\text{(e.g.)}  \Rightarrow
  \begin{bmatrix}
  \text{height} \\
  \text{weight} \\
  \vdots \\
  \text{age} \\
  \end{bmatrix}
$$

Further, by transposing $x_i$ we stack a set of $n$ observation vectors into a matrix $X$ of the form $x_{n \times d}$:

$$
\begin{align}
 \mathbf X_{n \times d}=
        \begin{bmatrix}
        -x_{1}^T - \\
        -x_{2}^T -\\
        \vdots \\
        -x_{n1}-\\
        \end{bmatrix}
        =
       \begin{bmatrix}
        x_{11} & x_{12} & \cdots & x_{1d} \\
        x_{21} & x_{22} & \cdots & x_{2d} \\
        \vdots & \vdots & \ddots & \vdots \\
        x_{n1} & x_{n2} & \cdots & x_{nd} \\
        \end{bmatrix}
\end{align}
$$

This matrix notation is very similar to a spreadsheet representation, where each row corresponds to an observation and each column to a feature. Please note that we assume that all features are continuous-valued $(x \in \mathbb R^d)$ and that there are more observations than dimensions $(n > d)$.

For linear regression (and classification) the intercept term $\beta_0$ does not interact with any element in the vector $x \in \mathbb R^d$. Thus it is convenient to populate the first column of the matrix $\mathbf X$ with ones:

$$
\begin{align}
 \mathbf X_{n \times d+1}=
       \begin{bmatrix}
        1 & x_{11} & x_{12} & \cdots & x_{1d} \\
        1 &x_{21} & x_{22} & \cdots & x_{2d} \\
        \vdots &\vdots & \vdots & \ddots & \vdots \\
        1 & x_{n1} & x_{n2} & \cdots & x_{nd} \\
        \end{bmatrix}
\end{align}
$$

Note that the matrix $\mathbf X$ is now of the size $n \times d+1$ and that $\beta\in \mathbb R^{d+1}$, and thus $[\beta_0, \beta_1,...,\beta_d]^T$ is of size $d+1 \times 1$. The vector $\mathbf y$ is denoted as **target vector** and the matrix $\mathbf X$ is called the **feature matrix**.

$$
\begin{align}
\mathbf y_{n \times 1}=
       \begin{bmatrix}
        y_{1} \\
        y_{2} \\
        \vdots \\
        y_{n} \\
        \end{bmatrix}\text{,}
\quad
\mathbf \beta_{d+1 \times 1}^T=
       \begin{bmatrix}
        \beta_0 \\
        \beta_1 \\
        \vdots \\
        \beta_d  \\
        \end{bmatrix}
\end{align}
$$

Recalling the basics of [**Linear Algebra**](https://en.wikipedia.org/wiki/Linear_algebra), we know that the [*dot product*](https://en.wikipedia.org/wiki/Dot_product), also referred to as the *scalar product* of a vector $\mathbf u_{1\times n}$ with a vector $\mathbf v_{n\times 1}$ results in a scalar $w$ ($w$ may be written as a matrix of size $1\times1$). The value of the scalar $w$ corresponds to the sum of the products of the elements in the vectors $\mathbf u$ and $\mathbf v$.

$$\mathbf u \cdot \mathbf v = \sum_{i=1}^n  u_1v_1+u_2v_2+...+ u_nv_n$$

### Objective function (The loss function)

Recall the equation for the multiple regression model

$$
y_i = \beta_0 + \sum_{j=1}^dx_{ij}\beta_j + \epsilon_i  \text{,}
\quad i = 1,2,...n\text{, } x\in \mathbb R^d
$$

In order to find a appropriate set of parameters $\beta$ we need to define an [**objective function**](https://en.wikipedia.org/wiki/Mathematical_optimization) $(\mathcal L)$.

One of the the most popular estimation method is the so-called [ordinary least squares (OSL) method](https://en.wikipedia.org/wiki/Ordinary_least_squares). Thereby the coefficients $\beta_0, \beta_1,\beta_2,...,\beta_{d}$ are picked by minimizing the residual sum of squares.

$$\mathcal L = \sum_{i=1}^n\epsilon_i^2=\sum_{i=1}^n(y_i - f(x_i; \beta))^2$$

By plugging in the regression model equation from above we get

$$\mathcal L = \sum_{i=1}^n(y_i - \beta_0 + \sum_{j=1}^dx_{ij}\beta_j)^2\text{,}$$
where $n$ corresponds to the number of observations and $d$ corresponds to the number of features of our data set.

We write a function to construct the target vector, `y` and the feature matrix, `X`.

```python
def model_matrix(data, y_col, x_col):
    if not isinstance(x_col, list):
        x_col = [x_col]

    # target vector
    y = data[y_col].values.reshape(-1,1)
    print("Target vector y: {}".format(y.shape))

    # feature matrix
    X = data.copy()
    X["beta0"]= 1
    X = X[["beta0"]+x_col].values
    print("Feature matrix X: {}".format(X.shape))
    return X, y
```

```python
X_train, y_train = model_matrix(train,
                                y_col="Volume",
                                x_col=["Height", "Girth"])

X_test, y_test = model_matrix(test,
                              y_col="Volume",
                              x_col=["Height", "Girth"])
```

```python
y_train
```

#### Optimization

Returning to our problem we see that we may rewrite the original least squares objective function

$$\mathcal L = \sum_{i=1}^n(y_i - \beta_0 + \sum_{j=1}^dx_{ij}\beta_j)^2\text{,}$$

by using the matrix notation as

$$\mathcal L = \sum_{i=1}^n(y_i - x_i^T \beta)^2$$

The representation in matrix form allows us to further simplify the equation by plugging in the response vector $\mathbf y$ and the model matrix $\mathbf X$:

$$\mathcal L = \sum_{i=1}^n(y_i - x_i^T \beta)^2 = \Vert\mathbf y - \mathbf X\beta\Vert^2 = (\mathbf y - \mathbf X\beta)^T(\mathbf y - \mathbf X\beta)$$
Now, in order to find the least squares solution we take the gradient with respect to $\beta$ and find a unique solution $\hat \beta$:

$$\nabla_\beta\mathcal L=2\mathbf X^T\mathbf X \beta-2\mathbf X^T\mathbf y \quad \Rightarrow \quad\hat \beta =  (\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y$$

> __Challenge:__ Implement the one step learning algorithm using `numpy`'s `dot` and `linalg.inv` functions.

```python
## your code here ...
```

```python
# %load ../src/_solutions/one_step_learning.py
```

```python
beta_hat = one_step_learning(X_train, y_train)
beta_hat
```

$$ y = \beta_0 + \beta_1x + \beta_2x $$

$$ \text{Volume} = -1.70 + 0.04 \times \text{Height} + 5.05 \times \text{Girth}$$

#### Predictions

Given new data $\mathbf X_{new}$, the least squares prediction for $\hat y$ is

$$\hat y = \mathbf X_{new}\hat \beta = \mathbf X_{new}(\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y$$

Be aware that the calculation of $\hat \beta = (\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y$ assumes $(\mathbf X^T\mathbf X)^{-1}$ exists. Therefore $\mathbf X^T\mathbf X$ has to be a full rank matrix. A matrix is full rank when the $n \times (d + 1)$ matrix $\mathbf X$ has at least $d + 1$ linearly independent rows. This means that any point in $\mathbb R^{d+1}$ can be reached by a weighted combination of $d + 1$ rows of $\mathbf X$. Thus, if $n < d + 1$ least squares fails as if $(\mathbf X^T\mathbf X)^{-1}$ does not exist, as there are an infinite number of possible solutions.

> __Challenge:__ Write a function to make predictions.

```python
## your code here ...
```

```python
# %load ../src/_solutions/predict.py
```

```python
h=22
g=1.2
predict(np.array([1, h, g]), beta_hat)
```

```python
XX = trees.sample(8, random_state=42).copy().reset_index(drop=True)
XX
```

```python
prediction_set, _ = model_matrix(XX, "Volume", ["Height", "Girth"])
prediction_set
```

```python
XX.loc[:,"Predicted Volume"] = pd.Series(predict(prediction_set, beta_hat).ravel())
XX
```

**Predictions on the test set, the so far not seen data**

```python
predictions = predict(X_test, beta_hat)
predictions
```

### Evaluation metric

Recall that out optimization approach is based on minimizing the **sum of squared errors (SSE)**:

$$SSE = \sum e^2 = \sum (\hat y - y)^2\text{.}$$

A widely applied metric to evaluate the goodness-of-fit of the model is the [**root-mean-square error (RMSE)**](https://en.wikipedia.org/wiki/Root-mean-square_deviation) defined by

$$RMSE = \sqrt{\frac{\sum_{i=1}^n (\hat y - y)^2}{n}}\text{.}$$

> __Challenge:__ Write a function to compute the RMSE.

```python
## your code here ...
```

```python
# %load ../src/_solutions/rmse.py
```

```python
rmse(predictions, y_test)
```

***
$
\begin{array}{lll}
\text{Summary} & \\
\hline
\text{Hypothesis} & \text{Linear model} & \bf{\hat y} = \bf x^T \beta \\
\text{Model parameters} & \beta_0, \beta_1, \beta_2 &\\
\text{Hyperparameters} & \text{None}  &\\
\text{Objective function} & \text{Sum of squared errors (SSE)} & \sum (\hat y - y)^2 \\
\text{Optimization} & \text{One step learning (analytic approach)} & \hat \beta =  (\mathbf X^T\mathbf X)^{-1}\mathbf X^T\mathbf y \\
\text{Evaluation metric} & \text{Root mean square error (RMSE)} &\sqrt{\frac{\sum_{i=1}^n (\hat y - y)^2}{n}} \\
\hline
\end{array}
$


# Lecture 8

## Complex Numbers

```python
import numpy as np
import sympy as sp
import scipy.integrate
sp.init_printing()
##################################################
##### Matplotlib boilerplate for consistency #####
##################################################
from ipywidgets import interact
from ipywidgets import IntSlider
from ipywidgets import FloatSlider
from matplotlib import pyplot as plt

import cmath

%matplotlib inline

from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')

global_fig_width = 8
global_fig_height = global_fig_width / 1.61803399
font_size = 12

plt.rcParams['axes.axisbelow'] = True
plt.rcParams['axes.edgecolor'] = '0.8'
plt.rcParams['axes.grid'] = True
plt.rcParams['axes.labelpad'] = 8
plt.rcParams['axes.linewidth'] = 2
plt.rcParams['axes.titlepad'] = 16.0
plt.rcParams['axes.titlesize'] = font_size * 1.4
plt.rcParams['figure.figsize'] = (global_fig_width, global_fig_height)
plt.rcParams['font.sans-serif'] = ['Computer Modern Sans Serif', 'DejaVu Sans', 'sans-serif']
plt.rcParams['font.size'] = font_size
plt.rcParams['grid.color'] = '0.8'
plt.rcParams['grid.linestyle'] = 'dashed'
plt.rcParams['grid.linewidth'] = 2
plt.rcParams['lines.dash_capstyle'] = 'round'
plt.rcParams['lines.dashed_pattern'] = [1, 4]
plt.rcParams['xtick.labelsize'] = font_size
plt.rcParams['xtick.major.pad'] = 4
plt.rcParams['xtick.major.size'] = 0
plt.rcParams['ytick.labelsize'] = font_size
plt.rcParams['ytick.major.pad'] = 4
plt.rcParams['ytick.major.size'] = 0
##################################################
```

## Imaginary numbers

**The Imaginary Number $i$:**

- The polynomial $~~x^2-1=0~~$ has two real roots:  $~~1~~$ and $~~-1~~$
- The polynomial $~~x^2+1=0~~$ has **no** real roots.

Consider solving:
$$x^2 = 1\qquad\mbox{and}\qquad x^2 = -1$$

- We introduce an "imaginary" number $~~i~~$ so that there are two solutions to $~~x^2 = -1~~$: $~~i~~$ and $~~-i~~$.  That is, $~~i~~$ is a number with the property that $~~i^2=-1$.

## Complex numbers

The complex numbers are  the set of all expressions of the form $a + bi$ where $i^2=-1$ and $a$ and $b$ are real numbers:

$$\mathbb{C}=\left\{a + bi~~\vert~~a,~b~\in\mathbb{R}\right\}$$

For $z=a+bi\in\mathbb{C}$ we define the *real* and *imaginary*
parts of $z$ to be ${\Re}(z) = a$ and $\Im(z)=b$.

The imaginary number $i$ has no home on the real number line.
Instead, we locate it on the **complex plane** at the point $(0,1)$.

- we can represent any complex number $z=a+bi$  as the
point $(a,b)$ in the complex plane.
- The coordinates $(a,b)$ are
usually called the *cartesian* coordinates for $z$.  (Named
after the mathematician and philosopher Rene Descartes).
- In this
plane, the real numbers lie on the horizontal axis.  We usually
refer to the horizontal axis of $\mathbb{C}$ as the **real axis**.

```python
def plot_imag_axis(a,b):
    plt.scatter([a],[b])
    plt.plot([0,a],[b,b],ls='--')
    plt.axvline(x=0,c='k')
    plt.axhline(y=0,c='k')
    plt.plot([0,a],[b,b],ls='--')
    plt.plot([a,a],[0,b],ls='--')
    plt.xlabel('Real Axis')
    plt.ylabel('Imaginary Axis')
    plt.xlim(left=-5, right=25)
    plt.ylim(bottom=-5, top=25)
```

```python
interact(plot_imag_axis, a = IntSlider(value=10, min=0, max=20, continuous_update=False), b = IntSlider(value=10, min=0, max=20, continuous_update=False))
```

## Complex conjugates

- The complex plane $\mathbb{C}$ contain **all** of the roots of **every** polynomial.

E.g.

$$x^2-8x+25 = 0 =ax^2 +bx +c \iff x= {-b \pm\sqrt{b^2-4ac}\over 2a} $$

$$={{8\pm\sqrt{64-100}}\over2}={{8\pm\sqrt{-36}}\over2}={{8\pm
6i}\over2}=4\pm3i$$

- Note that these two roots are reflections of one another through the
real axis. They are *conjugates* of one another.

- In general, let $z=a + bi$. The **conjugate** of $z$ is the complex number
  $\bar{z}=a-bi$.

We can also use Sympy's `solve` method to solve polynomials:

```python
x = sp.symbols('x')
sp.solve(x**2 - 8*x + 25)

```

## Modulus (size) of a complex number

- The distance to a point on the complex plane from 0 is called its **modulus**, and we find this by calculating the hypotenuse of the triange with base  ${\Re}(z)$ and height $\Im(z)$:

E.g. The modulus of $4\pm3i$ is $\sqrt{3^2+4^2}=\sqrt{9+16}=\sqrt{25}=5$

- In general, the **modulus** of $z=a+bi$ is the real number
  $|z|=\sqrt{a^2+b^2}$.

- The **modulus** is connected to the **conjugate** by means of the formula
  $z\cdot \bar{z}=|z|^2$.  Indeed:

\begin{align}
    z\cdot\bar{z}&=(a+bi)(a-bi)=a^2-(bi)^2=a^2-b^2\cdot i^2\\
                 &=a^2-b^2(-1)=a^2+b^2=|z|^2
\end{align}

## Complex numbers in Python

```python
x = 1 + 2j
print(f'x = {x}     Re(x) = {x.real}     Im(x) = {x.imag}     |x| = {abs(x)}')

```

Complex numbers using `Sympy`:

```python
x = 1 + 2 * sp.I
print(f'x = {x}     Re(x) = {sp.re(x)}     Im(x) = {sp.im(x)}     |x| = {sp.Abs(x)}')

```

## Addition and Subtraction:

Addition and subtraction of complex numbers work as you would expect:

$$(a+bi)\pm(c+di)=(a\pm c) + (b\pm d)i$$

and

$$-(a+bi)=-a-bi$$

Try adding: $(5+6i)+(1-i)$=

```python
print((5 + 6j) + (1 - 1j))

```

Try subtracting: $(5+6i)-(1-i)$=

```python
print((5 + 6j) - (1 - 1j))

```

## Multiplication:

Multiplication is not quite so convenient in cartesian coordinates:

\begin{align*}
    (a+bi)(c+di)&=ac + adi + bci + bidi \\ &= ac + adi + bci -bd \\ &=
    (ac-bd)+(ad+bc)i
    \end{align*}

Try multiplying: $(5+6i)(1-i)$=

```python
print((5 + 6j) * (1 - 1j))

```

## Division:

Division is even more awkward in cartesian
coordinates: we have to multiply the numerator and the denominator
by the complex conjugate of the denominator.

\begin{align*}
    {{a+bi}\over{c+di}}&={{(a+bi)(c-di)}\over{(c+di)(c-di)}}\\
    &={{(ac+bd)+(bc-ad)i}\over{c^2+d^2}}=\left({{ac+bd}\over{c^2+d^2}}\right)+
\left({{bc-ad}\over{c^2+d^2}}\right)i
    \end{align*}

Try dividing: $${(-4+7i)\over (2+3i)}=$$

```python
print((-4 + 7*sp.I) / (2 + 3*sp.I))

```

## Polar Coordinates

It's often convenient to represent  the complex number
$z = a + bi$ in terms of its polar coordinates $\langle r,\theta
\rangle$.
- The angle $\theta$ is called the *argument* of $z$.

- The real number $r=|z|$ is sometimes denoted mod$(z)$.

```python
ax = plt.subplot(121)
ax.plot([0,1],[0,1],'o-')
ax = plt.subplot(122,projection='polar')
ax.plot([0,np.pi/4.0],[0,np.sqrt(2)],'o-')
```

## Connection between cartesian and polar

Let $z=x+iy$.   If we are given the polar coordinates of
$z$ and want to express the cartesian coordinates use
$$\displaylines{x=r\cos\theta\cr
    y=r\sin\theta\cr
    z=r\cos\theta + ri\sin\theta=r(\cos\theta + i\sin\theta)\cr}$$

If we are given the cartesian coordinates and want to find the polar
coordinates use:

$$
r={\rm mod}(z)=|z|=\sqrt{x^2+y^2}
$$

\begin{align*}
\theta={\rm arg}(z)=\tan^{-1}{y\over x}=
\begin{cases}\pi/2,&\mbox{ if }~~ x=0,y>0 \\
    -\pi/2,&\mbox{ if }~~ x=0,y<0 \\
    \arctan\left({y\over x}\right),&\mbox{ if }~~ x>0\\
    \arctan\left({y\over x}\right)+\pi,&\mbox{ if }~~ x<0, y\geq 0\\
    \arctan\left({y\over x}\right)-\pi,&\mbox{ if }~~ x<0, y<0\\
   \end{cases}
    \end{align*}

<!--N.B. All of the fuss about the value of $\theta$ in the formula
above is to make sure that $z$ gets into the proper quadrant. Beware
of the sign of this tangent: depends on which quadrant you are in.
The positive $x$ axis is defined as having $\theta=0$ and positive
$\theta$ goes in an anticlockwise sense around the $xy$
plane.-->

## Some examples:

- Find the cartesian coordinates for the complex number $z$ with polar
  coordinates $r=2$ and $\theta=\pi/6$.

$$\Re(z)=x=r\cos\theta=2\cos(\pi/6)=2\left({{\sqrt{3}\over2}}\right)=\sqrt{3}$$
$$\Im(z)=y=r\sin\theta=2\sin(\pi/6)=2\left({{1\over2}}\right)=1$$
$${\rm Therefore\ } z = \sqrt{3} + i$$

```python
print(cmath.rect(2, np.pi/6))

```

## Some examples:
- Find the polar coordinates for the complex number $z= -3+4i$.

$$|z|=r = $$

$$\sqrt{(-3)^2+4^2}=\sqrt{25}=5$$

$${\rm arg}(z)=\theta=\arctan\left({{y}\over{x}}\right)=$$

$$-0.93+\pi{\rm ~radians}\approx 127^\circ$$

```python
print(cmath.polar(-3 + 4j))

```

## Some examples:
- Find the polar coordinates for the complex number $z= -2i$.

$${\rm mod}(z)=r = |z|=2$$

$${\rm arg}(z)=\theta=-{{\pi}\over2}$$

```python
print(cmath.polar(-2j))

```

## Multiplication in Polar Coordinates:

First a reminder of three useful and important identities:

$$\cos^2\theta + \sin^2\theta = 1$$

$$\cos(\theta_1+\theta_2)=\cos\theta_1\cos \theta_2 - \sin\theta_1\sin\theta_2$$

$$\sin(\theta_1+\theta_2)=\sin\theta_1\cos \theta_2 + \sin\theta_2\cos\theta_1$$

Now let $z_1=r_1\cos\theta_1+ir_1\sin\theta_1$ and $z_2=r_2\cos\theta_2+ir_2\sin\theta_2$.

We first compute the real part of the product $z_1\cdot z_2$:

\begin{align*}
\Re(z_1\cdot z_2) &= r_1\cos\theta_1\cdot r_2\cos\theta_2 - r_1\sin\theta_1\cdot
r_2\sin\theta_2\cr
&=r_1r_2(\cos\theta_1\cos\theta_2 - \sin\theta_1\sin\theta_2)\cr
&=r_1r_2\cos(\theta_1 + \theta_2)
\end{align*}

Note that for the real part the moduli have been multiplied and the
arguments added.

Now we compute the imaginary part of $z_1\cdot z_2$:

\begin{align*}
\Im(z_1\cdot z_2) &= r_1\sin\theta_1\cdot r_2\cos\theta_2 + r_2\sin\theta_2\cdot
    r_1\cos\theta_1\cr
&=r_1r_2(\sin\theta_1\cos\theta_2 - \sin\theta_2\cos\theta_1)\cr
&=r_1r_2\sin(\theta_1 + \theta_2)
    \end{align*}

For the imaginary part too, the  moduli multiply while the arguments
add.

This gives a relatively compact and highly geometric result
for the product:

$$z_1\cdot z_2 = r_1r_2(\cos(\theta_1 + \theta_2)+i\sin(\theta_1 + \theta_2))$$

It is **multiplicative** in the modulus and **additive** in the argument:

$$|z_1z_2|= |z_1\cdot |z_2|$$
$$\arg(z_1z_2)=\arg (z_1)+ \arg( z_2)$$

This means that when we multiply by $z$, we are **rotating** through the angle $\arg(z)$ and  **radially stretching** by a factor of $|z|$.

## A Remarkable Connection with $e^{i\theta}$:

First, think of $z=\cos\theta + i\sin\theta$ as a function of $\theta$ and differentiate with respect to $\theta$:

$${\rm~~(1)~~~~}\frac{{\rm d}z}{{\rm d}\theta}=\frac{{\rm d}}{{\rm d}\theta}\left(\cos\theta+i\sin\theta\right)=-\sin\theta+i\cos\theta$$

Next notice that the right-hand side is just the product $iz$:

$${\rm~~(2)~~~~}iz=i(\cos\theta+i\sin\theta)=i\cos\theta+i^2\sin\theta=-\sin\theta+i\cos\theta$$

Thus from (1) and (2) $$\frac{{\rm d}z}{{\rm d}\theta}=iz$$

This is a separable differential equation. Thus:
$$\int{dz\over iz}=\int d\theta~~~~~~~~~\Rightarrow~~~~~~{1\over
i}~\ln z =\theta +c~~~~\Rightarrow~~~~\ln z =i\theta +ic$$ In
exponential form:
$$z=e^{i\theta+ic}~~=~~e^{i\theta}~e^{ic}~~=~~Ae^{i\theta}~~~~{\rm~ with}~~~~
A=e^{ic}$$

When $\theta=0$, $z=1$ giving $A=1$, so:
$$z=\cos\theta + i \sin\theta=e^{i\theta}~~~~~~~~~~~~\rm (1)$$

Similarly we can show that:
$$z=\cos\theta - i \sin\theta=e^{-i\theta}~~~~~~~~~~~~\rm (2)$$

Adding  (1) and (2) and subtracting (1) and (2) gives:
$$\cos\theta ={e^{i\theta}+ e^{-i\theta}\over 2}~~~~~~~~~~~~~~~~~~~~~~~~~
\sin\theta ={ e^{i\theta}-e^{-i\theta}\over 2i}$$

Thus **any complex number** can now be written:
$$z=x+iy=r(\cos\theta + i\sin\theta)=r~e^{i\theta}$$

Several important consequences:

1. Any complex number can be written in the polar form $z = re^{i\theta}$ where $r=|z|$ and $\theta=\arg(z)$.

2. The unit circle in $\mathbb{C}$ consists exactly of those complex numbers with
   modulus equal to 1, that is the numbers $e^{i\theta}$.

3. Multiplication on the unit circle $r=1$ can be carried out by adding the
   angles:

$$e^{i\theta_1}\cdot e^{i\theta_2} = e^{i(\theta_1+\theta_2)}$$

$$z=x+iy=r(\cos\theta + i\sin\theta)=r~e^{i\theta}$$

Other important consequences:

4. Exponentiation on the unit circle $r=1$ can be done by multiplying the angle
   by the index:

$$\left(e^{i\theta}\right)^n = e^{i\theta n}=e^{i(n\theta)}$$

5. This result is known as **DeMoivre's Theorem**.  It is usually stated in its
   cartesian form:

$$(\cos\theta + i\sin\theta)^n=\cos(n\theta) + i\sin(n\theta)$$

6.  Finally, the famous identity by Leonhard Euler

$$e^{\pi i}+1=0$$


## 11. Differential Equations

[](https://www.youtube.com/watch?v=HReAo38LoM4&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy "Python Data Science")

Specific types of equations with differential terms arise from fundamental relationships such as conservation of mass, energy, and momentum. For example, the accumulation of mass $\frac{dm}{dt}$ in a control volume is equal to the mass inlet $\dot m_{in}$ minus mass outlet $\dot m_{out}$ from that volume.

$\frac{dm}{dt} = \dot m_{in} - \dot m_{out}$

Dynamic models can either be regressed (identified) from data or developed without data with fundamental relationships. Even fundamental relationships may have unknown or uncertain parameters. A combined approach for dynamic system modeling is to use fundamental physics-based relationships with data science. This approach uses the best features of both methods. It creates a model that aligns to measured values and extrapolates to regions where there is no or little data.

The first exercise is to [solve differential equations](https://apmonitor.com/pdc/index.php/Main/SolveDifferentialEquations) with several examples using `odeint`. The same examples are also [solved with Gekko](https://apmonitor.com/pdc/index.php/Main/PythonDifferentialEquations). For simulation, the two give equivalent results but Gekko is built to use differential equations in optimization or combined with machine learning. The `odeint` function has a specific purpose to solve ODEs. The function `odeint` solves the differential equation requires 3 inputs.

    y = odeint(model, y0, t)

1. `model` Function name that returns derivative values at requested `y` and `t` values as `dydt = model(y,t)`
2. `y0` Initial conditions of the differential states
3. `t` Time points where the solution is reported

### Solve Differential Equation

We solve the differential equation with initial condition $y(0) = 5$:

$ k \, \frac{dy}{dt} = -y$

where $k=10$. The solution of `y` is reported from an initial time `0` to final time `20` and with a plot of the result for $y(t)$ versus $t$. Notice how the equation is rearranged to return just the derivative value as `dydt = -(1.0/k) * y` from the function.

```python
import numpy as np
from scipy.integrate import odeint

# function that returns dy/dt
def model(y,t):
    k = 10.0
    dydt = -(1.0/k) * y
    return dydt

y0 = 5                 # initial condition
t = np.linspace(0,20)  # time points
y = odeint(model,y0,t) # solve ODE

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(t,y)
plt.xlabel('time'); plt.ylabel('y(t)')
plt.show()
```

### Solve Differential Equation with Gekko

[Python Gekko](https://gekko.readthedocs.io/en/latest/) solves the same differential equation problem. Gekko is built for large-scale problems. [Additional tutorials on Gekko](https://apmonitor.com/wiki/index.php/Main/GekkoPythonOptimization) show how to solve other types of equations and optimization problems.

```python
from gekko import GEKKO

m = GEKKO(remote=False)    # GEKKO model
m.time = np.linspace(0,20) # time points
y = m.Var(5.0); k = 10.0   # GEKKO variable and constant
m.Equation(k*y.dt()+y==0)  # GEKKO equation

m.options.IMODE = 4        # Dynamic simulation
m.solve(disp=False)        # Solve

plt.plot(m.time,y)
plt.xlabel('time'); plt.ylabel('y(t)')
plt.show()
```

### Differential Equation Activity

Solve the differential equation with initial condition $y(0) = 10$:

$ k \, \frac{dy}{dt} = -y$

Compare the five solutions of `y` from time `0` to `20` with `k=[1,2,5,10,20]`.

```python

```

### Symbolic Solution

Compact differential equation problems may have an analytic solution that can be expressed symbolically. A symbolic math package in Python is `sympy`. Sympy determines the analytic solution as $y(x)=C_1 \, \exp{\left(-\frac{x}{k}\right)}$. With the initial condition $y(0)=5$, the constant $C_1$ is equal to 5.

```python
from IPython.display import display
import sympy as sym
from sympy.abc import x, k
y = sym.Function('y')
ans = sym.dsolve(sym.Derivative(y(x), x) + y(x)/k, y(x))
display(ans)
```

### Solve Differential Equation with Input `u`

Differential equations can also have an input (feature) that changes from an external source (exogenous input) such as actively changed by a measurement sensor, a person (manually), or selected by a computer.

Calculate the response `y(t)` when the input `u` changes from `0` to `2` at `t=5`.

$2 \frac{dy(t)}{dt} + y(t) = u(t)$

The initial condition is `y(0)=1` and the solution should be calculated until `t=15`. **Tip**: The expression `y(t)` does not mean `y` multiplied by `t`. It indicates that `y` changes with time and is written as a function of time. There are additional examples for [ODEINT](https://apmonitor.com/pdc/index.php/Main/SolveDifferentialEquations) and [GEKKO](https://apmonitor.com/pdc/index.php/Main/PythonDifferentialEquations) if you need help.

```python

```

### TCLab Activity

### Data Collection

Turn on heater 1 to 100% and record $T_1$ every 5 seconds for 3 minutes. The data should include a total of 37 data points for each temperature sensor.

```python
import numpy as np
import pandas as pd
import tclab
import time
# collect data for 3 minutes, every 5 sec
n = 37
tm = np.linspace(0,180,n)
t1s = np.empty(n); t2s = np.empty(n)
with tclab.TCLab() as lab:
    lab.Q1(100); lab.Q2(0)
    print('Time T1  T2')
    for i in range(n):
        t1s[i] = lab.T1; t2s[i] = lab.T2
        print(tm[i],t1s[i],t2s[i])
        time.sleep(5.0)
# put into dataframe
data = pd.DataFrame(np.column_stack((tm,t1s,t2s)),\
                    columns=['Time','T1','T2'])
data.to_csv('11-data.csv',index=False)
```

### Solve Differential Equation

Use the parameters `a`, `b`, and `c` from [10. Solve Equations](https://github.com/APMonitor/data_science/blob/master/10.%20Solve_Equations.ipynb) or use the following default values.

| Parameter | Value |
|------|------|
| a    | 78.6 |
| b    | -50.3 |
| c    | -0.003677 |

Solve the ordinary differential equation (ODE) with these values.

$\frac{dT_1}{dt} = c (T_1-a)$

The initial condition for $T_1$ is $a + b$. Show the ODE solution at time points between the initial time `0` and the final time `180` sec. Plot the measured $T_1$ on the same plot as the ODE predicted temperature. Add appropriate labels to the plot.

```python

```


# Statistics

## 1.1 Sales for the past week was the following amounts: [3505, 2400, 3027, 2798, 3700, 3250, 2689]. Without using library functions, what is the mean, variance, and standard deviation of sales from last week? (for extra bonus points, write your own function that can calculate these three values for any sized list)

```
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
```

```
sales = [3505, 2400, 3027, 2798, 3700, 3250, 2689]

#Mean
def mean(data):
  return sum(data) / len(data)
print("Mean:", mean(sales))

#Variance
def variance(data):
  data = np.array(data)
  return ((data - mean(data))**2).sum() / len(data)
print("Variance:",variance(sales))

#Standard Deviation
def std(data):
  return variance(data)**(0.5)
print("STD:", std(sales))
```

    Mean: 3052.714285714286
    Variance: 183761.06122448976
    STD: 428.67360686714756

## 1.2 Find the covariance between last week's sales numbers and the number of customers that entered the store last week: [127, 80, 105, 92, 120, 115, 93] (you may use library functions for calculating the covariance since we didn't specifically talk about its formula)

```
num_of_cust = [127, 80, 105, 92, 120, 115, 93]

covar = np.cov(sales, num_of_cust, )
print(covar)
```

    [[214387.9047619    7604.35714286]
     [  7604.35714286    290.95238095]]

## 1.3 Find the standard deviation of customers who entered the store last week. Then, use the standard deviations of both sales and customers to standardize the covariance to find the correlation coefficient that summarizes the relationship between sales and customers. (You may use library functions to check your work.)

```
devs = [[std(sales)*std(sales), std(sales)*std(num_of_cust)],
       [std(sales)*std(num_of_cust), std(num_of_cust)*std(num_of_cust)]]

print(devs)

print('Correlation:', np.divide(np.cov(sales, num_of_cust), devs))

print(np.corrcoef(sales, num_of_cust))
```

    [[183761.06122448976, 6769.620265121536], [6769.620265121536, 249.3877551020408]]
    Correlation:
    [[1.16666667 1.12330631]
     [1.12330631 1.16666667]]
    Numpy corr:
    [[1.         0.96283398]
     [0.96283398 1.        ]]

## 1.4 Use pandas to import a cleaned version of the titanic dataset from the following link: [Titanic Dataset](https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv)

## Calculate the variance-covariance matrix and correlation matrix for the titanic dataset's numeric columns. (you can encode some of the categorical variables and include them as a stretch goal if you finish early)

```
titanic = pd.read_csv("https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv")

numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']

numtitanic = titanic.select_dtypes(include=numerics).drop(columns="Unnamed: 0")

numtitanic.cov()
```

      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      pclass
      0.701969
      -0.127248
      -3.954605
      0.053090
      0.013287
      -24.227788
      -2.876653
      -0.249992

      survived
      -0.127248
      0.236250
      -0.314343
      -0.014088
      0.034776
      6.146023
      0.000000
      0.061406

      age
      -3.954605
      -0.314343
      165.850021
      -2.559806
      -1.459378
      114.416613
      81.622922
      1.463138

      sibsp
      0.053090
      -0.014088
      -2.559806
      1.085052
      0.336833
      8.641768
      -8.708471
      -0.003946

      parch
      0.013287
      0.034776
      -1.459378
      0.336833
      0.749195
      9.928031
      4.237190
      0.013316

      fare
      -24.227788
      6.146023
      114.416613
      8.641768
      9.928031
      2678.959738
      -179.164684
      10.976961

      body
      -2.876653
      0.000000
      81.622922
      -8.708471
      4.237190
      -179.164684
      9544.688567
      3.625689

      has_cabin_number
      -0.249992
      0.061406
      1.463138
      -0.003946
      0.013316
      10.976961
      3.625689
      0.174613

```
numtitanic.corr()
```

      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      pclass
      1.000000
      -0.312469
      -0.366370
      0.060832
      0.018322
      -0.558629
      -0.034642
      -0.713857

      survived
      -0.312469
      1.000000
      -0.050199
      -0.027825
      0.082660
      0.244265
      NaN
      0.302250

      age
      -0.366370
      -0.050199
      1.000000
      -0.190747
      -0.130872
      0.171892
      0.059059
      0.271887

      sibsp
      0.060832
      -0.027825
      -0.190747
      1.000000
      0.373587
      0.160238
      -0.099961
      -0.009064

      parch
      0.018322
      0.082660
      -0.130872
      0.373587
      1.000000
      0.221539
      0.051099
      0.036806

      fare
      -0.558629
      0.244265
      0.171892
      0.160238
      0.221539
      1.000000
      -0.043110
      0.507253

      body
      -0.034642
      NaN
      0.059059
      -0.099961
      0.051099
      -0.043110
      1.000000
      0.083796

      has_cabin_number
      -0.713857
      0.302250
      0.271887
      -0.009064
      0.036806
      0.507253
      0.083796
      1.000000

# Orthogonality

## 2.1 Plot two vectors that are orthogonal to each other. What is a synonym for orthogonal?

```
vector_1 = [-3,3]
vector_2 = [3,3]

plt.arrow(0,0, vector_1[0], vector_1[1],head_width=.05, head_length=0.05, color ='red')
plt.arrow(0,0, vector_2[0], vector_2[1],head_width=.05, head_length=0.05, color ='green')
plt.xlim(-3,3)
plt.ylim(-3,3)
plt.title("Orthogonal Vectors")
plt.show()

```

### A synonym for orthogonal is perpendicualr

## 2.2 Are the following vectors orthogonal? Why or why not?

\begin{align}
a = \begin{bmatrix} -5 \\ 3 \\ 7 \end{bmatrix}
\qquad
b = \begin{bmatrix} 6 \\ -8 \\ 2 \end{bmatrix}
\end{align}

```
a = np.array([-5, 3, 7])
b = np.array([6, -8, 2])

dot = np.dot(a,b)
print(dot, "Not Orthogonal")
```

    -40 Not Orthogonal

## 2.3 Compute the following values: What do these quantities have in common?

## What is $||c||^2$?

## What is $c \cdot c$?

## What is $c^{T}c$?

\begin{align}
c = \begin{bmatrix} 2 & -15 & 6 & 20 \end{bmatrix}
\end{align}

```
from numpy import linalg as LA

c = np.array([2, -15, 6, 20])
# C norm squared
normC = LA.norm(c)**2
print(normC)

#C C
cdot = np.dot(c,c)
print(cdot)

#CT C
ctransposec = np.transpose(c)*c
print(ctransposec)\

# The quatites for the norm and the dot prodcut are the same
```

    665.0
    665
    [  4 225  36 400]

# Unit Vectors

## 3.1 Using Latex, write the following vectors as a linear combination of scalars and unit vectors:

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix}
\qquad
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix}
\end{align}

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix} = 7\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 12\begin{bmatrix} 0 \\ 1 \end{bmatrix} \end{align}

\begin{align}
e = \begin{bmatrix} 2 \\ 11 \\ -8 \end{bmatrix} = 2\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} + 11\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}  + -8\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \end{align}

## 3.2 Turn vector $f$ into a unit vector:

\begin{align}
f = \begin{bmatrix} 4 & 12 & 11 & 9 & 2 \end{bmatrix}
\end{align}

```
f = np.array([4, 12, 11,9, 2])

f_hat = f/LA.norm(f)
print(f_hat)
```

    [0.20908335 0.62725005 0.57497921 0.47043754 0.10454167]

# Linear Independence / Dependence

## 4.1 Plot two vectors that are linearly dependent and two vectors that are linearly independent (bonus points if done in $\mathbb{R}^3$).

# Span

## 5.1 What is the span of the following vectors?

\begin{align}
g = \begin{bmatrix} 1 & 2 \end{bmatrix}
\qquad
h = \begin{bmatrix} 4 & 8 \end{bmatrix}
\end{align}

```
g = np.array([1,2])
h = np.array([4,8])

# STILL NOT SURE

```

## 5.2 What is the span of $\{l, m, n\}$?

\begin{align}
l = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}
\qquad
m = \begin{bmatrix} -1 & 0 & 7 \end{bmatrix}
\qquad
n = \begin{bmatrix} 4 & 8  & 2\end{bmatrix}
\end{align}

```
#STILL NOT SURE
```

# Basis

## 6.1 Graph two vectors that form a basis for $\mathbb{R}^2$

```
a = np.array([0,2])
b = np.array([1,0])

vectors = [a,b]

fig = plt.figure()
for vec in vectors:
    plt.arrow(0,0,vec[0],vec[1], head_width=0.2,
              head_length=0.3, length_includes_head = True)

plt.xlim(-1,3)
plt.ylim(-1,3)

plt.show()
```

## 6.2 What does it mean to form a basis?

A set of vectors forms the basis of a vector space if they are linearly independent and their linear combinations fill that space.

# Rank

## 7.1 What is the Rank of P?

\begin{align}
P = \begin{bmatrix}
1 & 2 & 3 \\
 -1 & 0 & 7 \\
4 & 8  & 2
\end{bmatrix}
\end{align}

```
P = np.array([[1,2,3],[-1,0,7],[4,8,2]])
print('Rank: ' + str(np.linalg.matrix_rank(P)))
```

    Rank: 3

## 7.2 What does the rank of a matrix tell us?

The rank of a matrix is the dimension of the vector space that its columns span. In this case, it tells us that the columns of this matrix span all of 3D space, $\mathbb{R}^3$.

# Linear Projections

## 8.1 Line $L$ is formed by all of the vectors that can be created by scaling vector $v$
\begin{align}
v = \begin{bmatrix} 1 & 3 \end{bmatrix}
\end{align}

\begin{align}
w = \begin{bmatrix} -1 & 2 \end{bmatrix}
\end{align}

## find $proj_{L}(w)$

## graph your projected vector to check your work (make sure your axis are square/even)

```

```

# Stretch Goal

## For vectors that begin at the origin, the coordinates of where the vector ends can be interpreted as regular data points. (See 3Blue1Brown videos about Spans, Basis, etc.)

## Write a function that can calculate the linear projection of each point (x,y) (vector) onto the line y=x. run the function and plot the original points in blue and the new projected points on the line y=x in red.

## For extra points plot the orthogonal vectors as a dashed line from the original blue points to the projected red points.

```
import pandas as pd
import matplotlib.pyplot as plt

# Creating a dataframe for you to work with -Feel free to not use the dataframe if you don't want to.
x_values = [1, 4, 7, 3, 9, 4, 5 ]
y_values = [4, 2, 5, 0, 8, 2, 8]

data = {"x": x_values, "y": y_values}

df = pd.DataFrame(data)

df.head()

plt.scatter(df.x, df.y)
plt.show()
```

```

```


# Exercise 1
## JIT the pressure poisson equation
The equation we need to unroll is given by

\begin{equation}
p_{i,j}^{n} = \frac{1}{4}\left(p_{i+1,j}^{n}+p_{i-1,j}^{n}+p_{i,j+1}^{n}+p_{i,j-1}^{n}\right) - b
\end{equation}

and recall that `b` is already computed, so no need to worry about unrolling that.  We've also filled in the boundary conditions, so don't worry about those.  (don't forget to decorate your function!)

```python
import numpy
from numba import jit
```

```python
def pressure_poisson(p, b, l2_target=1e-4):
    I, J = b.shape

    iter_diff = l2_target + 1
    n = 0
    while iter_diff > l2_target and n <= 500:
        pn = p.copy()

        #Your code here

        #boundary conditions
        for i in range(I):
            p[i, 0] = p[i, 1]
            p[i, -1] = 0

        for j in range(J):
            p[0, j] = p[1, j]
            p[-1, j] = p[-2, j]

        if n % 10 == 0:
            iter_diff = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))

        n += 1

    return p
```

```python
import pickle
from snippets.ns_helper import cavity_flow, velocity_term, quiver_plot
```

```python
def run_cavity():
    nx = 41
    with open('../IC.pickle', 'rb') as f:
        u, v, p, b = pickle.load(f)

    dx = 2 / (nx - 1)
    dt = .005
    nt = 1000

    u, v, p = cavity_flow(u, v, p, nt, dt, dx,
                         velocity_term,
                         pressure_poisson,
                         rtol=1e-4)

    return u, v, p
```

```python
un, vn, pn = run_cavity()
```

```python
%timeit run_cavity()
```

```python
with open('../numpy_ans.pickle', 'rb') as f:
    u, v, p = pickle.load(f)
```

```python
assert numpy.allclose(u, un)
assert numpy.allclose(v, vn)
assert numpy.allclose(p, pn)
```

# Exercise 2 (optional)

Finish early?  Just want to try more stuff?

This line is not super efficient:

```python
iter_diff = numpy.sqrt(numpy.sum((p - pn)**2)/numpy.sum(pn**2))
```

Try rewriting it using a jitted function and see what kind of performance gain you can get.

```python

```


# Music Machine Learning - Probability distributions

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [quick recap](#recap) on simple probability concepts
2. An introduction to [probability distributions](#distributions)
3. An explanation on how to [sample](#sampling) from distributions ourselves

## Quick recap on probability

The field of probability aims to model random or uncertain events through random variables $X$ denoting a quantity that is uncertain, which take values $\omega$ in a sample space $\omega \in \Omega$. If we observe several occurrences of this variable $\{\mathbf{x}_{i}\}_{i=1}$, we might try to model the _probability distribution_ $p(\mathbf{x})$ of that variable. We recall here that the probability of an event $a$ is a real number $p(a)$, with $0 \leq p(a) \leq 1$, knowing that $p(\Omega)=1$ and $p(\{\})=0$. The probability of two events occuring simultaneously is defined as $p\left(a \cup b \right)$. Therefore, the probability of one event **or** the other occuring is defined as
$
\begin{equation}
p\left(a \cap b \right) = p(a) + p(b) - p\left(a \cup b \right)
\end{equation}
$

The *conditional probability* of an event $a$ occuring *given* another event $b$ is denoted $p \left(a \mid b \right)$ and is defined as
$$
\begin{equation}
p \left(a \mid b \right) = \frac{p \left(a , b \right)}{p \left(b \right)}
\end{equation}
$$

This can be understood as the probability of event $a$ to occur if we restrict the world of possibilities to event $b$. The *chain rule* defines the probabilities of a set of events to co-occur simultaneously
$$
\begin{equation}
p \left(x_{1},...,x_{n} \right) = \prod_{i=n}^{1}{p \left(x_{i}\mid x_{i-1},..., x_{1} \right)}
\end{equation}
$$

Finally, we say that two events are independent if $p(a\mid b) = p(a)$.

To understand these concepts graphically, we will rely on `PyTorch` and specifically the `distributions` package.

```python
import torch.distributions as distribution
import torch.distributions.transforms as transform
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from helper_plot import hdr_plot_style
hdr_plot_style()
```

### Probability distributions

Let $X$ be a random variable associated with a *probability distribution function* that assigns probabilities to the different outcomes $X$ can take in the sample space. We can divide random variables into three different types:

- **$X$ is discrete**: Discrete random variables may only assume values on a specified list.
- **$X$ is continuous**: Continuous random variable can take on arbitrarily exact values.
- **$X$ is mixed**: Mixed random variables assign probabilities to both discrete and continuous random variables, (i.e. a combination of the above two categories).

**Expected Value**
The expected value $\mathbb{E}\left[X\right]$ for a given probability distribution can be described as _"the mean expected value for many repeated samples from that distribution."_ As the number of repeated observation goes to infinity, the difference between the average outcome and the expected value becomes arbitrarily small. Formally, it is defined for discrete variables as

$$
\mathbb{E}\left[X\right] = \sum\limits_{i}\bx_i p(\bx=\bx_i)
$$

#### Discrete distributions

If $X$ is discrete, then its distribution is called a *probability mass function* (pmf), which measures the probability $X$ takes on the value $x_{i}$, denoted $P(X=x_{i})$. Let $\mathbf{x}$ be a discrete random variable with range $R_{X}=\{x_1,\cdots,x_n\}$ (finite or countably infinite). The function

$$
p_{X}(x_{i})=p(X=x_{i}), \forall i\in\{1,\cdots,n\}
$$

is called the probability mass function (PMF) of $X$.

Hence, the PMF defines the probabilities of all possible values for a random variable. The above notation allows to express that the PMF is defined for the random variable $X$, so that $p_{X}(1)$ gives the probability that $X=1$. For discrete random variables, the PMF is also called the \textit{probability distribution}. The PMF is a probability measure, therefore it satisfies all the corresponding properties
- $0 \leq p_{X}(x_i) < 1, \forall x_i$
- $\sum_{x_i\in R_{X}} p_{X}(x_i) = 1$
- $\forall A \subset R_{X}, p(X \in A)=\sum_{x_a \in A}p_{X}(x_a)$

A very simple example of discrete distribution is the `Bernoulli` distribution. With this distribution, _we can model a coin flip_, if it has equal probability. More formally, a Bernoulli distribution is defined as

$$
Bernoulli(x)= p^x (1-p)^{(1-x)}
$$

with $p$ controlling the probability of the two classes. Hence, a fair coin should have $p=0.5$, and if we throw the coin a very large number of times, we hope to see on average an equal amount of _heads_ and _tails_.

```python
bernoulli = distribution.Bernoulli(0.5)
samples = bernoulli.sample((10000,))
plt.figure(figsize=(10,8))
sns.distplot(samples)
plt.title("Samples from a Bernoulli (coin toss)")
plt.show()
```

However, we can also _sample_ from the distribution to have individual values of a single throw. In that case, we obtain a series of separate events that _follow_ the distribution

```python
vals = ['heads', 'tails']
samples = bernoulli.sample((10, ))
for s in samples:
    print('Coin is tossed on ' + vals[int(s)])
```

    Coin is tossed on tails
    Coin is tossed on tails
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on tails
    Coin is tossed on heads
    Coin is tossed on heads
    Coin is tossed on tails

Now, we can mess up our probability to model an unfair (loaded) coin, as shown in the following example (where we use a cheated coin that should give us a lot more of _heads_ than _tails_)

```python
bernoulli = distribution.Bernoulli(0.3)
samples = bernoulli.sample((10000,))
plt.figure(figsize=(10, 8)); sns.distplot(samples)
plt.title("Samples from an unfair coin"); plt.show()
vals = ['heads', 'tails']
samples = bernoulli.sample((10, ))
for s in samples:
    print('Coin is tossed on ' + vals[int(s)])
```

#### Poisson distribution

Let's introduce one of the (many) useful probability mass functions. We say $Z$ is *Poisson*-distributed if:

$$
\begin{equation}
P\left(Z = k\right) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k \in \mathbb{N^{+}}
\end{equation}
$$

$\lambda \in \mathbb{R}$ is a parameter of the distribution that controls its shape (usually termed the *intensity* of the Poisson distribution). By increasing $\lambda$, we add more probability to larger values. One can describe $\lambda$ as the *intensity* of the Poisson distribution. If a random variable $Z$ has a Poisson mass distribution, we denote it by

$$
\begin{equation}
Z \sim \text{Poi}(\lambda)
\end{equation}
$$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will plot the probability mass distribution for different $\lambda$ values.

### Continuous distributions

The same ideas apply to _continuous_ random variables, which can model for instance the height of human beings. If we try to guess the height of someone that we do not know, there is a higher probability that this person will be around 1m70, instead of 20cm or 3m. For the rest of this course, we will use the shorthand notation $p(\mathbf{x})$ for the distribution $p(\mathbf{x}=x_{i})$,  which expresses for a real-valued random variable $\mathbf{x}$, evaluated at $x_{i}$, the probability that $\mathbf{x}$ takes the value $x_i$.

One notorious example of such distributions is the Gaussian (or Normal) distribution, which is defined as
\begin{equation}
    p(x)=\mathcal{N}(\mu,\sigma)=\frac{1}{\sqrt{2\pi\sigma^{2}}}e^{-\frac{(x-\mu)^{2}}{2\sigma^{2}}}
\end{equation}

Similarly as before, we can observe the behavior of this distribution with the following code (in our height example)

```python
normal = distribution.Normal(150., 30.)
samples = normal.sample((10000, ))
sns.distplot(samples)
plt.title("Samples from a standard Normal")
plt.show()
```

If we have access to this complete probability distribution (its exact parameterization and function), we can generate samples (in this case "new humans") that follow the correct distribution. You can experiment with this in the following code space.

```python

######################
# YOUR CODE GOES HERE
######################

```

An example of continuous random variable is a random variable with *exponential density*

$$
\begin{equation}
f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0
\end{equation}
$$

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential*

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

### PyTorch distributions

Here, we rely on the [PyTorch distributions module](https://pytorch.org/docs/stable/_modules/torch/distributions/), which is defined in `torch.distributions`. Most notably, we are going to rely both on the `Distribution` and `Transform` objects.

```python
# Imports for plotting
import numpy as np
import matplotlib.pyplot as plt
# Define grids of points (for later plots)
x = np.linspace(-4, 4, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
```

Inside this toolbox, we can already find some of the major probability distributions that we are used to deal with

```python
p = distrib.Normal(loc=0, scale=1)
p = distrib.Bernoulli(probs=torch.tensor([0.5]))
p = distrib.Beta(concentration1=torch.tensor([0.5]), concentration0=torch.tensor([0.5]))
p = distrib.Gamma(concentration=torch.tensor([1.0]), rate=torch.tensor([1.0]))
p = distrib.Pareto(alpha=torch.tensor([1.0]), scale=torch.tensor([1.0]))
```

The interesting aspect of these `Distribution` objects is that we can both obtain some samples from it through the `sample` (or `sample_n`) function, but we can also obtain the analytical density at any given point through the `log_prob` function

```python
# Based on a normal
n = distrib.Normal(0, 1)
# Obtain some samples
samples = n.sample((1000, ))
# Evaluate true density at given points
density = torch.exp(n.log_prob(torch.Tensor(x))).numpy()
# Plot both samples and density
fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(15, 4))
ax1.hist(samples, 50, alpha=0.8);
ax1.set_title('Empirical samples', fontsize=18);
ax2.plot(x, density); ax2.fill_between(x, density, 0, alpha=0.5)
ax2.set_title('True density', fontsize=18);
```

Here you can experiment with different distributions, and try to compare how they behave depending on their parameters and also on how much _samples_ you draw from these.

```python

######################
# YOUR CODE GOES HERE
######################

```

## Sampling from distributions

The advantage of using probability distributions is that we can *sample* from these to obtain examples that follow the distribution. For instance, if we perform sampling repeatedly (up to infinity) from a Gaussian PDF, the different values will be distributed following the exact Gaussian distribution. However, although we know the PDF, we need to compute the *Cumulative Distribution Function* (CDF), and then its inverse to obtain the sampling function. Therefore, if we denote the PDF as $f_{X}(x)$, we need to compute the CDF

$$
\begin{equation}
F_{X}\left(x\right)=\intop_{\infty}^{x}f_{X}\left(t\right)dt
\end{equation}
$$

Then, the *inverse sampling method* consists in solving and applying the inverse CDF $F_{X}^{-1}\left(x\right)$. Here, we recall that the Exponential probability is defined with the following function.

$$
p_{\lambda}(x) = \lambda e^{-\lambda x}
$$

with $\lambda$ defining the _rate_ parameter. Therefore, to be able to define our own `sample` method, we need to solve for

$$
F^{-1}_Y(x) = \left(\int_0^x \lambda e^{-\lambda y} dy\right)^{-1}
$$

***

**Exercise**

1. Code the exponential probability density functions
2. Perform the inverse transform method on the exponential distribution PDF
3. Code the `sample_exponential` function to sample from the exponential
4. (optional) Perform the same operations for sampling from the Beta distribution

***

```python
from scipy.stats import expon
nb_samples = 500
nb_bins = 50

def sample_exponential(mu, n, m):

    ######################
    # YOUR CODE GOES HERE
    ######################

    ######################
    # Solution

    samples = np.random.uniform(0, 1, size=(m,n))
    samples = - np.log(1 - samples)
    samples *= mu

    ######################

    return samples

# Exponential distribution
mu = 2
samples = np.random.exponential(mu, nb_samples)
samples_ex = sample_exponential(mu, nb_samples, 1)
# Compute the PDF
X = np.linspace(0, np.max(samples), int(np.max(samples)) * 100)
y1 = expon.pdf(X) #* (nb_samples / nb_bins) * int(np.max(samples) * 1.5)
# Display both
plt.figure(figsize=(12, 8))
plt.subplot(2,1,1)
plt.hist(samples, 50, label='Samples')
plt.plot(X,y1,ls='--',c='r',linewidth=2, label='Exponential PDF')
plt.legend(loc=1)
plt.subplot(2,1,2)
plt.hist(samples_ex, 50, label='Samples')
plt.plot(X,y1,ls='--',c='r',linewidth=2, label='Our exponential sampples')
plt.legend(loc=1)
```

```python
def sampleBeta(a, b, M, N):

    ######################
    # YOUR CODE GOES HERE
    ######################

    ######################
    # Solution

    samples = np.random.beta(a, b, size=(M, N))

    ######################

    return samples

from scipy.stats import beta
# Beta distribution
a = 0.6
b = 1.5
samples = np.random.beta(a, b, nbSamples)
samplesBeta = sampleBeta(a, b, nbSamples, 1)
# Compute the PDF
X = np.linspace(0, 1, 100)
y1 = beta.pdf(X, a, b) * (nbSamples / nbBins) * (np.max(samples) * 1.5)
# Display both
plt.figure(figsize=(12, 8))
plt.subplot(2,1,1)
plt.hist(samples, 50, label='Samples')
plt.plot(X,y1,ls='--',c='r',linewidth=2, label='Beta PDF')
plt.legend(loc=1)
plt.subplot(2,1,2)
plt.hist(samplesBeta, 50, label='Samples')
plt.plot(X,y1,ls='--',c='r',linewidth=2, label='Beta PDF')
plt.legend(loc=1)
```

### Comparing distributions (KL divergence)
$
\newcommand{\R}{\mathbb{R}}
\newcommand{\bb}[1]{\mathbf{#1}}
\newcommand{\bx}{\bb{x}}
\newcommand{\by}{\bb{y}}
\newcommand{\bz}{\bb{z}}
\newcommand{\KL}[2]{\mathcal{D}_{\text{KL}}\left[#1 \| #2\right]}$
Originally defined in the field of information theory, the _Kullback-Leibler (KL) divergence_ (usually noted $\KL{p(\bx)}{q(\bx)}$) is a dissimilarity measure between two probability distributions $p(\bx)$ and $q(\bx)$. In the view of information theory, it can be understood as the cost in number of bits necessary for coding samples from $p(\bx)$ by using a code optimized for $q(\bx)$ rather than the code optimized for $p(\bx)$. In the view of probability theory, it represents the amount of information lost when we use $q(\bx)$ to approximate the true distribution $p(\bx)$. %that explicit the cost incurred if events were generated by $p(\bx)$ but charged under $q(\bx)$

Given two probability distributions $p(\bx)$ and $q(\bx)$, the Kullback-Leibler divergence of $q(\bx)$ _from_ $p(\bx)$ is defined to be
\begin{equation}
    \KL{p(\bx)}{q(\bx)}=\int_{\R} p(\bx) \log \frac{p(\bx)}{q(\bx)}d\bx
\end{equation}

Note that this dissimilarity measure is _asymmetric_, therefore, we have
\begin{equation}
    \KL{p(\bx)}{q(\bx)}\neq \KL{q(\bx)}{p(\bx)}
\end{equation}
This asymmetry also describes an interesting behavior of the KL divergence, depending on the order to which it is evaluated. The KL divergence can either be a _mode-seeking_ or _mode-coverage_ measure (we will come back to these notions in the _approximate inference_ course)

```python

```


# Blatt 2

## Aufgabe 5

```python
import numpy as np
from numpy import random
import matplotlib.pyplot as plt

uniform = random.uniform(size = 1000)
```

__a)__ Gleichverteilung auf dem Gebiet $[x_{min}, x_{max}]$:
$$f(x) = \frac{1}{x_{max} - x_{min}}$$
Bilde Verteilungsfunktion $F(x)$:
$$
F(x) = \int_{x_{min}}^{x}f(x') \, dx' = \frac{x - x_{min}}{x_{max} - x_{min}}
$$
Bilde Umkehrfunktion $F^{-1}(y)$:
$$
F^{-1}(y) = (x_{max} - x_{min}) y + x_{min}.
$$
Implementierung und Darstellung:

```python
def uniform_intervall(y, xmin, xmax):
    assert xmax >= xmin
    return xmin + y * (xmax - xmin)
```

```python
fig, ax = plt.subplots(1, 1)
ax.hist(uniform_intervall(uniform, 42, 44), bins = 20, histtype='step')
ax.set_xlabel('x')
ax.set_ylabel('Counts')
plt.show()
```

__b)__ Exponentialgesetz:
$$f(t) = N e^{-\frac{t}{\tau}} = \frac{1}{\tau}e^{-\frac{t}{\tau}}, \quad t \in [0, \infty)$$

Verteilungsfunkion:
$$F(t) = \int_{0}^{t} f(t') \, dt' = (1 - e^{-\frac{t}{\tau}}) $$

Umkehrfunktion:
$$F^{-1}(y) = \tau \, \text{ln}\left(\frac{1}{1 - y}\right) $$
Implementierung:

```python
def F(y, tau = 1):
    return tau * np.log(1 / (1 - y))
```

```python
fig, ax = plt.subplots(1, 1)
ax.hist(F(uniform), bins = 30, histtype='step')
ax.set_xlabel(r'$\frac{t}{\tau}$')
ax.set_ylabel('Counts')
plt.show()
```

__c)__ Potenzgesetz:
$$f(x) = N x^{-n} = \frac{1 - n}{x_{max}^{1-n} - x_{min}^{1-n} } x^{-n}, \quad x \in [x_{min}, x_{max}], n \geq 2$$
Verteilungsfunktion:
$$F(x) = \int_{x_{min}}^{x} f(x') \, dx' = \frac{x^{1-n} - x_{min}^{1-n}  }{ x_{max}^{1-n} - x_{min}^{1-n} }$$

Umkehrfunktion:
$$  F^{-1}(y) = \left\{ \left(x_{max}^{1-n} - x_{min}^{1 - n}\right)y + x_{min}^{1-n}  \right\}^{\frac{1}{1 - n}}  $$
Implementierung:

```python
def power(y, xmin, xmax, n):
    assert n >= 2
    return ( (xmax**(1-n) - xmin**(1-n)) * y +  xmin**(1-n) )**(1 / (1-n))
```

```python
fig, ax = plt.subplots(1, 1)
ax.hist(power(uniform, xmin = 1, xmax = 4, n = 5), bins = 30, histtype='step')
ax.set_xlabel('x')
ax.set_ylabel('Counts')
plt.show()
```

__d)__ Cauchy-Verteilung:
$$f(x) = \frac{1}{\pi} \frac{1}{1 + x^2}, \quad x \in (-\infty, \infty)$$
Verteilungsfunktion:
$$  F(x) = \int_{-\infty}^{x} f(x')\, dx'   = \frac{1}{\pi} \left[\text{arctan}(x) + \frac{\pi}{2} \right]$$
Umkehrfunktion:
$$F^{-1}(y) = \text{tan} \left( \pi y - \frac{\pi}{2} \right) $$
Implementierung:

```python
def cauchy(y):
    return np.tan(np.pi * (y - 1/2))
```

```python
fig, ax = plt.subplots(1, 1)
ax.hist(cauchy(uniform), bins = 500, histtype='step')
ax.set_xlim(-50, 50)
ax.set_xlabel('x')
ax.set_ylabel('Counts')
plt.show()
```

__e)__ Verteilung aus empirischem Histogramm mit der 'rejection sampling' Methode

```python
import pandas as pd
hist = pd.read_csv('empirisches_histogramm.csv')
binmids = hist['binmid']
counts = hist['counts']

fig, ax = plt.subplots(1, 1)
ax.errorbar(x = binmids, y = counts, xerr = 0.01, linestyle = '')
ax.set_xlabel('x')
ax.set_ylabel('Counts')
plt.show()
```

```python
norm = np.sum(counts * 0.2) #bin width is 0.2
norm_counts = counts / norm #normalize data
u1 = random.uniform(size = 10000) #
u2 = random.uniform(size = 10000) #two uniform samples
```

```python
def empirical(counts, binmids, u1, u2):
    binindex = u1 // 0.02 #get index of bin that includes u1 values
    return u1[ [ u2[i] <= counts[binindex[i]]
                for i in range(len(u1)) ] ]
y = empirical(norm_counts, binmids, u1, u2)
```

Von den erzeugten 20000 Zufallszahlen bleiben 4.905% übrig.

```python
fig, ax = plt.subplots(1, 1)
counts, binedges = np.histogram(y, bins = 50)
ax.errorbar(x = (binedges[:-1] + binedges[1:]) * 0.5, y = counts, xerr = np.diff(binedges) * 0.5, linestyle = '')
ax.set_xlabel('x')
ax.set_ylabel('Counts')
plt.show()
```

## Aufgabe 6

```python
np.random.seed(0)
def linGen(x_0, a, b, m):
    random_numbers = [x_0]
    for i in range(m):
        next_number = (a * random_numbers[i] + b) % m
        if (next_number in random_numbers):
            return len(random_numbers), np.array(random_numbers)
        else:
            random_numbers.append(next_number)
```

```python
a = np.linspace(0,100,101)
D = np.array([(linGen(x_0 = 4, a = n, b = 3, m = 1024))[0] for n in a])
#d = np.array([linGen(n,3,1024,4)[1] for n in a])
print("Die maximale Periodenlänge beträgt: ", np.max(D), '\n')
print("Für folgende Werte von a ist die Periodenlänge maximal: ",'\n', a[D == np.max(D)] )
```

```python
plt.plot(a[:100], D[:100], 'rx', label = r'Periodenlängen');
plt.xlabel('Parameter a')
plt.ylabel('Periodenlänge')
plt.legend(loc='upper right', bbox_to_anchor = (1,0.9))
```

__b)__

```python
def linGen2(a, b, m, x_0, length):
    random_numbers = [x_0]
    for i in range(length):
        random_numbers.append((a * random_numbers[i] + b) % m)
    return random_numbers
```

```python
Liste1 = linGen2(a = 1601, b = 3456, m = 10000, x_0 = 0,   length = 10000)
Liste2 = linGen2(a = 1601, b = 3456, m = 10000, x_0 = 500, length = 10000)
Liste3 = linGen2(a = 1601, b = 3456, m = 10000, x_0 = 750, length = 10000)
```

```python
%matplotlib inline

plt.rcParams.update({'figure.figsize': (6, 4), 'font.size': 12})
plt.hist(Liste1, bins = 500, range = [np.min(Liste1), np.max(Liste1)],  histtype = 'step' );
plt.show()
plt.hist(Liste2, bins = 500, range = [np.min(Liste2), np.max(Liste2)], histtype = 'step' );
plt.show()
plt.hist(Liste3, bins = 500, range = [np.min(Liste3), np.max(Liste3)], histtype = 'step' );
plt.show()
```

Die erhaltenen Zufallszahlen entsprechen nicht den Ansprüchen an einen guten Zufallsgenerator,
da keine Gleichverteilung der Zahlen vorhanden ist. In dem Histogram sieht man jeweils einen
Peak bei dem gewählten Startwert. Die Güte des Zufallsgenerators hängt jedoch nicht vom Startwert ab.

__c)__

```python
plt.figure(figsize = [10,10])
plt.scatter(Liste1[0:-2:2], Liste1[1::2])
```

```python
from mpl_toolkits .mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111 , projection ='3d')
ax.view_init(45, 30)
ax.scatter(
Liste1[0:-2:2], Liste1[1::2], Liste1[2::2],
lw=0,
s=5,
)
plt.show()
```

Der Spektraltest zeigt deutlich, dass es sich nicht um einen guten Zufallsgenerator handelt, da sowohl in
2D als auch in 3D ein starkes Muster zu erkennen ist.

__d)__

```python
Random = np.random.uniform(0,10000,10000)
plt.hist(Random, 500, range = [np.min(Random), np.max(Random)], histtype = 'step');
```

```python
plt.figure(figsize = [10,10])
plt.scatter(Random[0::2], Random[1::2], marker = '.')
```

```python
fig = plt.figure()
ax = fig.add_subplot(111 , projection ='3d')
ax.view_init(45, 30)
ax.scatter(
Random[0:-2:3], Random[1::3], Random[2::3],
lw=0,
s=5,
)
plt.show ()
```

__e)__

```python
x0 = np.linspace(0, 100, 201) # 0.5 steps
De = [(linGen(x_0 = n, a = 5, b = 3, m = 1024))[1].tolist() for n in x0]
# a = 5 is one of the values with max period
Liste05 = []
for n in De:
    if (0.5 in n) == True:
        #print(n)
        Liste05.append(x0[De.index(n)])
print('In jeder Liste kommt der Wert 0.5 maximal einmal vor, da bei einer Periodenlänge jeder Wert nur einmal vorkommen kann.','\n')
print('Der Wert 0.5 kann dabei für folgende Startwerte erhalten werden: ', Liste05[1:], '\n')
```

## Aufgabe 7

__a)__

```python
mu_x = 4
mu_y = 2
sigma_x = 3.5
sigma_y = 1.5
Cov = 4.2
```

```python
Korr = Cov / (sigma_x*sigma_y)
print("Der Korrelationskoeffizient beträgt:", Korr, '\n' )
```

```python
COV = [[sigma_x**2, Cov], [Cov, sigma_y**2]]
mean = [mu_x, mu_y]
gauss_2d = np.random.multivariate_normal(mean, COV, 15000)
```

__b)__

Verwende die Schreibweise aus Aufgabenteil e) :

$$ f(u_x, u_y) = \lambda \exp{(-\gamma\left[u_x^2 - 2\rho u_x u_y + u_y^2\right])} $$

$f(u_x,u_y)$ soll konstant sein, somit muss lediglich der Exponent betrachtet werden:

$$u_x^2 - 2\rho u_x u_y + u_y^2 = -\frac{1}{\gamma}konst.$$

Es handelt sich dabei um eine Ellipsengleichung, somit liegen Punkte gleicher Wahrscheinlichkeit immer auf einer Ellipse.

__c)__

```python
def gauss2d(x,y):
    return 1 / (2 * np.pi * sigma_x * sigma_y * np.sqrt(1-Korr**2)) * np.exp(-1 / (2 * (1 - Korr**2)) *  (((x - mu_x) / sigma_x)**2- 2 *  Korr * ((x - mu_x) / sigma_x) * ((y - mu_y) / sigma_y)+ ((y - mu_y)  /sigma_y)**2))

TestGaus = gauss2d(gauss_2d[:,0], gauss_2d[:,1]) # zugehörige Wahrscheinlichkeitswerte
e = gauss_2d[np.round(TestGaus,3) == np.round((1/np.sqrt(np.e))*np.max(TestGaus),3)]

plt.figure(figsize = [15,10])
plt.scatter(gauss_2d[:, 0], gauss_2d[:, 1], s=5, alpha=0.1, label='RandomGauß')
plt.scatter(e[:,0], e[:,1], s=20, alpha = 0.7, label = r'$\frac{1}{\sqrt{e}}$')
plt.errorbar(mean[0], mean[1], xerr = (sigma_x), yerr=(sigma_y),
             ecolor='black', label = r'$\pm\sigma_x, \pm\sigma_y$')
plt.scatter(mean[0],mean[1], s = 40, alpha = 1,
            label = r'$(\mu_x|\mu_y)$', color = 'red')
x = np.linspace(-15,25)
plt.plot(x, Korr*sigma_y/sigma_x*(x-mu_x) + mu_y, 'g-', label = r'$E(y|x)$')
Eyx = [Korr*sigma_x/sigma_y*(n-mu_y) + mu_x for n in x]
plt.plot(Eyx, x, 'r-', label=r'$E(x|y)$')
plt.xlim(-12,20)
plt.ylim(-4,9)

plt.legend(loc='best', fontsize = 20)
```

__d)__ Die Matrix $M$ ist hier die transponierte derjenigen Rotationsmatrix, die die Inverse der Kovarianzmatrix $B$ diagonalisiert. $B$ ist gegeben durch:
$$
B = Cov^{-1} = \frac{1}{\sigma_x^2 \sigma_y^2 - \text{Cov}(x, y) }
\begin{pmatrix}
    \sigma_y^2 & -\text{Cov}(x, y) \\
    -\text{Cov}(x, y) & \sigma_x^2
\end{pmatrix}.
$$
Das ist die Matrix, die in der quadratischen Form im Exponenten der mehrdim. Gaußverteilung steht. Eine Diagonalisierung entkoppelt also das Problem. Die Diagonalisierte Matrix hat die Eigenwerte auf der Diagonalen stehen. Hierbei handelt es sich um die gesuchten
$\frac{1}{{\sigma'}_x^2}$ und $\frac{1}{{\sigma'}_y^2}$. Die Eigenwerte ergeben sich aus
$$
\text{det}\left(B - \frac{1}{{\sigma'}_i^2} \cdot 1  \right) \overset{!}{=} 0
$$
durch unspektakuläre Umformungen zu
$$
\frac{1}{{\sigma'}_{x, y}^2} = \frac{1}{2[\sigma_x^2 \sigma_y^2 - \text{Cov}^2(x, y)]} \left\{\sigma_x^2 + \sigma_y^2 \pm  \left[ \left(\sigma_x^2 - \sigma_y^2\right)^2     + 4 \text{Cov}^2(x, y)\right]^{\frac{1}{2}}\right\}.
$$
Für die hier gegeben Werte kann man das dann ausrechnen:

```python
def newsig(sigx2, sigy2, cov2):
    return ((0.5 / (sigx2*sigy2 - cov2) * ( sigx2 + sigy2
                 + np.sqrt(
                     (sigx2 - sigy2)**2
                     + 4 * cov2
                  )))**(-0.5),
            (0.5 / (sigx2*sigy2 - cov2) * ( sigx2 + sigy2
                 - np.sqrt(
                     (sigx2 - sigy2)**2
                     + 4 * cov2
                  )))**(-0.5))
```

```python
mu_x = 4
mu_y = 2
sigx = 3.5
sigy = 1.5
cov = 4.2

gauss_2d = np.random.multivariate_normal([mu_x, mu_y],
        [[sigx**2, cov], [cov, sigy**2]], 10000)

newsigma = newsig(sigx**2, sigy**2, cov**2)
print(f'sig_y, six_x = {newsigma}')

```

Nun soll die Rotaionsmatrix $M$ bestimmt werden. Eine Rotationsmatrix in 2D ist allgemein gegeben durch
$$
R = \begin{pmatrix}
cos(\alpha) & -sin(\alpha) \\
sin(\alpha) & cos(\alpha)
\end{pmatrix}.
$$
Die Transformation von Matrizen und Vektoren sähe dann so aus
$$
S' = R^{T} S R, \quad \vec{x}' = R^{T} \vec{x}.
$$
Die Matrix $M$ ist in der Aufgabenstellung andersrum gewählt, sodass angesetzt werden kann
$$
M = \begin{pmatrix}
cos(\alpha) & sin(\alpha) \\
-sin(\alpha) & cos(\alpha)
\end{pmatrix}.
$$
Von $M$ kann man jetzt fordern, dass sie $B$ diagonalisiert
$$
\begin{pmatrix}
\frac{1}{{\sigma'}_y^2} & 0 \\
0 & \frac{1}{{\sigma'}_x^2}
\end{pmatrix}
\overset{!}{=}
M B M^{T}.
$$
Das gibt eins nices überbestimmtes Gleichungssystem für den Winkel $\alpha$. Rechnen nur mit dem Eintrag unten links. Das ergibt die Gleichung
$$
0 \overset{!}{=}  \sigma_y^2 sin(\alpha)cos(\alpha) + \text{Cov}(x, y)(cos^2(\alpha) - sin^2(\alpha)) - \sigma_x^2 sin(\alpha) cos(\alpha)
$$
Da kann man jetzt krasse Trigonometrie Skills rein stecken und erhält
$$
 0 = - \frac{1}{2}sin(2\alpha) (\sigma_x^2 - \sigma_y^2) - \text{Cov}(x, y)cos(2\alpha).
$$
Daraus dann
$$
\alpha = \frac{1}{2} \text{arctan} \left(-\frac{2\text{Cov}(x, y)}{\sigma_x^2 - \sigma_y^2}\right).
$$

```python
def alpha(cov, sigx2, sigy2):
    return 0.5 * np.arctan(- 2 * cov / (sigx2 - sigy2) )
```

```python
print(f'alpha ist ca. {alpha(cov, sigx**2, sigy**2) * 180 / np.pi:.3f} Grad')
```

```python
def rotate(x, alpha):
    #function that rotates a given vector x by an angle alpha.
    matrix = np.array([[np.cos(alpha), np.sin(alpha)],
                       [-np.sin(alpha), np.cos(alpha)]] )
    return matrix.dot(x)

old_x_axis = np.array([1, 0])
old_y_axis = np.array([0, 1])

new_x_axis = rotate(old_x_axis, alpha(cov, sigx**2, sigy**2))
new_y_axis = rotate(old_y_axis, alpha(cov, sigx**2, sigy**2))

def linear_function_from_vector(x, vec, center):
    # linear funtion along a vector x
    return vec[1] / vec[0] * (x - center[0]) + center[1]

```

```python
fig, ax = plt.subplots(1, 1, figsize = (10, 10))
ax.set_aspect('equal')

ax.scatter(gauss_2d[:, 0],gauss_2d[:, 1], s=5, alpha = 0.3, label = 'Data')

xlim = ax.get_xlim()
ylim = ax.get_ylim()
ax.set_xlim(xlim)
ax.set_ylim(ylim)
#plt.axes('equal')
xplot = np.linspace(xlim[0], xlim[1], 1000)

new_y_axis_values = linear_function_from_vector(xplot, new_y_axis, (mu_x, mu_y))
new_x_axis_values = linear_function_from_vector(xplot, new_x_axis, (mu_x, mu_y))

ax.plot(xplot,
        linear_function_from_vector(xplot, new_x_axis, (mu_x, mu_y)),
        color = 'r',
        linestyle = '--',
        label = 'Hauptachsen')
ax.plot(xplot,
        linear_function_from_vector(xplot, new_y_axis, (mu_x, mu_y)),
        color = 'r',
        linestyle = '--')

y = linear_function_from_vector(xplot, new_x_axis, (mu_x, mu_y))
mask = np.sqrt((xplot - mu_x)**2 + (y - mu_y)**2) <= newsigma[1]
ax.plot(xplot[mask], y[mask], linewidth = 5, color = 'k', label = '$\sigma\'$')

y = linear_function_from_vector(xplot, new_y_axis, (mu_x, mu_y))
mask = np.sqrt((xplot - mu_x)**2 + (y - mu_y)**2) <= newsigma[0]
ax.plot(xplot[mask], y[mask], linewidth = 5, color = 'k')

ax.legend()
plt.show()
```

__e)__

$$ f(y\,|\,x) = \frac{f(x,y)}{g(x)} \; mit \; g(x) = \int_{-\infty}^{\infty} f(x,y) \mathrm{dy}$$
$$f(x,y) = \frac{1}{2\pi\sigma_y\sigma_x\sqrt{1-\rho^2}} \exp{\left(-\frac{1}{2(1-\rho^2)} \left[\left(\frac{x-\mu_x}{\sigma_x}\right)^2 -2\rho\left(\frac{x-\mu_x}{\sigma_x}\right)\left(\frac{y-\mu_y}{\sigma_y}\right) +
\left(\frac{y-\mu_y}{\sigma_y}\right)^2\right]\right)} $$

$$\mathrm{nutze} \; u_i = \left( \frac{i-\mu_i}{\sigma_i} \right) , \lambda = \frac{1}{2\pi\sigma_y\sigma_x\sqrt{1-\rho^2}} \; \mathrm{und} \; \gamma = \frac{1}{2(1-\rho^2)} : $$
$$f(u_x, u_y) = \lambda \exp{(-\gamma\left[u_x^2 - 2\rho u_x u_y + u_y^2\right])} $$

\begin{align} g(x) = \lambda \sigma_y \exp{(-\gamma u_x^2)} \int_{-\infty}^{\infty} \exp{(-\gamma([u_y - \rho u_x]^2 -\rho^2u_x^2))} \,
\mathrm{d u_y} \\
= \lambda \sigma_y \exp{(-\gamma u_x^2)} \exp{(\gamma \rho^2 u_x^2)} \int_{-\infty}^{\infty} \exp{(-\gamma[u_y-\rho u_x]^2)} \,
\mathrm{du_y} \\
= \frac{\lambda}{\sqrt{\gamma}} \sigma_y \exp{(-u_x^2[\gamma-\gamma\rho^2])} \int_{-\infty}^{\infty} \exp{(-\theta^2)}
\,\mathrm{d\theta} \\
= \frac{\lambda}{\sqrt{\gamma}} \sigma_y \exp{(-u_x^2[\gamma-\gamma\rho^2])} \sqrt{\pi} \end{align}

$$ f(y\,|\,x) = \frac{\sqrt{\gamma}}{\sigma_y \sqrt{\pi}}\exp{(-\gamma[u_y^2-2\rho u_xu_y + \rho^2u_x^2])}$$

Analog ergeben sich auch $h(y)$ und $f(x\,|\,y)$

\begin{align} h(y) = \frac{\lambda}{\sqrt{\gamma}} \sigma_x \exp{(-u_y^2[\gamma-\gamma\rho^2])} \sqrt{\pi} \\
f(x\,|\,y) = \frac{\sqrt{\gamma}}{\sigma_x \sqrt{\pi}}\exp{(-\gamma[u_x^2-2\rho u_xu_y + \rho^2u_y^2])} \end{align}

__f)__

$$ \mathrm{E}(x\,|y\,) = \int_{-\infty}^{\infty} x \,  f(x\,|\,y) \, \mathrm{dx} $$

$$ u_x = \frac{x-\mu_x}{\sigma_x} \Leftrightarrow x = u_x\sigma_x + \mu_x $$

\begin{align} \mathrm{E}(x\,|\,y) = \int_{-\infty}^{\infty}(u_x\sigma_x + \mu_x)\frac{\sqrt{\gamma}}{\sigma_x\sqrt{\pi}} \exp{(-\gamma[u_x-\rho u_y]^2)} \sigma_x \, \mathrm{du_x} \\
= \int_{-\infty}^{\infty} u_x\sigma_x\sqrt{\frac{\gamma}{\pi}}\exp{(-\gamma[u_x-\rho u_y]^2)}\,\mathrm{du_x} + \int_{-\infty}^{\infty}\mu_x\sqrt{\frac{\gamma}{\pi}}\exp{(-\gamma[u_x-\rho u_y]^2)}\,\mathrm{du_x} \\
\mathrm{substituiere} \; \sqrt{\gamma}(u_x-\rho u_y) = z \\
= \frac{\sigma_x\sqrt{\gamma}}{\sqrt{\pi}}\int_{-\infty}^{\infty}(\frac{1}{\sqrt{\gamma}}z+\rho u_y)\exp{(-z^2)}\frac{1}{\sqrt{\gamma}}\mathrm{dz} + \frac{\mu_x}{\sqrt{\pi}}\int_{-\infty}^{\infty} \exp{(-z^2)}\mathrm{dz} \\
= \sigma_x\rho u_y + \mu_x = \rho \frac{\sigma_x}{\sigma_y}(y-\mu_y) + \mu_x  \end{align}

Analog wieder für $\mathrm{E}(y\,|\,x)$

$$\mathrm{E}(y\,|\,x) = \rho \frac{\sigma_y}{\sigma_x}{x-\mu_x}+\mu_x $$

Beide Geraden sind in Aufgabenteil c) bereits eingezeichnet


## Linear Representations

1. Homomorphism $\alpha: G \rightarrow GL(n,k)$ where n is the dimension of a vector space over field $k$. That is $n \times n $ matrices.
2. A group action $G \times X \rightarrow X:(g,x) \mapsto x$ where
 >1. $X$ is vector space and
 >2. $(gh)x) = g(hx)$
 >3. $ex=x$
 >4. $g(u + v) = gu + gv$
 >5. $g(ρv) = ρ(gv)$

---
A group action on a set of finite size is a subgroup of permutations, they may be represented as special matrices of size n

for example:  $\begin{pmatrix}
1 & 2 & 3\\
2 & 1 & 3
\end{pmatrix} \Leftrightarrow
\begin{pmatrix}
0 & 1 & 0\\
1 & 0 & 0\\
0 & 0 & 1
\end{pmatrix}$

---

A 1-dimensional representation is simply a homomorphism to multiplicative group of the field.

$g \mapsto 1$ is the trivial representation.

a 1-dimensional representation is constant on each conjugacy class

for $S_n$:

$g \mapsto \left
\{\begin{matrix}
1 & if \ g \ is \ even \\
-1 & if \ g \ is \ odd
\end{matrix}\right.$

for $D_4$:
$r^4=1, c^2=1, (rc)^2=1 \Rightarrow c=\pm1,r=\pm1$
there are 4 1-dimensional representations

2-dimensional representations of $S_3$: $\langle s, t \vert s^3,t^2, st=ts^2  \rangle$ over $\mathbb{C}$
suppose $s$ has a eigenvector $v_1$ with eigenvalue $\lambda \ne 1, \ \lambda^3=1, \ \lambda = \exp(i2\pi/3)$
$s(tv_1) = ts^2v_1=\lambda^2(tv_1) \Rightarrow s \mapsto \begin{pmatrix}
\lambda & 0 \\
0 & \lambda^2
\end{pmatrix}$ and the next eigenvector is $v_2==tv_1$
on the other hand $tv_1=v_2, tv_2=v_1 \Rightarrow t \mapsto \begin{pmatrix}
0 & 1 \\
1 & 0
\end{pmatrix}$

the other case is $s \mapsto \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$, then t will be free: $t \mapsto \begin{pmatrix} \lambda & 0 \\ 0 & \mu \end{pmatrix}$, $\lambda,\mu=\pm1$

The representation $\alpha$ of $G$ in the vector space $V$ over $k$ is said to
be **simple (irreducible)** if no proper subspace of $V$ is stable under $G$.

$dim \ \alpha \le \vert G \vert$

A simple representation of a finite abelian group over $\mathbb{C}$ is necessarily 1-dimensional. (commutative matrices can be diagonalised simultaneously)

**Arithmetic of Representations**

$\alpha:g \mapsto A(g), \  \beta:g \mapsto B(g), \ \alpha+\beta: g \mapsto \begin{pmatrix} A(g) & 0 \\ 0 & B(g)\end{pmatrix}$. The representation space is  $V \oplus W$

$\alpha:g \mapsto A(g), \  \beta:g \mapsto B(g), \ \alpha\beta: g \mapsto A(g) \otimes B(g)$. . The representation space is  $V \otimes W$

> $\alpha+\beta=\beta+ \alpha$
> $\alpha+(\beta+\gamma)=(\alpha+\beta)+ \gamma$
> $\alpha\beta=\beta \alpha$
> $\alpha(\beta\gamma)=(\alpha\beta) \gamma$
> $\alpha(\beta+\gamma)=\alpha\beta+ \alpha\gamma$

Conjugacy: if $\alpha$ is the representation of $G$ in $V$, $\alpha^*$ is the representation of $G$ in dual space $V^*$
$\alpha:g \mapsto A(g), \ \alpha^* \mapsto (A(g)^{-1})^\prime$

> $(\alpha^*)^*=\alpha$
> $(\alpha+\gamma)^*=\alpha^*+\beta^*$
> $(\alpha\beta)^*=\alpha^*\beta^*$

The represenation is said to be **semisimple** if it is expressible as a sum of simple representations:
$\alpha = \sigma_1+\sigma_2+ ... + \sigma_r$ .
Equivalently each stable subspace $U \subset V$ has at least one complementary stable subspace $W$, ie $V=U \oplus W$.

---

Let $Q$ be a positive (hermitian) form in $\mathbb{R}^n$ ($\mathbb{C}^n$). Avaraging it over group:
$P(u,v) =\frac{1}{\vert G\vert} \sum_ {g \in G} Q(gu,gv)$

$P(u,v)=0 \Rightarrow gu \perp gv \ for \ all \ g \in G$ and $u \in U \Rightarrow v \in U^{\bot}$

---

**Maschke’s Theorem**: Suppose α is a representation of the finite
group $G$ over $k$ ($\vert G \vert \ne 0 \ in \ k$). Then α is semisimple.

The expression for a semisimple representation $\alpha$ as a sum of simple parts $\alpha = \sigma_1+\sigma_2+ ... + \sigma_r$ is unique up to order.

The character $\chi_\alpha:G \rightarrow k$ is defined as $\chi(g)=tr(\alpha(g))$
> $\chi_{\alpha+\beta}(g)=\chi_\alpha(g)+\chi_\beta(g)$
> $\chi_{\alpha\beta}(g)=\chi_\alpha(g)\chi_\beta(g)$
> $\chi_{\alpha^*}(g)=\chi_{\alpha}(g^{-1})$
> $\chi_{1}(g)=1$
> $\chi_\alpha(e)=dim \alpha$

Two representations are equivalent if and only if their characters are equal:
> $\alpha = \beta \Leftrightarrow \chi_\alpha(g)=\chi_\beta(g) \ for \ all \ g\in G$

For representations of $G$ over $C$
>$\chi (I) = n$
>$\vert \chi(g) \vert \le n$,
>$\vert \chi(g) \vert = n$ implies $g = e^{iθ}I$
>$\chi$ is constant on any given conjugacy class of $G$
>$\chi (g^{−1}) = \chi^*(g)$
>$\chi(g)$ is an algebraic number for all $g$.

1. The number of simple representations of $G$ over $C$ is equal to the number of conjugacy classes in $G$
2. The dimensions of the simple representations $σ_1, . . . , σ_s$ of $G$ over $C$ satisfy the relation
>$dim^2 σ_1 + · · · + dim^2 σ_s = \vert G \vert$.
3. The dimension of each simple representation $σ_i$ divides the order of the group: $dim σ_i\> |\> \vert G \vert$.

$\alpha$ is representation of $G$ and $\beta$ is representation of $H$, then $\alpha \times \beta$ (over $U \otimes V$) is representation of $G \times H$. and $\alpha \times \beta$ is simple iff $\alpha$ and $\beta$ are simple.

Theorem. A representation of $G$ is irreducible if and only if $\sum_{g \in G} |\chi(g)|^2 = \vert G \vert$

The characters are orthogonal, that is: $\sum_{i=1}^{r} r_i (\chi^p_i)^* \chi^q_i = |G|\delta_{pq}$ and $\sum_{p=1}^r (\chi^p_i)^* \chi^q_j = \frac{|G|}{r_i}\delta_{ij}$
where $p,q$ denote representations, $\chi^p_i$ is the character of $i$th conjugacy calss in $p$th irreducable representation, and $r_i$ is size of $i$th conjugacy class.

Theorem. If $\rho$ is an arbitrary representation of $G$ with character $\chi$, and $\rho^p$ are the
inequivalent irreducible representations of $G$ with characters $\chi^p$, then $\rho = \bigoplus_p c_p \rho^p$, where
$c_p = \frac{1}{|G|} \sum_{i=1}^{r} r_i (\chi_i^p)^*\chi_i$

The regular representation contains $dim \rho^p$ instances of each irreducible representation: $\chi_i^R = \sum (dim \rho^p)\chi_i^p$

```python
# Z3 sympy exercises
import sympy.combinatorics.permutations as P
import sympy.combinatorics.named_groups as G
from sympy.matrices import eye
from sympy import *
init_printing(use_latex='mathjax')
P.Permutation.print_cyclic
c = G.CyclicGroup(3)
el = [e for e in c.elements]
el.sort(key=lambda x: c.coset_rank(x))
display(el)
m = eye(3)
regR = [m.permuteFwd(e) for e in el]
display(regR)
lamda = symbols('lamda')
poly=[factor(p.charpoly(lamda)) for p in regR]
display(poly)
(p1,d1) = regR[1].diagonalize()

(p2,d2) = regR[2].diagonalize()
#display(d1)

p1_=simplify(p1**-1)
display([simplify(p1_*regR[1]*p1),simplify(p1_*regR[2]*p1)])

```

$$\left [ \left( 2\right), \quad \left( 0\; 1\; 2\right), \quad \left( 0\; 2\; 1\right)\right ]$$

$$\left [ \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \quad \left[\begin{matrix}0 & 1 & 0\\0 & 0 & 1\\1 & 0 & 0\end{matrix}\right], \quad \left[\begin{matrix}0 & 0 & 1\\1 & 0 & 0\\0 & 1 & 0\end{matrix}\right]\right ]$$

$$\left [ \left(\lambda - 1\right)^{3}, \quad \left(\lambda - 1\right) \left(\lambda^{2} + \lambda + 1\right), \quad \left(\lambda - 1\right) \left(\lambda^{2} + \lambda + 1\right)\right ]$$

$$\left [ \left[\begin{matrix}1 & 0 & 0\\0 & - \frac{1}{2} - \frac{\sqrt{3} i}{2} & 0\\0 & 0 & - \frac{1}{2} + \frac{\sqrt{3} i}{2}\end{matrix}\right], \quad \left[\begin{matrix}1 & 0 & 0\\0 & - \frac{1}{2} + \frac{\sqrt{3} i}{2} & 0\\0 & 0 & - \frac{1}{2} - \frac{\sqrt{3} i}{2}\end{matrix}\right]\right ]$$

```python
w = exp(I*2*pi/3)
display(w,w.expand(complex=true))
m = Matrix([[0,-1],[1,-1]])
display([m,m.inv(),m*m])
```

$$e^{\frac{2 \pi}{3} i}$$

$$- \frac{1}{2} + \frac{\sqrt{3} i}{2}$$

$$\left [ \left[\begin{matrix}0 & -1\\1 & -1\end{matrix}\right], \quad \left[\begin{matrix}-1 & 1\\-1 & 0\end{matrix}\right], \quad \left[\begin{matrix}-1 & 1\\-1 & 0\end{matrix}\right]\right ]$$

3 representations over $C$: >$\{1,1,1\}$ $\{1,w,w^2\}$ $\{1,w^2,w\}$

not trivial representation over $N$: $\begin{bmatrix}
1 & 0\\
0 & 1
\end{bmatrix}$,
$\begin{bmatrix}
0 & -1\\
1 & -1
\end{bmatrix}$,
$\begin{bmatrix}
-1 & 1\\
-1 & 0
\end{bmatrix}$

not trivial orthogonal representation over $R$: $\begin{bmatrix}
1 & 0\\
0 & 1
\end{bmatrix}$,
$\begin{bmatrix}
-\frac{1}{2} & -\frac{\sqrt 3}{2}\\
\frac{\sqrt 3}{2} & -\frac{1}{2}
\end{bmatrix}$,
$\begin{bmatrix}
-\frac{1}{2} & \frac{\sqrt 3}{2}\\
-\frac{\sqrt 3}{2} & -\frac{1}{2}
\end{bmatrix}$

## Pauli matrices

$\{±I,±iI,±X,±iX,±Y,±iY,±Z,±iZ\}$
One of Generator Sets: $\langle iI,X,Y \rangle$

Center is $\{±I,±iI\}$

```python
e, X, Y, Z = symbols('e X Y Z')

t = [e,X,Y,Z]
pauliSymbols = [e,X,Y,Z]
pauliSymbols.extend([-l for l in t])
pauliSymbols.extend([I*l for l in t])
pauliSymbols.extend([-I*l for l in t])

display([(e,pauliSymbols[e]) for e in range(16)])
```

$$\left [ \left ( 0, \quad e\right ), \quad \left ( 1, \quad X\right ), \quad \left ( 2, \quad Y\right ), \quad \left ( 3, \quad Z\right ), \quad \left ( 4, \quad - e\right ), \quad \left ( 5, \quad - X\right ), \quad \left ( 6, \quad - Y\right ), \quad \left ( 7, \quad - Z\right ), \quad \left ( 8, \quad i e\right ), \quad \left ( 9, \quad i X\right ), \quad \left ( 10, \quad i Y\right ), \quad \left ( 11, \quad i Z\right ), \quad \left ( 12, \quad - i e\right ), \quad \left ( 13, \quad - i X\right ), \quad \left ( 14, \quad - i Y\right ), \quad \left ( 15, \quad - i Z\right )\right ]$$

```python
from sympy.combinatorics.perm_groups import PermutationGroup
e8 = P.Permutation([8,9,10,11, 12,13,14,15, 4,5,6,7,  0,1,2,3])
e1 = P.Permutation([1,0,11,14, 5,4,15,10,   9,8,7,2,  13,12,3,6])
e2 = P.Permutation([2,15,0,9,  6,11,4,13,   10,3,8,5, 14,7,12,1])
display(e8)
display(e1)
display(e2)

p = PermutationGroup(e8,e1,e2)
display(p.degree)
p.center()
```

$$\left( 0\; 8\; 4\; 12\right)\left( 1\; 9\; 5\; 13\right)\left( 2\; 10\; 6\; 14\right)\left( 3\; 11\; 7\; 15\right)$$

$$\left( 0\; 1\right)\left( 2\; 11\right)\left( 3\; 14\right)\left( 4\; 5\right)\left( 6\; 15\right)\left( 7\; 10\right)\left( 8\; 9\right)\left( 12\; 13\right)$$

$$\left( 0\; 2\right)\left( 1\; 15\right)\left( 3\; 9\right)\left( 4\; 6\right)\left( 5\; 11\right)\left( 7\; 13\right)\left( 8\; 10\right)\left( 12\; 14\right)$$

$$16$$

$$PermutationGroup([
    (15),
    (0 4)(1 5)(2 6)(3 7)(8 12)(9 13)(10 14)(11 15),
    (0 8 4 12)(1 9 5 13)(2 10 6 14)(3 11 7 15),
    (0 12 4 8)(1 13 5 9)(2 14 6 10)(3 15 7 11)])$$

```python
from sympy.combinatorics.free_groups import free_group
from sympy.combinatorics.fp_groups import FpGroup
from sympy.combinatorics.fp_groups import low_index_subgroups
F, i, x, y = free_group("i, x y")
G = FpGroup(F, [i**4, x**2, i*x*i**-1*x**-1, y**2, i*y*i**-1*y**-1, (x*y)**4, i*i*x*y*x**-1*y**-1])
print("Order is:",G.order())
l = low_index_subgroups(G,16)
print("SubGroup Count is:",len(l))
#for t in l:
#    print(len(t.table))
```

    Order is: 16
    SubGroup Count is: 20

## Fourier transforms

Let $f$ be a function which maps group elements to complex numbers. For an irreducible representation $\rho$ of $G$, of dimension $d_\rho$, we define the Fourier transform of $f$ to be: $$\widehat{f}(\rho) \equiv \sqrt{\frac{d_\rho}{|G|}} \sum_{g \in G} f(g) \rho(g)$$

Inverse fourier transform will be: $$f(g) = \frac{1}{\sqrt{|G|}} \sum_{\rho in \widehat{G}} \sqrt{d_{\rho}} tr(\widehat{f}(\rho) \rho(g^{-1}))$$

for $Z_n$ the formules simplify to usual finite discrete fourier transforms.

## Links

[breif intro to Representations of finite groups](https://www.maths.tcd.ie/pub/Maths/Courseware/GroupRepresentations/FiniteGroups.pdf)

[combinatorics, Permutation Group](http://www.di.fc.ul.pt/~jpn/r/combinatorics/combinatorics.html)

[EFFICIENT COMPUTATION OF THE FOURIER TRANSFORM ON FINITE GROUPS](https://www.ams.org/journals/jams/1990-03-02/S0894-0347-1990-1030655-4/S0894-0347-1990-1030655-4.pdf)

[RECENT RESULTS IN GENERALIZED FAST FOURIER TRANSFORMS](https://www.math.hmc.edu/seniorthesis/archives/2005/emalm/emalm-2005-expository.pdf)

https://en.wikipedia.org/wiki/Noncommutative_harmonic_analysis


## 8. Consider the point group 4mm. It can be generated from a 4-fold rotation axis about the c-axis and a coinciding mirror plane about the a-c plane.

## a. Write down the 3D matrix representations for the two generators in the tetragonal basis.

Solution:

The 4-fold rotation axis maps the $\mathbf{a}$ and $\mathbf{b}$ lattice vectors to $\mathbf{a}'$ and $\mathbf{b}'$, where $\mathbf{a}'$ = $\mathbf{b}$ and $\mathbf{b}'$ = -$\mathbf{a}$. Hence, the matrix is

```
D_4 = [[0, -1, 0],
       [1, 0, 0],
       [0, 0, 1]]
```

The mirror plane maps $\mathbf{b}$ lattice vector to $\mathbf{b}'$, where $\mathbf{b}'$ = -$\mathbf{b}$. Hence, the matrix is

```
D_m = [[1, 0, 0],
       [0, -1, 0],
       [0, 0, 1]]
```

## b. Determine the matrix representations for all symmetry elements in the group, and construct a multiplication table for the 4mm point group.

Solution: All symmetry elements can be obtained by the multiplication of the generators.

```
import numpy as np

D = {}
D["4"] = np.array(D_4)
D["m"] = np.array(D_m)
D["4_2"] = np.dot(D["4"], D["4"])
D["4_3"] = np.dot(D["4"], D["4_2"])
D["E"] = np.dot(D["4"], D["4_3"])
D["m_4"] = np.dot(D["m"], D["4"])
D["m_4_2"] = np.dot(D["m"], D["4_2"])
D["m_4_3"] = np.dot(D["m"], D["4_3"])

print "The complete set of symmetry elements are given by the following matrices:"
for k, v in D.items():
    print "%s = %s" % (k, str(v))
```

    The complete set of symmetry elements are given by the following matrices:
    E = [[1 0 0]
     [0 1 0]
     [0 0 1]]
    m = [[ 1  0  0]
     [ 0 -1  0]
     [ 0  0  1]]
    4 = [[ 0 -1  0]
     [ 1  0  0]
     [ 0  0  1]]
    m_4_2 = [[-1  0  0]
     [ 0  1  0]
     [ 0  0  1]]
    m_4_3 = [[0 1 0]
     [1 0 0]
     [0 0 1]]
    4_2 = [[-1  0  0]
     [ 0 -1  0]
     [ 0  0  1]]
    4_3 = [[ 0  1  0]
     [-1  0  0]
     [ 0  0  1]]
    m_4 = [[ 0 -1  0]
     [-1  0  0]
     [ 0  0  1]]

The multiplication table is given below:

```
import itertools
keys = ["E", "4", "4_2", "4_3", "m", "m_4", "m_4_2", "m_4_3"]
def find_key(m):
    for k, v in D.items():
        if np.all(v == m):
            return k
from prettytable import PrettyTable

t = PrettyTable([""] + keys)
headers = list(keys)
for k1 in keys:
    row = [k1]
    for k2 in keys:
        prod = np.dot(D[k1], D[k2])
        row.append(find_key(prod))
    t.add_row(row)
print t
```

    +-------+-------+-------+-------+-------+-------+-------+-------+-------+
    |       |   E   |   4   |  4_2  |  4_3  |   m   |  m_4  | m_4_2 | m_4_3 |
    +-------+-------+-------+-------+-------+-------+-------+-------+-------+
    |   E   |   E   |   4   |  4_2  |  4_3  |   m   |  m_4  | m_4_2 | m_4_3 |
    |   4   |   4   |  4_2  |  4_3  |   E   | m_4_3 |   m   |  m_4  | m_4_2 |
    |  4_2  |  4_2  |  4_3  |   E   |   4   | m_4_2 | m_4_3 |   m   |  m_4  |
    |  4_3  |  4_3  |   E   |   4   |  4_2  |  m_4  | m_4_2 | m_4_3 |   m   |
    |   m   |   m   |  m_4  | m_4_2 | m_4_3 |   E   |   4   |  4_2  |  4_3  |
    |  m_4  |  m_4  | m_4_2 | m_4_3 |   m   |  4_3  |   E   |   4   |  4_2  |
    | m_4_2 | m_4_2 | m_4_3 |   m   |  m_4  |  4_2  |  4_3  |   E   |   4   |
    | m_4_3 | m_4_3 |   m   |  m_4  | m_4_2 |   4   |  4_2  |  4_3  |   E   |
    +-------+-------+-------+-------+-------+-------+-------+-------+-------+

## c. What are the subgroups of the 4mm point group?

Solution:

4, m, m_4, m_4_2 and m_4_3 are subgroups of 4mm.

## d. Draw a stereographic projection for the point group.

Solutinon:

Refer to Structure of Materials Chapter 9, Fig 9.7.

## e. Determine the orbit for the general position (x, y, z).

Solution:

The orbit of the generalposition is given by:

```
from sympy import symbols
x, y, z = symbols("x y z")
p = [x, y, z]
for k, v in D.items():
    print np.dot(v, p)
```

    [x y z]
    [x -y z]
    [-y x z]
    [-x y z]
    [y x z]
    [-x -y z]
    [y -x z]
    [-y -x z]

## f. Identify possible special positions and determine the orbits of points lying on these special positions.

Solution:

There are special positions on the 4-fold axis (0, 0, z), the mirror plane (0, y, 0) and the diagonal mirror plane (x, x, z).

The orbit for the special position (0, 0, z) on the 4-fold axis is given by:

```
p = [0, 0, z]
orbit = []
for k, v in D.items():
    orbit.append(str(np.dot(v, p)))
for o in set(orbit):
    print o
```

    [0 0 z]

The orbit for the special position (0, y, 0) on the mirror plane on the a-c plane is given by:

```
p = [0, y, 0]
orbit = []
for k, v in D.items():
    orbit.append(str(np.dot(v, p)))
for o in set(orbit):
    print o
```

    [0 y 0]
    [0 -y 0]
    [-y 0 0]
    [y 0 0]

The orbit for the special position (x, x, z) on the diagonal mirror plane is given by:

```
p = [x, x, z]
orbit = []
for k, v in D.items():
    orbit.append(str(np.dot(v, p)))
for o in set(orbit):
    print o
```

    [-x x z]
    [x x z]
    [-x -x z]
    [x -x z]

**Note that all special positions have orbits that have < 8 elements.**