```python
%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_context('notebook', font_scale=1.5)
```

```python
import warnings
warnings.simplefilter('ignore', FutureWarning)
```

**1**. (25 points)

In this exercise, we will practice using Pandas dataframes to explore and summarize a data set `heart`.

This data contains the survival time after receiving a heart transplant, the age of the patient and whether or not the survival time was censored

- Number of Observations - 69
- Number of Variables - 3

Variable name definitions::

- survival - Days after surgery until death
- censors - indicates if an observation is censored. 1 is uncensored
- age - age at the time of surgery

Answer the following questions (5 points each) with respect to the `heart` data set:

- How many patients were censored?
- What is the correlation coefficient between age and survival for uncensored patients?
- What is the average age for censored and uncensored patients?
- What is the average survival time for censored and uncensored patients under the age of 45?
- What is the survival time of the youngest and oldest uncensored patient?

```python
import statsmodels.api as sm
heart = sm.datasets.heart.load_pandas().data
heart.head(n=6)
```

      survival
      censors
      age

      0
      15.0
      1.0
      54.3

      1
      3.0
      1.0
      40.4

      2
      624.0
      1.0
      51.0

      3
      46.0
      1.0
      42.5

      4
      127.0
      1.0
      48.0

      5
      64.0
      1.0
      54.6

```python
# How many patients were censored?

sum(heart.censors == 0)
```

    24

```python
# What is the correlation coefficient between age and survival for uncensored patients?

uncensored = heart[heart.censors == 1]
np.corrcoef(uncensored.age, uncensored.survival)[0,1]
```

    0.003256499283211926

```python
# What is the average age for censored and uncensored patients?

heart.groupby('censors')['age'].mean()
```

    censors
    0.0    41.729167
    1.0    48.484444
    Name: age, dtype: float64

```python
# What is the average survival time for censored and uncensored patients under the age of 45?

young = heart[heart.age < 45]
young.groupby('censors')['survival'].mean()
```

    censors
    0.0    712.818182
    1.0    169.909091
    Name: survival, dtype: float64

```python
# What is the survival time of the youngest and oldest uncensored patient?

uncensored.survival[np.argmin(uncensored.age)], uncensored.survival[np.argmax(uncensored.age)]
```

    (228.0, 60.0)

**2**. (35 points)

- Consider a sequence of $n$ Bernoulli trials with success probabilty $p$ per trial. A string of consecutive successes is known as a success *run*. Write a function that returns the counts for runs of length $k$ for each $k$ observed in a dictionary. (10 points)

For example: if the trials were [0, 1, 0, 1, 1, 0, 0, 0, 0, 1], the function should return
```
{1: 2, 2: 1})
```

Test that it does so.

- What is the probability of observing at least one run of length 5 or more when $n=100$ and $p=0.5$?. Estimate this from 100,000 simulated experiments. Is this more, less or equally likely than finding runs of length 7 or more when $p=0.7$? (10 points)

- There is an exact solution

$$
s_n = \sum_{i=1}^n{f_i} \\
f_n = u_n - \sum_{i=1}^{n-1} {f_i u_{n-i}} \\
u_n = p^k - \sum_{i=1}^{k-1} u_{n-i} p^i
$$

Implement the exact solution using caching to avoid re-calculations and calculate the same two probabilities found by simulation. (15 points)

**Part 1**

```python
from collections import Counter

def count_runs(xs):
    """Count number of success runs of length k."""
    ys = []
    count = 0
    for x in xs:
        if x == 1:
            count += 1
        else:
            if count: ys.append(count)
            count = 0
    if count: ys.append(count)
    return Counter(ys)
```

```python
count_runs([0, 1, 0, 1, 1, 0, 0, 0, 0, 1],)
```

```python
count_runs(np.random.randint(0,2,1000000))
```

**Part 2**

```python
def run_prob(expts, n, k, p):
    """Probability of observing at least one run of lenght k or more by simuulation."""
    xxs = np.random.choice([0,1], (expts, n), p=(1-p, p))
    return sum(max(d.keys()) >= k for d in map(count_runs, xxs))/expts
```

```python
run_prob(expts=100000, n=100, k=5, p=0.5)
```

```python
run_prob(expts=100000, n=100, k=7, p=0.7)
```

**Part 3**

```python
from functools import lru_cache

@lru_cache()
def s(n, k, p):
    """Probability of observing at least one run of lenght k or more."""
    return sum(f(i, k, p) for i in range(1, n+1))

@lru_cache()
def f(n, k, p):
    """Helper function for s."""
    return u(n, k, p) - sum(f(i, k, p) * u(n-i, k, p) for i in range(1, n))

@lru_cache()
def u(n, k, p):
    """Helper function for f."""
    if n < k:
        return 0
    return p**k - sum(u(n-i, k, p) * p**i for i in range(1, k))
```

```python
s(100, 5, 0.5)
```

    0.8101095991963579

```python
s(100, 7, 0.7)
```

    0.9491817984156692

**3**. (40 points)

Given matrix $M$
```python
      [[7, 8, 8],
       [1, 3, 8],
       [9, 2, 1]]
```

- Normalize the given matrix $M$ so that all rows sum to 1.0.  (5 points)
- The normalized matrix can then be considered as a transition matrix $P$ for a Markov chain. Find the stationary distribution of this matrix in the following ways using `numpy` and `numpy.linalg` (or `scipy.linalg`):
    - By repeated matrix multiplication of a random probability vector $v$ (a row vector normalized to sum to 1.0) with $P$ using matrix multiplication with `np.dot`. (5 points)
    - By raising the matrix $P$ to some large power until it doesn't change with higher powers (see `np.linalg.matrix_power`) and then calculating $vP$ (10 points)
    - From the equation for stationarity $wP = w$, we can see that $w$ must be a left eigenvector of $P$ with eigenvalue $1$ (Note: np.linalg.eig returns the right eigenvectors, but the left eigenvector of a matrix is the right eigenvector of the transposed matrix). Use this to find $w$ using `np.linalg.eig`. (20 points)

Suppose $w = (w_1, w_2, w_3)$. Then from $wP = w$, we have:
\begin{align}
w_1 P_{11} + w_2 P_{21} + w_3 P_{31} &= w_1 \\
w_1 P_{12} + w_2 P_{22} + w_3 P_{32} &= w_2 \\
w_1 P_{13} + w_2 P_{23} + w_3 P_{331} &= w_3 \\
\end{align}
This is a singular system, but we also know that $w_1 + w_2 + w_3 = 1$. Use these facts to set up a linear system of equations that can be solved with `np.linalg.solve` to find $w$.

**Part 1**

```python
M = np.array([7,8,8,1,3,8,9,2,1.0]).reshape(3,3)
M
```

    array([[7., 8., 8.],
           [1., 3., 8.],
           [9., 2., 1.]])

```python
# Normalize the matrix so that rows sum to 1
P = M/np.sum(M, 1)[:, np.newaxis]
P
```

    array([[0.30434783, 0.34782609, 0.34782609],
           [0.08333333, 0.25      , 0.66666667],
           [0.75      , 0.16666667, 0.08333333]])

**Part 2**

```python
v = np.random.random(3)
v /= v.sum()
v
```

    array([0.07208584, 0.47664814, 0.45126603])

```python
# By raising the matrix $P$ to some large power
P50 = np.linalg.matrix_power(P, 50)
P51 = np.dot(P50, P)
# check that P50 is stationary
np.testing.assert_allclose(P50, P51)
np.dot(v, P51)
```

    array([0.39862184, 0.2605972 , 0.34078096])

**Part 3**

```python
import scipy.linalg as la
```

```python
lam, vec = la.eig(P, left=True, right=False)
idx = np.argmin(np.abs(lam - 1))
w = np.real(vec[:, idx])
w/w.sum()
```

    array([0.39862184, 0.2605972 , 0.34078096])

```python
# Left eigenvector with eigenvalue 1
# note transpose of P to find left eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(P.T)
# find index of eigenvalue = 1
idx = np.argmin(np.abs(eigenvalues - 1))
w = np.real(eigenvectors[:, idx]).T
# remember to normalize eigenvector to get a probability distribution
w/np.sum(w)
```

    array([0.39862184, 0.2605972 , 0.34078096])

**Part 4**

We want to solve

\begin{align}
w_1 P_{11} + w_2 P_{21} + w_3 P_{31} &= w_1 \\
w_1 P_{12} + w_2 P_{22} + w_3 P_{32} &= w_2 \\
w_1 + w_2 + w_3 &= 1
\end{align}

In matrix form, we have

$$
\begin{pmatrix}
P_{11} - 1 & P_{21} & P_{31}\\
P_{12} & P_{22} -1 & P_{32} \\
1 & 1 & 1
\end{pmatrix}
\begin{pmatrix}
w_1 \\
w_2 \\
w_3
\end{pmatrix}
 =
\begin{pmatrix}
0 \\ 0 \\ 1
\end{pmatrix}
$$

Knowing this, it is just a matter of setting up the system for `np.linalg.solve` to work.

```python
# Solving linear system
# Use the first 2 rows of  the matrix P - I = 0
# and construnct the last row from $w_1 + w_2 + w_3 = 1$
A = P.T - np.eye(3)
A[2] = [1,1,1]
np.linalg.solve(A, [0,0,1])
```

    array([0.39862184, 0.2605972 , 0.34078096])

```python

```


```python
import os, sys
import h5py
import numpy as np
from scipy.io import loadmat
import cv2
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from numpy import matrix as mat
from sympy import *
from numpy import linalg as la
```

```python
def getFx(para, frame):   # para中为(13*frame+3)的一套参数，frame传进来是为了确定循环次数
    #先写出参数表达式，ABDCDEF六个点的齐次坐标
    K = Matrix([[1149.67569986785, 0.0, 508.848621645943],
                     [0.0, 1147.59161666764, 508.064917088557],
                     [0.0, 0.0, 1.0]])
    r11, r12, r13, r14, r21, r22, r23, r24, r31, r32, r33 = symbols('r11 r12 r13 r14 r21 r22 r23 r24 r31 r32 r33')
    Rt = Matrix([[r11, r12, r13, r14], [r21, r22, r23, r24], [r31, r32, r33, 1]])
    a, b, c, th, al = symbols('a b c th al')
    ua, va, wa, ub, vb, wb, uc, vc, wc, ud, vd, wd, ue, ve, we, uf, vf, wf = symbols('ua va wa ub vb wb uc vc wc ud vd wd ue ve we uf vf wf')
    f = Symbol('f')
    XA = Matrix([[-a * c * cos(th) * cos(al)], [c-a * c * sin(th)], [-a * c *  cos(th) * sin(al)], [1]])
    XB = Matrix([[0], [c], [0], [1]])
    XC = Matrix([[a * c * cos(th) * cos(al)], [c+a * c * sin(th)], [a * c * cos(th) * sin(al)], [1]])
    XD = Matrix([[-b * c], [0], [0], [1]])
    XE = Matrix([[0], [0], [0], [1]])
    XF = Matrix([[b * c], [0], [0], [1]])
    ua, va, wa = K[0,:] * (Rt * XA), K[1,:] * (Rt * XA), K[2,:] * (Rt * XA)
    ub, vb, wb = K[0,:] * (Rt * XB), K[1,:] * (Rt * XB), K[2,:] * (Rt * XB)
    uc, vc, wc = K[0,:] * (Rt * XC), K[1,:] * (Rt * XC), K[2,:] * (Rt * XC)
    ud, vd, wd = K[0,:] * (Rt * XD), K[1,:] * (Rt * XD), K[2,:] * (Rt * XD)
    ue, ve, we = K[0,:] * (Rt * XE), K[1,:] * (Rt * XE), K[2,:] * (Rt * XE)
    uf, vf, wf = K[0,:] * (Rt * XF), K[1,:] * (Rt * XF), K[2,:] * (Rt * XF)
    #根据每一帧的循环，提取出Rt的参数，K是公用的，代入参数写出3D坐标，并计算出u/w，v/w
    #写成f的形式，即按照六个点分块，每块里面有M帧
    getfx = mat(np.zeros((6*frame*2,1)))
    for i in range(6):
        for j in range(frame):
            if i == 0 :
                f = Matrix([ua/wa, va/wa])
            elif i == 1 :
                f = Matrix([ub/wb, vb/wb])
            elif i == 2 :
                f = Matrix([uc/wc, vc/wc])
            elif i == 3 :
                f = Matrix([ud/wd, vd/wd])
            elif i == 4 :
                f = Matrix([ue/we, ve/we])
            else:
                f = Matrix([uf/wf, vf/wf])
            f_value = f.subs({r11:para[13*j], r12:para[13*j+1], r13:para[13*j+2], r14:para[13*j+3],
                        r21:para[13*j+4], r22:para[13*j+5], r23:para[13*j+6], r24:para[13*j+7],
                        r31:para[13*j+8], r32:para[13*j+9], r33:para[13*j+10], th:para[13*j+11],
                        al:para[13*j+12], a:para[-3], b:para[-2], c:para[-1]})
            getfx[i*frame*2+j*2] = f_value[0]
            getfx[i*frame*2+j*2+1] = f_value[1]
    #返回getfx值，2*frame*6 by 1
    return getfx
```

```python
def getJacobian(point, frame, para):
    # 用参数表示K，R矩阵
    focalx, focaly, px, py = symbols('focalx focaly px py')
    r11, r12, r13, r14, r21, r22, r23, r24, r31, r32, r33 = symbols('r11 r12 r13 r14 r21 r22 r23 r24 r31 r32 r33')
    Rt = Matrix([[r11, r12, r13, r14], [r21, r22, r23, r24], [r31, r32, r33, 1]])
    K = Matrix([[focalx, 0, px], [0, focaly, py], [0, 0, 1]])
    # KRt = K * Rt
    # 用参数表示ABCDEF六个点坐标
    a, b, c, th, al = symbols('a b c th al')
    ua, va, wa, ub, vb, wb, uc, vc, wc, ud, vd, wd, ue, ve, we, uf, vf, wf = symbols('ua va wa ub vb wb uc vc wc ud vd wd ue ve we uf vf wf')
    f = Symbol('f')
    if point == 0 :
        XA = Matrix([[-a * c * cos(th) * cos(al)], [c-a * c * sin(th)], [-a * c *  cos(th) * sin(al)], [1]])
        ua, va, wa = K[0,:] * (Rt * XA), K[1,:] * (Rt * XA), K[2,:] * (Rt * XA)
        f = Matrix([ua/wa, va/wa])
    elif point == 1 :
        XB = Matrix([[0], [c], [0], [1]])
        ub, vb, wb = K[0,:] * (Rt * XB), K[1,:] * (Rt * XB), K[2,:] * (Rt * XB)
        f = Matrix([ub/wb, vb/wb])
    elif point == 2 :
        XC = Matrix([[a * c * cos(th) * cos(al)], [c+a * c * sin(th)], [a * c * cos(th) * sin(al)], [1]])
        uc, vc, wc = K[0,:] * (Rt * XC), K[1,:] * (Rt * XC), K[2,:] * (Rt * XC)
        f = Matrix([uc/wc, vc/wc])
    elif point == 3 :
        XD = Matrix([[-b * c], [0], [0], [1]])
        ud, vd, wd = K[0,:] * (Rt * XD), K[1,:] * (Rt * XD), K[2,:] * (Rt * XD)
        f = Matrix([ud/wd, vd/wd])
    elif point == 4 :
        XE = Matrix([[0], [0], [0], [1]])
        ue, ve, we = K[0,:] * (Rt * XE), K[1,:] * (Rt * XE), K[2,:] * (Rt * XE)
        f = Matrix([ue/we, ve/we])
    elif point == 5:
        XF = Matrix([[b * c], [0], [0], [1]])
        uf, vf, wf = K[0,:] * (Rt * XF), K[1,:] * (Rt * XF), K[2,:] * (Rt * XF)
        f = Matrix([uf/wf, vf/wf])
    args = Matrix([r11, r12, r13, r14, r21, r22, r23, r24, r31, r32, r33, th, al, a, b, c])
    f_X1 = f[0,:].jacobian(args)
    f_X2 = f[1,:].jacobian(args)
    JA = Matrix([f_X1, f_X2])   # 2 by 16 matrix
    JA_value = JA.subs({focalx:1149.676, focaly:1147.592, px:508.849, py:508.065, r11:para[13*frame], r12:para[13*frame+1],
                                 r13:para[13*frame+2], r14:para[13*frame+3], r21:para[13*frame+4], r22:para[13*frame+5],
                                 r23:para[13*frame+6], r24:para[13*frame+7], r31:para[13*frame+8], r32:para[13*frame+9],
                                 r33:para[13*frame+10], th:para[13*frame+11], al:para[13*frame+12], a:para[-3], b:para[-2], c:para[-1]})
    #JA_value = JA_value.subs({f:1149.68})
    return JA_value
```

```python
def getJ(para, frame):
    getj = mat(np.zeros((6*frame*2, 13*frame+3)))
    for m in range(6):
        for n in range(frame):
            JA_value = getJacobian(m, n, para)
            #print(JA_value)
            getj[2*(m*frame+n):2*(m*frame+n+1), 13*n:13*n+13] = JA_value[:, 0:13]
            getj[2*(m*frame+n):2*(m*frame+n+1), -3:] = JA_value[:, -3:]
    return getj
```

```python
def getE(getfx, frame, k, xkp):
    E = mat(np.zeros((6*frame*2,1)))
    for i in range(6):
        for j in range(frame):
            if i==0 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
            elif i==1 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
            elif i==2 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
            elif i==3 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
            elif i==4 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
            elif i==5 :
                E[(i*frame+j)*2] = getfx[i*frame*2+j*2] - xkp[i*3+k,0]
                E[(i*frame+j)*2+1] = getfx[i*frame*2+j*2+1] - xkp[i*3+k, 1]
    return E
```

```python
def LM_opti(frame, ith, x2d, x_para, u=1, v=2, step_max=500):
    J = mat(np.zeros((6*frame*2, 13*frame+3)))
    E = mat(np.zeros((6*frame*2,1)))   # E = f(X) - b ;
    E_temp = mat(np.zeros((6*frame*2,1)))   # E_temp compare with E in L-M
    x_k = mat(x_para.copy())   #parameter initialization
    step = 0   # iteration steps
    mse_last = 0   # mse value after iteration each time

    # L-M Algorithm obtain optimal parameters
    while(step < step_max):
        step += 1
        mse, mse_temp = 0, 0

        # generate Jacobian Matrix and calculate E
        getfx = mat(np.zeros((6*frame*2,1)))
        getfx = getFx(x_k, frame)
        E = getE(getfx, frame, ith, x2d)   #单帧模式，第几帧，GT

        for i in range(6*frame*2):
            mse += E[i]**2
        mse /= 6*frame*2

        # get new J
        J = mat(np.zeros((6*frame*2, 13*frame+3)))
        J = getJ(x_k, frame)

        # delta X = ...
        #print(J.T * J)
        dx = mat(np.zeros((13*frame+3,1)))
        LM = u * mat(np.eye(13*frame+3))
        dx = -(J.T * J + LM).I * J.T * E
        x_k_temp = x_k.copy()
        x_k_temp += dx

        #R的更新不能简单赋值
        #get R meet R.T*R=I
        #U * D * V.T = R --> R+ = U * V.T

        R_old = mat([[x_k_temp[0,0], x_k_temp[1,0], x_k_temp[2,0]],
                     [x_k_temp[4,0], x_k_temp[5,0], x_k_temp[6,0]],
                     [x_k_temp[8,0], x_k_temp[9,0], x_k_temp[10,0]]])
        U, sigma, VT = la.svd(R_old)
        R_new = U * VT
        x_k_temp[0,0], x_k_temp[1,0], x_k_temp[2,0] = R_new[0,0], R_new[0,1], R_new[0,2]
        x_k_temp[4,0], x_k_temp[5,0], x_k_temp[6,0] = R_new[1,0], R_new[1,1], R_new[1,2]
        x_k_temp[8,0], x_k_temp[9,0], x_k_temp[10,0] = R_new[2,0], R_new[2,1], R_new[2,2]

        ###########

        # calculate E_temp with x_k_temp
        # copy from E with x_k
        getfx_temp = mat(np.zeros((6*frame*2,1)))
        getfx_temp = getFx(x_k_temp, frame)
        E_temp = getE(getfx_temp, frame, ith, x2d)

        for i in range(6*frame*2):
            mse_temp += E_temp[i]**2
        mse_temp /= 6*frame*2

        # segma value to choose optimization model
        segma = (mse - mse_temp)/((dx.T * (u * dx - J.T * E))[0,0])

        # calculate new u
        if segma > 0:
            s = 1.0/3.0
            v = 2
            x_k = x_k_temp
            mse = mse_temp
            u = u * max(s, 1-pow(2*segma,3))
            u = u[0,0]
        else:
            u = u * v
            v = v * 2
            x_k = x_k_temp

        print("step = %d, abs(mse-mse_last) = %.8f" %(step, abs(mse-mse_last)))
        if abs(mse-mse_last)<0.000001:
            break
        mse_last = mse

    print("step = ", step)
    print("mse = ", mse_last)
    #print("parameter = ", x_k)

    return x_k
```

```python
frame = 3
m = loadmat("valid.mat")

# camera intrinsic matrix
K = m["annot"][0][0][4]
K_cam = K[0][0].tolist()

# key point 3D groundtruth
gt = m["annot"][0][0][3]
img_gt = np.zeros((frame, 3, 17))
img1_gt = gt[135]   # array 3 by 17
img2_gt = gt[136]   # array 3 by 17
img3_gt = gt[137]   # array 3 by 17
img_gt[0] = img1_gt
img_gt[1] = img2_gt
img_gt[2] = img3_gt
#print(img_gt)

# key point 2D groundtruth
kp = np.zeros((frame, 17, 2))
for m in range(frame):
    for i in range(17):
        u = K_cam[0] * mat([img_gt[m][0][i], img_gt[m][1][i], img_gt[m][2][i]]).T
        v = K_cam[1] * mat([img_gt[m][0][i], img_gt[m][1][i], img_gt[m][2][i]]).T
        w = K_cam[2] * mat([img_gt[m][0][i], img_gt[m][1][i], img_gt[m][2][i]]).T
        kp[m][i][0] = u/w
        kp[m][i][1] = v/w
#print(kp)

# show image
img1 = cv2.imread("S9_Posing_1.55011271_000676.jpg")
img2 = cv2.imread("S9_Posing_1.55011271_000681.jpg")
img3 = cv2.imread("S9_Posing_1.55011271_000686.jpg")
plt.figure(figsize = (15, 45)) # 图像窗口名称
plt.subplot(131)
plt.imshow(img1[:,:,[2,1,0]])
plt.subplot(132)
plt.imshow(img2[:,:,[2,1,0]])
plt.subplot(133)
plt.imshow(img3[:,:,[2,1,0]])
plt.axis('on') # 关掉坐标轴为 off
plt.title('image') # 图像题目
plt.show()

# visualize key points
"""
txt = ['1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17']
plt.figure()
plt.subplot(131)
img_kp = plt.scatter(kp[0][:,0], kp[0][:,1], s = 80, c = 'g', marker = 'X')
for i in range(17):
    plt.annotate(txt[i], xy = (kp[0][i,0], kp[0][i,1]), xytext = (kp[0][i,0]+0.1, kp[0][i,1]+0.1))
plt.subplot(132)
img_kp = plt.scatter(kp[1][:,0], kp[1][:,1], s = 80, c = 'g', marker = 'X')
for i in range(17):
    plt.annotate(txt[i], xy = (kp[1][i,0], kp[1][i,1]), xytext = (kp[1][i,0]+0.1, kp[1][i,1]+0.1))
plt.subplot(133)
img_kp = plt.scatter(kp[2][:,0], kp[2][:,1], s = 80, c = 'g', marker = 'X')
for i in range(17):
    plt.annotate(txt[i], xy = (kp[2][i,0], kp[2][i,1]), xytext = (kp[2][i,0]+0.1, kp[2][i,1]+0.1)) # 这里xy是需要标记的坐标，xytext是对应的标签坐标
plt.axis('on') # 关掉坐标轴为 off
plt.title('keypoint of image') # 图像题目
"""

# visualize torso joints A-F
plt.figure(figsize = (5,15))
plt.subplot(311)
for i in {0,1,4,8,11,14}:
    img_kp = plt.scatter(kp[0][i,0], kp[0][i,1], s = 80, c = 'g', marker = 'X')
plt.subplot(312)
for i in {0,1,4,8,11,14}:
    img_kp = plt.scatter(kp[1][i,0], kp[1][i,1], s = 80, c = 'g', marker = 'X')
plt.subplot(313)
for i in {0,1,4,8,11,14}:
    img_kp = plt.scatter(kp[2][i,0], kp[2][i,1], s = 80, c = 'g', marker = 'X')
plt.axis('on') # 关掉坐标轴为 off
plt.title('torso keypoint of image') # 图像题目
plt.show()

# save 2D keypoint coordinate
x2d = np.zeros((6 * frame,2))   #[AAABBBCCCDDDEEEFFF].T
for i in range(6):
    for j in range(frame):
        if i==0 :
            x2d[i*frame+j, :] = kp[j][14, :]
        elif i==1 :
            x2d[i*frame+j, :] = kp[j][8, :]
        elif i==2 :
            x2d[i*frame+j, :] = kp[j][11, :]
        elif i==3 :
            x2d[i*frame+j, :] = kp[j][4, :]
        elif i==4 :
            x2d[i*frame+j, :] = kp[j][0, :]
        elif i==5 :
            x2d[i*frame+j, :] = kp[j][1, :]
print(x2d)
```

```python
# parameter initialization for first frame
x_para = np.zeros((13+3,1))
x_para[0] = -1   # r11
x_para[5] = 1   # r22
x_para[10] = -1   # r33
x_para[3] = 0.0047   #tx(cal)
x_para[7] = -0.0997   #ty(cal)
x_para[11] = 0   # th
x_para[12] = 0   #al
x_para[-3] = 0.35   # a
x_para[-2] = 0.25   # b
x_para[-1] = -0.095  # c(cal)
getfx_ini = getFx(x_para, 1)
print(getfx_ini)
```

    [[476.02538041]
     [284.62882932]
     [514.25209744]
     [284.62882932]
     [552.47881446]
     [284.62882932]
     [486.94729956]
     [393.65003291]
     [514.25209744]
     [393.65003291]
     [541.55689531]
     [393.65003291]]

```python
# visualize first frame after initilization
plt.figure()
img_kp = plt.scatter(kp[0][14,0], kp[0][14,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[0,0], getfx_ini[1,0], s = 80, c = 'r', marker = 'X')
img_kp = plt.scatter(kp[0][8,0], kp[0][8,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[2,0], getfx_ini[3,0], s = 80, c = 'r', marker = 'X')
img_kp = plt.scatter(kp[0][11,0], kp[0][11,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[4,0], getfx_ini[5,0], s = 80, c = 'r', marker = 'X')
img_kp = plt.scatter(kp[0][4,0], kp[0][4,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[6,0], getfx_ini[7,0], s = 80, c = 'r', marker = 'X')
img_kp = plt.scatter(kp[0][0,0], kp[0][0,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[8,0], getfx_ini[9,0], s = 80, c = 'r', marker = 'X')
img_kp = plt.scatter(kp[0][1,0], kp[0][1,1], s = 80, c = 'g', marker = 'X')
img_kp = plt.scatter(getfx_ini[10,0], getfx_ini[11,0], s = 80, c = 'r', marker = 'X')
plt.axis('on') # 关掉坐标轴为 off
plt.title('image_kp_ABCDEF_original_opt') # 图像题目
plt.show()
```

```python
# optimization for each frame
X_para = np.zeros((13*frame+3,1))
X_para[-3] = 0.35   # a
X_para[-2] = 0.25   # b
X_para[-1] = -0.095  # c(cal)
para_data = np.zeros((frame,16))
x_k_para = x_para.copy()
for i in range(frame):
    x_k = LM_opti(1, i, x2d, x_k_para)   #单帧优化，第几帧，GT，初始化参数

    # R matrix adjustment
    '''
    R_old = mat([[x_k[0,0], x_k[1,0], x_k[2,0]],
             [x_k[4,0], x_k[5,0], x_k[6,0]],
             [x_k[8,0], x_k[9,0], x_k[10,0]]])
    U, sigma, VT = la.svd(R_old)
    R_new = U * VT
    x_k[0,0], x_k[1,0], x_k[2,0] = R_new[0,0], R_new[0,1], R_new[0,2]
    x_k[4,0], x_k[5,0], x_k[6,0] = R_new[1,0], R_new[1,1], R_new[1,2]
    x_k[8,0], x_k[9,0], x_k[10,0] = R_new[2,0], R_new[2,1], R_new[2,2]
    '''
    print()
    print("For the No.", i+1, " frame : x_k = ", x_k)
    para_data[i] = mat(x_k).T
    # for next frame initilization
    x_k_para = x_k
    X_para[13*i:13*i+13] = x_k[0:13,0]
print(para_data)
print(X_para)
```

    step = 1, abs(mse-mse_last) = 9.83445441
    step = 2, abs(mse-mse_last) = 0.07658973
    step = 3, abs(mse-mse_last) = 0.03711521
    step = 4, abs(mse-mse_last) = 0.00000000
    step =  4
    mse =  [[9.72074947]]

    For the No. 1  frame : x_k =  [[-0.9961843 ]
     [-0.06427425]
     [ 0.05903942]
     [ 0.00649174]
     [-0.06445466]
     [ 0.99791997]
     [-0.00115447]
     [-0.09968937]
     [-0.05884241]
     [-0.00495543]
     [-0.99825498]
     [ 0.09610256]
     [ 0.16333518]
     [ 0.38184965]
     [ 0.26152886]
     [-0.08737298]]
    step = 1, abs(mse-mse_last) = 10.59310710
    step = 2, abs(mse-mse_last) = 0.00000000
    step =  2
    mse =  [[10.5931071]]

    For the No. 2  frame : x_k =  [[-9.78834254e-01]
     [-4.44169043e-02]
     [ 1.99776478e-01]
     [ 3.32326833e-04]
     [-4.70953041e-02]
     [ 9.98852752e-01]
     [-8.67244145e-03]
     [-9.97689135e-02]
     [-1.99162082e-01]
     [-1.78974167e-02]
     [-9.79803117e-01]
     [ 1.68431204e-01]
     [ 5.06546257e-01]
     [ 4.63995894e-01]
     [ 2.69374608e-01]
     [-8.93099016e-02]]
    step = 1, abs(mse-mse_last) = 26.84820929
    step = 2, abs(mse-mse_last) = 0.00000000
    step =  2
    mse =  [[26.84820929]]

    For the No. 3  frame : x_k =  [[-0.93364624]
     [-0.04826612]
     [ 0.35492969]
     [-0.00729535]
     [-0.05890958]
     [ 0.99807798]
     [-0.01923581]
     [-0.10001772]
     [-0.35331907]
     [-0.0388682 ]
     [-0.93469508]
     [ 0.24836053]
     [ 0.90426588]
     [ 0.8733488 ]
     [ 0.23460038]
     [-0.09998521]]
    [[-9.96184304e-01 -6.42742503e-02  5.90394156e-02  6.49173516e-03
      -6.44546564e-02  9.97919969e-01 -1.15446921e-03 -9.96893652e-02
      -5.88424092e-02 -4.95542936e-03 -9.98254985e-01  9.61025564e-02
       1.63335179e-01  3.81849650e-01  2.61528863e-01 -8.73729846e-02]
     [-9.78834254e-01 -4.44169043e-02  1.99776478e-01  3.32326833e-04
      -4.70953041e-02  9.98852752e-01 -8.67244145e-03 -9.97689135e-02
      -1.99162082e-01 -1.78974167e-02 -9.79803117e-01  1.68431204e-01
       5.06546257e-01  4.63995894e-01  2.69374608e-01 -8.93099016e-02]
     [-9.33646238e-01 -4.82661180e-02  3.54929689e-01 -7.29534920e-03
      -5.89095806e-02  9.98077975e-01 -1.92358071e-02 -1.00017719e-01
      -3.53319068e-01 -3.88681981e-02 -9.34695084e-01  2.48360525e-01
       9.04265876e-01  8.73348804e-01  2.34600381e-01 -9.99852063e-02]]
    [[-9.96184304e-01]
     [-6.42742503e-02]
     [ 5.90394156e-02]
     [ 6.49173516e-03]
     [-6.44546564e-02]
     [ 9.97919969e-01]
     [-1.15446921e-03]
     [-9.96893652e-02]
     [-5.88424092e-02]
     [-4.95542936e-03]
     [-9.98254985e-01]
     [ 9.61025564e-02]
     [ 1.63335179e-01]
     [-9.78834254e-01]
     [-4.44169043e-02]
     [ 1.99776478e-01]
     [ 3.32326833e-04]
     [-4.70953041e-02]
     [ 9.98852752e-01]
     [-8.67244145e-03]
     [-9.97689135e-02]
     [-1.99162082e-01]
     [-1.78974167e-02]
     [-9.79803117e-01]
     [ 1.68431204e-01]
     [ 5.06546257e-01]
     [-9.33646238e-01]
     [-4.82661180e-02]
     [ 3.54929689e-01]
     [-7.29534920e-03]
     [-5.89095806e-02]
     [ 9.98077975e-01]
     [-1.92358071e-02]
     [-1.00017719e-01]
     [-3.53319068e-01]
     [-3.88681981e-02]
     [-9.34695084e-01]
     [ 2.48360525e-01]
     [ 9.04265876e-01]
     [ 3.50000000e-01]
     [ 2.50000000e-01]
     [-9.50000000e-02]]

```python
def visual2D(kp, getfx):
    plt.figure()
    img_kp = plt.scatter(kp[14,0], kp[14,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[0,0], getfx[1,0], s = 80, c = 'r', marker = 'X')
    img_kp = plt.scatter(kp[8,0], kp[8,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[2,0], getfx[3,0], s = 80, c = 'r', marker = 'X')
    img_kp = plt.scatter(kp[11,0], kp[11,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[4,0], getfx[5,0], s = 80, c = 'r', marker = 'X')
    img_kp = plt.scatter(kp[4,0], kp[4,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[6,0], getfx[7,0], s = 80, c = 'r', marker = 'X')
    img_kp = plt.scatter(kp[0,0], kp[0,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[8,0], getfx[9,0], s = 80, c = 'r', marker = 'X')
    img_kp = plt.scatter(kp[1,0], kp[1,1], s = 80, c = 'g', marker = 'X')
    img_kp = plt.scatter(getfx[10,0], getfx[11,0], s = 80, c = 'r', marker = 'X')
    plt.axis('on') # 关掉坐标轴为 off
    plt.title('image_kp_ABCDEF_original_opt') # 图像题目
    plt.show()
```

```python
visual = np.zeros((13+3,1))
for i in range(frame):
    visual = mat(para_data[i]).T
    getfx_visual = getFx(visual, 1)
    visual2D(kp[i], getfx_visual)
```

```python
def visual3D(img_gt, para):
    K = Matrix([[1149.67569986785, 0.0, 508.848621645943],
                 [0.0, 1147.59161666764, 508.064917088557],
                 [0.0, 0.0, 1.0]])
    r11, r12, r13, r14, r21, r22, r23, r24, r31, r32, r33 = symbols('r11 r12 r13 r14 r21 r22 r23 r24 r31 r32 r33')
    Rt = Matrix([[r11, r12, r13, r14], [r21, r22, r23, r24], [r31, r32, r33, 1]])
    a, b, c, th, al = symbols('a b c th al')
    ua, va, wa, ub, vb, wb, uc, vc, wc, ud, vd, wd, ue, ve, we, uf, vf, wf = symbols('ua va wa ub vb wb uc vc wc ud vd wd ue ve we uf vf wf')
    XA = Matrix([[-a * c * cos(th) * cos(al)], [c-a * c * sin(th)], [-a * c *  cos(th) * sin(al)], [1]])
    XB = Matrix([[0], [c], [0], [1]])
    XC = Matrix([[a * c * cos(th) * cos(al)], [c+a * c * sin(th)], [a * c * cos(th) * sin(al)], [1]])
    XD = Matrix([[-b * c], [0], [0], [1]])
    XE = Matrix([[0], [0], [0], [1]])
    XF = Matrix([[b * c], [0], [0], [1]])
    A3D = Rt * XA
    B3D = Rt * XB
    C3D = Rt * XC
    D3D = Rt * XD
    E3D = Rt * XE
    F3D = Rt * XF
    j = 0
    #A3D_value = A3D.subs({r11:para[13*j], r12:para[13*j+1], r13:para[13*j+2], r14:para[13*j+3],
    #            r21:para[13*j+4], r22:para[13*j+5], r23:para[13*j+6], r24:para[13*j+7],
    #            r31:para[13*j+8], r32:para[13*j+9], r33:para[13*j+10], th:para[13*j+11],
    #            al:para[13*j+12], a:para[-3], b:para[-2], c:para[-1]})
    Y = mat([img_gt[0][0], img_gt[1][0], img_gt[2][0]]).T
    s = Y[2,0]
    print(s)
    #A3D_value = getValue(A3D, para) * s
    A3D_value = A3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    B3D_value = B3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    C3D_value = C3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    D3D_value = D3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    E3D_value = E3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    F3D_value = F3D.subs({r11:para[0], r12:para[1], r13:para[2], r14:para[3],
        r21:para[4], r22:para[5], r23:para[6], r24:para[7],
        r31:para[8], r32:para[9], r33:para[10], th:para[11],
        al:para[12], a:para[-3], b:para[-2], c:para[-1]}) * s
    print(A3D_value,B3D_value,C3D_value,D3D_value,E3D_value,F3D_value)

    X1 = mat([img_gt[0][14], img_gt[1][14], img_gt[2][14]]).T
    X2 = mat([img_gt[0][8], img_gt[1][8], img_gt[2][8]]).T
    X3 = mat([img_gt[0][11], img_gt[1][11], img_gt[2][11]]).T
    X4 = mat([img_gt[0][4], img_gt[1][4], img_gt[2][4]]).T
    X5 = mat([img_gt[0][0], img_gt[1][0], img_gt[2][0]]).T
    X6 = mat([img_gt[0][1], img_gt[1][1], img_gt[2][1]]).T
    print(X1, X2, X3, X4, X5, X6)

    # 绘制散点图
    fig = plt.figure()
    ax = Axes3D(fig)
    ax.scatter(X1[0], X1[1], X1[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(X2[0], X2[1], X2[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(X3[0], X3[1], X3[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(X4[0], X4[1], X4[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(X5[0], X5[1], X5[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(X6[0], X6[1], X6[2], s = 80, c = 'g', marker = 'X')
    ax.scatter(A3D_value[0], A3D_value[1], A3D_value[2], s = 80, c = 'r', marker = 'X')
    ax.scatter(B3D_value[0], B3D_value[1], B3D_value[2], s = 80, c = 'r', marker = 'X')
    ax.scatter(C3D_value[0], C3D_value[1], C3D_value[2], s = 80, c = 'r', marker = 'X')
    ax.scatter(D3D_value[0], D3D_value[1], D3D_value[2], s = 80, c = 'r', marker = 'X')
    ax.scatter(E3D_value[0], E3D_value[1], E3D_value[2], s = 80, c = 'r', marker = 'X')
    ax.scatter(F3D_value[0], F3D_value[1], F3D_value[2], s = 80, c = 'r', marker = 'X')

    ax.set_zlabel('Z', fontdict={'size': 15, 'color': 'red'})
    ax.set_ylabel('Y', fontdict={'size': 15, 'color': 'red'})
    ax.set_xlabel('X', fontdict={'size': 15, 'color': 'red'})
    plt.show()

    # 计算3D error
    d1 = A3D_value-X1
    e1 = sum(map(lambda x:x**2, d1))
    d2 = B3D_value-X2
    e2 = sum(map(lambda x:x**2, d2))
    d3 = C3D_value-X3
    e3 = sum(map(lambda x:x**2, d3))
    d4 = D3D_value-X4
    e4 = sum(map(lambda x:x**2, d4))
    d5 = E3D_value-X5
    e5 = sum(map(lambda x:x**2, d5))
    d6 = F3D_value-X6
    e6 = sum(map(lambda x:x**2, d6))
    error = (e1**(0.5)+e2**(0.5)+e3**(0.5)+e4**(0.5)+e5**(0.5)+e6**(0.5))/6
    print(error)
    print(d1,d2,d3,d4,d5,d6)
```

```python
visual3d = np.zeros((13+3,1))
for i in range(frame):
    visual3d = mat(para_data[i]).T
    visual3D(img_gt[i], visual3d)
```

```python

```

annot/BA and LM --- mF6P (m=3) #2-1.ipynb


# Neuromatch Academy: Week 1, Day 3, Tutorial 2
# Model Fitting: Linear regression with MLE

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil  with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Madineh Sarvestani, Patrick Mineault, Ella Batty, Michael Waskom

---
#Tutorial Objectives

This is Tutorial 2 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will use a different approach to fit linear models that incorporates the random 'noise' in our data.
- Learn about probability distributions and probabilistic models
- Learn how to calculate the likelihood of our model parameters
- Learn how to implement the maximum likelihood estimator, to find the model parameter with the maximum likelihood

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper Functions
def plot_density_image(x, y, theta, sigma=1, ax=None):
  """ Plots probability distribution of y given x, theta, and sigma

  Args:

    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta (float): Slope parameter
    sigma (float): standard deviation of Gaussian noise

  """

  # plot the probability density of p(y|x,theta)
  if ax is None:
    fig, ax = plt.subplots()

  xmin, xmax = np.floor(np.min(x)), np.ceil(np.max(x))
  ymin, ymax = np.floor(np.min(y)), np.ceil(np.max(y))
  xx = np.linspace(xmin, xmax, 50)
  yy = np.linspace(ymin, ymax, 50)

  surface = np.zeros((len(yy), len(xx)))
  for i, x_i in enumerate(xx):
    surface[:, i] = stats.norm(theta * x_i, sigma).pdf(yy)

  ax.set(xlabel='x', ylabel='y')

  return ax.imshow(surface, origin='lower', aspect='auto', vmin=0, vmax=None,
            cmap=plt.get_cmap('Wistia'),
            extent=[xmin, xmax, ymin, ymax])

def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter.

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  theta_hat = (x.T @ y) / (x.T @ x)
  return theta_hat
```

---
# Section 1: Maximum Likelihood Estimation (MLE)

```python
#@title Video 1: Maximum Likelihood Estimation
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="8mpNmzLKNfU", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=8mpNmzLKNfU

In the previous tutorial we made the assumption that the data was drawn from a linear relationship with noise added, and found an effective approach for estimating model parameters based on minimizing the mean squared error.

In that case we treated the noise as simply a nuisance, but what if we factored it directly into our model?

Recall our linear model:

\begin{align}
y = \theta x + \epsilon.
\end{align}

The noise component $\epsilon$ is often modeled as a random variable drawn from a Gaussian distribution (also called the normal distribution).

The Gaussian distribution is described by its [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (pdf)
\begin{align}
\mathcal{N}(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{align}

and is dependent on two parameters: the mean $\mu$ and the variance $\sigma^2$. We often consider the noise signal to be Gaussian "white noise", with zero mean and unit variance:

\begin{align}
\epsilon \sim \mathcal{N}(0, 1).
\end{align}

## Interactive Demo: Gaussian Distribution Explorer

Use the explorer widget below to see how varying the $\mu$ and $\sigma$ parameters change the location and shape of the samples.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(mu=widgets.FloatSlider(0.0, min=-2.0, max=2.0),
                  sigma=widgets.FloatSlider(1.0, min=0.5, max=2.0))
def plot_normal_dist(mu=0, sigma=1):

  # Generate pdf & samples from normal distribution with mu/sigma
  rv = stats.norm(mu, sigma)
  x = np.linspace(-5, 5, 100)
  y = rv.pdf(x)
  samples = rv.rvs(1000)

  # Plot
  fig, ax = plt.subplots()
  ax.hist(samples, 20, density=True, color='g', histtype='stepfilled', alpha=0.8,
          label='histogram')
  ax.plot(x, y, color='orange', linewidth=3, label='pdf')
  ax.vlines(mu, 0, rv.pdf(mu), color='y', linewidth=3, label='$\mu$')
  ax.vlines([mu-sigma, mu+sigma], 0, rv.pdf([mu-sigma, mu+sigma]), colors='red',
            color='b', linewidth=3, label='$\sigma$')
  ax.set(xlabel='x', ylabel='probability density', xlim=[-5, 5], ylim=[0, 1.0])
  ax.legend()
```

    interactive(children=(FloatSlider(value=0.0, description='mu', max=2.0, min=-2.0), FloatSlider(value=1.0, desc…

## Section 1.1: Probabilistic Models

Now that we have a model of our noise component $\epsilon$ as random variable, how do we incorporate this back into our original linear model from before? Consider again our simplified model $y = \theta x + \epsilon$ where the noise has zero mean and unit variance $\epsilon \sim \mathcal{N}(0, 1)$. We can now also treat $y$ as a random variable drawn from a Gaussian distribution where $\mu = \theta x$ and $\sigma^2 = 1$:

\begin{align}
y \sim \mathcal{N}(\theta x, 1)
\end{align}

which is to say that the probability of observing $y$ given $x$ and parameter $\theta$ is
\begin{align}
p(y|x,\theta) = \frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}(y-\theta x)^2}
\end{align}

---

Let's revisit our original sample dataset where the true underlying model has $\theta = 1.2$.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence

np.random.seed(121)
theta = 1.2
n_samples = 30
x = 10 * np.random.rand(n_samples) # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples) # sample from a standard normal distribution
y = theta * x + noise
```

This time we can plot the density of $p(y|x,\theta=1.2)$ and see how $p(y)$ changes for different values of $x$.

```python
#@title
#@markdown Execute this cell to visualize p(y|x, theta=1.2)

fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 4))

# Invokes helper function to generate density image plots from data and parameters
im = plot_density_image(x, y, 1.2, ax=ax1)
plt.colorbar(im, ax=ax1)
ax1.axvline(8, color='k')
ax1.set(title=r'p(y | x, $\theta$=1.2)')

# Plot pdf for given x
ylim = ax1.get_ylim()
yy = np.linspace(ylim[0], ylim[1], 50)
ax2.plot(yy, stats.norm(theta * 8, 1).pdf(yy), color='orange', linewidth=2)
ax2.set(
    title=r'p(y|x=8, $\theta$=1.2)',
    xlabel='y',
    ylabel='probability density');
```

## Section 1.2: Likelihood Estimation

Now that we have our probabilistic model, we turn back to our original challenge of finding a good estimate for $\theta$ that fits our data. Given the inherent uncertainty when dealing in probabilities, we talk about the [likelihood](https://en.wikipedia.org/wiki/Likelihood_function) that some estimate $\hat \theta$ fits our data. The likelihood function $\mathcal{L(\theta)}$ is equal to the probability density function parameterized by that $\theta$:

\begin{align}
\mathcal{L}(\theta|x,y) = p(y|x,\theta) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2\sigma^2}(y-\theta x)^2}
\end{align}

### Exercise 1: Likelihood Function

In this exercise you will implement the likelihood function $\mathcal{L}(\theta|x,y)$ for our linear model where $\sigma = 1$.

After implementing this function, we can produce probabilities that our estimate $\hat{\theta}$ generated the provided observations. We will try with one of the samples from our dataset.

TIP: Use `np.exp` and `np.sqrt` for the exponential and square root functions, respectively.

```python
def likelihood(theta_hat, x, y):
  """The likelihood function for a linear model with noise sampled from a
    Gaussian distribution with zero mean and unit variance.

  Args:
    theta_hat (float): An estimate of the slope parameter.
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    ndarray: the likelihood values for the theta_hat estimate
  """
  sigma = 1
  ##############################################################################
  ## TODO for students: implement the likelihood function
  # Fill out function and remove
#   raise NotImplementedError("Student exercise: implement the likelihood function")
  ##############################################################################

  # Compute Gaussian likelihood
  pdf = 1 / np.sqrt(2 * np.pi * sigma**2) * np.exp(-(y - theta_hat * x)**2 / (2 * sigma**2))

  return pdf

# Uncomment below to test your function
print(likelihood(1.0, x[1], y[1]))
```

    0.11344443599846923

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial2_Solution_87e35553.py)

We should see that $\mathcal{L}(\theta=1.0|x=2.1,y=3.7) \approx 0.11$. So far so good, but how does this tell us how this estimate is better than any others?

When dealing with a set of data points, as we are with our dataset, we are concerned with their joint probability -- the likelihood that all data points are explained by our parameterization. Since we have assumed that the noise affects each output independently, we can factorize the likelihood, and write:

\begin{align}
\mathcal{L}(\theta|X,Y) = \prod_{i=1}^N \mathcal{L}(\theta|x_i,y_i),
\end{align}

where we have $N$ data points $X = \{x_1,...,x_N\}$ and $Y = \{y_1,...,y_N\}$.

In practice, such a product can be numerically unstable. Indeed multiplying small values together can lead to [underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow), the situation in which the digital representation of floating point number reaches its limit. This problem can be circumvented by taking the logarithm of the likelihood because the logarithm transforms products into sums:

\begin{align}
\operatorname{log}\mathcal{L}(\theta|X,Y) = \sum_{i=1}^N \operatorname{log}\mathcal{L}(\theta|x_i,y_i)
\end{align}

We can take the sum of the log of the output of our `likelihood` method applied to the full dataset to get a better idea of how different $\hat\theta$ compare. We can also plot the different distribution densities over our dataset and see how they line up qualitatively.

```python
#@title
#@markdown Execute this cell to visualize different distribution densities
theta_hats = [0.5, 1.0, 2.2]
fig, axes = plt.subplots(ncols=3, figsize=(16, 4))
for theta_hat, ax in zip(theta_hats, axes):
  ll = np.sum(np.log(likelihood(theta_hat, x, y)))  # log likelihood
  im = plot_density_image(x, y, theta_hat, ax=ax)
  ax.scatter(x, y)
  ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat}, log likelihood: {ll:.2f}')
plt.colorbar(im, ax=ax);
```

Using the log likelihood calculation, we see that $\mathcal{L}(\theta=1.0) > \mathcal{L}(\theta=0.5) > \mathcal{L}(\theta=2.2)$.

This is great: now we have a way to compare estimators based on likelihood. But like with the MSE approach, we want an analytic solution to find the best estimator. In this case, we want to find the estimator that maximizes the likelihood.

## Section 1.3: Finding the Maximum Likelihood Estimator

We want to find the parameter value $\hat\theta$ that makes our data set most likely:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \mathcal{L}(\theta|X,Y)
\end{align}

We discussed how taking the logarithm of the likelihood helps with numerical stability, the good thing is that it does so without changing the parameter value that maximizes the likelihood. Indeed, the $\textrm{log}()$ function is *monotonically increasing*, which means that it preserves the order of its inputs. So we have:

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} \sum_{i=1}^m \textrm{log} \mathcal{L}(\theta|x_i,y_i)
\end{align}

Now substituting our specific likelihood function and taking its logarithm, we get:
\begin{align}
\hat{\theta}_{\textrm{MLE}} = \underset{\theta}{\operatorname{argmax}} [-\frac{N}{2} \operatorname{log} 2\pi\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^N (y_i-\theta x_i)^2].
\end{align}

Note that maximizing the log likelihood is the same as minimizing the negative log likelihood (in practice optimization routines are developed to solve minimization not maximization problems). Because of the convexity of this objective function, we can take the derivative of our negative log likelihhood, set it to 0, and solve - just like our solution to minimizing MSE.

\begin{align}
\frac{\partial\operatorname{log}\mathcal{L}(\theta|x,y)}{\partial\theta}=\frac{1}{\sigma^2}\sum_{i=1}^N(y_i-\theta x_i)x_i = 0
\end{align}

This looks remarkably like the equation we had to solve for the optimal MSE estimator, and, in fact, we arrive to the exact same solution!

\begin{align}
\hat{\theta}_{\textrm{MLE}} = \hat{\theta}_{\textrm{MSE}} = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

```python
# Compute theta_hat_MLE
theta_hat_mle = (x @ y) / (x @ x)

```

```python
#@title
#@markdown Execute this cell to visualize density with theta_hat_mle

# Plot the resulting distribution density
fig, ax = plt.subplots()
ll = np.sum(np.log(likelihood(theta_hat_mle, x, y))) # log likelihood
im = plot_density_image(x, y, theta_hat_mle, ax=ax)
plt.colorbar(im, ax=ax);
ax.scatter(x, y)
ax.set(title=fr'$\hat{{\theta}}$ = {theta_hat_mle:.2f}, log likelihood: {ll:.2f}');
```

---
# Summary

- Likelihood vs probability
    - $\mathcal{L}(\theta|x, y) = p(y|\theta, x)$
    - $p(y|\theta, x)$ -> "probability of observing the response $y$ given parameter $\theta$ and input $x$"
    - $\mathcal{L}(\theta|x, y)$ -> "likelihood model that parameters $\theta$ produced response $y$ from input $x$"
- Log-likelihood maximization
    - We take the $\textrm{log}$ of the likelihood function for computational convenience
    - The parameters $\theta$ that maximize $\textrm{log}\mathcal{L}(\theta|x, y)$ are the model parameters that maximize the probability of observing the data.
- **Key point**:
  - the log-likelihood is a flexible cost function, and is often used to find model parameters that best fit the data.

---
# Appendix

We can also see $\mathrm{p}(\mathrm{y} | \mathrm{x}, \theta)$ as a function of $x$. This is the stimulus likelihood function, and it is useful in case we want to decode the input $x$ from observed responses $y$. This is what is relevant from the point of view of a neuron that does not have access to the outside world and tries to infer what's out there from the responses of other neurons!


# UCL AI Society Machine Learning Tutorials
### Session 01. Introduction to Numpy, Pandas and Matplotlib

### Contents
1. Numpy
2. Pandas
3. Matplotlib
4. EDA(Exploratory Data Analysis)

### Aim
At the end of this session, you will be able to:
- Understand the basics of numpy.
- Understand the basics of pandas.
- Understand the basics of matplotlib.
- Perform a simple EDA using libraries above.

## 3. Matplotlib
Matplotlib is a Python data visualisation library. Its plotting system is similar to that of MATLAB.

### 3.1 Basics of Matplotlib

```python
# run this cell if you haven't installed matplotlib
!pip install matplotlib
```

- `%matplotlib inline` is only available for Jupyter Notebook and Jupyter QtConsole. With this backend, the output of your command will be displayed inline with frontends, directly below the code cell that produces it.
- `%matplotlib tk` is also only available for Jupyter Notebook. The output of your command will be displayed on a new broswer.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
# %matplotlib tk
```

Plot a **y=sin(x)** graph.

```python
# declare x and y
x = np.arange(0, 3 * np.pi, 0.1)
y = np.sin(x)
print(x)
print(y)
assert len(x)==len(y)
```

```python
# plot a graph of sin(x)
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.title('y = sin(x)')
plt.show()
# plt.savefig('./image/sinGraph.png')
```

Plot multiple functions: **y = x**, **y = x^2**, and **y = x^3** in one graph.

```python
# TODO: plot multiple graphs in one graph
x = np.arange(10)
x_linear = None
x_square = None
x_cubic = None

plt.plot(x, None)
plt.plot(x, None)
plt.plot(x, None)
plt.xlabel('x axis')
plt.ylabel('y axis')
plt.grid()
plt.legend(['y = x', 'y = x^2', 'y = x^3'])
plt.title('y = x | y = x^2 | y = x^3')
# TODO: print out the graph under this cell
plt.None()
```

### 3.2 Various Types of Plots
Matplotlib library supports various types of graphs such as bar graph, histogram, scatter plot, area plot and pie plot. Let's use IMDB-Movie-Data again to get a better understanding of the data. Visualising data is a crucial part of EDA, which you'll get hands-on experience soon!

```python
movie = pd.None("./data/IMDB-Movie-Data.csv")
movie.columns
```

Let's see if there is a positive correlation between `Rating` and `Revenue (Millions)` using `scatter` function. Parameter `s` controls the diameter / size of the scattered dots and `alpha` controls the degree of their transparency.

#### 3.2.1 Scatter plot

```python
movie.plot.scatter(x = 'Rating', y = 'Revenue (Millions)', s = 10, alpha = 1)
```

They seem to have a bit of positive relationship.

Now, let's try to use different colors and markers to distinguish different groups of data and their centroids.

```python
# Read the dataset and print the first 20 samples.
scatters = pd.None("./data/scatters_data.csv",index_col=0)
scatters['class']=list(map(int,scatters['class']))
print(scatters.head(20))

# Split the datasets according to class index
clas=[0,1,2]
for clsNum in clas:
    tempData= scatters[scatters['class'].isin([clsNum])]
    exec("class%s=tempData"%clsNum)
# Store them into a dictionary
class_dict={'0':class0,'1':class1,'2':class2}
```

Firstly, calculate centroids for each class. Centroid ($c_x$,$c_y$) of a set of equally weighted points in 2D space is simply given by:

\begin{align}
\begin{array}
\$c_x  = \frac{\sum_{i=1}^n{x_i}}{n}
\end{array} ,
\begin{array}
\$c_y  = \frac{\sum_{i=1}^n{y_i}}{n}
\end{array}
\end{align}

```python
# TODO: Calculate Cx, Cy for each class
cx=[None,None,None] # x coordinates for centroids of class 0, 1 and 2
cy=[None,None,None] # y coordinates for centroids of class 0, 1 and 2
```

Let's plot scatters for data points by assigning different color and marker for different group. And then plot their centroids with corresponding color but same marker **X**.

```python
# Set potential colors and markers
color=['r','g','b']
marker=['o','s','v']
# TODO: plot scatters for data points and their centroids.
for i in clas:
    #plot the groups of data
    plt.None(class_dict[str(i)][None],class_dict[str(i)][None],color=color[i],marker=marker[i],s=100)
    #plot centroid of each group you calculate above
    plt.None(cx[i],cy[i],color=color[i],marker='x',s=200)
```

#### 3.2.2 Bar plot
Let's look into the Most Used Programming, Scripting, and Markup Languages in 2018.
Use `plt.bar` and `plt.xticks` to plot a 'language against Percentage' graph.

`plt.xticks` has two parameters: `ticks` accepts an array to set the space between scales, `labels` accepts an arary or a list to indicate the name of each scale.

```python
# https://insights.stackoverflow.com/survey/2018#most-popular-technologies
language = ["JS", "HTML", "SQL", "Java", "Shell", "Python", "C#"] # Name of each scale
percentage = [69.8, 68.5, 57.0, 45.3, 39.8, 38.8, 34.4]

# Generating the x positions as the scales on x axis.
x_positions = np.array(range(len(percentage)))

# TODO: Create a bar plot
plt.bar(None, None)
plt.xticks(None, None)
plt.xlabel('language')
plt.ylabel('Percentage (%)')
plt.title("Most Used Programming, Scripting, and Markup Languages in 2018")
plt.show()
```

Then, let's learn how to adjust the width of the bars.
`plt.bar()` has a parameter called `width` which can control the width of the bars and between each bar.

```python
# Set the bar width(any float between 0 and 1)
bar_width=None
# TODO: Create a bar plot with different bar width
plt.None(None, None,width=bar_width)
plt.xticks(None, None)
plt.xlabel('language')
plt.ylabel('Percentage (%)')
plt.title("Most Used Programming, Scripting, and Markup Languages in 2018")
plt.show()

# TODO: what if the bar_width is set bigger than 1?

```

Bar charts with multiple groups are particularly useful. Let's compare the Most Used Programming, Scripting, and Markup Languages in 2020 with that in 2018.
On this occasion, we are going to add labels, grids and each bar's value. Mind that the attributes used for `AxeSubplot` object may partially differ from these in the previous cases of directly using `pyplot` module.

`set_xticks()` is to set the ticks' position on x axis.

`set_xticklabels()` maps elements in `language` to each tick position.

`yaxis.grid()` sets the grids in y direction

`legend()` places a legend, where the parameter `loc='best'` indicates that the location of the legend card is determined automatically.

`annotate()` add text/annotation in the chart. Parameter `test` is the annotation content, `xy` is position of annotating point and `xytext` is the postion of annotation content.

```python
# language = ["JS", "HTML", "SQL", "Java", "Shell", "Python", "C#"]

# The first group (2018) of language popularity
percentage_2018 = percentage
# The second group (2020) of language popularity
percentage_2020 = [67.8, 63.5, 54.4, 41.1, 36.6, 41.7, 31.0]

fig,ax=plt.subplots()
# set bar width to 0.4, then it gives 0.1 bar padding.
barwidth=0.4
# Plot the first group of data
ax.bar(x_positions, None, width=barwidth, label='2018')
# Plot the second group of data
ax.bar(x_positions+barwidth, None, width=barwidth, label='2020')

# TODO: set proper ticks, labels, grids, and bars' value
ax.set_xticks(ticks= None) # Set the ticks in the center of the two bars
ax.set_xticklabels(language)
ax.set_yticks(ticks=range(0,100,10)) # Set the ticks on y axis by 10% spacing
ax.yaxis.grid(True)

# add the value of each bar
for xy in zip(x_positions, percentage_2018):
    ax.annotate(text="%s" % xy[1], xy=xy, xytext=(-15, 10), textcoords='offset points')
for xy in zip(x_positions+barwidth, percentage_2020):
    ax.annotate(text="%s" % xy[1], xy=xy, xytext=(-10, 10), textcoords='offset points')
ax.legend(loc='best')
ax.set_xlabel('language')
ax.set_ylabel('Percentage (%)')
fig.suptitle("Comparison of Most Used Programming Languages in 2018 and 2020")
```

It is often useful to use horizontal bars, particularly if there are a lot of them. The transformation from vertical chart to horizontal chart only requires replacing `bar` method with its horizontal counterpart `barh`, and simply switch the axis labels, grids, etc.

```python
# TODO: Rewrite the counterpart horizontal bar chart by referring to the previous case.
y_positions=np.array(range(len(percentage)))
fig,ax=plt.subplots()
# Very similar to the previous example
# Plot the first group of data
ax.barh(None)
# Plot the second group of data
ax.barh(None)
ax.set_yticks(ticks=None)
ax.set_yticklabels(None)
ax.set_xticks(None)
ax.xaxis.grid(None)
for xy in zip(percentage_2018,y_positions):
    ax.annotate(None)
for xy in zip(percentage_2020,y_positions+barwidth):
    ax.annotate(None)
ax.legend(loc=None)
ax.set_ylabel('language')
ax.set_xlabel('Percentage (%)')
fig.suptitle("Comparison of Most Used Programming Languages in 2018 and 2020")
```

### 3.3 (Advanced) Matplotlib Exercise

```python
# TODO: Draw 4 graphs in total in one figure
# Hint: https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplots.html
# You can see great example code from the url above

fig, axes = plt.None(None, None)  # this is where you need to use subplots

# TODO: Scatter Graph (Upper left)
x = np.random.randn(50)
y = np.random.randn(50)
colors = np.random.randint(0, 100, 50)
sizes = 500 * np.pi * np.random.rand(50) ** 2
axes[None, None].scatter(None)

# TODO: Bar Graph (Upper right)
x = np.arange(10)
axes[None, None].bar(x, x ** 2)

# TODO: Multi-Bar Graph (Lower left) -> Understand how it works!!
x = np.random.rand(3)
y = np.random.rand(3)
z = np.random.rand(3)
data = [x, y, z]

x_ax = np.arange(3)
for i in x_ax:
    axes[1, 0].bar(x_ax, data[i], bottom=np.sum(data[:i], axis=0))
axes[1, 0].set_xticks(x_ax)
axes[1, 0].set_xticklabels(['A', 'B', 'C'])

# TODO: Histogram Graph (Lower right)
data = np.random.randn(1000)
axes[None][None].hist(data, bins=40)

# TODO: Either show the image or save it to png file
None
```

### What to do next?
Below websites would be helpful for your further study on matplotlib:
- [DataCamp Matplotlib Tutorial: Python Plotting](https://www.datacamp.com/community/tutorials/matplotlib-tutorial-python)
- [Matplotlib official website](https://matplotlib.org/#)
- [Python Plotting With Matplotlib (Guide)](https://realpython.com/python-matplotlib-guide/)
- [Different plotting using pandas and matplotlib](https://www.geeksforgeeks.org/different-plotting-using-pandas-and-matplotlib/)
- [Matplotlib tutorial for beginner](https://github.com/rougier/matplotlib-tutorial)


Sympy Derivatives

Dr. Joshua Vaughan

joshua.vaughan@louisiana.edu

http://www.ucs.louisiana.edu/~jev9637/

```
# Import the sympy module
import sympy
```

```
# Make sympy print "pretty"
sympy.init_printing(use_latex=True)
```

```
# Define symbols
x, y, z, t = sympy.symbols(('x', 'y', 'z', 't'), real=True)
c, k = sympy.symbols(('c', 'k'), real=True, positive=True)

xdot, ydot, zdot = sympy.symbols(('\dot{x}', '\dot{y}', '\dot{z}'))

xddot, yddot, zddot = sympy.symbols(('\ddot{x}', '\ddot{y}', '\ddot{z}'))
```

```
# Define the function
f = xddot + c*xdot**2*ydot + k * ydot * zddot * x

f # Show f
```

Now, let's take some partial derivatives, as we would need to do during linearization.

First let's look at:

$$\frac{\delta f}{\delta\dot{x}}$$

```
# Now let's take some partial derivatives of f

# first df/dxdot
dfdxdot = sympy.diff(f, xdot)

dfdxdot
```

Now:

$$\frac{\delta f}{\delta x}$$

```
# df/dx
dfdx = sympy.diff(f, x)

dfdx
```

Next:

$$\frac{\delta ^2f}{\delta x\delta \dot{y}}$$

```
dfdxdy = sympy.diff(f, x, ydot)

dfdxdy
```

### Now, let's try it on a mass-spring-system (Ask Dare for details on the system)

```
x1, x2, x3, y1, y2, y3, x, y, z = sympy.symbols('x1 x2 x3 y1 y2 y3 x y z', Real=True)
l1, l2, m, k, c, g  = sympy.symbols('l1 l2 m k c g', Real=True, positive=True)

x1_dot, x1_ddot = sympy.symbols(('\dot{x}_1','\ddot{x}_1'))
x2_dot, x2_ddot = sympy.symbols(('\dot{x}_2','\ddot{x}_2'))
x3_dot, x3_ddot = sympy.symbols(('\dot{x}_3','\ddot{x}_3'))

y1_dot, y1_ddot = sympy.symbols(('\dot{y}_1','\ddot{y}_1'))
y2_dot, y2_ddot = sympy.symbols(('\dot{y}_2','\ddot{y}_2'))
y3_dot, y3_ddot = sympy.symbols(('\dot{y}_3','\ddot{y}_3'))

# Shouldn't l1 and l2 be a constant? It's the equilibrium length of the springs, right?
l1_dot, l1_ddot = sympy.symbols(('\dot{l}_1','\ddot{l}_1'))
l2_dot, l2_ddot = sympy.symbols(('\dot{l}_2','\ddot{l}_2'))

theta1 = sympy.atan2((y3 - y1),(x3 - x1))
theta2 = sympy.atan2((y2 - y3),(x2 - x3))

# deflection from equilibrium
d1 = sympy.sqrt((y1 - y3)**2 + (x1 - x3)**2) - l1
d1_dot = (((y1 - y3)*(y1_dot - y3_dot) + (x1 - x3)*(x1_dot - x3_dot)) / (sympy.sqrt((y1 - y3)**2 + (x1 - x3)**2))) - l1_dot
d2 = sympy.sqrt((y2 - y3)**2 + (x2 - x3)**2) - l2
d2_dot = (((y2 - y3)*(y2_dot - y3_dot) + (x2 - x3)*(x2_dot - x3_dot)) / (sympy.sqrt((y2 - y3)**2 + (x2 - x3)**2))) - l2_dot

x_direction = m*x3_ddot + k*d1*sympy.cos(theta1) + c*sympy.cos(theta1)*d1_dot - k*d2*sympy.cos(theta2) - c*sympy.cos(theta2)*d2_dot

y_direction = m*y3_ddot + k*d1*sympy.sin(theta1) + c*sympy.sin(theta1)*d1_dot - k*d2*sympy.sin(theta2) - c*sympy.sin(theta2)*d2_dot + m*g

```

```
# Defining K Matrix
k1_x = sympy.diff(x_direction, x3)
k1_y = sympy.diff(x_direction, y3)
k2_x = sympy.diff(y_direction, x3)
k2_y = sympy.diff(y_direction, y3)

K = sympy.Matrix([[k1_x, k1_y],[k2_x, k2_y]])

K_eq = K.subs([(m,1.75),(g,9.81),(l1,0.707),(l2,0.707),(k,50.0),(c,0.35),(x1,0.00),(x1_dot,0.0),(x2,2.00), \
           (x2_dot,0.00),(x3,1.0),(x3_dot,0.0),(y1,0.0),(y1_dot,0.0),(y2,0.0),(y2_dot,0.0),(y3,-1.29),(y3_dot,0.0), \
           (l1_dot,0.0),(l2_dot,0.0)])

K_eq
```

```
# Defining the C Matrix
c1_x = sympy.diff(x_direction, x3_dot)
c1_y = sympy.diff(x_direction, y3_dot)
c2_x = sympy.diff(y_direction, x3_dot)
c2_y = sympy.diff(y_direction, y3_dot)

C = sympy.Matrix([[c1_x, c1_y],[c2_x, c2_y]])

C_eq = C.subs([(m,1.75),(g,9.81),(l1,0.707),(l2,0.707),(k,50.0),(c,0.35),(x1,0.00),(x1_dot,0.0),(x2,2.00), \
           (x2_dot,0.00),(x3,1.0),(x3_dot,0.0),(y1,0.0),(y1_dot,0.0),(y2,0.0),(y2_dot,0.0),(y3,-1.29),(y3_dot,0.0), \
           (l1_dot,0.0),(l2_dot,0.0)])

C_eq
```

#### Licenses
Code is licensed under a 3-clause BSD style license. See the licenses/LICENSE.md file.

Other content is provided under a [Creative Commons Attribution-NonCommercial 4.0 International License](http://creativecommons.org/licenses/by-nc/4.0/), CC-BY-NC 4.0.

```
# Ignore this cell - We just update the CSS to make the notebook look a little bit better and easier to read

# Improve the notebook styling -- Run this first
from IPython.core.display import HTML
css_file = '../styling/CRAWLAB_IPythonNotebook.css'
HTML(open(css_file, "r").read())
```


# Smith Sphere

The [smith chart](http://en.wikipedia.org/wiki/Smith_chart) is a nomogram used frequently in RF/Microwave Engineering. Since its inception it has been recognised that projecting the chart onto the reimen sphere [1].

[1]H. . Wheeler, “Reflection Charts Relating to Impedance Matching,” IEEE Transactions on Microwave Theory and Techniques, vol. 32, no. 9, pp. 1008–1021, Sep. 1984.

```
from IPython.display import SVG
SVG('pics/smith_sphere.svg')
```

```
from GA.ga import Ga

from sympy import *

(ga,er,ex,es) = Ga.build('e_r e_x e_s',g=[1,1,1])
(gaz,zr,zx) = Ga.build('z_r z_x',g=[1,1])

Bz = er^ex # impedance plance
Bs = es^ex # reflection coefficient plane
Bx = er^es #
ZERO = er|ex # this is dumb
I = ga.I()

def down(p, N):
    '''
    stereographically project a vector in G3 downto the bivector N
    '''
    n= -1*N.dual()
    return -(n^p)*(n-n*(n|p)).inv()

def up(p):
    '''
    stereographically project a vector in G2 upto the space G3
    '''
    if p^Bz == ZERO:
        N = Bz
    elif  p^Bs == ZERO:
        N = Bs

    n = -N.dual()

    return   n + 2*(p*p + 1).inv()*(p-n)

a,b,c,z,s,n = [ga.mv(k,'vector') for k in ['a','b','c','z','s' ,'n']]
```

Starting with an impedance vector $z$, defined by a vector in the impedance plane  $B_z$, this vector has two scalar components ( $z^r$, $z^x$) known as resistance and reactance

```
Bz.dual()
```

\begin{equation*} -1 \boldsymbol{e_{s}} \end{equation*}

```
z = z.proj([er,ex])
z
```

\begin{equation*} z^{r} \boldsymbol{e_{r}} + z^{x} \boldsymbol{e_{x}} \end{equation*}

stereographically  up-projecting this onto the sphere to point $p$,

```
p = up(z)
p
```

\begin{equation*} \frac{2 z^{r}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{r}} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{x}} + \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{s}} \end{equation*}

```
p.norm2()
```

\begin{equation*} 1 \end{equation*}

Stereographically down-projecting this back onto the impedance plane

```
down(p, Bz)
```

\begin{equation*} z^{r} \boldsymbol{e_{r}} + z^{x} \boldsymbol{e_{x}} \end{equation*}

Rotating about the reactive plane

```
p= up(z)
R=((pi/4)*Bx).exp()

down(R*p*R.rev(),Bz)
```

\begin{equation*} \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{r}} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{x}} \end{equation*}

```
(z-er)*(z+er).inv()
```

\begin{equation*} \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1}  - \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + 2 z^{r} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{r}\wedge e_{x}} \end{equation*}

```
p
```

\begin{equation*} \frac{2 z^{r}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{r}} + \frac{2 z^{x}}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{x}} + \frac{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} - 1}{{\left ( z^{r} \right )}^{2} + {\left ( z^{x} \right )}^{2} + 1} \boldsymbol{e_{s}} \end{equation*}

```

R*er*R.rev()
```

\begin{equation*} -1 \boldsymbol{e_{s}} \end{equation*}

```
def z2p(z):
    z = real(z)*er+imag(z)*ex
    return up(z)
def p2z(p):
    z = down(p,Bz)
    return (er|z).scalar() + (ex|z).scalar()*1j
```

```
z = rand()+ rand()*1j
s = (z-1)/(z+1)
s
```

    (-0.4105211018197388+0.4168573043670432j)

```
p2z(R*z2p(z)*R.rev())
```

    -0.410521101819739 + 0.416857304367043*I

```
import skrf as rf
rf.zl_2_Gamma0(z0=1,zl=z)
```

    array([-0.4105211+0.4168573j])

```

```


# Integración Numérica

Python en Jupyter Notebook

Creado por Giancarlo Ortiz para el curso de Métodos Numéricos

## Solución de integrales por análisis numérico
la integración numérica constituye una amplia gama de algoritmos para calcular el valor numérico de una integral definida

## Agenda
1. Generalidades
1. Método del punto medio
1. Método del trapecio
1. Regla Simpson

```python
# Importar módulos al cuaderno de jupyter
import math as m
import numpy as np
import pylab as plt
from sympy import *
```

## 1. Generalidades
---
En un sistema de coordenadas cartesianas, la integral de una función $\color{#a78a4d}{f(x)}$ representa el área limitada por la gráfica de la función y el eje de las abscisas $\color{#a78a4d}{x}$ en un intervalo $\color{#a78a4d}{(a,b)}$, agregando el signo positivo cuando la función toma valores positivos o está definida por encima del eje $\color{#a78a4d}{x}$ y agregando el signo negativo cuando la función toma valores negativos o está definida por debajo del eje $\color{#a78a4d}{x}$.

\begin{equation*}
\tag{1} \int_{a}^{b} f(x) \,dx \\
\end{equation*}

Resolver esta integral consiste en calcular el área total bajo la función $\color{#a78a4d}{f(x)}$ en un intervalo definido, del [teorema fundamental del calculo](https://es.wikipedia.org/wiki/Teorema_fundamental_del_c%C3%A1lculo) se obtiene la solucion exacta evaluando la función primitiva $\color{#a78a4d}{F(x)}$ en los limites de la integral definida.

\begin{equation*}
\tag{1} \int_{a}^{b} f(x) \,dx = F(b) - F(a) \\
\end{equation*}

    Prueba:

\begin{equation}
\begin{split}
A & = \frac{\pi r^2}{2} \\
 & = \frac{1}{2} \pi r^2
\end{split}
\end{equation}

\begin{align*}
2x - 5y &=  8 \\
3x + 9y &=  -12
\end{align*}

\begin{align*}
x&=y           &  w &=z              &  a&=b+c\\
2x&=-y         &  3w&=\frac{1}{2}z   &  a&=b\\
-4 + 5x&=2+y   &  w+2&=-1+w          &  ab&=cb
\end{align*}

| $\tag{Texto}$ | $\tag{Formula}$ |
| ------------: | :-------------- |
|                             | $\tag{1} y - f(a) = \frac{ f(b)-f(a) }{ b-a } \left( x - a \right)$ |
| Despejando de $x$ que es uno| $\tag{2} y - f(a) = \frac{ f(b)-f(a) }{ b-a } \left( x - a \right)$ |

    Definitivo:

```python
fig, ax = plt.subplots(1, 2)
fig.set_size_inches(16, 4)
```

---
## Mas Recursos

- [Integración numérica](https://es.wikipedia.org/wiki/Integraci%C3%B3n_num%C3%A9rica) (Wikipedia)
- [Suma de Riemann](https://es.wikipedia.org/wiki/Suma_de_Riemann) (Wikipedia)
- [Regla del trapecio](https://es.wikipedia.org/wiki/Regla_del_trapecio) (Wikipedia)
- [Regla de Simpson](https://es.wikipedia.org/wiki/Regla_de_Simpson) (Wikipedia)


# Sampling of Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Ideal Sampling and Reconstruction

[Digital signal processors](https://en.wikipedia.org/wiki/Digital_signal_processor) and general purpose processors can only perform arithmetic operations within a limited number range. So far we considered continuous signals which are continuous with respect to time and its amplitude values. Such signals cannot be handled by processors in a straightforward manner. In order to obtain a digital representation of a continuous signal, a discretization has to be performed both in time and amplitude. The former is known as [*sampling*](https://en.wikipedia.org/wiki/Sampling_%28signal_processing%29), the latter as [*quantization*](https://en.wikipedia.org/wiki/Quantization_%28signal_processing%29). Sampling refers to the process of picking amplitude values from a continuous signal at discrete time-instants. The sampled signal is referred to as *discrete signal*. Quantization refers to the process of mapping a continuous amplitude to a countable set of amplitude values. The quantized signal is referred to as *quantized signal*. A signal which is discrete and quantized is termed a *digital signal*. This is illustrated in the following

The order of sampling and quantization can be exchanged under the assumption that both are memoryless processes. Only digital signals can be handled by digital signal or general purpose processors. The sampling of signals is discussed as a first step towards a digital signal.

### Model of Ideal Sampling

A continuous signal $x(t)$ is sampled by taking its amplitude values at given time-instants. These time-instants can be chosen arbitrary in time, but most common are equidistant sampling schemes. The process of sampling is modeled by multiplying the continuous signal with a series of Dirac impulses. This constitutes an idealized model since Dirac impulses cannot be realized in practice.

For equidistant sampling of a continuous signal $x(t)$ with sampling interval $T$, the sampled signal $x_\text{s}(t)$ reads

\begin{equation}
x_\text{s}(t) = \sum_{k = - \infty}^{\infty} x(t) \cdot \delta(t - k T) = \sum_{k = - \infty}^{\infty} x(k T) \cdot \delta(t - k T)
\end{equation}

where the [multiplication property](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) of the Dirac impulse was used for the last equality. The sampled signal is composed from a series of equidistant Dirac impulse which are weighted by the amplitude values of the continuous signal taken at their time-instants.

The series of Dirac impulse is represented conveniently by the [Dirac comb](../periodic_signals/spectrum.ipynb#The-Dirac-Comb). Rewriting the sampled signal yields

\begin{equation}
x_\text{s}(t) = x(t) \cdot \frac{1}{T} {\bot \!\! \bot \!\! \bot} \left( \frac{t}{T} \right)
\end{equation}

The process of sampling can be modeled by multipyling the continuous signal $x(t)$ with a Dirac comb. The samples $x(k T)$ for $k \in \mathbb{Z}$ of the continuous signal constitute the [discrete (-time) signal](https://en.wikipedia.org/wiki/Discrete-time_signal) $x[k] := x(k T)$. The question arises if and under which conditions the samples $x[k]$ fully represent the continuous signal and allow for a reconstruction of the analog signal. In order to investigate this, the spectrum of the sampled signal is derived.

### Spectrum of Sampled Signal

The spectrum $X_\text{s}(j \omega) = \mathcal{F} \{ x_\text{s}(t) \}$ of the sampled signal $x_\text{s}(t)$ is derived by applying the [multiplication theorem](../fourier_transform/theorems.ipynb#Multiplication-Theorem) to the representation of the sampled signal using the Dirac comb

\begin{equation}
\begin{split}
X_\text{s}(j \omega) &= \frac{1}{2 \pi} X(j \omega) * {\bot \!\! \bot \!\! \bot} \left( \frac{\omega}{\omega_\text{s}} \right) \\
&= \frac{1}{2 \pi} X(j \omega) * \frac{2 \pi}{T} \sum_{\mu = - \infty}^{\infty} \delta(\omega - \mu \omega_\text{s}) \\
&= \frac{1}{T} \sum_{\mu = - \infty}^{\infty} X \left(j (\omega - \mu \omega_\text{s}) \right)
\end{split}
\end{equation}

where $X(j \omega) = \mathcal{F} \{ x(t) \}$ denotes the Fourier transform of the continuous signal, $\omega_\text{s} = 2 \pi \, f_\text{s}$ the angluar sampling frequency and $f_\text{s} = \frac{1}{T}$ the sampling frequency. The second equality results from the definition of the Dirac comb and the scaling property of the Dirac impulse, the third from its sifting property. The spectrum of the sampled signal consists of a superposition of shifted copies of the spectrum of the continuous signal. The resulting spectrum is periodic with a period of $\omega_\text{s}$. It can be concluded, that equidistant sampling generates repetitions of the spectrum of the continuous signal.

The spectrum $X_\text{s}(j \omega)$ of a sampled signal is illustrated at the example of a real-valued low-pass signal. A low-pass signal $x(t)$ is a signal with band-limited spectrum

\begin{equation}
X(j \omega) = 0 \qquad \text{for } |\omega| > \omega_\text{u}
\end{equation}

where $\omega_\text{u}$ denotes its upper frequency limit. For ease of illustration, the generic spectrum of a continuous real-valued low-pass signal is depicted by a triangle-shaped spectrum

The spectrum of the sampled signal is constructed by superimposing shifted copies of the spectrum of the continuous low-pass signal $X(j \omega)$ at multiples of $\omega_\text{s}$

It can be concluded from the illustration, that

* the shifted copies of $X(j \omega)$ do not overlap if $\omega_\text{u} < \frac{\omega_\text{s}}{2}$. For $|\omega| < \omega_\text{u}$ the spectrum of the continuous signal is not affected by overlapping
* for $\omega_\text{u} > \frac{\omega_\text{s}}{2}$ overlapping occurs which changes the spectrum of the continuous signal within $|\omega| < \omega_\text{u}$

### Ideal Reconstruction

The question arises if and under which conditions the continuous signal can be recovered from the sampled signal. Above consideration revealed that the spectrum $X_\text{s}(j \omega)$ of the sampled signal contains the unaltered spectrum of the continuous signal $X(j \omega)$ if $\omega_\text{u} < \frac{\omega_\text{s}}{2}$. Hence, the continuous signal can be reconstructed from the sampled signal by extracting the spectrum of the continuous signal from the spectrum of the sampled signal. This can be done by applying an [ideal low-pass](../system_properties/idealized_systems.ipynb#Ideal-Low-Pass) with cut-off frequency $\omega_\text{c} = \frac{\omega_{s}}{2}$. This is illustrated in the following

where the blue line represents the spectrum of the sampled signal and the red line the spectrum of the ideal low-pass. The transfer function $H(j \omega)$ of the low-pass reads

\begin{equation}
H(j \omega) = T \cdot \text{rect} \left( \frac{\omega}{\omega_\text{s}} \right)
\end{equation}

Its impulse response $h(t)$ is yielded by inverse Fourier transform of the transfer function

\begin{equation}
h(t) = \text{sinc} \left( \frac{\pi t}{T} \right)
\end{equation}

The reconstructed signal $y(t)$ is given by convolving the sampled signal $x_\text{s}(t)$ with the impulse response of the low-pass filter. This yields

\begin{align}
y(t) &= x_\text{s}(t) * h(t) \\
&= \left( \sum_{k = - \infty}^{\infty} x(k T) \cdot \delta(t - k T) \right) * \text{sinc} \left( \frac{\pi t}{T} \right) \\
&= \sum_{k = - \infty}^{\infty} x(k T) \cdot \text{sinc} \left( \frac{\pi}{T} (t - k T) \right)
\end{align}

where for the last equality the fact was exploited that $x(k T)$ is independent of the time $t$ for which the convolution is performed. The reconstructed signal is given by a weighted superposition of shifted sinc functions. Their weights are given by the samples $x(k T)$ of the continuous signal. The reconstruction is illustrated in the following figure

The black boxes show the samples $x(k T)$ of the continuous signal, the blue line the reconstructed signal $y(t)$, the gray lines the weighted sinc functions. The sinc function for $k = 0$ is highlighted in red. The amplitudes $x(k T)$ at the sampled positions are reconstructed perfectly since

\begin{equation}
\text{sinc} ( \frac{\pi}{T} (t - k T) ) = \begin{cases}
\text{sinc}(0) = 1 & \text{for } t=k T \\
\text{sinc}(n \pi) = 0 & \text{for } t=(k+n) T \quad , n \in \mathbb{Z} \notin \{0\}
\end{cases}
\end{equation}

The amplitude values in between the sampling positions $t = k T$ are given by superimposing the shifted sinc functions. The process of computing values in between given sampling points is termed [*interpolation*](https://en.wikipedia.org/wiki/Interpolation). The reconstruction of the sampled signal is performed by interpolating the discrete amplitude values $x(k T)$. The sinc function is the optimal interpolator for band-limited signals.

### Aliasing

So far the case was discussed when no overlaps occur in the spectrum of the sampled signal. Hence when the upper frequency limit $\omega_\text{u}$ of the real-valued low-pass signal is lower than $\frac{\omega_\text{s}}{2}$. Here a perfect reconstruction of the continuous signal $x(t)$ from its discrete counterpart $x[k]$ is possible. However when this condition is not met, the repetitions of the spectrum of the continuous signal overlap. This is illustrated in the following

In this case no perfect reconstruction of the continuous signal by low-pass filtering (interpolation) of the sampled signal is possible. The spectrum within the pass-band of the low-pass contains additional contributions from the repeated spectrum of the continuous signal. These contributions are known as [aliasing](https://en.wikipedia.org/wiki/Aliasing). It becomes evident from above discussion of ideal reconstruction that the amplitude values are reconstructed correctly at the time-instants $k T$. However, in between these time-instants the reconstructed signal $y(t)$ differs from the sampled signal $x(t)$ if aliasing is present.

### Sampling Theorem for Low-Pass Signals

It can be concluded from above discussion of sampling, that a sufficient condition for the perfect reconstruction of a real-valued low-pass signal $x(t)$ is given as

\begin{equation}
\omega_\text{s} \geq 2 \cdot \omega_\text{c}
\end{equation}

The minimum sampling frequency has to be chosen as double the highest frequency present in the continuous signal. This condition is known as [*Nyquist–Shannon sampling theorem*](https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem). Only if this condition is fulfilled, all information contained in a low-pass signal $x(t)$ is represented by its samples $x[k] = x(k T)$.

Depending on the relation between the sampling frequency $\omega_\text{s}$ and the upper frequency limit $\omega_\text{u}$ of the low-pass signal, three different cases can be distinguished

* oversampling $\omega_\text{s} > 2 \cdot \omega_\text{c}$
* critical sampling $\omega_\text{s} = 2 \cdot \omega_\text{c}$
* undersampling $\omega_\text{s} < 2 \cdot \omega_\text{c}$

In practical applications sampling is always oversampled to some degree since the ideal low-pass used to reconstruct the continuous signal cannot be realized. Examples for sampling rates in audio are

| Application | Sampling frequency $f_\text{s}$ |
|---|:---:|
| Telephone service | Narrowband: 8 kHz, Wideband: 16 kHz |
| [Compact Disc (CD)](https://en.wikipedia.org/wiki/Compact_disc) | [44.1 kHz](https://en.wikipedia.org/wiki/44,100_Hz) |
| [DVD-Audio](https://en.wikipedia.org/wiki/DVD-Audio) | 44.1, 48, 88.2, 96, 176.4, 192 kHz |

### Example: Ideal Sampling and Reconstruction of a Cosine Signal

The ideal sampling and reconstruction of an analog signal $x(t)$ is illustrated in the following. For ease of illustration a cosine signal $x(t) = \cos(\omega_0 t)$ is considered. First, two `Python` functions are defined which ideally sample the signal $x(t)$ at equidistant time-instants $t = k T$ and compute the reconstructed signal $y(t)$ from the samples by applying an ideal low-pass.

```python
import sympy as sym
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
sym.init_printing()

t = sym.symbols('t', real=True)
k = sym.symbols('k', integer=True)

def ideal_sampling(x, k, w_s):
    kappa = sym.symbols('kappa')
    xs = sym.lambdify(kappa, x.subs(t, kappa * 2 * sym.pi / w_s))
    return [xs(kappa) for kappa in k]

def ideal_reconstruction(xs, k, w_s):
    T = 2*sym.pi/w_s
    return sum(xs[n] * sym.sinc(sym.pi / T * (t - k[n] * T)) for n in range(len(k)))
```

Furthermore a helper function for plotting of the sampled and reconstructed signal is defined.

```python
def plot_signals(xs, y, w_s, k):

    plt.stem(k*2*np.pi/w_s, xs)
    plt.xlabel('$t$ in s')
    plt.ylabel('$x_s[k] = x_s(kT)$')
    plt.axis([0, 5, -1.2, 1.2])

    sym.plot(y, (t, 0, 5), xlabel='$t$', ylabel='$y(t)$', ylim=(-1.2, 1.2))
```

Now the continuous signal $x(t) = \cos(\omega_0 t)$ to be sampled and reconstructed is defined and plotted for $\omega_0 = 5$.

```python
w_0 = 5
x = sym.cos(w_0 * t)

sym.plot(x, (t, 0, 5), xlabel=r'$t$', ylabel=r'$x(t)$')
```

    <sympy.plotting.plot.Plot at 0x107d70748>

First the case of oversampling $\omega_\text{s} > 2 \cdot \omega_0$ with $\omega_\text{s} = 50$ is illustrated

```python
k = np.arange(-100, 100)

w_s = 50
xs = ideal_sampling(x, k, w_s)
y = ideal_reconstruction(xs, k, w_s)

plot_signals(xs, y, w_s, k)
```

Then the case of critical sampling $\omega_\text{s} = 2 \cdot \omega_0$ with $\omega_\text{s} = 10$ is illustrated

```python
w_s = 10
xs = ideal_sampling(x, k, w_s)
y = ideal_reconstruction(xs, k, w_s)

plot_signals(xs, y, w_s, k)
```

Finally the case of undersampling $\omega_\text{s} < 2 \cdot \omega_0$ with $\omega_\text{s} = 7$ is illustrated

```python
w_s = 7
xs = ideal_sampling(x, k, w_s)
y = ideal_reconstruction(xs, k, w_s)

plot_signals(xs, y, w_s, k)
```

**Exercise**

* Derive the spectrum of the reconstructed signal for the sampling of $x = \cos(\omega_0 t)$ with sampling frequency $\omega_s$ by calculating

    * the spectrum $X(j \omega)$ of the continuous signal
    * the spectrum $X_\text{s}(j \omega)$ of the sampled signal
    * the spectrum $Y(j \omega)$ of the reconstructed signal for the case of over-, critial- and undersampling
    * the reconstructed signal $y(t)$ for the case of over-, critial- and undersampling

* Reevaluate above example with $x(t) = \text{rect}(t - \frac{3}{2})$.

    * Hint: Define the signal by `x = sym.Heaviside(t-1) - sym.Heaviside(t-2)`
    * Is a perfect reconstruction possible? If not, why?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.