# TSSL Lab 2 - Structural model, Kalman filtering and EM
We will continue to work with the Global Mean Sea Level (GMSL) data that we got acquainted with in lab 1. The data is taken from https://climate.nasa.gov/vital-signs/sea-level/ and is available on LISAM in the file `sealevel.csv`.

In this lab we will analyse this data using a structural time series model. We will first set up a model and implement a Kalman filter to infer the latet states of the model, as well doing long-term prediction. We will then implement a disturbance smoother and an expectation maximization algorithm to tune the parameters of the model.

We load a few packages that are useful for solving this lab assignment.

```python
import pandas  # Loading data / handling data frames
import numpy as np
import scipy
from scipy import  linalg
import matplotlib.pyplot as plt
plt.rcParams["figure.figsize"] = (12,8)  # Increase default size of plots
```

## 2.1 Setting up a structural state space model

We start by loading and plotting data to remind ourselves what it looks like.

```python
data=pandas.read_csv('sealevel.csv',header=0)
```

```python
y = data['GMSL'].values
u = data['Year'].values
ndata = len(y)
plt.plot(u,y)
plt.xlabel('Year')
plt.ylabel('GMSL deviation (mm)')
plt.title("Data")
plt.show()
```

In this lab we will use a structural time series model to analys this data set. Specifically, we assume that the data $\{y_t\}_{t\geq 1}$ is generated by

\begin{align}
    y_t = \mu_t + \gamma_t + \varepsilon_t
\end{align}

where $\mu_t$ is a trend component, $\gamma_t$ is a seasonal component, and $\varepsilon_t$ is an observation noise. The model is expressed using a state space representation,

\begin{align}
    \alpha_{t+1} &= T \alpha_t + R\eta_t, & \eta_t&\sim N(0,Q), \\
    y_t &= Z \alpha_t + \varepsilon_t, & \varepsilon_t&\sim N(0,\sigma_\varepsilon^2).
\end{align}

**Q0:** Let $d = \dim(\alpha_t)$ denote the _state dimension_ and $d_\eta = \dim(\eta_t)$ denote the dimension of the state noise. Then, what are the dimenisons of the matrices $T$, $R$, and $Z$ of the state space model?

**A:**

Dimensions is given as  Row * Column and it is given that d = dim($\alpha_{t}$) and $d_\eta = \dim(\eta_t)$

T = d * d

R = $d_\eta$ * 2

Z = 1 * d

**Q1:** Create the state space matrices $T_{[\mu]}$, $R_{[\mu]}$, and $Z_{[\mu]}$ corresponding to the trend component $\mu_t$. We assume a local linear trend (that is, of order $k=2$).

_Hint:_ Use **2-dimensional** `numpy.ndarray`s of the correct sizes to represent all the matrices.

```python
T_mu = np.array([[2,-1],[1,0]])
R_mu = np.array([[1],[0]])
Z_mu = np.array([[1 ,0 ]])
```

```python
print(T_mu.shape)
print(R_mu.shape)
print(Z_mu.shape)
```

    (2, 2)
    (2, 1)
    (1, 2)

**Q2:** There is a yearly seasonal pattern present in the data. What should we set the periodicity $s$ of the seasonal component to, to capture this pattern?

_Hint:_ Count the average number of observations per (whole) year and round to the closest integer.

```python
#s = 37 manual check
#coverting sample to year unit only
sampleYearly = u.astype(int)
unique_elements, counts_elements = np.unique(sampleYearly, return_counts=True)
#because of presence of last year count is decreasing by 1 so removing it as avg value is coming as 36 while without that it is 37 so going ahead with 37
#s = round(np.mean(np.delete(counts_elements , -1)))
s = round(np.mean(counts_elements))
# Removed assumptions
print(f"Average number of observations per (whole) year and round to the closest integer is {s}")
# print(f"Year   : {unique_elements}")
# print(f"count per year : {counts_elements} ")
# print( f"avg of count {s}")
```

    Average number of observations per (whole) year and round to the closest integer is 36.0

**Q3:** What is the _state dimension_ of a seasonal component with periodicity $s$? That is, how many states are needed in the corresponding state space representation?

**A:** For S as 36 seasonal component, state dimension will be S-1 * S-1 i.e 35*35

**Q4:** Create the state space matrices $T_{[\gamma]}$, $R_{[\gamma]}$, and $Z_{[\gamma]}$ corresponding to the seasonal component $\gamma_t$.

_Hint:_ Use **2-dimensional** `numpy.ndarray`s of the correct sizes to represent all the matrices.

```python
T_gamma = -1 * np.ones((1,35) , dtype=int)
T_gamma = np.append(T_gamma , np.append(np.identity(34) , np.zeros((34 ,1)) , axis = 1 ) , axis= 0)
R_gamma = np.zeros((35,1) , dtype= int)
R_gamma[0,0] = 1
Z_gamma = np.zeros((1,35) , dtype = int)
Z_gamma[0,0] = 1
```

```python
print(T_gamma.shape)
print(R_gamma.shape)
print(Z_gamma.shape)
```

    (35, 35)
    (35, 1)
    (1, 35)

**Q5:** Using the matrices that you have constructed above, create the state space matrices for the complete structural time series model. Print out the shapes of the resulting system matrices and check that they correspond to what you expect (cf **Q0**).

_Hint:_ Use `scipy.linalg.block_diag` and `numpy.concatenate`.

```python
T = scipy.linalg.block_diag(T_mu , T_gamma)
R = scipy.linalg.block_diag(R_mu , R_gamma)
Z = np.concatenate((Z_mu , Z_gamma) , axis = 1 )
```

```python
print(f"Shape for T is {T.shape}")
print(f"Shape for R is {R.shape}")
print(f"Shape for Z is {Z.shape}")
```

    Shape for T is (37, 37)
    Shape for R is (37, 2)
    Shape for Z is (1, 37)

```python
#Z
```

Looking at above shape we can say that they corresponds to Q0 as dimensions are same as expected.

We also need to specify the variances of the process noise $\eta_t$ and measurement noise $\varepsilon_t$. Below, we will estimate (two of) these variances from data, but for now we set them arbitrarily to get an initial model to work with.

```python
# Some arbitrary noise values for now
sigma_trend = 0.01
sigma_seas = 1
sigma_eps = 1
Q = np.array([[sigma_trend**2, 0.], [0., sigma_seas**2]])  # Process noise covariance matrix
```

Finally, to complete the model we need to specify the distribution of the initial state. This encodes our _a priori_ belief about the actual values of the trend and seasonality, i.e., before observing any data.

**Q6:** Set up the mean vector of the initial state $a_1 = \mathbb{E}[\alpha_1]$ such that:
* The trend component starts at the first observation, $\mathbb{E}[\mu_1] = y_1$,
* The slope of the trend is _a priori_ zero in expectation, $\mathbb{E}[\mu_1 - \mu_0] = 0$,
* The initial mean of all states related to the seasonal component are zero.

Also, create an initial state covariance matrix $P_1 = \text{Cov}(\alpha_1)$ as an identity matrix of the correct dimension, multiplied with a large value (say, 100) to represent our uncertainty about the initial state.

```python
# a1 = y[0,]
# a1 = np.append(a1.reshape(1,1) , np.zeros((36,1)) , axis=0)
```

```python
a1 = np.zeros(37).reshape(37,1)
a1[0] = y[0]
```

```python
P1 = 100 * np.identity(n = 37)
```

```python
P1.shape
```

    (37, 37)

We have now defined all the matrices etc. that make up the structural state space model. For convenience, we can create an object of the class `LGSS` available in the module `tssltools_lab2` as a container for these quantities.

```python
from tssltools_lab2 import LGSS  # Module available in LISAM
model = LGSS(T, R, Q, Z, sigma_eps**2, a1, P1)

help(model.get_params)
```

    Help on method get_params in module tssltools_lab2:

    get_params() method of tssltools_lab2.LGSS instance
        Return all model parameters.

        T, R, Q, Z, H, a1, P1 = model.get_params()

## 2.2 Kalman filtering for the structural model
Now we have the data and a model available. Next, we will turn our attention to the inference problem, which is a central task when analysing time series data using the state space framework.

State inference is the problem of estimating the unknown (latent) state variables given the data. For the time being we assume that the _model parameters_ are completely specified, according to above, and only consider how to estimate the states using the Kalman filter.

In the questions below we will treat the first $n=800$ time steps as training data and the remaining $m=197$ observations as validation data.

```python
n = 800
m = ndata-n
```

**Q7:** Complete the Kalman filter implementation below. The function should be able to handle missing observations, which are encoded as "not a number", i.e. `y[t] = np.nan` for certain time steps `t`.

_Hint:_ The Kalman filter involves a lot of matrix-matrix and matrix-vector multiplications. It turns out to be convient to store sequences of vectors (such as the predicted and filtered state estimates) as `(d,1,n)` arrays, instead of `(d,n)` or `(n,d)` arrays. In this way the matrix multiplications will result in 2d-arrays of the correct shapes without having to use a lot of explicit `reshape`. However, clearly, this is just a matter of coding style preferences!

```python
from tssltools_lab2 import kfs_res  # Module available in LISAM. kfs_res is a container class for storing the result.

def kalman_filter(y, model: LGSS):
    """Kalman filter for LGSS model with one-dimensional observation.

    :param y: (n,) array of observations. May contain nan, which encodes missing observations.
    :param model: LGSS object with the model specification.

    :return kfs_res: Container class with member variables,
        alpha_pred: (d,1,n) array of predicted state means.
        P_pred: (d,d,n) array of predicted state covariances.
        alpha_filt: (d,1,n) array of filtered state means.
        P_filt: (d,d,n) array of filtered state covariances.
        y_pred: (n,) array of means of p(y_t | y_{1:t-1})
        F_pred: (n,) array of variances of p(y_t | y_{1:t-1})
    """

    n = len(y)
    d = model.d  # State dimension
    alpha_pred = np.zeros((d, 1, n))
    P_pred = np.zeros((d, d, n))
    alpha_filt = np.zeros((d, 1, n))
    P_filt = np.zeros((d, d, n)) #renamed
    y_pred = np.zeros(n)
    F_pred = np.zeros(n)

    #print(alpha_filt.shape)

    T, R, Q, Z, H, a1, P1 = model.get_params()  # Get all model parameters (for brevity)

    for t in range(n):
        # Time update (predict)
        if t == 0:  # Initialize predictions at first time step
            alpha_pred[:,:,t] = a1
            P_pred[:,:,t] = P1
        else:  # All consecutive time steps
            #@ > dot product
            alpha_pred[:,:,t] = T @ alpha_filt[:,:,(t-1)]
            #alpha_pred[:,:,t] = alpha_filt[:,:,(t-1)]
            P_pred[:,:,t] = (T @ P_filt[:,:,(t-1)]) @ np.transpose(T) + R @ Q @ np.transpose(R)
        # Compute prediction of current output
        y_pred[t] = Z @ alpha_pred[:,:,t]
        F_pred[t] = ((Z @ P_pred[:,:,t]) @ np.transpose(Z)) +  H
        #F_pred[t] = Z @ P_pred[:,:,t] @ np.transpose(Z)

        # Measurement update (correct)
        if np.isnan(y[t]):  # Handle missing data
          alpha_filt[:,:,t] = alpha_pred[:,:,t]
          P_filt[:,:,t] = P_pred[:,:,t]
        else:
          K_Gain = (P_pred[:,:,t] @ np.transpose(Z))/F_pred[t]
          alpha_filt[:,:,t] = alpha_pred[:,:,t] + (K_Gain * (y[t] - y_pred[t]))
          #print(alpha_filt.shape)
          P_filt[:,:,t] = (np.identity(d) - (K_Gain @ Z)) @ P_pred[:,:,t]

    # Container for storing all results
    kf = kfs_res(alpha_pred, P_pred, alpha_filt, P_filt, y_pred, F_pred)
    return kf
```

**Q8:** Use the Kalman filter to infer the states of the structural time series applied to the sealevel data. Run the filter on the training data (i.e., first $n=700$ time steps), followed by a long-range prediction of $y_t$ for the remaining time points.

Generate a plot which shows:
1. The data $y_{1:n+m}$,
2. The one-step predictions $\hat y_{t|t-1} \pm 1 $ standard deviation for the training data, i.e., $t = 1,...,n$,
3. The long-range predictions $\hat y_{t|n} \pm 1 $ standard deviation for the validation data, i.e., $t= n+1,...,n+m$,
4. A vertical line indicating the switch between training and validation data, using, e.g., `plt.axvline(x=u[n])`.

_Hint:_ It is enough to call the `kalman_filter` function once. Make use of the missing data functionality!

```python
#it looks like we have some issue with n consideration as in Q7 it is asked to use 800 while in Q8 it is asked to use 700
#Going with Q7 way with n as 800 datapoints
n = 800
m = ndata-n
yTrain = y.copy()
#implemented such that for we can use it for long range prediction(i.e. adding Nan for future values)
yTrain[801:] = np.nan
kf = kalman_filter(yTrain , model)

sd1 = kf.y_pred - np.sqrt(kf.F_pred)
sd2 = kf.y_pred + np.sqrt(kf.F_pred)
```

```python
#Data Plot
plt.plot(u,y ,label = "Original Data")
plt.plot(u , kf.y_pred , label = "Kalman Pred")
#plt.scatter(u , sd1 , label = "Standard Deviation" , color = "green")
#plt.scatter(u , sd2 , label = "Standard Deviation" , color = "green")
plt.plot(u , sd1 , label = "Standard Deviation" , color = "green")
plt.plot(u , sd2 , label = "Standard Deviation" , color = "green")
plt.axvline(x=u[n])
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("Kalman Filter")
plt.legend()
plt.show()
```

**Q9:** Based on the output of the Kalman filter, compute the training data log-likelihood $\log p(y_{1:n})$.

```python
#logLikLi = (-1/2)*(np.sum(np.log(np.absolute(kf.F_pred[:801,])) + (np.transpose(yTrain[:801,] - kf.y_pred[:801,]) / kf.F_pred[:801,]) @ (yTrain[:801,] - kf.y_pred[:801,])))
# -2085686.7746347892 old value kept for comparision
logLikLi = (-1/2)*(np.sum(np.log(np.abs(kf.F_pred[:801,])) + (yTrain[:801,] - kf.y_pred[:801,])**2 / kf.F_pred[:801,]))
print(f"Log-Likelihood for Training Data is {logLikLi}")

```

    Log-Likelihood for Training Data is -3199.819037286188

## 2.3 Identifying the noise variances using the EM algorithm
So far we have used fixed model parameters when running the filter. In this section we will see how the model parameters can be learnt from data using the EM algorithm. Specifically, we will try to learn the variance of the state noise affecting the seasonal component as well as the variance of the observation noise,

\begin{align}
    \theta = (\sigma_\gamma^2, \sigma_\varepsilon^2).
\end{align}

For brevity, the variance of the trend component $\sigma_\mu^2$ is fixed to the value $\sigma_\mu^2 = 0.01^2$ as above. (See Appendix A below for an explanation.)

Recall that we consider $y_{1:n}$ as the training data, i.e., we will estimate $\theta$ using only the first $n=800$ observations.

**Q10:** Which optimization problem is it that the EM algorithm is designed to solve? Complete the line below!

**A:** $\hat\theta = \arg\max_{\theta} Q(\theta | \theta_{known}$)

**Q11:** Write down the updating equations on closed form for the M-step in the EM algorithm.

_Hint: Look at Exercise Session 2_

**A:**

$\sigma_{\epsilon} ^ 2 $ = 1/n * $\sum_{t =1}^{n}$ ($\hat{\epsilon_{t}^2} + Var(\epsilon_{t} |Y_{n})  $)

$\sigma_{\eta} ^ 2$ = 1/n * $\sum_{t =1}^{n}$trace(($\hat{\eta_{t}^2}$ + Cov($\eta_t|Y_n$)

To implement the EM algorithm we need to solve a _smoothing problem_. The Kalman filter that we implemented above is based only on a forward propagation of information. The _smoother_ complements the forward filter with a backward pass to compute refined state estimates. Specifically, the smoothed state estimates comprise the mean and covariances of

\begin{align}
    &p(\alpha_t \mid y_{1:n}), & t=&1,\dots,n
\end{align}

Furthermore, when implementing the EM algorithm it is convenient to work with the (closely related) smoothed estimates of the disturbances, i.e., the state and measurement noise,

\begin{align}
    &p(\eta_t \mid y_{1:n}), & t=&1,\dots,n-1 \\
    &p(\varepsilon_t \mid y_{1:n}), & t=&1,\dots,n
\end{align}

An implementation of a state and disturbance smoother is available in the `tssltools_lab2` module. You may use this when implementing the EM algorithm below.

```python
from tssltools_lab2 import kalman_smoother
help(kalman_smoother)
```

    Help on function kalman_smoother in module tssltools_lab2:

    kalman_smoother(y, model:tssltools_lab2.LGSS, kf:tssltools_lab2.kfs_res)
        Kalman (state and disturbance) smoother for LGSS model with one-dimensional observation.

        :param y: (n,) array of observations. May contain nan, which encodes missing observations.
        :param model: LGSS object with the model specification.
        :parma kf: kfs_res object with result from a Kalman filter foward pass.

        :return kfs_res: Container class. The original Kalman filter result is augmented with the following member variables,
            alpha_sm: (d,1,n) array of smoothed state means.
            V: (d,d,n) array of smoothed state covariances.
            eps_hat: (n,) array of smoothed means of observation disturbances.
            eps_var: (n,) array of smoothed variances of observation disturbances.
            eta_hat: (deta,1,n) array of smoothed means of state disturbances.
            eta_cov: (deta,deta,n) array of smoothed covariances of state disturbances.

**Q12:** Implement an EM algorithm by completing the code below. Run the algorithm for 100 iterations and plot the traces of the parameter estimates, i.e., the values $\theta_r$, for $r = 0,\dots,100$.

_Note:_ When running the Kalman filter as part of the EM loop you should only filter the _training data_ (i.e. excluding the prediction for validation data).

```python
num_iter = 100
yTrain = y.copy()
yTrain = yTrain[:801]
# Some arbitrary noise values for now
sigma_trend = 0.01
sigma_seas = 1
#sigma_eps = 1
Q = np.array([[sigma_trend**2, 0.], [0., sigma_seas**2]])  # Process noise covariance matrix

def compute_theta(ks):
    """Implements the M-step of the EM-algorithm, based on the Kalman smoother results (ks). See Q11."""
    n = len(ks.eps_hat)
    sigma_epsilon2  = (1/n) * np.sum(ks.eps_hat**2 + ks.eps_var) #observation
    #need to be calculated along diagonal only
    #sigma_neta2 = (1/(n-1)) * np.sum(ks.eta_hat**2 - ks.eta_cov) #seasonal

    traceMatSum = 0
    #t_Mat = (ks.eta_hat.T @  ks.eta_hat + ks.eta_cov)    # dimensinal error because of grouping
    #  traceMatSum = np.trace(t_Mat)
    for t in range(n):
      t_Mat = (ks.eta_hat[:,:,t] @  ks.eta_hat[:,:,t].T + ks.eta_cov[:,:,t])
      traceMatSum = traceMatSum + t_Mat[1,1]

    sigma_neta2 = (1/n) * traceMatSum
    theta_new = (sigma_epsilon2 , sigma_neta2)
    return theta_new

minChange = 0.001 #custom threshold set, could be changed based on expected diff
logLikeLi = []
sigma_eps_Arr = []
sigma_seas_Arr = []
#Q = np.array([[sigma_trend**2, 0.], [0., sigma_seas**2]])  # Process noise covariance matrix #from above

sigma_eps2 = 1**2 #given

#changing r to 0 in-order to python indexing
for r in range(0,num_iter):
    print(f"Number of iteration is : {r}")
    # E-step
    model = LGSS(T, R , Q, Z , sigma_eps2 , a1 , P1) #model Parameter
    kf = kalman_filter(yTrain , model)#Call Kalman filter with updated model
    ks = kalman_smoother(yTrain, model, kf)

    #calculating log likelihood while ignoring constant as it is same accross all

    logLikeLi.append((-1/2)*(np.sum(np.log(np.abs(kf.F_pred)) + (yTrain - kf.y_pred)**2 / kf.F_pred)))

    if r > 0:
      if (abs(logLikeLi[r] - logLikeLi[(r-1)]) > minChange):
        print(f"LogLikeLihood is {logLikeLi[r]}")
      else:
        print("Min change does not occur thus converging and loop break is applied")
        break;

    # M-step
    sigma_eps2 , sigma_seas2 = compute_theta(ks)
    sigma_eps_Arr.append(np.sqrt(sigma_eps2))
    sigma_seas_Arr.append(np.sqrt(sigma_seas2))
    Q = np.array([[sigma_trend**2, 0.], [0., sigma_seas2]])

```

    Number of iteration is : 0
    Number of iteration is : 1
    LogLikeLihood is -1866.8309059076453
    Number of iteration is : 2
    LogLikeLihood is -1737.1008966208346
    Number of iteration is : 3
    LogLikeLihood is -1705.5712890569646
    Number of iteration is : 4
    LogLikeLihood is -1693.6708986444391
    Number of iteration is : 5
    LogLikeLihood is -1686.6694873942026
    Number of iteration is : 6
    LogLikeLihood is -1681.1852264737176
    Number of iteration is : 7
    LogLikeLihood is -1676.4161295286615
    Number of iteration is : 8
    LogLikeLihood is -1672.1401988475586
    Number of iteration is : 9
    LogLikeLihood is -1668.2648409078197
    Number of iteration is : 10
    LogLikeLihood is -1664.7316664581126
    Number of iteration is : 11
    LogLikeLihood is -1661.4961324673172
    Number of iteration is : 12
    LogLikeLihood is -1658.5220925431386
    Number of iteration is : 13
    LogLikeLihood is -1655.7795117158917
    Number of iteration is : 14
    LogLikeLihood is -1653.2430877948818
    Number of iteration is : 15
    LogLikeLihood is -1650.8912800822036
    Number of iteration is : 16
    LogLikeLihood is -1648.7055856072982
    Number of iteration is : 17
    LogLikeLihood is -1646.669985593203
    Number of iteration is : 18
    LogLikeLihood is -1644.7705149168219
    Number of iteration is : 19
    LogLikeLihood is -1642.9949227213378
    Number of iteration is : 20
    LogLikeLihood is -1641.3324017506982
    Number of iteration is : 21
    LogLikeLihood is -1639.7733702059081
    Number of iteration is : 22
    LogLikeLihood is -1638.3092942185172
    Number of iteration is : 23
    LogLikeLihood is -1636.932542070822
    Number of iteration is : 24
    LogLikeLihood is -1635.6362634759316
    Number of iteration is : 25
    LogLikeLihood is -1634.4142888217982
    Number of iteration is : 26
    LogLikeLihood is -1633.2610444590969
    Number of iteration is : 27
    LogLikeLihood is -1632.1714809890707
    Number of iteration is : 28
    LogLikeLihood is -1631.141012167675
    Number of iteration is : 29
    LogLikeLihood is -1630.1654625446358
    Number of iteration is : 30
    LogLikeLihood is -1629.2410223401075
    Number of iteration is : 31
    LogLikeLihood is -1628.3642083589943
    Number of iteration is : 32
    LogLikeLihood is -1627.5318299741548
    Number of iteration is : 33
    LogLikeLihood is -1626.740959390622
    Number of iteration is : 34
    LogLikeLihood is -1625.988905546466
    Number of iteration is : 35
    LogLikeLihood is -1625.2731911189207
    Number of iteration is : 36
    LogLikeLihood is -1624.5915321959878
    Number of iteration is : 37
    LogLikeLihood is -1623.9418202463671
    Number of iteration is : 38
    LogLikeLihood is -1623.3221060799156
    Number of iteration is : 39
    LogLikeLihood is -1622.730585539519
    Number of iteration is : 40
    LogLikeLihood is -1622.1655867042268
    Number of iteration is : 41
    LogLikeLihood is -1621.6255584166215
    Number of iteration is : 42
    LogLikeLihood is -1621.1090599742745
    Number of iteration is : 43
    LogLikeLihood is -1620.6147518477114
    Number of iteration is : 44
    LogLikeLihood is -1620.141387306021
    Number of iteration is : 45
    LogLikeLihood is -1619.6878048476037
    Number of iteration is : 46
    LogLikeLihood is -1619.2529213464259
    Number of iteration is : 47
    LogLikeLihood is -1618.8357258361204
    Number of iteration is : 48
    LogLikeLihood is -1618.4352738634138
    Number of iteration is : 49
    LogLikeLihood is -1618.0506823515466
    Number of iteration is : 50
    LogLikeLihood is -1617.6811249204864
    Number of iteration is : 51
    LogLikeLihood is -1617.3258276180336
    Number of iteration is : 52
    LogLikeLihood is -1616.9840650203664
    Number of iteration is : 53
    LogLikeLihood is -1616.6551566657492
    Number of iteration is : 54
    LogLikeLihood is -1616.3384637891636
    Number of iteration is : 55
    LogLikeLihood is -1616.033386328767
    Number of iteration is : 56
    LogLikeLihood is -1615.7393601789618
    Number of iteration is : 57
    LogLikeLihood is -1615.4558546665437
    Number of iteration is : 58
    LogLikeLihood is -1615.1823702299414
    Number of iteration is : 59
    LogLikeLihood is -1614.9184362827227
    Number of iteration is : 60
    LogLikeLihood is -1614.66360924502
    Number of iteration is : 61
    LogLikeLihood is -1614.417470727909
    Number of iteration is : 62
    LogLikeLihood is -1614.179625857425
    Number of iteration is : 63
    LogLikeLihood is -1613.9497017259837
    Number of iteration is : 64
    LogLikeLihood is -1613.7273459603666
    Number of iteration is : 65
    LogLikeLihood is -1613.5122253964896
    Number of iteration is : 66
    LogLikeLihood is -1613.3040248517893
    Number of iteration is : 67
    LogLikeLihood is -1613.1024459873001
    Number of iteration is : 68
    LogLikeLihood is -1612.9072062520027
    Number of iteration is : 69
    LogLikeLihood is -1612.718037902721
    Number of iteration is : 70
    LogLikeLihood is -1612.534687093556
    Number of iteration is : 71
    LogLikeLihood is -1612.3569130292942
    Number of iteration is : 72
    LogLikeLihood is -1612.1844871776377
    Number of iteration is : 73
    LogLikeLihood is -1612.0171925358984
    Number of iteration is : 74
    LogLikeLihood is -1611.8548229477094
    Number of iteration is : 75
    LogLikeLihood is -1611.6971824659572
    Number of iteration is : 76
    LogLikeLihood is -1611.5440847585412
    Number of iteration is : 77
    LogLikeLihood is -1611.3953525536604
    Number of iteration is : 78
    LogLikeLihood is -1611.2508171216207
    Number of iteration is : 79
    LogLikeLihood is -1611.1103177905493
    Number of iteration is : 80
    LogLikeLihood is -1610.9737014934562
    Number of iteration is : 81
    LogLikeLihood is -1610.8408223444826
    Number of iteration is : 82
    LogLikeLihood is -1610.7115412420712
    Number of iteration is : 83
    LogLikeLihood is -1610.5857254972204
    Number of iteration is : 84
    LogLikeLihood is -1610.463248485114
    Number of iteration is : 85
    LogLikeLihood is -1610.3439893182465
    Number of iteration is : 86
    LogLikeLihood is -1610.2278325399295
    Number of iteration is : 87
    LogLikeLihood is -1610.1146678363139
    Number of iteration is : 88
    LogLikeLihood is -1610.0043897659975
    Number of iteration is : 89
    LogLikeLihood is -1609.896897505997
    Number of iteration is : 90
    LogLikeLihood is -1609.7920946126192
    Number of iteration is : 91
    LogLikeLihood is -1609.6898887968396
    Number of iteration is : 92
    LogLikeLihood is -1609.590191712637
    Number of iteration is : 93
    LogLikeLihood is -1609.4929187578161
    Number of iteration is : 94
    LogLikeLihood is -1609.3979888862357
    Number of iteration is : 95
    LogLikeLihood is -1609.3053244310327
    Number of iteration is : 96
    LogLikeLihood is -1609.2148509376677
    Number of iteration is : 97
    LogLikeLihood is -1609.1264970066754
    Number of iteration is : 98
    LogLikeLihood is -1609.0401941450857
    Number of iteration is : 99
    LogLikeLihood is -1608.9558766261744

```python
# for t in range(5):
#   print((ks.eta_hat[:,:,t] @  ks.eta_hat[:,:,t].T + ks.eta_cov[:,:,t]).shape)
# #ks.eta_cov.shape
```

```python
plt.plot(range(0,len(sigma_eps_Arr)) , (sigma_eps_Arr) , label = "sigma_eps")
plt.plot(range(0,len(sigma_seas_Arr)) , (sigma_seas_Arr) , label = "sigma_eta")
plt.xlabel("Index")
plt.ylabel("Q")
plt.title("Traces of the parameter estimates")
plt.legend()
plt.show()
```

```python
#likelihood plot

plt.plot(range(0,len(logLikeLi)) , logLikeLi , label = "Loglikelihood")
plt.xlabel("Iteration")
plt.ylabel("Log Likelihood")
plt.title("Log Likelihood Vs Number of Iteration")
plt.legend()
plt.show()
```

## 2.4 Further analysing the data
We will now fix the model according to the final output from the EM algorithm and further analyse the data using this model.

**Q13:** Rerun the Kalman filter to compute a _long range prediction for the validation data points,_ analogously to **Q8** (you can copy-paste code from that question). That is, generate a plot which shows:
1. The data $y_{1:n+m}$,
2. The one-step predictions $\hat y_{t|t-1} \pm 1 $ standard deviation for the training data, i.e., $t = 1,...,n$,
3. The long-range predictions $\hat y_{t|n} \pm 1 $ standard deviation for the validation data, i.e., $t= n+1,...,n+m$,
4. A vertical line indicating the switch between training and validation data, using, e.g., `plt.axvline(x=u[n])`.

Furthermore, compute the training data log-likelihood $\log p(y_{1:n})$ using the estimated model (cf. **Q9**).

```python
#create model with new parameters
#since last values will be used to direectly creating model with last Var values
model = LGSS(T, R , Q, Z , sigma_eps2 , a1 , P1)
n = 800
m = ndata-n
yTrain = y.copy()
#implemented such that for we can use it for long range prediction(i.e. adding Nan for future values)
yTrain[801:] = np.nan
kf = kalman_filter(yTrain , model)

sd1 = kf.y_pred - np.sqrt(kf.F_pred)
sd2 = kf.y_pred + np.sqrt(kf.F_pred)

```

```python
#Data Plot
plt.plot(u,y ,label = "Original Data")
plt.plot(u , kf.y_pred , label = "Kalman Pred with EM Parameter")
#plt.scatter(u , sd1 , label = "Standard Deviation" , color = "green")
#plt.scatter(u , sd2 , label = "Standard Deviation" , color = "green")
plt.plot(u , sd1 , label = "Standard Deviation" , color = "green" , ls = "-.")
plt.plot(u , sd2 , label = "Standard Deviation" , color = "green" , ls = "-.")
plt.axvline(x=u[n])
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("Kalman filter for long range prediction")
plt.legend()
plt.show()
```

```python
#Traning Data Log Likelihood
logLikLi = (-1/2)*(np.sum(np.log(np.abs(kf.F_pred[:801,])) + (yTrain[:801,] - kf.y_pred[:801,])**2 / kf.F_pred[:801,]))
print(f"Log-Likelihood for Training Data is {logLikLi}")
```

    Log-Likelihood for Training Data is -1608.8734813570054

Note that we can view the model for the data $y_t$ as being comprised of an underlying "signal", $s_t = \mu_t + \gamma_t$ plus observation noise $\varepsilon_t$

\begin{align}
    y_t = s_t + \varepsilon_t
\end{align}

We can obtain refined, _smoothed,_ estimates of this signal by conditioning on all the training data $y_{1:n}$.

**Q14:** Run a Kalman smoother to compute smoothed estimates of the signal, $\mathbb{E}[s_t | y_{1:n}]$, conditionally on all the _training data_. Then, similarly to above, plot the following:
1. The data $y_{1:n+m}$,
2. The smoothed estimates $\mathbb{E}[s_t | y_{1:n}] \pm 1 $ standard deviation for the training data, i.e., $t = 1,...,n$,
3. The predictions $\mathbb{E}[s_t | y_{1:n}] \pm 1 $ standard deviation for the validation data, i.e., $t = n+1,...,n+m$,
4. A vertical line indicating the switch between training and validation data, using, e.g., `plt.axvline(x=u[n])`.

_Hint:_ Express $s_t$ in terms of $\alpha_t$. Based on this expression, compute the smoothed mean and variance of $s_t$ based on the smoothed mean and covariance of $\alpha_t$.

```python
ksSE = kalman_smoother(yTrain , model , kf)
```

```python
#since s_t = z * alpha_sum

n = len(yTrain)
s_tMean = []
s_tVar = []
for t in range(0,n):
  s_tMean.append(Z @ ksSE.alpha_sm[:,:,t])
  s_tVar.append(((Z @ ksSE.V[:,:,t]) @ np.transpose(Z)) + sigma_eps)

from pandas.core.common import flatten
s_tMean = list(flatten(s_tMean))
s_tVar = list(flatten(s_tVar))

sd1KS = s_tMean - np.sqrt(s_tVar)
sd2KS = s_tMean + np.sqrt(s_tVar)
```

```python
#Data Plot
from pandas.core.common import flatten
s_tMean = list(flatten(s_tMean))
s_tVar = list(flatten(s_tVar))

sd1KS = s_tMean - np.sqrt(s_tVar)
sd2KS = s_tMean + np.sqrt(s_tVar)

plt.plot(u,y ,label = "Original Data")
plt.plot(u , s_tMean , label = "Kalman Smoothing Mean")
plt.scatter(u , sd1KS , label = "Standard Deviation" , color = "green" , ls = ":")
plt.scatter(u , sd2KS , label = "Standard Deviation" , color = "green" , ls = ":")
plt.axvline(x=u[801])
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("Kalman smoother for computed smoothed estimates")
plt.legend()
plt.show()
```

**Q15:** Explain, using a few sentences, the qualitative differences (or similarities) between the Kalman filter predictions plotted in **Q13** and the smoothed signal estimates plotted in **Q14** for,
1. Training data points, $t \leq n$
2. Validation data points, $t > n$

**A:**
To predict Kalman filter uses data till time t while in case of smoother we use all available (i.e. complete data).

**Training Data Points**

It can be seen that for Kalman filter prediction are changing rapidly since it only uses t-1 data for prediction while Kalman Smoother prediction is smoother since it is using future values to correct error in filtering prediction.

**Validation data points**

Now while predicting validation data points, all future values are Nan so both predictions are similar as both access to the same data points only which is past values.

We can shed additional light on the properties of the process under study by further decomposing the signal into its trend and seasonal components.

**Q16:** Using the results of the state smoother, compute and plot the _smoothed estimates_ of the two signal components, i.e.:

1. Trend: $\hat \mu_{t|n} = \mathbb{E}[\mu_t | y_{1:n}]$ for $t = 1,\dots,n$
2. Seasonal: $\hat \gamma_{t|n} = \mathbb{E}[\gamma_t | y_{1:n}]$ for $t = 1,\dots,n$

_(You don't have to include confidence intervals here if don't want to, for brevity.)_

```python
#ksSE.alpha_sm.shape
#ksSE.alpha_sm[0:2,0,:].sum(axis = 0).shape

plt.scatter(u , ksSE.alpha_sm[0:2,0,:].sum(axis = 0) , label = "Tread" )
plt.scatter(u , ksSE.alpha_sm[2:37,0,:].sum(axis = 0) , label = "Seasonal" )
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("smoothed estimates signal components")
plt.legend()
plt.show()
```

## 2.5 Missing data
We conclude this section by illustrating one of the key merits of the state space approach to time series analysis, namely the simplicity of handling missing data. To this end we will assume that a chunk of observations in the middle of the training data is missing.

**Q17:** Let the values $y_{t}$ for $ 300 < t \leq 400$ be missing (set to `np.nan`). Modify the data and rerun the Kalman filter and smoother. Plot,
1. The Kalman filter predictions, analogously to **Q8**
2. The Kalman smoother predictions, analogously to **Q13**

Comment on the qualitative differences between the filter and smoother estimates and explain what you see (in a couple of sentences).

```python
yNew = y.copy()
yNew[300:400] = np.nan
```

```python
#Filter
kfNew = kalman_filter(yNew , model)

#Smoother
ksNew = kalman_smoother(yNew , model , kfNew)

n = len(yNew)
s_tMeanNew = []

for t in range(0,n):
  s_tMeanNew.append(Z @ ksSE.alpha_sm[:,:,t])

from pandas.core.common import flatten
s_tMeanNew = list(flatten(s_tMeanNew))
```

```python
plt.plot(u,y ,label = "Original Data")
plt.plot(u , kfNew.y_pred , label = "Kalman Filter")
plt.plot(u , s_tMeanNew , label = "Kalman Smoother")
plt.axvline(x=u[300])
plt.axvline(x=u[400])
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("Kalman Filter and Kalman Smoother with missing data")
plt.legend()
plt.show()
```

```python
plt.plot(u[290:410],y[290:410] ,label = "Original Data")
plt.plot(u[290:410] , kfNew.y_pred[290:410] , label = "Kalman Filter")
plt.plot(u[290:410] , s_tMeanNew[290:410] , label = "Kalman Smoother")
plt.axvline(x=u[300] , color = "black")
plt.axvline(x=u[400] , color = "black")
plt.xlabel("Time")
plt.ylabel("GMSL")
plt.title("Kalman Filter and Kalman Smoother with missing data with subsection view")
plt.legend()
plt.show()
```

It has had been seen while predicting missing values, Filter predictions are rapidly changing which can predict changing amplitude, but after a point, it is not able to follow data properly.

Now smoother can follow trend and seasonality in data comparatively better when we have access better amount of past and future datapoints.

## Appendix A. Why didn't we learn the trend noise variance as well?
In the assignment above we have fixed $\sigma_\mu$ to a small value. Conceptually it would have been straightforward to learn also this parameter with the EM algorithm. However, unfortunately, the maximum likelihood estimate of $\sigma_\mu$ often ends up being too large to result in accurate _long term predictions_. The reason for this issue is that the structural model

\begin{align}
    y_t = \mu_t + \gamma_t + \varepsilon_t
\end{align}

is not a perfect description of reality. As a consequence, when learning the parameters the mismatch between the model and the data is compensated for by increasing the noise variances. This results in a trend component which does not only capture the long term trends of the data, but also seemingly random variations due to a model misspecification, possibly resulting in poor _long range predictions_.

Kitagawa (Introduction to Time Series Modeling, CRC Press, 2010, Section 12.3) discusses this issue and proposes two solutions. The first is a simple and pragmatic one: simply fix $\sigma_\mu^2$ to a value smaller than the maximum likelihood estimate. This is the approach we have taken in this assignment. The issue is of course that in practice it is hard to know what value to pick, which boild down to manual trial and error (or, if you are lucky, the designer of the lab assignment will tell you which value to use!).

The second, more principled, solution proposed by Kitagawa is to augment the model with a stationary AR component as well. That is, we model

\begin{align}
    y_t = \mu_t + \gamma_t + \nu_t + \varepsilon_t
\end{align}

where $\nu_t \sim$ AR$(p)$. By doing so, the stationary AR component can compensate for the discrepancies between the original structural model and the "true data generating process". It is straightforward to include this new component in the state space representation (how?) and to run the Kalman filter and smoother on the resulting model. Indeed, this is one of the beauties with working with the state space representation of time series data! However, the M-step of the EM algorithm becomes a bit more involved if we want to use the method to estimate also the AR coefficients of the $\nu$-component, which is beyond the scope of this lab assignment.


It was proposed by Christian Goldbach that every odd composite number can be written as the sum of a prime and twice a square.

$$
9 = 7 + 2 \cdot 1^2 \\
15 = 7 + 2 \cdot 2^2 \\
21 = 3 + 2 \cdot 3^2 \\
25 = 7 + 2 \cdot 3^2 \\
27 = 19 + 2 \cdot 2^2 \\
33 = 31 + 2 \cdot 1^2
$$

It turns out that the conjecture was false.

What is the smallest odd composite that cannot be written as the sum of a prime and twice a square?

### Version 1: Brute force

Formally stated, Goldbach's Other Conjecture says that all odd composite numbers $n$ can be expressed as

$$
n = 2k^2 + p
$$

for some integer $k$ and prime number $p$. Let

$$
S_n = \{ n - 2k^2 : k = 1, 2, \dotsc, \lfloor \sqrt{\frac{n}{2}} \rfloor \}
$$

If any element $n - 2k^2$ of $S_n$ is prime, then we call $k$ a *witness* to Goldbach's Other Conjecture for $n$. We are asked to find the smallest $n$ that has no witness, i.e. such that *all* elements of $S_n$ are composite.

Let $P_n$ be the set of all prime numbers stricly smaller than $n$, then our algorithm searches for the smallest $n$ such that $P_n \cap S_n = \emptyset$, providing a counterexample to Goldbach's Other Conjecture.

<!-- TEASER_END -->

```python
from IPython.display import Math, display

from collections import defaultdict
from itertools import count
```

We modify and augment a very space-efficient implementation (due to David Eppstein, UC Irvine) of the Sieve of Erastothenes to generate *composite* numbers and keep track of all prime numbers below it. In the outermost-loop below, `primes` will always be the list of all prime number strictly below $n$, i.e. it is equivalent to $P_n$. Note that if $n$ is odd and composite, then the largest prime below it is no greater than $n-2$, since $n-1$ is even. Since $\max S_n = n - 2$ searching through $P_n$ is sufficient (i.e. if $n-2$ is prime, then $n-2 \in P_n$.) The loop terminates when we encounter an $n$ with no witnesses.

```python
factors = defaultdict(list)
witness = {}
primes = set()
for n in count(2):
    if factors[n]:
        # n is composite
        for m in factors.pop(n):
            factors[n+m].append(m)
        if n % 2:
            # n is odd and composite
            for k in range(1, int((n/2)**.5)+1):
                p = n - 2*k**2
                if p in primes:
                    # TODO: `in` is O(len(primes))
                    # could optimize by using set
                    witness[n] = k
                    break
            if not n in witness:
                break
    else:
        factors[n*n].append(n)
        primes.add(n)
```

```python
n
```

    5777

The answer is 5777.

Note that not only is this implementation space-efficient, it is also very time efficient, since we incrementally build our list of primes, composites and witnesses incrementally from bottom-up. If we hadn't augmented the implementation of the prime sieve, we would have had to use the prime sieve to obtain all odd composites, and perform a primality test on all elements of $S_n$, which would (usually) require a prime factorization algorithm, which in turn (usually) requires a prime sieve, not to mention the fact that there would be many overlapping subproblems, i.e. many $n < m$ such that $S_n \cap S_m \ne \emptyset$, so we'd have to use dynamic programming, by, for example memoizing the primality testing function or some other optimization - just a whole mess of redundancies and inefficiencies that we happily avoided with this method :)

Let's list out the witnesses to the first 100 numbers.

```python
lines = [r'{0} &= {1} + 2 \cdot {2}^2 \\'.format(n, n - 2*witness[n]**2, witness[n]) for n in sorted(witness)]
```

```python
Math(r"""
     \begin{{align}}
     {body}
     \end{{align}}
     """.format(body='\n'.join(lines[:100])))
```

$$
     \begin{align}
     9 &= 7 + 2 \cdot 1^2 \\
15 &= 13 + 2 \cdot 1^2 \\
21 &= 19 + 2 \cdot 1^2 \\
25 &= 23 + 2 \cdot 1^2 \\
27 &= 19 + 2 \cdot 2^2 \\
33 &= 31 + 2 \cdot 1^2 \\
35 &= 17 + 2 \cdot 3^2 \\
39 &= 37 + 2 \cdot 1^2 \\
45 &= 43 + 2 \cdot 1^2 \\
49 &= 47 + 2 \cdot 1^2 \\
51 &= 43 + 2 \cdot 2^2 \\
55 &= 53 + 2 \cdot 1^2 \\
57 &= 7 + 2 \cdot 5^2 \\
63 &= 61 + 2 \cdot 1^2 \\
65 &= 47 + 2 \cdot 3^2 \\
69 &= 67 + 2 \cdot 1^2 \\
75 &= 73 + 2 \cdot 1^2 \\
77 &= 59 + 2 \cdot 3^2 \\
81 &= 79 + 2 \cdot 1^2 \\
85 &= 83 + 2 \cdot 1^2 \\
87 &= 79 + 2 \cdot 2^2 \\
91 &= 89 + 2 \cdot 1^2 \\
93 &= 61 + 2 \cdot 4^2 \\
95 &= 23 + 2 \cdot 6^2 \\
99 &= 97 + 2 \cdot 1^2 \\
105 &= 103 + 2 \cdot 1^2 \\
111 &= 109 + 2 \cdot 1^2 \\
115 &= 113 + 2 \cdot 1^2 \\
117 &= 109 + 2 \cdot 2^2 \\
119 &= 101 + 2 \cdot 3^2 \\
121 &= 113 + 2 \cdot 2^2 \\
123 &= 73 + 2 \cdot 5^2 \\
125 &= 107 + 2 \cdot 3^2 \\
129 &= 127 + 2 \cdot 1^2 \\
133 &= 131 + 2 \cdot 1^2 \\
135 &= 127 + 2 \cdot 2^2 \\
141 &= 139 + 2 \cdot 1^2 \\
143 &= 71 + 2 \cdot 6^2 \\
145 &= 137 + 2 \cdot 2^2 \\
147 &= 139 + 2 \cdot 2^2 \\
153 &= 151 + 2 \cdot 1^2 \\
155 &= 137 + 2 \cdot 3^2 \\
159 &= 157 + 2 \cdot 1^2 \\
161 &= 89 + 2 \cdot 6^2 \\
165 &= 163 + 2 \cdot 1^2 \\
169 &= 167 + 2 \cdot 1^2 \\
171 &= 163 + 2 \cdot 2^2 \\
175 &= 173 + 2 \cdot 1^2 \\
177 &= 127 + 2 \cdot 5^2 \\
183 &= 181 + 2 \cdot 1^2 \\
185 &= 167 + 2 \cdot 3^2 \\
187 &= 179 + 2 \cdot 2^2 \\
189 &= 181 + 2 \cdot 2^2 \\
195 &= 193 + 2 \cdot 1^2 \\
201 &= 199 + 2 \cdot 1^2 \\
203 &= 131 + 2 \cdot 6^2 \\
205 &= 197 + 2 \cdot 2^2 \\
207 &= 199 + 2 \cdot 2^2 \\
209 &= 191 + 2 \cdot 3^2 \\
213 &= 211 + 2 \cdot 1^2 \\
215 &= 197 + 2 \cdot 3^2 \\
217 &= 199 + 2 \cdot 3^2 \\
219 &= 211 + 2 \cdot 2^2 \\
221 &= 149 + 2 \cdot 6^2 \\
225 &= 223 + 2 \cdot 1^2 \\
231 &= 229 + 2 \cdot 1^2 \\
235 &= 233 + 2 \cdot 1^2 \\
237 &= 229 + 2 \cdot 2^2 \\
243 &= 241 + 2 \cdot 1^2 \\
245 &= 227 + 2 \cdot 3^2 \\
247 &= 239 + 2 \cdot 2^2 \\
249 &= 241 + 2 \cdot 2^2 \\
253 &= 251 + 2 \cdot 1^2 \\
255 &= 223 + 2 \cdot 4^2 \\
259 &= 257 + 2 \cdot 1^2 \\
261 &= 229 + 2 \cdot 4^2 \\
265 &= 263 + 2 \cdot 1^2 \\
267 &= 139 + 2 \cdot 8^2 \\
273 &= 271 + 2 \cdot 1^2 \\
275 &= 257 + 2 \cdot 3^2 \\
279 &= 277 + 2 \cdot 1^2 \\
285 &= 283 + 2 \cdot 1^2 \\
287 &= 269 + 2 \cdot 3^2 \\
289 &= 281 + 2 \cdot 2^2 \\
291 &= 283 + 2 \cdot 2^2 \\
295 &= 293 + 2 \cdot 1^2 \\
297 &= 199 + 2 \cdot 7^2 \\
299 &= 281 + 2 \cdot 3^2 \\
301 &= 293 + 2 \cdot 2^2 \\
303 &= 271 + 2 \cdot 4^2 \\
305 &= 233 + 2 \cdot 6^2 \\
309 &= 307 + 2 \cdot 1^2 \\
315 &= 313 + 2 \cdot 1^2 \\
319 &= 317 + 2 \cdot 1^2 \\
321 &= 313 + 2 \cdot 2^2 \\
323 &= 251 + 2 \cdot 6^2 \\
325 &= 317 + 2 \cdot 2^2 \\
327 &= 277 + 2 \cdot 5^2 \\
329 &= 311 + 2 \cdot 3^2 \\
333 &= 331 + 2 \cdot 1^2 \\
     \end{align}
     $$


# Multiclass classification

The multiclass classification problem is a regression problem from an input $x \in {\cal X}$ to discrete labels $y\in {\cal Y}$, where ${\cal Y}$ is a discrete set of size $C$ bigger than two (for $C=2$ it is the more usual binary classification).

Labels are encoded in a one-hot fashion, that is if $C=4$ and $y=2$, we note $\bar{y} = [0,1,0,0]$.

The generative model for this problem consists of:

* $C$ latent functions $\mathbf{f} = [f_1,...,f_C]$ with an independent Gaussian Process prior
* a deterministic function that builds a discrete distribution $\pi(\mathbf{f}) = [\pi_1(f_1),...,\pi_C(f_C)]$ from the latents such that $\sum_c \pi_c(f_c) = 1$
* a discrete likelihood $p(y|\mathbf{f}) = Discrete(y;\pi(\mathbf{f})) = \prod_c \pi_c(f_c)^{\bar{y}_c}$

A typical example of $\pi$ is the softmax function:

\begin{equation}
\pi_c (f_c) \propto \exp( f_c)
\end{equation}

Another convenient one is the robust max:
\begin{equation}
\pi_c(\mathbf{f}) = \begin{cases} 1 - \epsilon, & \mbox{if } c = \arg \max_c f_c \\
 \epsilon /(C-1), & \mbox{ otherwise} \end{cases}
\end{equation}

```python
import numpy as np
import tensorflow as tf

import warnings

warnings.filterwarnings("ignore")  # ignore DeprecationWarnings from tensorflow

import matplotlib.pyplot as plt

%matplotlib inline

import gpflow

from gpflow.utilities import print_summary, set_trainable
from gpflow.ci_utils import ci_niter

from multiclass_classification import plot_posterior_predictions, colors

# reproducibility:
np.random.seed(0)
tf.random.set_seed(123)
```

## Sampling from the GP multiclass generative model

### Declaring model parameters and input

```python
# Number of functions and number of data points
C = 3
N = 100

# Lengthscale of the SquaredExponential kernel (isotropic -- change to `[0.1] * C` for ARD)
lengthscales = 0.1

# Jitter
jitter_eye = np.eye(N) * 1e-6

# Input
X = np.random.rand(N, 1)
```

### Sampling

```python
# SquaredExponential kernel matrix
kernel_se = gpflow.kernels.SquaredExponential(lengthscales=lengthscales)
K = kernel_se(X) + jitter_eye

# Latents prior sample
f = np.random.multivariate_normal(mean=np.zeros(N), cov=K, size=(C)).T

# Hard max observation
Y = np.argmax(f, 1).flatten().astype(int)

# One-hot encoding
Y_hot = np.zeros((N, C), dtype=bool)
Y_hot[np.arange(N), Y] = 1

data = (X, Y)
```

### Plotting

```python
plt.figure(figsize=(12, 6))
order = np.argsort(X.flatten())

for c in range(C):
    plt.plot(X[order], f[order, c], ".", color=colors[c], label=str(c))
    plt.plot(X[order], Y_hot[order, c], "-", color=colors[c])

plt.legend()
plt.xlabel("$X$")
plt.ylabel("Latent (dots) and one-hot labels (lines)")
plt.title("Sample from the joint $p(Y, \mathbf{f})$")
plt.grid()
plt.show()
```

## Inference

Inference here consists of computing the posterior distribution over the latent functions given the data $p(\mathbf{f}|Y, X)$.

You can use different inference methods. Here we perform variational inference.
For a treatment of the multiclass classification problem using MCMC sampling, see [Markov Chain Monte Carlo (MCMC)](../advanced/mcmc.ipynb).

### Approximate inference: Sparse Variational Gaussian Process

#### Declaring the SVGP model (see [GPs for big data](../advanced/gps_for_big_data.ipynb))

```python
# sum kernel: Matern32 + White
kernel = gpflow.kernels.Matern32() + gpflow.kernels.White(variance=0.01)

# Robustmax Multiclass Likelihood
invlink = gpflow.likelihoods.RobustMax(C)  # Robustmax inverse link function
likelihood = gpflow.likelihoods.MultiClass(3, invlink=invlink)  # Multiclass likelihood
Z = X[::5].copy()  # inducing inputs

m = gpflow.models.SVGP(
    kernel=kernel,
    likelihood=likelihood,
    inducing_variable=Z,
    num_latent_gps=C,
    whiten=True,
    q_diag=True,
)

# Only train the variational parameters
set_trainable(m.kernel.kernels[1].variance, False)
set_trainable(m.inducing_variable, False)
print_summary(m, fmt="notebook")
```

name                               	class    	transform  	prior  	trainable  	shape  	dtype  	value

SVGP.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	1.0

SVGP.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	1.0

SVGP.kernel.kernels[1].variance    	Parameter	Softplus   	       	False      	()     	float64	0.009999999999999998

SVGP.likelihood.invlink.epsilon    	Parameter	Sigmoid    	Beta   	False      	()     	float64	0.0010000000000000005

SVGP.inducing_variable.Z           	Parameter	Identity   	       	False      	(20, 1)	float64	[[0.5488135...

SVGP.q_mu                          	Parameter	Identity   	       	True       	(20, 3)	float64	[[0., 0., 0....

SVGP.q_sqrt                        	Parameter	Softplus   	       	True       	(20, 3)	float64	[[1., 1., 1....

#### Running inference

```python
opt = gpflow.optimizers.Scipy()

opt_logs = opt.minimize(
    m.training_loss_closure(data), m.trainable_variables, options=dict(maxiter=ci_niter(1000))
)
print_summary(m, fmt="notebook")
```

name                               	class    	transform  	prior  	trainable  	shape  	dtype  	value

SVGP.kernel.kernels[0].variance    	Parameter	Softplus   	       	True       	()     	float64	156886.68783887077

SVGP.kernel.kernels[0].lengthscales	Parameter	Softplus   	       	True       	()     	float64	0.17427842722718734

SVGP.kernel.kernels[1].variance    	Parameter	Softplus   	       	False      	()     	float64	0.009999999999999998

SVGP.likelihood.invlink.epsilon    	Parameter	Sigmoid    	Beta   	False      	()     	float64	0.0010000000000000005

SVGP.inducing_variable.Z           	Parameter	Identity   	       	False      	(20, 1)	float64	[[0.5488135...

SVGP.q_mu                          	Parameter	Identity   	       	True       	(20, 3)	float64	[[-2.32589301e-01, 6.14343482e-01, -3.81754181e-01...

SVGP.q_sqrt                        	Parameter	Softplus   	       	True       	(20, 3)	float64	[[0.08266212, 0.07959608, 0.12088343...

```python
plot_posterior_predictions(m, X, Y)
```


Make sure you remove `raise NotImplementedError()` and fill in any place that says `# YOUR CODE HERE`, as well as your `NAME`, `ID`, and `LAB_SECTION` below:

```python
NAME = "FAHMID BIN KIBRIA"
ID = "19201063"
LAB_SECTION = "02"
```

---

# Part 1: Hermite Interpolation
---
Hermite Interpolation is an example of a variant of the interpolation problem, where the interpolant matches one or more **derivatives of $f$**  at each of the nodes, in addition to the function values.

## Importing the necessary libraries

```python
import numpy as np
import matplotlib.pyplot as plt
from itertools import combinations
from numpy.polynomial import Polynomial
```

## Creating the components for Hermite interpolation

For the case of Hermite Interpolation, we look for a polynomial that matches both $f'(x_i)$ and $f(x_i)$ at the nodes $x_i = x_0,\dots,x_n$. Say you have $n+1$ data points, $(x_0, y_0), (x_1, y_1), x_2, y_2), \dots, (x_n, y_n)$ and you happen to know the first-order derivative at all of these points, namely, $(x_0, y_0 ^\prime ), (x_1, y_1 ^\prime ), x_2, y_2 ^\prime ), \dots ,(x_n, y_n ^\prime )$. According to hermite interpolation, since there are $2n + 2$ conditions; $n+1$ for $f(x_i)$ plus $n+1$ for $f'(x_i)$; you can fit a polynomial of order $2n+1$.

General form of a $2n+1$ degree Hermite polynomial:

$$p_{2n+1} = \sum_{k=0}^{n} \left(f(x_k)h_k(x) + f'(x_k)\hat{h}_k(x)\right), \tag{1}$$

where $h_k$ and $\hat{h}_k$ are defined using Lagrange basis functions by the following equations:

$$h_k(x) = (1-2(x-x_k)l^\prime_k(x_k))l^2_k(x_k), \tag{2}$$

and

$$\hat{h}_k(x) = (x-x_k)l^2_k(x_k), \tag{3}$$

where the Lagrange basis function being:

$$l_k(x) = \prod_{j=0, j\neq k}^{n} \frac{x-x_j}{x_k-x_j}. \tag{4}$$

**Note** that, we can rewrite Equation $(2)$ in this way,

\begin{align}
h_k(x) &= \left(1-2(x-x_k)l^\prime_k(x_k) \right)l^2_k(x_k) \\
&= \left(1 - 2xl^\prime_k(x_k) + 2x_kl^\prime_k(x_k) \right)l^2_k(x_k) \\
&= \left(1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x \right) l^2_k(x_k) \tag{5}
\end{align}
Replacing $l^\prime_k(x_k)$ with $m$, we get:
$$h_k(x) = (1 - 2xm + 2x_km)l^2_k(x_k). \tag{6}$$

# Tasks:

* The functions: `l(k, x)`, `h(k, x)` and `h_hat(k, x)` calculate the corresponding $l_k$, $h_k$, and $\hat{h}_k$, respectively.

* Function `l(k, x)` has already been defined for you. Your task is to complete the `h(k, x)`, `h_hat(k, x)`, and `hermit(x, y, y_prime)` functions.

* Later we will draw some plots to check if the code is working.

---

### Part 1: Calculate $l_k$
This function uses the following equation to calculate $l_k(x)$ and returns a polynomial:

$$l_k(x) = \prod_{j=0, j\neq k}^{n} \frac{x-x_j}{x_k-x_j}.$$

```python
# Already written for you.

def l(k, x):
    n = len(x)
    assert (k < len(x))

    x_k = x[k]
    x_copy = np.delete(x, k)

    denominator = np.prod(x_copy - x_k)

    coeff = []

    for i in range(n):
        coeff.append(sum([np.prod(x) for x in combinations(x_copy, i)]) * (-1)**(i) / denominator)

    coeff.reverse()

    return Polynomial(coeff)
```

### Part 2: Calculate $h_k$
This function calculates $h_k(x)$ using the following equation:
$$h_k(x) = \left(1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x \right) l^2_k(x_k).$$

This equation is basically a multiplication of two polynomials.

First polynomial: $1 + 2x_kl^\prime_k(x_k) - 2l'_k(x_k)x$.

Second polynomial: $l^2_k(x_k)$.

The `coeff` variable should contain a python list of coefficient values for the **first** polynomial of the equation. These coefficient values are used to create a polynomial `p`.

```python
def h(k, x):
    # initialize with None. Replace with appropriate values/function calls
    # initialize with None. Replace with appropriate values/function calls
    l_k = None
    l_k_sqr = None
    l_k_prime = None
    coeff = None
    p = None

    # --------------------------------------------
    # YOUR CODE HERE
    l_k = l(k,x)
    l_k_sqr = l_k**2
    l_k_prime = l_k.deriv()
    coeff = [ 1 + 2*x[k]*l_k_prime(x[k]) , -2*l_k_prime(x[k])]
    p = Polynomial(coeff)
    # --------------------------------------------

    return p * l_k_sqr
```

```python
# Test case for the h(k, x) function
x = [3, 5, 7, 9]
k = 2
h_test = h(k, [3, 5, 7, 9])
h_result = Polynomial([-2.5, 0.5]) * (l(k, x) ** 2)
```

### Part 3: Calculate $\hat{h}_k$
This function calculates $\hat{h}_k(x)$ using the following equation:

$$\hat{h}_k(x) = (x-x_k)l^2_k(x_k).$$

This equation is also a multiplication of two polynomials.

First polynomial: $x-x_k$.

Second polynomial:  $l^2_k(x_k)$.

The `coeff` variable should contain a python list of coefficient values for the **first** polynomial of the equation. These coefficient values are used to create a polynomial `p`.

```python
def h_hat(k, x):
    # Initialize with none
    l_k = None
    l_k_sqr = None
    coeff = None
    p = None

    # --------------------------------------------
    # YOUR CODE HERE
    l_k = l(k,x)
    l_k_sqr = l_k**2
    coeff = [-x[k] , 1]
    p = Polynomial(coeff)

    # --------------------------------------------

    return p * l_k_sqr
```

```python
# Test case for the h(k, x) function
x = [3, 5, 7, 9]
k = 2
h_test = h_hat(k, [3, 5, 7, 9])
h_result = Polynomial([-7, 1]) * (l(k, x) ** 2)
```

### Part 4: The Hermite Polynomial
This function uses the following equation:

$$p_{2n+1} = \sum_{k=0}^{n} \left(f(x_k)h_k(x) + f'(x_k)\hat{h}_k(x)\right).$$

The polynomial denoted by the equation is calculated by the variable `f`.

```python
def hermit(x, y, y_prime):
    assert len(x) == len(y)
    assert len(y) == len(y_prime)

    f = 0
    # --------------------------------------------
    # YOUR CODE HERE
    for i in range(len(y)):
      f += y[i]*h(i,x) + y_prime[i]*h_hat(i,x)

    print(f)
    # --------------------------------------------
    return f
```

## Testing our methods by plotting graphs.

**Note:**

* For each of the 5 plots, there will be 2 curves plotted: one being the original function, and the other being the interpolated curve.

* The original functions are displayed in orange color, while the hermite interpolated curves are in blue.

* `x`, `y`, and `y_prime` contain $x_i$, $f(x_i)$, and $f'(x_i)$ of the given nodes of the original function $f$.

Upon calling the `hermit()` function, it returns a polynomial `f`. For example, for plot 1, it is called `f3`.

In general, a polynomial may look like the following: $f = 1 + 2x + 3x^2$. Next, we pass in a number of $x$ values to the polynomial by calling the `.linspace()` function on the polynomial object using `f.linspace()`. This function outputs a tuple, which is stored in a variable called `data`. First element of `data` contains a 1D numpy array of $x_i$ values generated by `linspace()`, and the second element of `data` contains a 1D numpy array of the corresponding $y_i$ values outputted by our example polynomial:
$f = 1 + 2x + 3x^2$.

Using `test_x`, we generate a range of $x_i$ values to plot the original function, and `test_y` contains the corresponding $y_i$ values of the original function. For the first plot, our original function is the *sine curve*.

For all the plots:

`plt.plot(test_x, test_y)` plots the original function.

`plt.plot(data[0], data[1])` plots the interpolated polynomial.

```python
pi      = np.pi
x       = np.array([0.0, pi/2.0,  pi, 3.0*pi/2.0])
y       = np.array([0.0,    1.0, 0.0,       -1.0])
y_prime = np.array([1.0,    0.0, 1.0,        0.0])
```

**Plot 1:** trying to interpolate a sine curve (`np.sin()`) using first 2 nodes in `x` and `y`, and their corresponding derivative in `y_prime`.

```python
n      = 1
f3     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f3.linspace(n=50, domain=[-3, 3])
test_x = np.linspace(-3, 3, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()
```

**Plot 2:** trying to interpolate a sine curve (`np.sin()`) using first 3 nodes in `x` and `y` and their corresponding derivative in `y_prime`.

```python
n      = 2
f5     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f5.linspace(n=50, domain=[-0.7, 3])
test_x = np.linspace(-2*pi, 2*pi, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(test_x, test_y) # 25-
plt.plot(data[0], data[1]) # 10-33
plt.show()

```

**Plot 3:** trying to interpolate a sine curve (`np.sin()`) using first 4 nodes in `x` and `y` and their corresponding derivative in `y_prime`.

```python
n      = 3
f7     = hermit(x[:(n+1)], y[:(n+1)], y_prime[:(n+1)])
data   = f7.linspace(n=50, domain=[-0.3, 3])
test_x = np.linspace(-2*pi, 2*pi, 50, endpoint=True)
test_y = np.sin(test_x)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()

```

**Plot 4:** trying to interpolate an exponential curve (`np.exp()`) using all nodes in `x` and `y` and their corresponding derivatives in `y_prime`.

```python
#defining new set of given node information: x, y and y'
x       = np.array([0.0, 1.0,          2.0       ])
y       = np.array([1.0, 2.71828183,  54.59815003])
y_prime = np.array([0.0, 5.43656366, 218.39260013])

f7      = hermit( x, y, y_prime)
data    = f7.linspace(n=50, domain=[-0.5, 2.2])
test_x  = np.linspace(-0.5, 2.2, 50, endpoint=True)
test_y  = np.exp(test_x**2)

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()

```

**Plot 5:** trying to interpolate $y = (x-3)^2 + 1$ using all nodes in `x` and `y` and their corresponding derivatives in `y_prime`.

For this plot you might be able to see only one curve due to the two curves overlapping. This means that our polynomial is accurately interpolating the original function.

```python
#defining new set of given node information: x, y and y'
x       = np.array([1.0, 3.0, 5.0])
y       = np.array([5.0, 1.0, 5.0])
y_prime = np.array([-4.0, 0.0, 4.0])

f7      = hermit( x, y, y_prime)
data    = f7.linspace(n=50, domain=[-10, 10])
test_x  = np.linspace(-10, 10, 50, endpoint=True)
test_y  = (test_x-3)**2 + 1

plt.plot(data[0], data[1])
plt.plot(test_x, test_y)
plt.show()
```

## Part 2: Polynomial Interpolation Using Newton's Divided Difference Form
---

### Newton's Divided Difference Form

Newton form of a $n$ degree polynomial:

$$p_n(x) = \sum_{k=0}^{n} a_kn_k(x),$$
where the basis is:
$$n_k(x) = \prod_{j=0}^{k-1}(x-x_j),$$
$$ n_0(x)=1,$$

and the coefficients are: $$a_k = f[x_0, x_1, ..., x_k],$$

where the notation $f[x_0, x_1,\dots,x_k]$ denotes the divided difference.

By expanding the Newton form, we get:

$$p(x) = f [x_0] + (x-x_0) f[x_0,x_1] + (x-x_0) (x-x_1) f[x_0,x_1,x_2] + \dots + (x-x_0) (x-x_1) \dots (x-x_{k-1}) f[x_0, x_1, \dots, x_k]$$

### Tasks:
1. Complete the `calc_div_diff(x,y)` function which takes input `x` and `y`, and calculates all the divided differences. You may use the lambda function `difference()` inside the `calc_div_diff(x,y)` function to calculate the divided differences.

2. Complete the `__call__()` function which takes an input `x`, and calculates `y` using all the difference coefficients. `x` can be a single value or a numpy. In this case, it is a numpy array.

`res` variable must contain all results (corresponding y for x).

```python
class Newtons_Divided_Differences:

    def __init__(self, differences):
        self.differences = differences

    def __call__(self, x):
        '''
        this function is for calculating y from given x using all the difference coefficients
        x can be a single value or a numpy
        the formula being used:
        f(x) = f [x0] + (x-x0) f[x0,x1] + (x-x0) (x-x1) f[x0,x1,x2] + . . . + (x-x0) (x-x1) . . . (x-xk-1) f[x0, x1, . . ., xk]

        work on this after implementing 'calc_div_diff'. Then you should have
        f[x0], f[x0,x1]. . . . . ., f[x0, x1, . . ., xk] stored in self.differences

        'res' variable must return all the results (corresponding y for x)
        '''

        res = np.zeros(len(x)) #Initialization to avoid runtime error. You can change this line if you wish

        #----------------------------------------------
        # YOUR CODE HERE
        for k in range(0, len(x)):
          res[k]=self.differences[0]
          for i in range(1, len(self.differences)):
            product = 1
            for j in range(i):
              product = product * (x[k] - data_x[j])
            res[k] = res[k] + product * self.differences[i]

        #----------------------------------------------

        return res

```

```python
# basic rule for calculating the difference, implanted in the lambda function.
# You may use it if you wish
difference = lambda y2, y1, x2, x1: (y2-y1)/(x2-x1)

def calc_div_diff(x,y):
    assert(len(x)==len(y))
    #write this function to calculate all the divided differences in the list 'b'
    b = []  #initializing
    #----------------------------------------------
    # YOUR CODE HERE
    table=np.array([[0.000000000]*len(y)]*len(y))
    for i in range(0,len(y)-1):
        table[i, 0]=y[i]

    for i in range(1, len(y)):
        for j in range(len(y) - i):
            table[j, i] = difference(table[j, i - 1], table[j + 1, i - 1], x[j], x[i + j])
    b = [0.0000000]*len(table)
    for i in range(0,len(table)):
        b[i]=table[0, i]
    #----------------------------------------------
    return b
```

```python
# Test case for the calc_div_diff(x,y) function.

data_x = [-3.,-2.,-1.,0.,1.,3.,4.]
data_y = [-60.,-80.,6.,1.,45.,30.,16.]

test = calc_div_diff(data_x, data_y)

assert len(test) == len(data_x)

```

### Plotting the polynomial
* `data_x` and `data_y` are the coordinates of the given nodes.

* `differences` is a list which contains the divided differences as each of its elements: $f[x_0], f[x_0,x_1], f[x_0,x_1,x_2], \dots$

* `obj` is an object of type `Newtons_Divided_Differences`. Creating the object runs the constructor of the class where the `difference` are stored in `self.differences`.

* `X` contains $x_i$ values through which we want to plot our polynomial.

* Calling the object using `obj(X)` executes the `__call__()` function of the class, which returns a numpy array containing the corresponding $y_i$ values, and storing them in variable `F`.

* Using `plt.plot(X,F)`, we plot the $(x_i, y_i)$ pairs of the polynomial.

```python
import numpy as np
import matplotlib.pyplot as plt

data_x = np.array([-3.,-2.,-1.,0.,1.,3.,4.])
data_y = np.array([-60.,-80.,6.,1.,45.,30.,16.])
differences = calc_div_diff(list(data_x), list(data_y))
p = Newtons_Divided_Differences(list(differences))
test_x = np.linspace(-3, 4, 50, endpoint=True)
test_y = p(test_x)

#generating 50 points from -3 to 4 in order to create a smooth line
plt.plot(test_x, test_y)
plt.plot(data_x, data_y, 'ro')
plt.show()

```

```python

```


```python
from sympy import *
init_session()
```

    IPython console for SymPy 1.9 (Python 3.9.2-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.9/

```python
(x**3 + y)**2
```

$\displaystyle \left(x^{3} + y\right)^{2}$

```python
solve(x**4 - 1,x)
```

$\displaystyle \left[ -1, \  1, \  - i, \  i\right]$

```python
f(x)
```

$\displaystyle f{\left(x \right)}$

```python
# 微分方程式
f(x).diff(x,x) + f(x)
```

$\displaystyle f{\left(x \right)} + \frac{d^{2}}{d x^{2}} f{\left(x \right)}$

```python
dsolve(f(x).diff(x,x) + f(x),f(x))
```

$\displaystyle f{\left(x \right)} = C_{1} \sin{\left(x \right)} + C_{2} \cos{\left(x \right)}$

```python

```


# Clase 5: SymPy

 __SymPy es una biblioteca de Python para matemática simbólica__. Apunta a convertirse en un sistema de algebra computacional (__CAS__) con todas sus prestaciones manteniendo el código tan simple como sea posible para manterlo comprensible y fácilmente extensible. SymPy está __escrito totalmente en Python y no requiere bibliotecas adicionales__. _Este proyecto comenzó en 2005, fue lanzado al público en 2007 y a él han contribuido durante estos años cientos de personas._

_ Otros CAS conocidos son Mathematica y Maple, sin embargo ambos son software privativo y de pago. [Aquí](https://github.com/sympy/sympy/wiki/SymPy-vs.-Maple) puedes encontrar una comparativa de SymPy con Maple. _

Hoy veremos cómo:

* Crear símbolos y expresiones.
* Manipular expresiones (simplificación, expansión...)
* Calcular derivadas e integrales.
* Límites y desarrollos en serie.
* Resolución de ecuaciones.
* Resolción de EDOs.
* Matrices

Sin embargo, SymPy no acaba aquí ni mucho menos...

## Documentación & SymPy Live Shell

```python
from IPython.display import HTML
HTML('')
```

## SymPy Gamma

```python
HTML('')
```

## Creación de símbolos

Lo primero, como siempre, es importar aquello que vayamos a necesitar:

```python
# Importación
from sympy import init_session
```

```python
init_session(use_latex='matplotlib')
```

    IPython console for SymPy 0.7.6.1 (Python 3.5.1-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://www.sympy.org

Lo primero que vemos es que el comando `init_session` ha llevado a cabo algunas acciones por nostros:

* Gracias a `use_latex=True` obtenemos la salida en $\LaTeX$.
* __Ha creado una serie de variables__ para que podamos ponernos a trabajar en el momento.

Nota:
En Python, no se declaran las variables, sin embargo, no puedes usar una hasta que no le hayas asignado un valor. Si ahora intentamos crear una variable `a` que sea `a = 2 * b`, veamos qué ocurre:

```python
# Intentamos usar un símbolo que no hemos creado
a = 2 * b
```

Como en `b` no había sido creada, Python no sabe qué es `b`.

Esto mismo nos ocurre con los símbolos de SymPy. __Antes de usar una variable, debo decir que es un símbolo y asignárselo:__

```python
# Creamos el símbolo a
a = symbols('a')
a
```

```python
# Número pi
(a + pi) ** 2
```

```python
# Unidad imaginaria
a + 2 * I
```

```python
# Número e
E
```

```python
# Vemos qué tipo de variable es a
type(a)
```

    sympy.core.symbol.Symbol

Ahora ya podría crear `b = 2 * a`:

```python
b = 2 * a
b
```

```python
type(b)
```

    sympy.core.mul.Mul

¿Qué está ocurriendo? Python detecta que a es una variable de tipo `Symbol` y al multiplicarla por `2` devuelve una variable de Sympy.

Como Python permite que el tipo de una variable cambie, __si ahora le asigno a `a` un valor float deja de ser un símbolo.__

```python
a = 2.26492
a
```

```python
type(a)
```

    float

---
__Las conclusiones son:__

* __Si quiero usar una variable como símbolo debo crearla previamente.__
* Las operaciones con símbolos devuelven símbolos.
* Si una varibale que almacenaba un símbolo recibe otra asignación, cambia de tipo.

---

__Las variables de tipo `Symbol` actúan como contenedores en los que no sabemos qué hay (un real, un complejo, una lista...)__. Hay que tener en cuenta que: __una cosa es el nombre de la variable y otra el símbolo con el que se representa__.

```python
#creación de símbolos
coef_traccion = symbols('c_T')
coef_traccion
```

Incluso puedo hacer cosas raras como:

```python
# Diferencia entre variable y símbolo
a = symbols('b')
a
```

Además, se pueden crear varos símbolos a la vez:

```python
x, y, z, t = symbols('x y z t')
```

y símbolos griegos:

```python
w = symbols('omega')
W = symbols('Omega')
w, W
```

_Fuente: Documentación oficial de SymPy_

__Por defecto, SymPy entiende que los símbolos son números complejos__. Esto puede producir resultados inesperados ante determinadas operaciones como, por ejemplo, lo logaritmos. __Podemos indicar que la variable es real, entera... en el momento de la creación__:

```python
# Creamos símbolos reales
x, y, z, t = symbols('x y z t', real=True)
```

```python
# Podemos ver las asunciones de un símbolo
x.assumptions0
```

    {'commutative': True,
     'complex': True,
     'hermitian': True,
     'imaginary': False,
     'real': True}

## Expresiones

Comencemos por crear una expresión como: $\cos(x)^2+\sin(x)^2$

```python
expr = cos(x)**2 + sin(x)**2
expr
```

### `simplify()`

Podemos pedirle que simplifique la expresión anterior:

```python
simplify(expr)
```

En este caso parece estar claro lo que quiere decir más simple, pero como en cualquier _CAS_ el comando `simplify` puede no devolvernos la expresión que nosotros queremos. Cuando esto ocurra necesitaremos usar otras instrucciones.

### `.subs()`

En algunas ocasiones necesitaremos sustituir una variable por otra, por otra expresión o por un valor.

```python
expr
```

```python
# Sustituimos x por y ** 2
expr.subs(x, y**2)
```

```python
# ¡Pero la expresión no cambia!
expr
```

```python
# Para que cambie
expr = expr.subs(x, y**2)
expr
```

Cambia el `sin(x)` por `exp(x)`

```python
expr.subs(sin(x), exp(x))
```

Particulariza la expresión $sin(x) + 3 x $ en $x = \pi$

```python
(sin(x) + 3 * x).subs(x, pi)
```

__Aunque si lo que queremos es obtener el valor numérico lo mejor es `.evalf()`__

```python
(sin(x) + 3 * x).subs(x, pi).evalf(25)
```

```python
#ver pi con 25 decimales
pi.evalf(25)
```

```python
#el mismo resultado se obtiene ocn la función N()
N(pi,25)
```

# Simplificación

SymPy ofrece numerosas funciones para __simplificar y manipular expresiones__. Entre otras, destacan:

* `expand()`
* `factor()`
* `collect()`
* `apart()`
* `cancel()`

Puedes consultar en la documentación de SymPy lo que hace cada una y algunos ejemplos. __Existen también funciones específicas de simplificación para funciones trigonométricas, potencias y logaritmos.__ Abre [esta documentación](http://docs.sympy.org/latest/tutorial/simplification.html) si lo necesitas.

##### ¡Te toca!

Pasaremos rápidamente por esta parte, para hacer cosas "más interesantes". Te proponemos algunos ejemplos para que te familiarices con el manejor de expresiones:

__Crea las expresiones de la izquierda y averigua qué función te hace obtener la de la derecha:__

expresión 1| expresión 2
:------:|:------:
$\left(x^{3} + 3 y + 2\right)^{2}$    |    $x^{6} + 6 x^{3} y + 4 x^{3} + 9 y^{2} + 12 y + 4$
$\frac{\left(3 x^{2} - 2 x + 1\right)}{\left(x - 1\right)^{2}} $ | $3 + \frac{4}{x - 1} + \frac{2}{\left(x - 1\right)^{2}}$
$x^{3} + 9 x^{2} + 27 x + 27$         |    $\left(x + 3\right)^{3}$
$\sin(x+2y)$                          |    $\left(2 \cos^{2}{\left (y \right )} - 1\right) \sin{\left (x \right )} + 2 \sin{\left (y \right )} \cos{\left (x \right )} \cos{\left (y \right )}$

```python
#1
expr1 = (x ** 3 + 3 * y + 2) ** 2
expr1
```

```python
expr1_exp = expr1.expand()
expr1_exp
```

```python
#2
expr2 = (3 * x ** 2 - 2 * x + 1) / (x - 1) ** 2
expr2
```

```python
expr2.apart()
```

```python
#3
expr3 = x ** 3 + 9 * x ** 2 + 27 * x + 27
expr3
```

```python
expr3.factor()
```

```python
#4
expr4 = sin(x + 2 * y)
expr4
```

```python
expand(expr4)
```

```python
expand_trig(expr4)
```

```python
expand(expr4, trig=True)
```

# Derivadas e integrales

Puedes derivar una expresion usando el método `.diff()` y la función `dif()`

```python
#creamos una expresión
expr = cos(x)

#obtenemos la derivada primera con funcion
diff(expr, x)
```

```python
#utilizando método
expr.diff(x)
```

__¿derivada tercera?__

```python
expr.diff(x, x, x)
```

```python
expr.diff(x, 3)
```

__¿varias variables?__

```python
expr_xy = y ** 3 * sin(x) ** 2 + x ** 2 * cos(y)
expr_xy
```

```python
diff(expr_xy, x, 2, y, 2)
```

__Queremos que la deje indicada__, usamos `Derivative()`

```python
Derivative(expr_xy, x, 2, y)
```

__¿Será capaz SymPy de aplicar la regla de la cadena?__

```python
# Creamos una función F
F = Function('F')
F(x)
```

```python
# Creamos una función G
G = Function('G')
G(x)
```

$$\frac{d}{d x} F{\left (G(x) \right )} $$

```python
# Derivamos la función compuesta F(G(x))
F(G(x)).diff(x)
```

    d        ⎛ d        ⎞│
    ──(G(x))⋅⎜───(F(ξ₁))⎟│
    dx       ⎝dξ₁       ⎠│ξ₁=G(x)

En un caso en el que conocemos las funciones:

```python
# definimos una f
f = 2 * y * exp(x)
f
```

```python
# definimos una g(f)
g = f **2 * cos(x) + f
g
```

```python
#la derivamos
diff(g,x)
```

##### Te toca integrar

__Si te digo que se integra usando el método `.integrate()` o la función `integrate()`__. ¿Te atreves a integrar estas casi inmediatas...?:

$$\int{\cos(x)^2}dx$$
$$\int{\frac{dx}{\sin(x)}}$$
$$\int{\frac{dx}{(x^2+a^2)^2}}$$

```python
int1 = cos(x) ** 2
integrate(int1)
```

```python
int2 =  1 / sin(x)
integrate(int2)
```

```python
x, a = symbols('x a', real=True)

int3 = 1 / (x**2 + a**2)**2
integrate(int3, x)
```

# Límites

Calculemos este límite sacado del libro _Cálculo: definiciones, teoremas y resultados_, de Juan de Burgos:

$$\lim_{x \to 0} \left(\frac{x}{\tan{\left (x \right )}}\right)^{\frac{1}{x^{2}}}$$

Primero creamos la expresión:

```python
x = symbols('x', real=True)
expr = (x / tan(x)) ** (1 / x**2)
expr
```

Obtenemos el límite con la función `limit()` y si queremos dejarlo indicado, podemos usar `Limit()`:

```python
limit(expr, x, 0)
```

# Series

Los desarrollos en serie se pueden llevar a cabo con el método `.series()` o la función `series()`

```python
#creamos la expresión
expr = exp(x)
expr
```

```python
#la desarrollamos en serie
series(expr)
```

Se puede especificar el número de términos pasándole un argumento `n=...`. El número que le pasemos será el primer término que desprecie.

```python
# Indicando el número de términos
series(expr, n=10)
```

Si nos molesta el $\mathcal{O}(x^{10})$ lo podemos quitar con `removeO()`:

```python
series(expr, n=10).removeO()
```

```python
series(sin(x), n=8, x0=pi/3).removeO().subs(x, x-pi/3)
```

---

## Resolución de ecuaciones

Como se ha mencionado anteriormente las ecuaciones no se pueden crear con el `=`

```python
#creamos la ecuación
ecuacion = Eq(x ** 2 - x, 3)
ecuacion
```

```python
# También la podemos crear como
Eq(x ** 2 - x -3)
```

```python
#la resolvemos
solve(ecuacion)
```

Pero la gracia es resolver con símbolos, ¿no?
$$a e^{\frac{x}{t}} = C$$

```python
# Creamos los símbolos y la ecuación
a, x, t, C = symbols('a, x, t, C', real=True)
ecuacion = Eq(a * exp(x/t), C)
ecuacion
```

```python
# La resolvemos
solve(ecuacion ,x)
```

Si consultamos la ayuda, vemos que las posibilidades y el número de parámetros son muchos, no vamos a entrar ahora en ellos, pero ¿se ve la potencia?

## Ecuaciones diferenciales

Tratemos de resolver, por ejemplo:

$$y{\left (x \right )} + \frac{d}{d x} y{\left (x \right )} + \frac{d^{2}}{d x^{2}}  y{\left (x \right )} = \cos{\left (x \right )}$$

```python
x = symbols('x')
y = Function('y')
ecuacion_dif = Eq(y(x).diff(x,2) + y(x).diff(x) + y(x), cos(x))
ecuacion_dif
```

```python
#resolvemos
dsolve(ecuacion_dif, f(x))
```

# Matrices

```python
#creamos una matriz llena de símbolos
a, b, c, d = symbols('a b c d')
A = Matrix([
            [a, b],
            [c, d]
           ])
A
```

```python
#sacamos autovalores
A.eigenvals()
```

```python
#inversa
A.inv()
```

```python
#elevamos al cuadrado la matriz
A ** 2
```

---

_ Esto ha sido un rápido recorrido por algunas de las posibilidades que ofrece SymPy . El cálculo simbólico es un terreno díficil y este joven paquete avanza a pasos agigantados gracias a un grupo de desarrolladores siempre dispuestos a mejorar y escuchar sugerencias. Sus posibilidades no acaban aquí. En otros cursos puedes encontrar también una clase sobre el módulo `mechanics`, pero además cuenta con herramientas para geometría, mecánica cuántica, teoría de números, combinatoria... Puedes echar un ojo [aquí](http://docs.sympy.org/latest/modules/index.html). _

---

Clase en vídeo, parte del [Curso de Python para científicos e ingenieros](http://cacheme.org/curso-online-python-cientifico-ingenieros/) grabado en la Escuela Politécnica Superior de la Universidad de Alicante.

```python
from IPython.display import YouTubeVideo

YouTubeVideo("OGQRcYVys1Q", width=560, height=315, list="PLGBbVX_WvN7as_DnOGcpkSsUyXB1G_wqb")
```

---

Si te ha gustado esta clase:

Tweet

---

####
¡Síguenos en Twitter!

###### Follow @AeroPython

#####
Curso AeroPython por Juan Luis Cano Rodriguez y Alejandro Sáez Mollejo se distribuye bajo una Licencia Creative Commons Atribución 4.0 Internacional.

#####

---
_Las siguientes celdas contienen configuración del Notebook_

_Para visualizar y utlizar los enlaces a Twitter el notebook debe ejecutarse como [seguro](http://ipython.org/ipython-doc/dev/notebook/security.html)_

    File > Trusted Notebook

```python
%%html
<a href="https://twitter.com/AeroPython" class="twitter-follow-button" data-show-count="false">Follow @AeroPython</a>

```

```python
# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = '../static/styles/style.css'
HTML(open(css_file, "r").read())
```


# "Oh, the things we can do with a 2D nonlinear system"
> "We're using sympy and a few custom features to carry out traditional analyses of a nonlinear ODE system"

- toc: true
- branch: master
- badges: true
- comments: true
- categories: [MATH280]

In my differential equations class, we use python for symbolic computation by default, explicitly calling on other libraries as needed.  We've written and collected a collection of utilities in the [MATH280 module](https://github.com/ejbarth/MATH280) for some tasks that weren't implemented (to our knowledge) or felt awkward.  We'll test drive a few of those features here as we explore the behavior of a 2-dimensional nonlinear equation.

```python
from sympy import *
```

## Some traditional symbolic analysis

We'll first define the symbolic system of nonlinear equations and use the symbolic `solve()` to find equilibrium solutions

```python
x,y=symbols("x y")
f1=-2*x-y**2
f2=x-2*y

eqs=solve([f1,f2],[x,y])
eqs
```

    [(-8, -4), (0, 0)]

We can symbolically generate the linearization matrix by manipulating our original functions $f_1(x,y)$ and $f_2(x,y)$ and check the eigenvalues at the equilibria, noting that $(0,0)$  is stable and $(8, -4)$ is an unstable saddle.

```python
L=Matrix([[f1.diff(x),f1.diff(y)],[f2.diff(x),f2.diff(y)]])
L
```

$\displaystyle \left[\begin{matrix}-2 & - 2 y\\1 & -2\end{matrix}\right]$

```python
L.subs({x:eqs[1][0],y:eqs[1][1]}).eigenvals()
```

    {-2: 2}

```python
L.subs({x:eqs[0][0],y:eqs[0][1]}).eigenvals()
```

    {-2*sqrt(2) - 2: 1, -2 + 2*sqrt(2): 1}

Especially when we can't find equilibrium solutions symbolically, zero isoclines can be really helpful in understanding qualitative behavior of solutions, and the task requires less from `solve()`

```python
isocline1=solve(f1,y)
isocline2=solve(f2,y)

isocline1, isocline2
```

    ([-sqrt(2)*sqrt(-x), sqrt(2)*sqrt(-x)], [x/2])

We can use the sympy `plot()` to draw the graphs of the symbolic zero isocline expressions returned by `solve()`

```python
plot(isocline1[0],
     isocline1[1],
     isocline2[0],
     xlim=(-10,1),line_color="blue")
```

## Direction Fields for Symbolic Expressions

In the [MATH280 module](https://github.com/ejbarth/MATH280), we've included a direction field plotter `drawdf()` that bridges the symbolic/numeric divide.  `drawdf()` takes a symbolic expression and draws a direction field plot using numerical data generated behind the scenes.  An axis handle is returned so that further changes to the plot can be made.  Below, we draw the direction field for the 2D system, along with a few representative solutions using the `soln_at` option.  Then the zero isoclines are plotted on the same set of axes.

```python
import MATH280
from numpy import linspace, sqrt

xx=linspace(-10,0,100)

ax=MATH280.drawdf([f1,f2],[x,-10,1],[y,-6,1],soln_at=[[-8,1],[2,-5],[-4,-6]])
ax.plot(xx,xx/2,"b--")
ax.plot(xx,-sqrt(2)*sqrt(-xx),"r-.",xx,sqrt(2)*sqrt(-xx),"r-.")
```

## Numerical Solution for Symbolic Equations

We'd like to have the ability to easily call for a numerical solution of symbolic equation.  In the [MATH280 module](https://github.com/ejbarth/MATH280), we've included two numerical solvers (based on `scipy.integrate.solve_ivp()`) that, like:  `drawdf()` bridge the symbolic/numeric divide:
* `rkf45(symbolic_rhs_expression,dep_vars_list,dep_vars_initvals,ind_var_list)`: for general use
* `BDF(symbolic_rhs_expression,dep_vars_list,dep_vars_initvals,ind_var_list)`: for equations known to be stiff

```python
t=symbols("t")
ns=MATH280.rkf45([f1,f2],[x,y],[-3,2],[t,0,10])
```

Having jumped the gap from symbolics and numerics, the output `ns` is a list of numerical arrays. We'll plot with `matplotlib`

```python
import matplotlib.pyplot as plt
plt.plot(ns[0],ns[1][0],ns[0],ns[1][1])
```

Of course, we could call `solve_ivp()` directly with a defined right-hand-side function:

```python
def vrhs(t,xy):
    return [-2*xy[0]-xy[1]**2, xy[0] - 2*xy[1]]

from scipy.integrate import solve_ivp

nsol=solve_ivp(vrhs,[0, 5],[2,-2])
```

```python
plt.plot(nsol.t,nsol.y[0])
plt.plot(nsol.t,nsol.y[1])
```


Equation of ray is [1],

$$P(t) = E + tD$$

where $E$ is the origin or "eye" point and $D$ is the direction vector. In component form,

$$
\begin{bmatrix}
x(t) \\
y(t) \\
z(t) \\
\end{bmatrix} =
\begin{bmatrix}
x_E + t x_D \\
y_E + t y_D \\
z_E + t z_D\\
\end{bmatrix}
$$

The equation of cylinder aligned along the z direction is,

$$
x^2 + y^2 = R^2
$$

where $R$ is the radius of the cylinder.

Substituting the equation of the ray into the equation of the cylinder,

$$
(x_E + t x_D)^2 + (y_E + t y_D)^2 = R^2
$$

and after grouping the $t^2$ and $t$ terms,

$$
t^2\left(x_D^2 + y_D^2\right) + t \left(2 x_E x_D + 2 y_E y _D \right) + \left( x_E^2 + y_E^2 - R^2 \right) = 0
$$

which is a standard quadratic equation,

$$
at^2 + bt + c = 0
$$

Solution of this equation give two values $\left( t_1, t_2 \right)$ which give the ray's distance to intersection points. To be ahead on the ray's path $\left( t_1, t_2 \right) >= 0$ and to be real intersection points the values must be finite and have imaginary component of zero.

The intersection with the cylinder caps is found by intersecting the ray with two infinite planes at $z=0$ and $z=L$, where $L$ is the length of the cylinder. The ray-plane intersection is given by [2],

$$
t = \frac{(Q - P) \cdot n}{D \cdot n}
$$

where $t$ is the distance from the ray origin to the intersection point, $Q$ is a point on the plane and $n$ the **outward** facing surface normal at that point. As before $P$ is the origin of the ray and $D$ is the ray's direction unit vector.

For the bottom cap at $z=0$,

$$
    t_{\text{bot}} =
    \frac{
    \left(
        \begin{bmatrix}
        0 \\
        0 \\
        0 \\
        \end{bmatrix} -
    \begin{bmatrix}
        x_E \\
        y_E \\
        z_E \\
    \end{bmatrix}
    \right) \cdot
    \begin{bmatrix}
        0 \\
        0 \\
        -1 \\
    \end{bmatrix}
    }{
    \begin{bmatrix}
        x_D \\
        y_D \\
        z_D \\
    \end{bmatrix} \cdot
    \begin{bmatrix}
        0 \\
        0 \\
        -1 \\
    \end{bmatrix}
    }
$$

and for the top cap at $z=L$,

$$
    t_{\text{bot}} =
    \frac{
    \left(
        \begin{bmatrix}
        0 \\
        0 \\
        L \\
        \end{bmatrix} -
    \begin{bmatrix}
        x_E \\
        y_E \\
        z_E \\
    \end{bmatrix}
    \right) \cdot
    \begin{bmatrix}
        0 \\
        0 \\
        1 \\
    \end{bmatrix}
    }{
    \begin{bmatrix}
        x_D \\
        y_D \\
        z_D \\
    \end{bmatrix} \cdot
    \begin{bmatrix}
        0 \\
        0 \\
        1 \\
    \end{bmatrix}
    }
$$

The intersection points with $t<0$ and points not contained inside the circle of the end cap are rejected using $(x^2 + y^2) < R$, where $x$ and $y$ are the components of the candidate intersection point.

```python
import numpy as np
from sympy import *
init_printing()
```

```python
xE, yE, xD, yD, t, R = var("x_E y_E x_D y_D t R")
sols = solve(Eq((xE + t*xD)**2 + (yE + t*yD)**2, R**2), t)
sols
```

```python
pos = np.array([0, 0, 0.5])
uvec = np.array([0.9, 0, 0.01])
uvec = uvec / np.linalg.norm(uvec)
radius = 1.0
tsols = [float(asol.subs({xE: pos[0], yE: pos[1], xD: uvec[0], yD: uvec[1], R: radius}))
 for asol in sols]
tsols
```

```python
L = 1
ipoints = [np.array(pos + t * uvec) for t in tsols if t > 0]
ipoints = [point for point in ipoints if point[2] > 0 and point[2] < L]
ipoints
```

    [array([1.        , 0.        , 0.51111111])]

For cylinder with length L filter solutions inside range, $0 < z < L$. This assumes the cylinder starts at $z=0$ and ends at $z=L$.

The intersection with ends caps.

We will work out the intersection with infinite planes at the end cap positions and then see if the intersection point is inside a circle on the surface of the plane.

The intersection with ray with a infinite xy-plane is,

$$
t = \frac{C_z-E_z}{D_z}
$$

where $C_z$ is the $z$ location of the plane.

This comes from the more general intersection formular described here https://www.scratchapixel.com/lessons/3d-basic-rendering/minimal-ray-tracer-rendering-simple-shapes/ray-plane-and-ray-disk-intersection

If the intersection point lies inside a cap circle then we include the intersection point in the list.

```python
cap_ipoints = []
for z in (0.0, L):
    with np.errstate(divide='ignore'):
        t = (z - pos[2])/uvec[2]
        if np.isfinite(t):
            point = np.array(pos + t * uvec)
            if np.sqrt(point[0]**2 + point[1]**2) < radius:
                cap_ipoints.append(point)
cap_ipoints
```

```python
def ray_z_cylinder(length, radius, ray_origin, ray_direction):
    """ Returns ray-cylinder intersection points for a cylinder aligned
        along the z-axis with centre at (0, 0, 0).

        Parameters
        ----------
        length : float
            The length of the cylinder
        radius : float
            The radius of the cylinder
        ray_origin : tuple of float
            The origin of th ray like, e.g. :math:`\left((0.0, 1.0, 2.0 \right)`
        ray_direction : tuple of float
            The direction **unit** vector of the ray like, e.g. :math:`(n_x, n_y, n_z)`.

        Returns
        -------
        points: tuple of points
            Returns a tuple of tuple like ((0.0, 1.0, 2.0), ...) where each item is an
            intersection point. The tuple is sorted by distance from the ray origin.

        Notes
        -----

        Equation of ray is [1],

        :math:`P(t) = E + t`

        where :math:`E` is the origin or "eye" point and :math:`D` is the direction vector.
        In component form,

        .. math::

            \begin{bmatrix}
            x(t) \\
            y(t) \\
            z(t) \\
            \end{bmatrix} =
            \begin{bmatrix}
            x_E + t x_D \\
            y_E + t y_D \\
            z_E + t z_D\\
            \end{bmatrix}

        The equation of cylinder aligned along the z direction is,

        .. math::

            x^2 + y^2 = R^2

        where :math`R` is the radius of the cylinder.

        Substituting the equation of the ray into the equation of the cylinder,

        .. math::

            (x_E + t x_D)^2 + (y_E + t y_D)^2 = R^2

        and after grouping the :math:`t^2` and :math:`t` terms,

        .. math::

            t^2\left(x_D^2 + y_D^2\right) +
            t \left(2 x_E x_D + 2 y_E y _D \right) +
            \left( x_E^2 + y_E^2 - R^2 \right) = 0

        which is a standard quadratic equation,

        .. math::

            at^2 + bt + c = 0

        Solution of this equation give two values :math:`\left( t_1, t_2 \right)` which
        give the ray's distance to intersection points. To be ahead on the ray's path
        $\left( t_1, t_2 \right) >= 0$ and to be real intersection points the values
        must be finite and have imaginary component of zero.

        The intersection with the cylinder caps is found by intersecting the ray with
        two infinite planes at $z=0$ and $z=L$, where $L$ is the length of the cylinder.
        The ray-plane intersection is given by [2],

        .. math::

            t = \frac{(Q - P) \cdot n}{D \cdot n}

        where :math:`t` is the distance from the ray origin to the intersection point,
        :math:`Q` is a point on the plane and :math:`n` the **outward** facing surface
        normal at that point. As before :math:`P` is the origin of the ray and :math:`D`
        is the ray's direction unit vector.

        For the bottom cap at :math:`z=0`,

        .. math::

            t_{\text{bot}} =
            \frac{
            \left(
                \begin{bmatrix}
                0 \\
                0 \\
                -0.5 L \\
                \end{bmatrix} -
            \begin{bmatrix}
                x_E \\
                y_E \\
                z_E \\
            \end{bmatrix}
            \right) \cdot
            \begin{bmatrix}
                0 \\
                0 \\
                -1 \\
            \end{bmatrix}
            }{
            \begin{bmatrix}
                x_D \\
                y_D \\
                z_D \\
            \end{bmatrix} \cdot
            \begin{bmatrix}
                0 \\
                0 \\
                -1 \\
            \end{bmatrix}
            }

        and for the top cap at :math:`z=L`,

        .. math::
            t_{\text{bot}} =
            \frac{
            \left(
                \begin{bmatrix}
                0 \\
                0 \\
                0.5 L \\
                \end{bmatrix} -
            \begin{bmatrix}
                x_E \\
                y_E \\
                z_E \\
            \end{bmatrix}
            \right) \cdot
            \begin{bmatrix}
                0 \\
                0 \\
                1 \\
            \end{bmatrix}
            }{
            \begin{bmatrix}
                x_D \\
                y_D \\
                z_D \\
            \end{bmatrix} \cdot
            \begin{bmatrix}
                0 \\
                0 \\
                1 \\
            \end{bmatrix}
            }

        The intersection points with :math:`t<0` and points not contained inside the circle
        of the end cap are rejected using $(x^2 + y^2) < R$, where $x$ and $y$ are the
        components of the candidate intersection point.

        References
        ----------
        [1] https://www.cl.cam.ac.uk/teaching/1999/AGraphHCI/
        [2] https://www.scratchapixel.com/lessons/3d-basic-rendering/minimal-ray-tracer-rendering-simple-shapes/ray-plane-and-ray-disk-intersection

    """
    p0 = np.array(ray_origin)
    n0 = np.array(ray_direction)
    xe, ye, ze = p0
    xd, yd, zd = n0

    # Look for intersections on the cylinder surface
    a = xd**2 + yd**2
    b = 2 * (xe*xd + ye*yd)
    c = xe**2 + ye**2 - radius**2
    tcyl = [t for t in np.roots([a, b, c]) if np.isfinite(t) and np.isreal(t) and t >= 0]

    # Look for intersections on the cap surfaces
    with np.errstate(divide='ignore'):
        # top cap
        point = np.array([0.0, 0.0, 0.5*length])
        normal = np.array([0.0, 0.0, 1.0]) # outward facing at z = length
        ttopcap = (point - p0).dot(normal) / n0.dot(normal)
        # bottom cap
        point = np.array([0.0, 0.0, -0.5*length])
        normal = np.array([0.0, 0.0, -1.0]) # outward facing at z = 0
        tbotcap = (point - p0).dot(normal) / n0.dot(normal)
        tcap = [t for t in (tbotcap, ttopcap) if np.isfinite(t) and t >= 0.0]

    # Reject point cap points which are not in the cap's circle radius
    # and cylinder points which outside the length.
    cap_candidates = [(p0 + t * n0, t) for t in tcap]
    cap_candidates = [(point, t) for (point, t) in cap_candidates
                      if np.sqrt(point[0]**2 + point[1]**2) < radius]
    cyl_candidates = [(p0 + t * n0, t) for t in tcyl]
    cyl_candidates = [(point, t) for (point, t) in cyl_candidates if point[2] > -0.5*length and point[2] < 0.5*length]
    intersection_info = tuple(cyl_candidates) + tuple(cap_candidates)
    intersection_info = sorted(intersection_info, key=lambda pair: pair[1])
    points = tuple([tuple(p.tolist()) for p in list(zip(*intersection_info))[0]])
    return points
```

```python
import numpy as np
import meshcat
import meshcat.geometry as g
import meshcat.transformations as tf
vis = meshcat.visualizer.Visualizer()

def norm(v):
    return v / np.linalg.norm(v)

vis.delete()
# touching
ray_origin = (0.0, 0.0, -1.5)
ray_direction = norm((0.0, 1.0, 1.0))
expected = ((0.0, 1.0, -0.5),)
touching = (ray_origin, ray_direction, expected)
# end caps only
ray_origin = (0.2, 0.2, -1)
ray_direction = norm((0.0, 0.0, 1.0))
expected = ((0.2, 0.2, -0.5), (0.2, 0.2, 0.5))
end_caps_only = (ray_origin, ray_direction, expected)
# surface and bottom
ray_origin = (-2, 0.2, 0.0)
ray_direction = norm((1.0, 0.2, -0.2))
expected = (
    (-0.9082895433880116, 0.41834209132239775, -0.2183420913223977),
    (0.5, 0.7, -0.5)
)
surface_and_bottom = (ray_origin, ray_direction, expected)
# surface and top
ray_origin = (-2, 0.2, 0.0)
ray_direction = norm((1.0, 0.2, 0.2))
expected = (
    (-0.9082895433880116, 0.41834209132239775, 0.2183420913223977),
    (0.5, 0.7, 0.5)
)
surface_and_top = (ray_origin, ray_direction, expected)
tests = (touching, end_caps_only, surface_and_bottom, surface_and_top)

# Place cylinder in scene
material = g.MeshLambertMaterial(reflectivity=1.0, sides=0)
vis['cyl'].set_object(
        g.Cylinder(length, radius),
        material)
vis['cyl'].set_transform(
    tf.translation_matrix([0.0, 0.0, 0.0]).dot(tf.rotation_matrix(np.radians(-90), [1, 0, 0]))
)

# Visualise test rays
for idx, (ray_origin, ray_direction, expected) in enumerate(tests):
    ray_inf = np.array(ray_origin) + 5.0 * np.array(ray_direction)
    vertices = np.column_stack((ray_origin, ray_inf))

    red_material = g.MeshLambertMaterial(reflectivity=1.0, sides=0, color=0xff0000)
    vis['line_{}'.format(idx)].set_object(g.Line(g.PointsGeometry(vertices)))
    points = ray_z_cylinder(length, radius, ray_origin, ray_direction)
    for ptidx, pt in enumerate(points):
        vis['point_{}_{}'.format(idx, ptidx)].set_object(
                g.Sphere(0.05),
                red_material)
        vis['point_{}_{}'.format(idx, ptidx)].set_transform(
            tf.translation_matrix(pt)
        )
    if np.allclose(points, expected, atol=1e-15):
        print(points)
        print("OK!")
vis.jupyter_cell()
```

```python

```