# ARMA models

2019, Jérémie DECOCK

## Lag operator

Definition of the *backward shift operator* $B$:

$$B y_t = y_{t-1}$$

$$B^i y_t = y_{t-i}$$

Definition of the *forward shift operator* $F$:

$$  F y_t = B^{-1} y_t = y_{t+1}$$

$$F^i y_t = B^{-i} y_t = y_{t+i}$$

## Lag polynomials

Polynomials of the lag operator are commonly used to simplify ARMA models notation.
These polynomials are called *lag polynomials*.

### Example

$a_{(p)} (B)$ is a *lag polynomials*:
$$
\begin{align}
a_{(p)} (B) &= 1 + \sum_{i=1}^p a_i B^i \\
            &= 1 + a_1 B + a_2 B^2 + \dots + a_p B^p
\end{align}
$$

### Usage example in time series analysis

$$
\begin{align}
a_{(p)} (B)y_t &= \left( 1 + \sum_{i=1}^p a_i B^i \right) ~ y_t \\
               &= y_t + a_1 B y_t + a_2 B^2 y_t + \dots + a_p B^p y_t \\
               &= y_t + a_1 y_{t-1} + a_2 y_{t-2} + \dots + a_p y_{t-p}
\end{align}
$$

## Difference operator

Definition of the *difference operator*:
$$\nabla^i y_t = (1-B)^i ~ y_t$$

### Example: first order difference operator (i.e. $i=1$)

$$
\begin{align}
\nabla y_t &= (1-B) ~ y_t \\
           &= y_t - B ~ y_t \\
           &= y_t - y_{t-1}
\end{align}
$$

### Example: second order difference operator (i.e. $i=2$)

$$
\begin{align}
\nabla^2 y_t &= (1-B)^2 ~ y_t \\
           &= (1-B)(1-B) ~ y_t \\
           &= \left( 1 - 2B + B^2 \right) ~ y_t \\
           &= y_t - 2y_{t-1} + y_{t-2}
\end{align}
$$

### Example: first order difference operator with a lag of 12

$$
\begin{align}
\left( 1-B^{12} \right) ~ y_t &= y_t - B^{12} ~ y_t \\
                              &= y_t - y_{t-12}
\end{align}
$$

## Autoregressive Model (AR)

Using this model, the observation $y_t$ of a time series at time $t$ is defined as a linear combination of its $p$ past observations $y_{t-1}, \dots, y_{t-p}$

$$
\begin{align}
AR(\color{\red}{p}): \quad\quad\quad \color{\red}{\underbrace{\varphi_{(p)} (B) y_t}_{\text{AR}}}
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ \left( 1 + \sum_{i=1}^p \varphi_i B^i \right) y_t }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^p \varphi_i B^i y_t }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^p \varphi_i y_{t-i} }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{y_t}
&= c + \color{\green}{\varepsilon_t} + \color{\red}{\sum_{i=1}^p \phi_i y_{t-i}}
\end{align}
$$

with $\varphi_i = -\phi_i$.

$c$ is a constant used to ensure the series is centered to 0 ($c$ is the average of the series).

$\color{\green}{\varepsilon_t}$ is the noise component of $y_t$.

### Example: Autoregressive order 3 process (i.e. $p=3$) noted AR(3)

$$
\begin{align}
AR(\color{\red}{3}): \quad\quad\quad \color{\red}{\underbrace{\varphi_{(3)} (B) y_t}_{\text{AR}}}
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ \left( 1 + \sum_{i=1}^3 \varphi_i B^i \right) y_t }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^3 \varphi_i B^i y_t }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^3 \varphi_i y_{t-i} }
&= c + \color{\green}{\varepsilon_t} \\
\color{\red}{y_t}
&= c + \color{\green}{\varepsilon_t} + \color{\red}{\sum_{i=1}^3 \phi_i y_{t-i}} \\
\color{\red}{y_t}
&= c + \color{\green}{\varepsilon_t} + \color{\red}{\phi_1 y_{t-1}} + \color{\red}{\phi_2 y_{t-2}} + \color{\red}{\phi_3 y_{t-3}}
\end{align}
$$

with $\varphi_i = -\phi_i$.

## Moving Average Model (MA)

Combinaison linéaire des $q$ erreurs passées
Using this model, the observation $y_t$ of a time series at time $t$ is defined as the output of a linear filter with transfer function $\theta_{(q)}(B)$ when the input is white noise $\varepsilon_t$.

$$
\begin{align}
MA(\color{\green}{q}): \quad\quad\quad
\color{\red}{y_t} &= c + \color{\green}{\underbrace{\theta_{(q)} (B) \varepsilon_t}_{MA}} \\
\color{\red}{y_t} &= c + \color{\green}{\left( 1 + \sum_{j=1}^q \theta_j B^j \right) ~ \varepsilon_t} \\
\color{\red}{y_t} &= c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j B^j \varepsilon_t} \\
\color{\red}{y_t} &= c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j ~ \varepsilon_{t-j}}
\end{align}
$$

### Example: Moving average order 3 process (i.e. $q=3$) noted MA(3)

$$
\begin{align}
MA(\color{\green}{3}): \quad\quad\quad
\color{\red}{y_t} &= c + \color{\green}{\underbrace{\theta_{(3)} (B) \varepsilon_t}_{MA}} \\
\color{\red}{y_t} &= c + \color{\green}{\left( 1 + \sum_{j=1}^3 \theta_j B^j \right) ~ \varepsilon_t} \\
\color{\red}{y_t} &= c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^3 \theta_j B^j \varepsilon_t} \\
\color{\red}{y_t} &= c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^3 \theta_j ~ \varepsilon_{t-j}} \\
\color{\red}{y_t} &= c + \color{\green}{\varepsilon_t} + \color{\green}{\theta_1 ~ \varepsilon_{t-1}} + \color{\green}{\theta_2 ~ \varepsilon_{t-2}} + \color{\green}{\theta_3 ~ \varepsilon_{t-3}}
\end{align}
$$

### AutoRegressive Moving Average (ARMA)

An ARMA(p,q) process of order $p$ and $q$ is a combination of an AR(p) and an MA(q) process.

$$
\begin{align}
ARMA(\color{\red}{p}, \color{\green}{q}): \quad\quad\quad
\color{\red}{\underbrace{\varphi_{(p)} (B) y_t}_{\text{AR}}}
&=
c + \color{\green}{\underbrace{\theta_{(q)} (B) \varepsilon_t}_{MA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^p \varphi_i B^i \right) y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^q \theta_j B^j \right) ~ \varepsilon_t}
\\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^p \varphi_i B^i y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j B^j \varepsilon_t}
\\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^p \varphi_i y_{t-i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j ~ \varepsilon_{t-j}}
\\
\color{\red}{y_t}
&=
c
+ \color{\red}{\underbrace{\sum_{i=1}^p \phi_i y_{t-i}}_{\text{AR model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^q \theta_j \varepsilon_{t-j}}_{\text{MA model}}}
\end{align}
$$

with $\varphi_i = -\phi_i$

#### Example: ARMA(3,2)

$$
\begin{align}
ARMA(\color{\red}{3}, \color{\green}{2}): \quad\quad\quad
\color{\red}{\underbrace{\varphi_{(3)} (B) y_t}_{\text{AR}}}
&=
c + \color{\green}{\underbrace{\theta_{(2)} (B) \varepsilon_t}_{MA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^3 \varphi_i B^i \right) y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^2 \theta_j B^j \right) ~ \varepsilon_t}
\\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^3 \varphi_i B^i y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \theta_j B^j \varepsilon_t}
\\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^3 \varphi_i y_{t-i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \theta_j ~ \varepsilon_{t-j}}
\\
\color{\red}{y_t}
&=
c
+ \color{\red}{\underbrace{\sum_{i=1}^3 \phi_i y_{t-i}}_{\text{AR model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^2 \theta_j \varepsilon_{t-j}}_{\text{MA model}}}
\\
\color{\red}{y_t}
&=
c
+ \underbrace{ \color{\red}{\phi_1 y_{t-1}} + \color{\red}{\phi_2 y_{t-2}} + \color{\red}{\phi_3 y_{t-3}} }_{\text{AR model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{ \color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}} }_{\text{MA model}}
\end{align}
$$

with $\varphi_i = -\phi_i$

## AutoRegressive Integrated Moving Average (ARIMA)

An ARIMA process is an *integrated* ARMA process.
It's used on non-stationary time series.
An initial transformation step (the "integrated" part of the model) is applied to eliminate the non-stationarity.
This transformation is usually operated by the *difference operator* described above.

An $ARIMA(p,d,q)$ process produce series that can be modeled as a stationary $ARMA(p,q)$ process after being differenced $d$ times.

$$
\begin{align}
ARIMA(\color{\red}{p}, \color{\orange}{d}, \color{\green}{q}): \quad\quad\quad
\color{\red}{\underbrace{\varphi_{(p)} (B) \color{\orange}{\nabla^d} y_t}_{\text{AR}}}
&=
c + \color{\green}{\underbrace{\theta_{(q)} (B) \varepsilon_t}_{MA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^p \varphi_i B^i \right)} \color{\orange}{\nabla^d} \color{\red}{ y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^q \theta_j B^j \right) ~ \varepsilon_t}
\\
\color{\orange}{\nabla^d} \color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^p \varphi_i B^i } \color{\orange}{\nabla^d} \color{\red}{ y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j B^j \varepsilon_t}
\\
\color{\orange}{\nabla^d} \color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^p \varphi_i } \color{\orange}{\nabla^d} \color{\red}{ y_{t-i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^q \theta_j ~ \varepsilon_{t-j}}
\\
\color{\orange}{\nabla^d} \color{\red}{y_t}
&=
c
+ \underbrace{
    \color{\red}{\sum_{i=1}^p \phi_i}
    \color{\orange}{\nabla^d}
    \color{\red}{y_{t-i}}
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\sum_{j=1}^q \theta_j \varepsilon_{t-j}}}_{\text{MA model}}
\end{align}
$$

with

$$\nabla = (1-B)$$
$$\nabla^2 = (1-B)^2$$
$$\dots$$
$$\nabla^d = (1-B)^d$$

#### Examples for $d=1$, $d=2$ and $d=3$

If $d=0$:
$$
\begin{align}
\nabla^0 y_t &:= (1-B)^0 y_t \\
              &= y_t
\end{align}
$$

If $d=1$:
$$
\begin{align}
\nabla y_t &:= (1-B) y_t \\
              &= y_t - B y_t \\
              &= y_t - y_{t-1}
\end{align}
$$

If $d=2$:
$$
\begin{align}
\nabla^2 y_t &:= (1-B)^2 y_t \\
              &= (1-B)(1-B) ~ y_t \\
              &= (1 - 2B + B^2) y_t \\
              &= y_t - 2B y_t + B^2 y_t \\
              &= y_t - 2 y_{t-1} + y_{t-2}
\end{align}
$$

The above approach generalises to the i-th difference operator $\nabla^i y_t = (1-B)^i ~ y_t$

#### Example: ARIMA(3, 1, 2)

$$
\begin{align}
ARIMA(\color{\red}{3}, \color{\orange}{1}, \color{\green}{2}): \quad\quad\quad
\color{\red}{\underbrace{\varphi_{(3)} (B) \color{\orange}{\nabla} y_t}_{\text{AR}}}
&=
c + \color{\green}{\underbrace{\theta_{(2)} (B) \varepsilon_t}_{MA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^3 \varphi_i B^i \right)} \color{\orange}{\nabla} \color{\red}{ y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^2 \theta_j B^j \right) ~ \varepsilon_t}
\\
\color{\orange}{\nabla} \color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^3 \varphi_i B^i } \color{\orange}{\nabla} \color{\red}{ y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \theta_j B^j \varepsilon_t}
\\
\color{\orange}{\nabla} \color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^3 \varphi_i } \color{\orange}{\nabla} \color{\red}{ y_{t-i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \theta_j ~ \varepsilon_{t-j}}
\\
\color{\orange}{\nabla} \color{\red}{y_t}
&=
c
+ \underbrace{
    \color{\red}{\sum_{i=1}^3 \phi_i}
    \color{\orange}{\nabla}
    \color{\red}{y_{t-i}}
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\sum_{j=1}^2 \theta_j \varepsilon_{t-j}}}_{\text{MA model}}
\\
\color{\orange}{\nabla} \color{\red}{y_t}
&=
c
+ \underbrace{
      \color{\red}{\phi_1} \color{\orange}{\nabla} \color{\red}{y_{t-1}}
    + \color{\red}{\phi_2} \color{\orange}{\nabla} \color{\red}{y_{t-2}}
    + \color{\red}{\phi_3} \color{\orange}{\nabla} \color{\red}{y_{t-3}}
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}}}_{\text{MA model}}
\\
\color{\orange}{(1-B)} \color{\red}{y_t}
&=
c
+ \underbrace{
      \color{\red}{\phi_1} \color{\orange}{(1-B)} \color{\red}{y_{t-1}}
    + \color{\red}{\phi_2} \color{\orange}{(1-B)} \color{\red}{y_{t-2}}
    + \color{\red}{\phi_3} \color{\orange}{(1-B)} \color{\red}{y_{t-3}}
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}}}_{\text{MA model}}
\\
\color{\red}{y_t} - \color{\orange}{B y_t}
&=
c
+ \underbrace{
      \color{\red}{\phi_1} (\color{\red}{y_{t-1}} - \color{\orange}{B y_{t-1}})
    + \color{\red}{\phi_2} (\color{\red}{y_{t-2}} - \color{\orange}{B y_{t-2}})
    + \color{\red}{\phi_3} (\color{\red}{y_{t-3}} - \color{\orange}{B y_{t-3}})
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}}}_{\text{MA model}}
\\
\color{\red}{y_t} - \color{\orange}{y_{t-1}}
&=
c
+ \underbrace{
      \color{\red}{\phi_1} (\color{\red}{y_{t-1}} - \color{\orange}{y_{t-2}})
    + \color{\red}{\phi_2} (\color{\red}{y_{t-2}} - \color{\orange}{y_{t-3}})
    + \color{\red}{\phi_3} (\color{\red}{y_{t-3}} - \color{\orange}{y_{t-4}})
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}}}_{\text{MA model}}
\\
\color{\red}{y_t}
&=
c
+ \color{\orange}{y_{t-1}}
+ \underbrace{
      \color{\red}{\phi_1 y_{t-1}} - \color{\orange}{\phi_1 y_{t-2}}
    + \color{\red}{\phi_2 y_{t-2}} - \color{\orange}{\phi_2 y_{t-3}}
    + \color{\red}{\phi_3 y_{t-3}} - \color{\orange}{\phi_3 y_{t-4}}
  }_{\text{ARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\theta_1 \varepsilon_{t-1}} + \color{\green}{\theta_2 \varepsilon_{t-2}}}_{\text{MA model}}
\end{align}
$$

## AutoRegressive Moving Average Including Exogenous Covariates (ARMAX)

$$
\begin{align}
\color{\red}{\underbrace{\varphi (B) y_t}_{AR}}
&= c
+ \color{\green}{\underbrace{\theta (B) \varepsilon_t}_{MA}}
+ \color{\purple}{\underbrace{\beta_1 x_{1,~t} + \beta_2 x_{2,~t} + \dots + \beta_r x_{r,~t}}_{X}}
\end{align}
$$

$x_{1,~t}, x_{2,~t}, \dots, x_{r,~t}$ is the value of the $r$ exogenous variables at time $t$, with $\beta_1, \beta_2, \dots, \beta_r$ the $r$ corresponding coefficients.

$$
ARMAX(\color{\red}{p}, \color{\green}{q}): \quad
\color{\red}{y_t} = c
+ \color{\red}{\underbrace{\sum_{i=1}^p \varphi_i y_{t-i}}_{\text{AR model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^q \theta_j \varepsilon_{t-j}}_{\text{MA model}}}
+ \color{\purple}{\underbrace{\sum_{k=1}^r \beta_k x_{k,~t}}_{\text{Exogenous}}}
$$

## Seasonal AutoRegressive Moving Average (SARMA)

#### Purely seasonal processes

$$
\begin{align}
ARMA(\color{\red}{P}, \color{\green}{Q})_{\color{\red}{s}, \color{\green}{s'}}: \quad\quad\quad
\color{\red}{\underbrace{\Phi_{(P)} (B^s) y_t}_{\text{SAR}}}
&=
c + \color{\green}{\underbrace{\Theta_{(Q)} (B^{s'}) \varepsilon_t}_{SMA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^P \Phi_i B^{si} \right) y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^Q \Theta_j B^{s'j} \right) ~ \varepsilon_t}
\\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^P \Phi_i B^{si} y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^Q \Theta_j B^{s'j} \varepsilon_t}
\\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^P \Phi_i ~ y_{t-si} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^Q \Theta_j ~ \varepsilon_{t-s'j}}
\\
\color{\red}{y_t} &= c
+ \color{\red}{\underbrace{\sum_{i=1}^P \varPhi_i ~ y_{t-si}}_{\text{SAR model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^Q \Theta_j ~ \varepsilon_{t-s'j}}_{\text{SMA model}}}
\end{align}
$$

with $\varPhi = -\Phi$ and :

- $s$  is the number of time steps for a single seasonal period for the AR process
- $s'$ is the number of time steps for a single seasonal period for the MA process
- $P$ is the seasonal AR process order
- $Q$ is the seasonal MA process order

Remark: most of the time, $s = s'$. In this case, the process is simply noted $ARMA(\color{\red}{P}, \color{\green}{Q})_{s}$

#### Example: $ARMA(3,2)_{4, 5}$

$$
\begin{align}
ARMA(\color{\red}{3}, \color{\green}{2})_{4, 5}: \quad\quad\quad
\color{\red}{\underbrace{\Phi_{(3)} (B^4) y_t}_{\text{SAR}}}
&=
c + \color{\green}{\underbrace{\Theta_{(2)} (B^5) \varepsilon_t}_{SMA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^3 \Phi_i B^{4i} \right) y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^2 \Theta_j B^{5j} \right) ~ \varepsilon_t}
\\
\color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^3 \Phi_i B^{4i} y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \Theta_j B^{5j} \varepsilon_t}
\\
\color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^3 \Phi_i y_{t-4i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \Theta_j ~ \varepsilon_{t-5j}}
\\
\color{\red}{y_t}
&=
c
+ \color{\red}{\underbrace{\sum_{i=1}^3 \varPhi_i y_{t-4i}}_{\text{SAR model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^2 \Theta_j \varepsilon_{t-5j}}_{\text{SMA model}}}
\\
\color{\red}{y_t}
&=
c
+ \underbrace{ \color{\red}{\varPhi_1 y_{t-4}} + \color{\red}{\varPhi_2 y_{t-8}} + \color{\red}{\varPhi_3 y_{t-12}} }_{\text{SAR model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{ \color{\green}{\Theta_1 \varepsilon_{t-5}} + \color{\green}{\Theta_2 \varepsilon_{t-10}} }_{\text{SMA model}}
\end{align}
$$

with $\varPhi_i = -\Phi_i$

### Seasonal and non seasonal processes : additive process $ARMA(p,q)+ARMA(P,Q)_{s, s'}$

These processes are rarely used $ARMA(p,q)+ARMA(P,Q)_{s, s'}$.

$$
\begin{align}
ARMA(p,q)+ARMA(P,Q)_{s, s'}: \quad
\left(
\underbrace{\varphi_p \left(B\right)}_{AR} +
\underbrace{\Phi_P \left(B^s\right)}_{SAR} -
1
\right) ~  y_t
&=
c +
\left(
\underbrace{\theta_q \left(B\right)}_{MA} +
\underbrace{\Theta_Q \left(B^{s'}\right)}_{SMA}
- 1 \right) ~ \varepsilon_t \\
\underbrace{\varphi_p (B) ~ y_t}_{AR} +
\underbrace{\Phi_P \left(B^s\right) ~ y_t}_{SAR} -
y_t
&=
c +
\underbrace{\theta_q \left(B\right) ~ \varepsilon_t}_{MA} +
\underbrace{\Theta_Q \left(B^{s'}\right) ~ \varepsilon_t}_{SMA} -
\varepsilon_t \\
\underbrace{\left( 1 + \sum_{i=1}^p \varphi_i B^i \right) ~ y_t}_{AR} +
\underbrace{\left( 1 + \sum_{k=1}^P \Phi_k B^{sk} \right) ~ y_t}_{SAR} -
y_t
&=
c +
\underbrace{\left( 1 + \sum_{j=1}^q \theta_j B^j \right) ~ \varepsilon_t}_{MA} +
\underbrace{\left( 1 + \sum_{l=1}^Q \Theta_l B^{s'l} \right) ~ \varepsilon_t}_{SMA} -
\varepsilon_t \\
\underbrace{y_t + \sum_{i=1}^p \varphi_i B^i y_t}_{AR} +
\underbrace{y_t + \sum_{k=1}^P \Phi_k B^{sk} y_t}_{SAR} -
y_t
&=
c +
\underbrace{\varepsilon_t + \sum_{j=1}^q \theta_j B^j \varepsilon_t}_{MA} +
\underbrace{\varepsilon_t + \sum_{l=1}^Q \Theta_l B^{s'l} \varepsilon_t}_{SMA} -
\varepsilon_t \\
y_t +
\sum_{i=1}^p \varphi_i B^i y_t +
\sum_{k=1}^P \Phi_k B^{sk} y_t
&=
c +
\varepsilon_t + \sum_{j=1}^q \theta_j B^j \varepsilon_t +
\sum_{l=1}^Q \Theta_l B^{s'l} \varepsilon_t \\
y_t +
\sum_{i=1}^p \varphi_i y_{t-i} +
\sum_{k=1}^P \Phi_k y_{t-sk}
&=
c +
\varepsilon_t +
\sum_{j=1}^q \theta_j \varepsilon_{t-j} +
\sum_{l=1}^Q \Theta_l \varepsilon_{t-s'l} \\
y_t
&=
c
+ \underbrace{\sum_{i=1}^p \phi_i y_{t-i}}_{\text{AR model}}
+ \underbrace{\sum_{k=1}^{P} \varPhi_k y_{t-sk}}_{\text{SAR model}}
+ \varepsilon_t
+ \underbrace{\sum_{j=1}^q \theta_j \varepsilon_{t-j}}_{\text{MA model}}
+ \underbrace{\sum_{l=1}^{Q} \Theta_l \varepsilon_{t-s'l}}_{\text{SMA model}}
\end{align}
$$

with $\varPhi = -\Phi$ and $\varphi = -\phi$

Remark: most of the time, $s = s'$. In this case, the process is simply noted $ARMA(\color{\red}{p}, \color{\green}{q}) + ARMA(\color{\red}{P}, \color{\green}{Q})_{s}$

### Seasonal and non seasonal processes : multiplicative process $ARMA(p,q) \times ARMA(P,Q)_{s, s'}$

These process are more often used than previously described additive processes.

Seasonal series are characterized by a strong serial correlation at the seasonal lag (and possibly multiples thereof).

$$
\begin{align}
ARMA(\color{\red}{p},\color{\green}{q}) \times ARMA(\color{\red}{P},\color{\green}{Q})_{\color{\red}{s}, \color{\green}{s'}}: \quad
\color{\red}{
\underbrace{\varphi_{(p)} (B)}_{AR} ~
\underbrace{\Phi_{(P)} (B^s)}_{SAR} ~
y_t
}
&=
c +
\color{\green}{
\underbrace{\theta_{(q)} (B)}_{MA} ~
\underbrace{\Theta_{(Q)} (B^{s'})}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{\left( 1 + \sum_{i=1}^p \varphi_i B^i \right)}_{AR} ~
\underbrace{\left( 1 + \sum_{k=1}^P \Phi_k B^{sk} \right)}_{SAR} ~
y_t
}
&=
c +
\color{\green}{
\underbrace{\left( 1 + \sum_{j=1}^q \theta_j B^j \right)}_{MA} ~
\underbrace{\left( 1 + \sum_{l=1}^Q \Theta_l B^{s'l} \right)}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{
\left( 1 + \varphi_1 B + \varphi_2 B^2 + \dots + \varphi_p B^p \right)
\left( 1 + \Phi_1 B^{s} + \Phi_2 B^{s2} + \dots + \Phi_P B^{sP} \right)
y_t
}_{\text{Order of the equivalent AR process} ~=~ p + Ps}
}
&=
c +
\color{\green}{
\underbrace{
\left( 1 + \theta_1 B + \theta_2 B^2 + \dots + \theta_q B^q \right)
\left( 1 + \Theta_1 B^{s'} + \Theta_2 B^{s'2} + \dots + \Theta_Q B^{s'Q} \right)
\varepsilon_t
}_{\text{Order of the equivalent MA process} ~=~ q + Qs'}
}
\\
\end{align}
$$

with $s > p$ and $s' > q$.

$ARMA(p,q) \times ARMA(P,Q)_{s, s'}$ can be defined as an $ARMA(p + Ps, ~ q + Qs')$ with some parameters fixed to 0.

Remark: most of the time, $s = s'$. In this case, the process is simply noted $ARMA(\color{\red}{p}, \color{\green}{q}) \times ARMA(\color{\red}{P}, \color{\green}{Q})_{s}$

#### Example: $ARMA(3,0) \times ARMA(2,0)_{6, 0}$

$$
\begin{align}
ARMA(\color{\red}{3},0) \times ARMA(\color{\pink}{2},0)_{6,0}: \quad
\color{\red}{
\underbrace{\varphi_{(3)} (B)}_{AR} ~
}
\color{\pink}{
\underbrace{\Phi_{(2)} (B^6)}_{SAR} ~
}
y_t
=~&
c + \varepsilon_t
\\
\color{\red}{
\underbrace{\left( 1 + \sum_{i=1}^3 \varphi_i B^i \right)}_{AR} ~
}
\color{\pink}{
\underbrace{\left( 1 + \sum_{k=1}^2 \Phi_k B^{6k} \right)}_{SAR} ~
}
y_t
=~&
c + \varepsilon_t
\\
\underbrace{
    \color{\red}{
    \left( 1 + \varphi_1 B + \varphi_2 B^2 + \varphi_3 B^3 \right)
    }
\color{\pink}{
    \left( 1 + \Phi_1 B^{6} + \Phi_2 B^{12} \right)
    }
y_t
}_{\text{Order of the equivalent AR process} ~=~ 3 + 2 \times 6 = 15}
=~&
c + \varepsilon_t
\\
\left(
1 + \Phi_1 B^{6} + \Phi_2 B^{12}
+ \color{\red}{\varphi_1} B    + \color{\red}{\varphi_1} \color{\pink}{\Phi_1} B^{7} + \color{\red}{\varphi_1} \color{\pink}{\Phi_2} B^{13}
+ \color{\red}{\varphi_2} B^2  + \color{\red}{\varphi_2} \color{\pink}{\Phi_1} B^{8} + \color{\red}{\varphi_2} \color{\pink}{\Phi_2} B^{14}
+ \color{\red}{\varphi_3} B^3  + \color{\red}{\varphi_3} \color{\pink}{\Phi_1} B^{9} + \color{\red}{\varphi_3} \color{\pink}{\Phi_2} B^{15}
\right)
y_t
=~&
c + \varepsilon_t
\\
y_t =
  & ~ c \\
  & + \color{\red}{\phi_1} y_{t-1} + \color{\red}{\phi_2} y_{t-2} + \color{\red}{\phi_3} y_{t-3} \\
  & + \color{\pink}{\varPhi_1}   y_{t-6} + \color{\red}{\phi_1} \color{\pink}{\varPhi_1} y_{t-7}  + \color{\red}{\phi_2} \color{\pink}{\varPhi_1}  y_{t-8} + \color{\red}{\phi_3} \color{\pink}{\varPhi_1} y_{t-9} \\
  & + \color{\pink}{\varPhi_2}  y_{t-12} + \color{\red}{\phi_1} \color{\pink}{\varPhi_2} y_{t-13} + \color{\red}{\phi_2} \color{\pink}{\varPhi_2} y_{t-14} + \color{\red}{\phi_3} \color{\pink}{\varPhi_2} y_{t-15} \\
  & + \varepsilon_t
\end{align}
$$

<center></center>

#### Example: $ARMA(3,2) \times ARMA(2,1)_{6, 12}$

$$
\begin{align}
ARMA(\color{\red}{3},\color{\green}{2}) \times ARMA(\color{\red}{2},\color{\green}{1})_{\color{\red}{6},\color{\green}{12}}: \quad
\color{\red}{
\underbrace{\varphi_{(3)} (B)}_{AR} ~
\underbrace{\Phi_{(2)} (B^6)}_{SAR} ~
y_t
}
=~&
c +
\color{\green}{
\underbrace{\theta_{(2)} (B)}_{MA} ~
\underbrace{\Theta_{(1)} (B^{12})}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{\left( 1 + \sum_{i=1}^3 \varphi_i B^i \right)}_{AR} ~
\underbrace{\left( 1 + \sum_{k=1}^2 \Phi_k B^{6k} \right)}_{SAR} ~
y_t
}
=~&
c +
\color{\green}{
\underbrace{\left( 1 + \sum_{j=1}^2 \theta_j B^j \right)}_{MA} ~
\underbrace{\left( 1 + \sum_{l=1}^1 \Theta_l B^{12l} \right)}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{
\left( 1 + \varphi_1 B + \varphi_2 B^2 + \varphi_3 B^3 \right)
\left( 1 + \Phi_1 B^{6} + \Phi_2 B^{12} \right)
y_t
}_{\text{Order of the equivalent AR process} ~=~ 3 + 2 \times 6 = 15}
}
=~&
c +
\color{\green}{
\underbrace{
\left( 1 + \theta_1 B + \theta_2 B^2 \right)
\left( 1 + \Theta_1 B^{12} \right)
\varepsilon_t
}_{\text{Order of the equivalent MA process} ~=~ 2 + 1 \times 12 = 14}
}
%
%\\
%\left(
%1 + \Phi_1 B^{6} + \Phi_2 B^{12}
%+ \varphi_1 B    + \varphi_1 B   \times \Phi_1 B^{6} + \varphi_1 B   \times \Phi_2 B^{12}
%+ \varphi_2 B^2  + \varphi_2 B^2 \times \Phi_1 B^{6} + \varphi_2 B^2 \times \Phi_2 B^{12}
%+ \varphi_3 B^3  + \varphi_3 B^3 \times \Phi_1 B^{6} + \varphi_3 B^3 \times \Phi_2 B^{12}
%\right)
%y_t
%=~&
%c +
%\left(
%1 + \Theta_1 B^{12}
%+ \theta_1 B   + \theta_1 B   \times \Theta_1 B^{12}
%+ \theta_2 B^2 + \theta_2 B^2 \times \Theta_1 B^{12}
%\right)
%\varepsilon_t
%
\\
\color{\red}{
\left(
1 + \Phi_1 B^{6} + \Phi_2 B^{12}
+ \varphi_1 B    + \varphi_1 \Phi_1 B^{7} + \varphi_1 \Phi_2 B^{13}
+ \varphi_2 B^2  + \varphi_2 \Phi_1 B^{8} + \varphi_2 \Phi_2 B^{14}
+ \varphi_3 B^3  + \varphi_3 \Phi_1 B^{9} + \varphi_3 \Phi_2 B^{15}
\right)
y_t
}
=~&
c +
\color{\green}{
\left(
1 + \Theta_1 B^{12}
+ \theta_1 B   + \theta_1 \Theta_1 B^{13}
+ \theta_2 B^2 + \theta_2 \Theta_1 B^{14}
\right)
\varepsilon_t
}
\\
\color{\red}{y_t} =
  & ~ c \\
  & \color{\red}{+ \varPhi_1 y_{t-1} + \varPhi_2 y_{t-2} + \varPhi_3 y_{t-3}} \\
  & \color{\red}{+ \phi_1 y_{t-6} + \varPhi_1 \phi_1 y_{t-7} + \varPhi_2 \phi_1 y_{t-8} + \varPhi_3 \phi_1 y_{t-9}} \\
  & \color{\red}{+ \phi_2 y_{t-12} + \varPhi_1 \phi_2 y_{t-13} + \varPhi_2 \phi_2 y_{t-14} + \varPhi_3 \phi_2 y_{t-15}} \\
  & \color{\green}{+ \varepsilon_t} \\
  & \color{\green}{+ \theta_1 \varepsilon_{t-1} + \theta_2 \varepsilon_{t-2}} \\
  & \color{\green}{+ \Theta_1 \varepsilon_{t-12} + \theta_1 \Theta_1 \varepsilon_{t-13} + \theta_2 \Theta_1 \varepsilon_{t-14}}
\end{align}
$$

### Seasonal AutoRegressive Moving Average (SARIMA) : $ARIMA(P,D,Q)_{s,s'',s'}$ with seasonal differencing

$$
\begin{align}
ARIMA(\color{\red}{P}, \color{\orange}{D}, \color{\green}{Q})_{\color{\red}{s}, \color{\orange}{s''}, \color{\green}{s'}}: \quad\quad\quad
\color{\red}{\underbrace{\Phi_{(P)} (B^s) \color{\orange}{\nabla^D_{s''}} y_t}_{\text{SARI}}}
&=
c + \color{\green}{\underbrace{\Theta_{(Q)} (B^{s'}) \varepsilon_t}_{SMA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^P \Phi_i B^{si} \right) \color{\orange}{\nabla^D_{s''}} y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^Q \Theta_j B^{s'j} \right) ~ \varepsilon_t}
\\
\color{\orange}{\nabla^D_{s''}} \color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^P \Phi_i B^{si} \color{\orange}{\nabla^D_{s''}} y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^Q \Theta_j B^{s'j} \varepsilon_t}
\\
\color{\orange}{\nabla^D_{s''}} \color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^P \Phi_i ~ \color{\orange}{\nabla^D_{s''}} y_{t-si} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^Q \Theta_j ~ \varepsilon_{t-s'j}}
\\
\color{\orange}{\nabla^D_{s''}} \color{\red}{y_t} &= c
+ \color{\red}{\underbrace{\sum_{i=1}^P \varPhi_i ~ \color{\orange}{\nabla^D_{s''}} y_{t-si}}_{\text{SARI model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^Q \Theta_j ~ \varepsilon_{t-s'j}}_{\text{SMA model}}}
\end{align}
$$

with

$$\nabla_{s''} = (1-B^{s''})$$
$$\nabla_{s''}^2 = (1-B^{s''})^2$$
$$\dots$$
$$\nabla_{s''}^d = (1-B^{s''})^d$$

Remark: most of the time, $s = s' = s''$. In this case, the process is simply noted $ARIMA(\color{\red}{P}, \color{\orange}{D}, \color{\green}{Q})_{s}$

#### Example: $ARIMA(3, 1, 2)_{24, 24, 24}$

$$
\begin{align}
ARIMA(\color{\red}{3}, \color{\orange}{1}, \color{\green}{2})_{\color{\red}{24}, \color{\orange}{24}, \color{\green}{24}}: \quad\quad\quad
\color{\red}{\underbrace{\Phi_{(3)} (B^{24}) \color{\orange}{\nabla_{24}} y_t}_{\text{SARI}}}
&=
c + \color{\green}{\underbrace{\Theta_{(2)} (B^{24}) \varepsilon_t}_{SMA}}
\\
\color{\red}{ \left( 1 + \sum_{i=1}^3 \Phi_i B^{24i} \right) \color{\orange}{\nabla_{24}} y_t }
&=
c + \color{\green}{\left( 1 + \sum_{j=1}^2 \Theta_j B^{24j} \right) ~ \varepsilon_t}
\\
\color{\orange}{\nabla_{24}} \color{\red}{ y_t } +  \color{\red}{\sum_{i=1}^3 \Phi_i B^{24i} \color{\orange}{\nabla_{24}} y_t }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \Theta_j B^{24j} \varepsilon_t}
\\
\color{\orange}{\nabla_{24}} \color{\red}{ y_t } + \color{\red}{ \sum_{i=1}^3 \Phi_i ~ \color{\orange}{\nabla_{24}} y_{t-24i} }
&=
c + \color{\green}{\varepsilon_t} + \color{\green}{\sum_{j=1}^2 \Theta_j ~ \varepsilon_{t-24j}}
\\
\color{\orange}{\nabla_{24}} \color{\red}{y_t} &= c
+ \color{\red}{\underbrace{\sum_{i=1}^3 \varPhi_i ~ \color{\orange}{\nabla_{24}} y_{t-24i}}_{\text{SARI model}}}
+ \color{\green}{\varepsilon_t}
+ \color{\green}{\underbrace{\sum_{j=1}^2 \Theta_j ~ \varepsilon_{t-24j}}_{\text{SMA model}}}
\\
\color{\orange}{\nabla_{24}} \color{\red}{y_t}
&=
c
+ \underbrace{
      \color{\red}{\varPhi_1} \color{\orange}{\nabla_{24}} \color{\red}{y_{t-24}}
    + \color{\red}{\varPhi_2} \color{\orange}{\nabla_{24}} \color{\red}{y_{t-48}}
    + \color{\red}{\varPhi_3} \color{\orange}{\nabla_{24}} \color{\red}{y_{t-72}}
  }_{\text{SARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\Theta_1 \varepsilon_{t-24}} + \color{\green}{\Theta_2 \varepsilon_{t-48}}}_{\text{MA model}}
\\
\color{\orange}{(1 - B^{24})} \color{\red}{y_t}
&=
c
+ \underbrace{
      \color{\red}{\varPhi_1} \color{\orange}{(1 - B^{24})} \color{\red}{y_{t-24}}
    + \color{\red}{\varPhi_2} \color{\orange}{(1 - B^{24})} \color{\red}{y_{t-48}}
    + \color{\red}{\varPhi_3} \color{\orange}{(1 - B^{24})} \color{\red}{y_{t-72}}
  }_{\text{SARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\Theta_1 \varepsilon_{t-24}} + \color{\green}{\Theta_2 \varepsilon_{t-48}}}_{\text{MA model}}
\\
\color{\red}{y_t} - \color{\orange}{B^{24} y_t}
&=
c
+ \underbrace{
      \color{\red}{\varPhi_1} (\color{\red}{y_{t-24}} - \color{\orange}{B^{24} y_{t-24}})
    + \color{\red}{\varPhi_2} (\color{\red}{y_{t-48}} - \color{\orange}{B^{24} y_{t-48}})
    + \color{\red}{\varPhi_3} (\color{\red}{y_{t-72}} - \color{\orange}{B^{24} y_{t-72}})
  }_{\text{SARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\Theta_1 \varepsilon_{t-24}} + \color{\green}{\Theta_2 \varepsilon_{t-48}}}_{\text{MA model}}
\\
\color{\red}{y_t} - \color{\orange}{y_{t-24}}
&=
c
+ \underbrace{
      \color{\red}{\varPhi_1} (\color{\red}{y_{t-24}} - \color{\orange}{y_{t-48}})
    + \color{\red}{\varPhi_2} (\color{\red}{y_{t-48}} - \color{\orange}{y_{t-72}})
    + \color{\red}{\varPhi_3} (\color{\red}{y_{t-72}} - \color{\orange}{y_{t-96}})
  }_{\text{SARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\Theta_1 \varepsilon_{t-24}} + \color{\green}{\Theta_2 \varepsilon_{t-48}}}_{\text{MA model}}
\\
\color{\red}{y_t}
&=
c
+ \color{\orange}{y_{t-24}}
+ \underbrace{
      \color{\red}{\varPhi_1 y_{t-24}} - \color{\orange}{\varPhi_1 y_{t-48}}
    + \color{\red}{\varPhi_2 y_{t-48}} - \color{\orange}{\varPhi_2 y_{t-72}}
    + \color{\red}{\varPhi_3 y_{t-72}} - \color{\orange}{\varPhi_3 y_{t-96}}
  }_{\text{SARI model}}
+ \color{\green}{\varepsilon_t}
+ \underbrace{\color{\green}{\Theta_1 \varepsilon_{t-24}} + \color{\green}{\Theta_2 \varepsilon_{t-48}}}_{\text{MA model}}
\end{align}
$$

### Seasonal AutoRegressive Moving Average (SARIMA) : $ARIMA(p,d,q) \times ARIMA(P,D,Q)_{s,s'',s'}$ with seasonal and non-seasonal differencing

$$
\begin{align}
ARIMA(\color{\red}{p}, \color{\orange}{d}, \color{\green}{q}) \times ARIMA(\color{\red}{P}, \color{\orange}{D}, \color{\green}{Q})_{\color{\red}{s}, \color{\orange}{s'}, \color{\green}{s''}}: \quad\quad\quad
\color{\red}{
\underbrace{\varphi_{(p)} (B)}_{AR} ~
\underbrace{\Phi_{(P)} (B^s)}_{SAR} ~
}
\color{\orange}{\nabla^D_{s''}} \color{\orange}{\nabla^d}
\color{\red}{y_t}
&=
c +
\color{\green}{
\underbrace{\theta_{(q)} (B)}_{MA} ~
\underbrace{\Theta_{(Q)} (B^{s'})}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{\left( 1 + \sum_{i=1}^p \varphi_i B^i \right)}_{AR} ~
\underbrace{\left( 1 + \sum_{k=1}^P \Phi_k B^{sk} \right)}_{SAR} ~
}
\color{\orange}{\nabla^D_{s''}} \color{\orange}{\nabla^d}
\color{\red}{y_t}
&=
c +
\color{\green}{
\underbrace{\left( 1 + \sum_{j=1}^q \theta_j B^j \right)}_{MA} ~
\underbrace{\left( 1 + \sum_{l=1}^Q \Theta_l B^{s'l} \right)}_{SMA} ~
\varepsilon_t
}
\\
\underbrace{
\color{\red}{
\left( 1 + \varphi_1 B + \varphi_2 B^2 + \dots + \varphi_p B^p \right)
\left( 1 + \Phi_1 B^{s} + \Phi_2 B^{s2} + \dots + \Phi_P B^{sP} \right)
}
\color{\orange}{\nabla^D_{s''}} \color{\orange}{\nabla^d}
\color{\red}{y_t}
}
&=
c +
\underbrace{
\color{\green}{
\left( 1 + \theta_1 B + \theta_2 B^2 + \dots + \theta_q B^q \right)
\left( 1 + \Theta_1 B^{s'} + \Theta_2 B^{s'2} + \dots + \Theta_Q B^{s'Q} \right)
\varepsilon_t
}
}
\\
\end{align}
$$

Remark: $\color{\orange}{\nabla^D_{s''}} \color{\orange}{\nabla^d} y_t = \color{\orange}{\nabla^d} \color{\orange}{\nabla^D_{s''}} y_t$

#### Example: $ARMA(3,1,2) \times ARMA(2,1,1)_{24, 24, 24}$

$$
\begin{align}
ARIMA(\color{\red}{3}, \color{\orange}{1}, \color{\green}{2}) \times ARIMA(\color{\red}{2}, \color{\orange}{1}, \color{\green}{1})_{\color{\red}{24}, \color{\orange}{24}, \color{\green}{24}}: \quad\quad\quad
\color{\red}{
\underbrace{\varphi_{(3)} (B)}_{AR} ~
\underbrace{\Phi_{(2)} \left( B^{24} \right)}_{SAR} ~
}
\color{\orange}{\nabla_{24}} \color{\orange}{\nabla}
\color{\red}{y_t}
&=
c +
\color{\green}{
\underbrace{\theta_{(2)} (B)}_{MA} ~
\underbrace{\Theta_{(1)} \left( B^{24} \right)}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\underbrace{\left( 1 + \sum_{i=1}^3 \varphi_i B^i \right)}_{AR} ~
\underbrace{\left( 1 + \sum_{k=1}^2 \Phi_k B^{24 k} \right)}_{SAR} ~
}
\color{\orange}{\nabla_{24}} \color{\orange}{\nabla}
\color{\red}{y_t}
&=
c +
\color{\green}{
\underbrace{\left( 1 + \sum_{j=1}^2 \theta_j B^j \right)}_{MA} ~
\underbrace{\left( 1 + \sum_{l=1}^1 \Theta_l B^{24 l} \right)}_{SMA} ~
\varepsilon_t
}
\\
\color{\red}{
\left( 1 + \varphi_1 B + \varphi_2 B^2 + \varphi_3 B^3 \right)
\left( 1 + \Phi_1 B^{24} + \Phi_2 B^{48} \right)
}
\color{\orange}{\nabla_{24}} \color{\orange}{\nabla}
\color{\red}{y_t}
&=
c +
\color{\green}{
\left( 1 + \theta_1 B + \theta_2 B^2 \right)
\left( 1 + \Theta_1 B^{24} \right)
\varepsilon_t
}
\\
\color{\red}{
\left(
1 + \Phi_1 B^{24} + \Phi_2 B^{48}
+ \varphi_1 B    + \varphi_1 \Phi_1 B^{25} + \varphi_1 \Phi_2 B^{49}
+ \varphi_2 B^2  + \varphi_2 \Phi_1 B^{26} + \varphi_2 \Phi_2 B^{50}
+ \varphi_3 B^3  + \varphi_3 \Phi_1 B^{27} + \varphi_3 \Phi_2 B^{51}
\right)
}
\color{\orange}{\nabla_{24}} \color{\orange}{\nabla}
\color{\red}{y_t}
=~&
c +
\color{\green}{
\left(
1 + \Theta_1 B^{24}
+ \theta_1 B   + \theta_1 \Theta_1 B^{25}
+ \theta_2 B^2 + \theta_2 \Theta_1 B^{26}
\right)
\varepsilon_t
}
\\
\color{\red}{y_t} =
  & ~ c \\
  & \color{\red}{+ \varPhi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-1} + \varPhi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-2} + \varPhi_3 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-3}} \\
  & \color{\red}{+ \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-24}  + \varPhi_1 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-25} + \varPhi_2 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-26} + \varPhi_3 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-27}} \\
  & \color{\red}{+ \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-48} + \varPhi_1 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-49} + \varPhi_2 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-50} + \varPhi_3 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-51}} \\
  & \color{\green}{+ \varepsilon_t} \\
  & \color{\green}{+ \theta_1 \varepsilon_{t-1} + \theta_2 \varepsilon_{t-2}} \\
  & \color{\green}{+ \Theta_1 \varepsilon_{t-24} + \theta_1 \Theta_1 \varepsilon_{t-25} + \theta_2 \Theta_1 \varepsilon_{t-26}}
\\
\end{align}
$$

As we have:

$$
\begin{align}
\nabla                  &= (1-B) \\
\nabla_{24}             &= (1-B^{24}) \\
\nabla \nabla_{24}      &= (1-B)(1-B^{24}) \\
                        &= 1-B^{24}-B+B^{25} \\
\nabla \nabla_{24} y_t  &= (1-B^{24}-B+B^{25}) y_t \\
                        &= y_t - B^{24} y_t - B y_t   + B^{25} y_t \\
                        &= y_t - y_{t-24}   - y_{t-1} + y_{t-25} \\
\end{align}
$$

Thus:

$$
\begin{align}
\color{\red}{y_t} =
  & ~ c \\
  & \color{\red}{+ \varPhi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-1} + \varPhi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-2} + \varPhi_3 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-3}} \\
  & \color{\red}{+ \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-24}  + \varPhi_1 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-25} + \varPhi_2 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-26} + \varPhi_3 \phi_1 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-27}} \\
  & \color{\red}{+ \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-48} + \varPhi_1 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-49} + \varPhi_2 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-50} + \varPhi_3 \phi_2 \color{\orange}{\nabla_{24}} \color{\orange}{\nabla} y_{t-51}} \\
  & \color{\green}{+ \varepsilon_t} \\
  & \color{\green}{+ \theta_1 \varepsilon_{t-1} + \theta_2 \varepsilon_{t-2}} \\
  & \color{\green}{+ \Theta_1 \varepsilon_{t-24} + \theta_1 \Theta_1 \varepsilon_{t-25} + \theta_2 \Theta_1 \varepsilon_{t-26}}
\\
= & ...
\\
\end{align}
$$


# Determine derivative of Jacobian from angular velocity to exponential rates

Peter Corke 2021

SymPy code to deterine the time derivative of the mapping from angular velocity to exponential coordinate rates.

```python
from sympy import *
```

A rotation matrix can be expressed in terms of exponential coordinates (also called Euler vector)

$
\mathbf{R} = e^{[\varphi]_\times}
$
where $\mathbf{R} \in SO(3)$ and $\varphi \in \mathbb{R}^3$.

The mapping from angular velocity $\omega$ to exponential coordinate rates $\dot{\varphi}$ is

$
\dot{\varphi} = \mathbf{A} \omega
$

where $\mathbf{A}$ is given by (2.107) of [Robot Dynamics Lecture Notes, Robotic Systems Lab, ETH Zurich, 2018](https://ethz.ch/content/dam/ethz/special-interest/mavt/robotics-n-intelligent-systems/rsl-dam/documents/RobotDynamics2018/RD_HS2018script.pdf)

$
\mathbf{A} = I_{3 \times 3} - \frac{1}{2} [v]_\times + [v]^2_\times \frac{1}{\theta^2} \left( 1 - \frac{\theta}{2} \frac{\sin \theta}{1 - \cos \theta} \right)
$
where $\theta = \| \varphi \|$ and $v = \hat{\varphi}$

We simplify the equation as

$
\mathbf{A} = I_{3 \times 3} - \frac{1}{2} [v]_\times + [v]^2_\times \Theta
$

where
$
\Theta = \frac{1}{\theta^2} \left( 1 - \frac{\theta}{2} \frac{\sin \theta}{1 - \cos \theta} \right)
$

We can find the derivative using the chain rule

$
\dot{\mathbf{A}} = - \frac{1}{2} [\dot{v}]_\times + 2 [v]_\times [\dot{v}]_\times \Theta + [v]^2_\times \dot{\Theta}
$

We start by defining some symbols

```python
Theta, theta, theta_dot, t = symbols('Theta theta theta_dot t', real=True)
```

We start by finding an expression for $\Theta$ which depends on $\theta(t)$

```python
theta_t = Function(theta)(t)
```

```python
Theta = 1 / theta_t ** 2 * (1 - theta_t / 2 * sin(theta_t) / (1 - cos(theta_t)))
Theta
```

$\displaystyle \frac{1 - \frac{\theta{\left(t \right)} \sin{\left(\theta{\left(t \right)} \right)}}{2 \left(1 - \cos{\left(\theta{\left(t \right)} \right)}\right)}}{\theta^{2}{\left(t \right)}}$

and now determine the derivative

```python
T_dot = Theta.diff(t)
T_dot
```

$\displaystyle - \frac{2 \left(1 - \frac{\theta{\left(t \right)} \sin{\left(\theta{\left(t \right)} \right)}}{2 \left(1 - \cos{\left(\theta{\left(t \right)} \right)}\right)}\right) \frac{d}{d t} \theta{\left(t \right)}}{\theta^{3}{\left(t \right)}} + \frac{- \frac{\theta{\left(t \right)} \cos{\left(\theta{\left(t \right)} \right)} \frac{d}{d t} \theta{\left(t \right)}}{2 \left(1 - \cos{\left(\theta{\left(t \right)} \right)}\right)} - \frac{\sin{\left(\theta{\left(t \right)} \right)} \frac{d}{d t} \theta{\left(t \right)}}{2 \left(1 - \cos{\left(\theta{\left(t \right)} \right)}\right)} + \frac{\theta{\left(t \right)} \sin^{2}{\left(\theta{\left(t \right)} \right)} \frac{d}{d t} \theta{\left(t \right)}}{2 \left(1 - \cos{\left(\theta{\left(t \right)} \right)}\right)^{2}}}{\theta^{2}{\left(t \right)}}$

which is a somewhat complex expression that depends on $\theta(t)$ and $\dot{\theta}(t)$.

We will remove the time dependency and generate code

```python
T_dot = T_dot.subs([(theta_t.diff(t), theta_dot), (theta_t, theta)])
```

```python
pycode(T_dot)
```

    '(-1/2*theta*theta_dot*math.cos(theta)/(1 - math.cos(theta)) + (1/2)*theta*theta_dot*math.sin(theta)**2/(1 - math.cos(theta))**2 - 1/2*theta_dot*math.sin(theta)/(1 - math.cos(theta)))/theta**2 - 2*theta_dot*(-1/2*theta*math.sin(theta)/(1 - math.cos(theta)) + 1)/theta**3'

In order to evaluate the line above we need an expression for $\theta$ and $\dot{\theta}$.  $\theta$ is the norm of $\varphi$ whose elements are functions of time

```python
phi_names = ('varphi_0', 'varphi_1', 'varphi_2')
phi = []  # names of angles, eg. theta
phi_t = []  # angles as function of time, eg. theta(t)
phi_d = []  # derivative of above, eg. d theta(t) / dt
phi_n = []  # symbol to represent above, eg. theta_dot
for i in phi_names:
    phi.append(symbols(i, real=True))
    phi_t.append(Function(phi[-1])(t))
    phi_d.append(phi_t[-1].diff(t))
    phi_n.append(i + '_dot')
```

Compute the norm

```python
theta = Matrix(phi_t).norm()
theta
```

$\displaystyle \sqrt{\varphi_{0}^{2}{\left(t \right)} + \varphi_{1}^{2}{\left(t \right)} + \varphi_{2}^{2}{\left(t \right)}}$

and find its derivative

```python
theta_dot = theta.diff(t)
theta_dot
```

$\displaystyle \frac{\varphi_{0}{\left(t \right)} \frac{d}{d t} \varphi_{0}{\left(t \right)} + \varphi_{1}{\left(t \right)} \frac{d}{d t} \varphi_{1}{\left(t \right)} + \varphi_{2}{\left(t \right)} \frac{d}{d t} \varphi_{2}{\left(t \right)}}{\sqrt{\varphi_{0}^{2}{\left(t \right)} + \varphi_{1}^{2}{\left(t \right)} + \varphi_{2}^{2}{\left(t \right)}}}$

and now remove the time dependenices

```python
theta_dot = theta_dot.subs(a for a in zip(phi_d, phi_n))
theta_dot = theta_dot.subs(a for a in zip(phi_t, phi))
theta_dot
```

$\displaystyle \frac{\varphi_{0} \varphi_{0 dot} + \varphi_{1} \varphi_{1 dot} + \varphi_{2} \varphi_{2 dot}}{\sqrt{\varphi_{0}^{2} + \varphi_{1}^{2} + \varphi_{2}^{2}}}$

which is simply the dot product over the norm.


# Tutorial

We will solve the following problem using a computer to assist with the technical aspects:

```{admonition} Problem

Consider the function $f(x)= \frac{24 x \left(a - 4 x\right) + 2 \left(a - 8 x\right) \left(b - 4 x\right)}{\left(b - 4 x\right)^{4}}$

1. Given that $\frac{df}{dx}|_{x=0}=0$, $\frac{d^2f}{dx^2}|_{x=0}=-1$ and that $b>0$ find the values of $a$ and $b$.
2. For the specific values of $a$ and $b$ find:
    1. $\lim_{x\to 0}f(x)$;
    2. $\lim_{x\to \infty}f(x)$;
    3. $\int f(x) dx$;
    4. $\int_{5}^{20} f(x) dx$.

```

Sympy is once again the library we will use for this.

We will start by creating a variable `expression` that has the value of the expression of $f(x)$:

```python
import sympy as sym

x = sym.Symbol("x")
a = sym.Symbol("a")
b = sym.Symbol("b")
expression = (24 * x * (a - 4 * x) + 2 * (a - 8 * x) * (b - 4 * x)) / ((b - 4 * x) ** 4)
expression
```

$\displaystyle \frac{24 x \left(a - 4 x\right) + \left(2 a - 16 x\right) \left(b - 4 x\right)}{\left(b - 4 x\right)^{4}}$

now we can will use `sympy.diff` to calculate the derivative. This tool takes two inputs:

- the first is the expression we are differentiating. Essentially this is the numerator of $\frac{df}{dx}$.
- the first is the variable we are differentiating for. Essentially this is the denominator of $\frac{df}{dx}$.

```{attention}
We have imported `import sympy as sym` so we are going to write `sym.diff`:
```

```python
derivative = sym.diff(expression, x)
derivative
```

$\displaystyle \frac{16 a - 16 b - 64 x}{\left(b - 4 x\right)^{4}} + \frac{16 \left(24 x \left(a - 4 x\right) + \left(2 a - 16 x\right) \left(b - 4 x\right)\right)}{\left(b - 4 x\right)^{5}}$

Let us factorise that to make it slightly clearer:

```python
sym.factor(derivative)
```

$\displaystyle \frac{16 \left(- 3 a b - 12 a x + b^{2} + 16 b x + 16 x^{2}\right)}{\left(- b + 4 x\right)^{5}}$

We will now create the first equation, which is obtained by substituting $x=0$
in to the value of the derivative and equating that to $0$:

```python
first_equation = sym.Eq(derivative.subs({x: 0}), 0)
first_equation
```

$\displaystyle \frac{32 a}{b^{4}} + \frac{16 a - 16 b}{b^{4}} = 0$

We will factor that equation:

```python
sym.factor(first_equation)
```

$\displaystyle \frac{16 \left(3 a - b\right)}{b^{4}} = 0$

Now we are going to create the second equation, substituting $x=0$ in to the
value of the second derivative. We calculate the second derivative by passing a
third (optional) input to `sym.diff`:

```python
second_derivative = sym.diff(expression, x, 2)
second_derivative
```

$\displaystyle \frac{64 \left(-1 - \frac{8 \left(- a + b + 4 x\right)}{b - 4 x} + \frac{10 \left(12 x \left(a - 4 x\right) + \left(a - 8 x\right) \left(b - 4 x\right)\right)}{\left(b - 4 x\right)^{2}}\right)}{\left(b - 4 x\right)^{4}}$

We equate this expression to $-1$:

```python
second_equation = sym.Eq(second_derivative.subs({x: 0}), -1)
second_equation
```

$\displaystyle \frac{64 \left(\frac{10 a}{b} - 1 - \frac{8 \left(- a + b\right)}{b}\right)}{b^{4}} = -1$

Now to solve the first equation to obtain a value for $a$:

```python
sym.solveset(first_equation, a)
```

$\displaystyle \left\{\frac{b}{3}\right\}$

Now to substitute that value for $a$ and solve the second equation for $b$:

```python
second_equation = second_equation.subs({a: b / 3})
second_equation
```

$\displaystyle - \frac{192}{b^{4}} = -1$

```python
sym.solveset(second_equation, b)
```

$\displaystyle \left\{- 2 \sqrt{2} \sqrt[4]{3}, 2 \sqrt{2} \sqrt[4]{3}, - 2 \sqrt{2} \sqrt[4]{3} i, 2 \sqrt{2} \sqrt[4]{3} i\right\}$

Recalling the question we know that $b>0$ thus: $b = 2\sqrt{2}\sqrt[4]{3}$ and
$a=\frac{2\sqrt{2}\sqrt[4]{3}}{3}$.

We will substitute these values back and finish the question:

```python
expression = expression.subs(
    {a: 2 * sym.sqrt(2) * sym.root(3, 4) / 3, b: 2 * sym.sqrt(2) * sym.root(3, 4)}
)
expression
```

$\displaystyle \frac{24 x \left(- 4 x + \frac{2 \sqrt{2} \sqrt[4]{3}}{3}\right) + \left(- 16 x + \frac{4 \sqrt{2} \sqrt[4]{3}}{3}\right) \left(- 4 x + 2 \sqrt{2} \sqrt[4]{3}\right)}{\left(- 4 x + 2 \sqrt{2} \sqrt[4]{3}\right)^{4}}$

```{attention}
We are using the `sym.root` command for the generic $n$th root.
```

We can confirm our findings:

```python
sym.diff(expression, x).subs({x: 0})
```

$\displaystyle 0$

```python
sym.diff(expression, x, 2).subs({x: 0})
```

$\displaystyle -1$

Now we will calculate the limits using `sym.limit`, this takes 3 inputs:

- The expression we are taking the limit of.
- The variable that is changing.
- The value that the variable is tending towards.

```python
sym.limit(expression, x, 0)
```

$\displaystyle \frac{\sqrt{3}}{36}$

```python
sym.limit(expression, x, sym.oo)
```

$\displaystyle 0$

Now we are going to calculate the **indefinite** integral using
`sympy.integrate`. This tool takes 2 inputs as:

- the first is the expression we're integrating. This is the $f$ in $\int_a^b f
  dx$.
- the second is the remaining information needed to calculate the integral: $x$.

```python
sym.factor(sym.integrate(expression, x))
```

$\displaystyle \frac{x \left(6 x - \sqrt{2} \sqrt[4]{3}\right)}{12 \left(4 x^{3} - 6 \sqrt{2} \sqrt[4]{3} x^{2} + 6 \sqrt{3} x - \sqrt{2} \cdot 3^{\frac{3}{4}}\right)}$

If we want to calculate a **definite** integral then instead of passing the
single variable we pass a tuple which contains the variable as well as the
bounds of integration:

```python
sym.factor(sym.integrate(expression, (x, 5, 20)))
```

$\displaystyle - \frac{5 \left(- 5000 \sqrt{2} \sqrt[4]{3} - 1200 \sqrt{3} + 75 \sqrt{2} \cdot 3^{\frac{3}{4}} + 119997\right)}{2 \left(-32000 - 120 \sqrt{3} + \sqrt{2} \cdot 3^{\frac{3}{4}} + 2400 \sqrt{2} \sqrt[4]{3}\right) \left(-500 - 30 \sqrt{3} + \sqrt{2} \cdot 3^{\frac{3}{4}} + 150 \sqrt{2} \sqrt[4]{3}\right)}$


```python
import warnings
warnings.filterwarnings('ignore')

from sympy import sqrt, Matrix, init_printing
init_printing()

from fractions import Fraction

from typing import Iterable

class Vector:
    def __init__(self, *components, dim=None):
        if dim is None:
            if isinstance(components[0], Iterable):
                self.components = tuple(components[0])
            else:
                self.components = components
            self.dim = len(self.components)
        else:
            self.components = tuple(0 for _ in range(dim))
            self.dim = dim

    def __add__(self, other):
        if isinstance(other, int):
            return self
        elif isinstance(other, Vector):
            if self.dim != other.dim:
                raise ValueError(f'Cannot add vector of dimension {self.dim} to vector of dimension {other.dim}')
            return Vector(i + j for i, j in zip(self.components, other.components))
        else:
            raise TypeError(f'Cannot add vector to object of type {type(other)}')

    __radd__ = __add__

    def __sub__(self, other):
        if self.dim != other.dim:
            raise ValueError(f'Cannot subtract vector of dimension {self.dim} from vector of dimension {other.dim}')
        return Vector(i - j for i, j in zip(self.components, other.components))

    def __mul__(self, scalar):
        return Vector(scalar * c for c in self.components)

    __rmul__ = __mul__

    def __truediv__(self, scalar):
        return self * (1 / scalar)

    def dot(self, other) -> int:
        if self.dim != other.dim:
            raise ValueError(f'Cannot dot product vector of dimension {self.dim} with vector of dimension {other.dim}')
        return sum(i * j for i, j in zip(self.components, other.components))

    def __abs__(self):
        return sqrt(self.dot(self))

    def __repr__(self):
        return f"[{', '.join(str(c) for c in self.components)}]"

    @property
    def pretty(self):
        return Matrix(self.components)

```

```python
# Given n vectors, which are a basis for a subspace W. Use the Gram-Schmidt process to find an orthogonal basis for the subspace W.
def gram_schmidt(*vectors):
    basis = []
    for v in vectors:
        if len(basis) == 0:
            basis.append(v)
        else:
            basis.append(v - sum(b * Fraction(v.dot(b), b.dot(b)) for b in basis))
    return basis

```

# Normalize vectors

```python
v1 = Vector(1, -1, 0, 1, 1)
v2 = Vector(-2, 2, 4, 2, 2)
v3 = Vector(3, 3, 0, -3, 3)

[(v1/abs(v1)).pretty, (v2/abs(v2)).pretty, (v3/abs(v3)).pretty]
```

$\displaystyle \left[ \left[\begin{matrix}\frac{1}{2}\\- \frac{1}{2}\\0\\\frac{1}{2}\\\frac{1}{2}\end{matrix}\right], \  \left[\begin{matrix}- \frac{\sqrt{2}}{4}\\\frac{\sqrt{2}}{4}\\\frac{\sqrt{2}}{2}\\\frac{\sqrt{2}}{4}\\\frac{\sqrt{2}}{4}\end{matrix}\right], \  \left[\begin{matrix}\frac{1}{2}\\\frac{1}{2}\\0\\- \frac{1}{2}\\\frac{1}{2}\end{matrix}\right]\right]$

# Distance from vector to vector

```python
u = Vector(12, -1, 1, -8)
v = Vector(11, -3, -4, -5)

abs(u-v)
```

# Distance from vector to plane

```python
y =  Vector(3, -7, 5)
u1 = Vector(-2, -5, 1)
u2 = Vector(-3, 2, 4)

abs(y - (u1 * Fraction(y.dot(u1), u1.dot(u1)) + u2 * Fraction(y.dot(u2), u2.dot(u2))))
```

# Gram-Schmidt Orthogonalization

```python
u1 = Vector(1, -1, -1, 1, 1)
u2 = Vector(4, 1, 6, -6, 4)
u3 = Vector(7, -5, -4, 8, 1)

[v.pretty for v in gram_schmidt(u1, u2, u3)]
```

$\displaystyle \left[ \left[\begin{matrix}1\\-1\\-1\\1\\1\end{matrix}\right], \  \left[\begin{matrix}5\\0\\5\\-5\\5\end{matrix}\right], \  \left[\begin{matrix}3\\0\\2\\2\\-3\end{matrix}\right]\right]$

# QR Decomposition

```python
v1 = Vector(1, -1, 0, 1, 1)
v2 = Vector(-2, 2, 4, 2, 2)
v3 = Vector(2, 3, -1, -2, 3)

A = Matrix([[1, 2, 4],
            [-1, -2, 1],
            [0, 4, 2],
            [1, 6, 3],
            [1, 6, 8]])

Q = Matrix([(v1/abs(v1)).components, (v2/abs(v2)).components, (v3/abs(v3)).components]).T # insert orthonormal vectors as list row vectors and transpose
R = Q.T*A
R
```

$\displaystyle \left[\begin{matrix}2 & 8 & 7\\0 & 4 \sqrt{2} & 3 \sqrt{2}\\0 & 0 & 3 \sqrt{3}\end{matrix}\right]$

# Orthogonal Diagonalization

```python
A = Matrix([[-2, -2, -3],
            [-2, -3, -2],
            [-3, -2, -2]])

A.eigenvects()
```

$\displaystyle \left[ \left( -7, \  1, \  \left[ \left[\begin{matrix}1\\1\\1\end{matrix}\right]\right]\right), \  \left( -1, \  1, \  \left[ \left[\begin{matrix}1\\-2\\1\end{matrix}\right]\right]\right), \  \left( 1, \  1, \  \left[ \left[\begin{matrix}-1\\0\\1\end{matrix}\right]\right]\right)\right]$

```python
u1 = Vector(1, 1, 1)
u2 = Vector(1, -2, 1)
u3 = Vector(-1, 0, 1)

[(u1/abs(u1)).pretty, (u2/abs(u2)).pretty, (u3/abs(u3)).pretty]
```

$\displaystyle \left[ \left[\begin{matrix}\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\\\frac{\sqrt{3}}{3}\end{matrix}\right], \  \left[\begin{matrix}\frac{\sqrt{6}}{6}\\- \frac{\sqrt{6}}{3}\\\frac{\sqrt{6}}{6}\end{matrix}\right], \  \left[\begin{matrix}- \frac{\sqrt{2}}{2}\\0\\\frac{\sqrt{2}}{2}\end{matrix}\right]\right]$


## Lie groups and representations

A (complex) **Lie group** is a finite dimensional smooth (complex) manifold $G$ together with a group structure on $G$, such that the multiplication $G \times G \rightarrow G$ and the attaching of an inverse $g \mapsto g^1:G \rightarrow G$ are smooth maps.

A **matrix Lie** group is any subgroup $H$ of $GL(n;C)$. $H$ is a closed subset: if $A_n$ is any sequence of matrices in $H$, and $A_n$ converges to some matrix $A$, then either $A \in H$, or $A$ is not invertible.

### Counterexamples.
Set of all $n \times n$ invertible matrices all of whose entries are rational. This is in fact a subgroup of $GL(n;\mathbb{C})$, but not a closed subgroup.
Subgroup of $GL(n;\mathbb{C})$ : Let $a$ be an irrational real number, and let
$H =
\begin{Bmatrix}
\begin{pmatrix}
e^{it} & 0\\
0 & e^{ita}
\end{pmatrix}
\vert t \in \mathbb{R}
\end{Bmatrix}
$

We can find a sequence wich converges to $-I$, but it does not belong to $H$. The closure of H is a 2 dimensional torus: $H =
\begin{Bmatrix}
\begin{pmatrix}
e^{it} & 0\\
0 & e^{is}
\end{pmatrix}
\vert t,s \in \mathbb{R}
\end{Bmatrix}
$

### Examples.
General linear groups: $GL(n;\mathbb{R}) \lt GL(n;\mathbb{C})$

Special linear groups: $SL(n;\mathbb{R})$ and $SL(n;\mathbb{C})$, matrices having determinant one.

orthogonal and special orthogonal groups: $O(n), SO(n)$. real matrices
> column vectors that make up $A$ are orthonormal, that is, $\sum_{i=1}^nA_{ij}A_{ik} = \delta_{jk}$
> $\langle x,y \rangle = \langle Ax,Ay \rangle$
> $A^{tr} A = I$
$det(A) = \pm 1$

unitary and special unitary groups, $U(n)$ and $SU(n)$: $A^\dagger A = I$
$\vert det(A) \vert = 1$

complex orthogonal groups, $O(n;\mathbb{C})$ and $SO(n;\mathbb{C})$

generalized orthogonal $O(n;k)$
> Define a symmetric bilinear form $[...]_{n+k}$ on $\mathbb{R}^{n+k}$: $$[x,y]_{n,k} = x_1 y_1 + ... + x_n y_n - x_{n+1} y_{n+1} - ... - x_{n+k} y_{n+k}$$
> $A \in O(n;k) \Longrightarrow [Ax,Ay]=[x,y] for all x,y \in \mathbb{R}^{n+k}$
> $A^{tr} g A = g$ where $g$ is diagonal matrix (metric tensor) with 1 in first $n$ positions and -1 in last $k$.
$det(A) = \pm 1$

Lorentz group: $O(3; 1)$

symplectic groups $Sp(n;\mathbb{R})$, $Sp(n;\mathbb{C})$, and $Sp(n)$
> Consider the skew-symmetric bilinear form $B$ on $\mathbb{R}^{2n}$ defined as: $B[x,y] = \sum_{i=1}^n x_i y_{n+i} - x_{n+i} y_i$
> $B[Ax,Ay] = B[x,y]$ for all $x,y$
> if $J(2n \times 2n) =
\begin{pmatrix}
0 & I\\
-I & 0
\end{pmatrix}$, then $A^{tr} J A = J$
> $Sp(n) = Sp(n;\mathbb{C}) \cap U(2n)$
> $det(A) = 1$

The Heisenberg group $H$:  $A =
\begin{pmatrix}
1 & a & b\\
0 & 1 & c\\
0 & 0 & 1
\end{pmatrix}
$; $A^{-1} =
\begin{pmatrix}
1 & -a & ac-b\\
0 & 1 & -c\\
0 & 0 & 1
\end{pmatrix}
$;
$H$ is subgroup of GL(n;\mathbb{R})$

$\mathbb{R}^* \simeq GL(1;\mathbb{R})$, $\mathbb{C}^* \simeq GL(1;\mathbb{C})$, $S^1 \simeq U(1)$,
$\mathbb{R}^n \simeq \begin{pmatrix}
e^{x_1} &   & 0\\
  & ... &  \\
0 &   & e^{x_n}
\end{pmatrix}$

### Euclidean and Poincar´e groups.

Euclidean group $E(n)$ is by definition the group of all one-to-one, onto, distance-preserving maps of $\mathbb{R}^n$
to itself, that is, maps $f : \mathbb{R}^n \rightarrow \mathbb{R}^n$ such that $d (f (x) , f (y)) = d (x, y)$ for all
$x, y \in \mathbb{R}^n$.$E(n)$ is subgroup of $GL(n+1;\mathbb{R})$. $T = T_x R$ where $R \in O(n)$ and $T_x$ is translation by $x$
$T = \begin{pmatrix}
 &  &  & x_1\\
 & R &  & \vdots\\
 &  &  & x_n\\
0 & ...  & 0 & 1
\end{pmatrix}$

Poincar´e group P(n; 1). affine transformations of $\mathbb{R}^{n+1}$ which preserve the Lorentz distance $d_L(x, y) = (x_1 − y_1)^2 + · · · + (x_n − y_n)^2 − (x_{n+1} − y_{n+1})^2$. $T = T_x R$ where $R \in O(n;1)$

A matrix Lie group $G$ is said to be **compact** if it is closed ($A_n \leftarrow A \in G$) and bounded ($A_{ij} \lt Constant$)

A matrix Lie group $G$ is said to be **connected** if given any two matrices $A$ and $B$ in $G$, there exists a continuous path $A(t), a \le t \le b$, lying in $G$ with $A(a) = A$, and $A(b) = B$.
For matrix Lie groups topological definitions of path-connected and connected are identical.

If $G$ is a matrix Lie group, then the component of $G$ containing the identity is a subgroup of $G$.

A connected matrix Lie group $G$ is said to be simply connected if every loop in $G$ can be shrunk continuously to a point in $G$.
More precisely, $G$ is **simply connected** if given any continuous path $A(t), 0 ≤ t ≤ 1$, lying in $G$ with $A(0) = A(1)$, there exists a continuous function $A(s, t), 0 \le s, t \le 1$, taking values in $G$ with the following properties:
>1) $A(s, 0) = A(s, 1)$ for all $s$
>2) $A(0, t) = A(t)$
>3) $A(1, t) = A(1, 0)$ for all $t$

$$\begin{matrix}
 & Components & Simply \space connected & dim & \\
GL(n;\mathbb{C}) & 1 & no & n^2 & \\
SL(n;\mathbb{C}) & 1  & yes & n^2 & \\
GL(n;\mathbb{R}) & 2 & no & n^2 & \\
SL(n;\mathbb{R}) & 1 & no & n^2-1 & \\
O(n) & 2 &  & \frac{n (n-1)}{2} & \\
SO(n) & 1 & no & \frac{n (n-1)}{2} & \\
U(n) & 1 & no & n^2 & \\
SU(n) & 1 & yes & n^2-1 & \\
O(n;1) & 4 &  &  & \\
SO(n;1) & 2 & n=1 yes; n \gt 1 no &  & \\
Heisenberg & 1 & yes &  & \\
E(n) & 2 &  &  & \\
P(n;1) & 4 &  &  & \\
 &  &  &  &
\end{matrix}$$

Theorem: Every matrix Lie group is a Lie group.

Theorem: Let $G$ and $H$ be Lie groups, and $φ$ a group homomorphism from $G$ to $H$. Then if $φ$ is continuous it is also smooth.

## Lie Algebras and the Exponential Mapping

For any $n \times n$ real or complex matrix $X$, the series $e^X = \sum_{m=0}^\infty \frac{X^m}{m!}$ converges. The matrix exponential $e^X$ is a continuous function of $X$.

$\Vert A \Vert = sup_{x \neq 0} \frac{\Vert Ax \Vert}{\Vert x \Vert}$ $\Leftrightarrow$ $\Vert A \Vert$ is the smallest $\lambda$, $\Vert Ax \Vert \le \lambda \Vert x \Vert$ for $x \in $\mathbb{C}^n$

$\sum_{m=0}^\infty \Vert \frac{X^m}{m!} \Vert \le \sum_{m=0}^\infty \frac{\Vert X \Vert ^m}{m!} = e^{\Vert X \Vert}$. (The sum converges absolutely)

Let $X, Y$ be arbitrary $n \times n$ matrices. Then
> $e^0 = I$
> $(e^X)^{-1} = e^{-X}$
> $e^{(\alpha + \beta)X} = e^{\alpha X} e^{\beta X}$
> if $XY = YX$, then $e^{X+Y} = e^X e^Y = e^y e^x$
> $e^{C X C^{-1}} = C e^X C^{-1}$
> $\Vert e^X \Vert \le e^{\Vert X \Vert}$

$e^{tX}$ is a smooth curve in $\mathbb{C}^{n \times n}$. And $\frac{d}{dt} e^{tX} = X e^{tX} = e^{tX} X$.
And particularly $\left . \frac{d}{dt} \right |_{t=0} e^{tX} = X e^{tX} = e^{tX} X$.

A Jordan–Chevalley decomposition of $X$ is an expression of it as a sum:
$X = X_{ss} + X_n$, where $X_{ss}$ is semisimple, $X_n$ is nilpotent, and $X_{ss}$ and $X_n$ commute. If such a decomposition exists it is unique, and $X_{ss}$ and $X_n$ are in fact expressible as polynomials in $X$.

The function $log(z) = \sum_{m=1}^\infty (-1)^{m+1} \frac{(z-1)^m}{m}$ is defined and analytic in a circle of radius one about $z=1$.
for all $\vert z-1 \vert \lt 1$ $e^{logz} = z$
for all $\vert u \vert \lt log2$, $\vert e^u-1 \vert \lt 1$ and $log e^u = u$.

The function $log(A) = \sum_{m=1}^\infty (-1)^{m+1} \frac{(A-I)^m}{m}$ is defined and continuous on the set of all $n \times n$ complex matrices $A$ with $\Vert A-I \Vert \lt 1$, and $log A$ is real if $A$ is real.
for all $\Vert A-I \Vert \lt 1$ $e^{logA} = A$
for all $\Vert X \Vert \lt log2$, $\Vert e^A-1 \Vert \lt 1$ and $log e^X = X$.

Lie Product Formula (Trotter formula): $e^{X+Y} = \lim_{m \to \infty} (e^{\frac{X}{m}}e^{\frac{Y}{m}})^m$

Theorem: $det(e^X)=e^{tr(X)}$

$A : \mathbb{R} \to GL(n;\mathbb{C})$ is called a one-parameter group if
> $A$ is continuous
> $A(0) = I$
> $A(t+s) = A(t) A(s)$ for all $t,s \in \mathbb{R}$

If $A$ is a one-parameter group in $GL(n;\mathbb{C})$, then there exists unique $X$, such that $A(t) = e^{tX}$

Let $G$ be a matrix Lie group. Then the **Lie algebra** of $G$, denoted $\mathfrak{g}$, is the set of all matrices $X$ such that $e^{tX}$ is in $G$ for all real numbers $t$. (In physics usually $e^{itX}$ is used).

$GL(n;\mathbb{C}) \Longrightarrow gl(n;\mathbb{C})$, all complex matrices.
$GL(n;\mathbb{R}) \Longrightarrow gl(n;\mathbb{g})$, all real matrices.
$SL(n;\mathbb{C}) \Longrightarrow sl(n;\mathbb{C})$, all complex matrices with trace 0.
$SL(n;\mathbb{R}) \Longrightarrow sl(n;\mathbb{R})$, all real matrices with trace 0.
$U(n) \Longrightarrow u(n)$, all complex matrices that $X^\dagger = -X$
$su(n)$, complex matrices that $X^\dagger = -X$ and $tr(X)=0$
$O(n)$ and $SO(n) \Longrightarrow so(n)$, all real such that $X^{tr} = -X$
$o(n;k), so(n;k)$ all real such that $gX^{tr}g = X$
$sp(n;\mathbb{R})$ all that $J X^{tr} J = X$
Heisenberg group: $X = \begin{pmatrix}
0 & \alpha & \beta \\
0 & 0 & \gamma \\
0 & 0 & 0
\end{pmatrix}$ and $e^{tX} = \begin{pmatrix}
1 & t\alpha & t\beta + t^2 \alpha \gamma \frac{1}{2} \\
0 & 1 & t\gamma \\
0 & 0 & 1
\end{pmatrix}$

Let $G$ be a matrix Lie group, and $X$ an element of its Lie
algebra. Then $e^X$ is an element of the identity component of $G$.

Let $G$ be a matrix Lie group with Lie algebra $\mathfrak{g}$. $X,Y \in \mathfrak{g}$ and $A \in G$.
> $AXA^{-1} \in \mathfrak{g}$
> $sX \in \mathfrak{g}$ for all real s
> $X+Y \in \mathfrak{g}$
> $XY - YX \in \mathfrak{g}$. (for physics convention $-i(XY - YX) \in \mathfrak{g}$)

**main theorem**
Let $G$ and $H$ be matrix Lie groups, with Lie algebras $\mathfrak{g}$ and $\mathfrak{h}$, respectively. Suppose that $\phi : G \rightarrow H$ be a Lie group homomorphism. Then there exists a unique real linear map $\tilde{\phi} : \mathfrak{g} \rightarrow \mathfrak{h}$ such that $\phi(e^X) = e^{\tilde{\phi}(X)}$ for all $X \in \mathfrak{g}$ .
> $\tilde{\phi}(AXA^{-1}) = \phi(A) \tilde{\phi}(X) \phi(A)^{-1}$
> $\tilde{\phi}([X,Y]) = [\tilde{\phi}(X), \tilde{\phi}(Y)]$
> $\tilde{\phi}(X) = \left . \frac{d}{dx} \right \vert_{t=0} \phi(e^{tX})$

If $G, H, K$ are matrix Lie groups and $\phi : G \rightarrow H$, $\psi : G \rightarrow H$, then $\widetilde{\phi \circ \psi} = \tilde{\psi} \circ \tilde{\psi}$

Adjoint Mapping: the linear map $Ad A : \mathfrak{g} \rightarrow \mathfrak{g}$, defined as $Ad A(X) = AXA^{-1}$.  $Ad : G \rightarrow GL(\mathfrak{g})$ is group homomorphism.

Let $G$ be a matrix Lie group with Lie algebra $\mathfrak{g}$. Then there
exist a neighborhood $U$ of zero in $\mathfrak{g}$ and a neighborhood $V$ of $I$ in $G$ such that the
exponential mapping takes $U$ homeomorphically onto $V$ .

If $G$ is a connected matrix Lie group, then every element $A$ of $G$ can be written in the form $A = e^{X_1}e^{X_2} · · · e^{X_n}$ for some $X1,X2, · · ·Xn \in \mathfrak{g}$.

A finite-dimensional real or complex **Lie algebra** is a finite-dimensional real or complex vector space $\mathfrak{g}$, together with a map $[ ]$ from $\mathfrak{g} \times \mathfrak{g} \to \mathfrak{g}$ , with the following properties:
> $[]$ is bilinear.
> $[X, Y ] = −[Y,X]$ for all $X, Y \in \mathfrak{g}$.
> $[X, [Y,Z]] + [Y, [Z,X]] + [Z, [X, Y ]] = 0$ for all $X, Y,Z \in \mathfrak{g}$ (Jacobi identity).

The space $\mathfrak{g}l(n;R)$ of all $n \times n$ real matrices is a real Lie algebra.

Linear maps of $V$ into itself $\mathfrak{g}l(V)$ becomes a real or complex Lie algebra with the bracket operation $[A,B] = AB − BA$.

Every finite-dimensional real Lie algebra is isomorphic to a subalgebra of $\mathfrak{g}l(n;R)$. Every finite-dimensional complex Lie algebra is isomorphic to a (complex) subalgebra of \mathfrak{g}l(n;C)$. (This is
in contrast to the situation for Lie groups, where most but not all Lie groups are
matrix Lie groups.)

Let $\mathfrak{g}$ be a finite-dimensional real or complex Lie algebra, and let $X_1, · · · ,X_n$ be a basis for $\mathfrak{g}$ (as a vector space). Then for each $i, j$ $[X_i,X_j ]$ can be written uniquely in the form $[X_i,X_j ] = \sum_{k=1}^n c_{ijk}X_k$. $c_{ijk}$ are called **structure constants**.

$V$ is a finite-dimensional real vector space, then the **complexification** of $V$ , denoted $V_C$, is the space of formal linear combinations: $v_1 + iv_2$. With with $v_1, v_2 \in V$.  $V$ is the real subspace of $V_C$.

Let $\mathfrak{g}$ be a finite-dimensional real Lie algebra, and $\mathfrak{g}_C$ its complexification (as a real vector space). Then the bracket operation on $\mathfrak{g}$ has a unique extension to $\mathfrak{g}_C$ which makes $\mathfrak{g}_C$ into a complex Lie algebra. The complex Lie algebra $\mathfrak{g}_C$ is called the complexification of the real Lie algebra $\mathfrak{g}$. $$[X_1 + iX_2, Y_1 + iY_2] = ([X_1, Y_1] − [X_2, Y_2]) + i ([X_1, Y_2] + [X_2, Y_1])$$

$\mathfrak{gl}(n;\mathbb{R})_{\mathbb{C}} \simeq \mathfrak{gl}(n;\mathbb{C})$;
$u(n)_{\mathbb{C}} \simeq \mathfrak{g}l(n;\mathbb{C})$;
$\mathfrak{sl}(n;\mathbb{R})_{\mathbb{C}} \simeq \mathfrak{sl}(n;\mathbb{C})$;
$so(n)_{\mathbb{C}} \simeq so(n;\mathbb{C})$

A given complex Lie algebra may have several non-isomorphic real forms.

**main theorem (inverse)**
1. Let $G$ and $H$ be matrix Lie groups, let $\phi_1,\phi_2 : G \rightarrow H$ be Lie group homomorphisms. And let $\tilde{\phi_1},\tilde{\phi_2} : \mathfrak{g} \rightarrow \mathfrak{h}$ be the associated Lie algebras homomorphisms. If $G$ is connected and $\tilde{\phi_1} = \tilde{\phi_2}$ then $\phi_1 = \phi_2$

2. Let $G$ and $H$ be matrix Lie groups with Lie algebras $\mathfrak{g},\mathfrak{h}$. Let $\tilde{\phi} : \mathfrak{g} \rightarrow \mathfrak{h}$ be a Lie algebra homomorphism. If $G$ is connected and simply connected, then there exists a unique Lie group homomorphism $\phi : G \rightarrow H$.
> $\tilde{\phi}(AXA^{-1}) = \phi(A) \tilde{\phi}(X) \phi(A)^{-1}$
> $\tilde{\phi}([X,Y]) = [\tilde{\phi}(X), \tilde{\phi}(Y)]$
> $\tilde{\phi}(X) = \left . \frac{d}{dx} \right \vert_{t=0} \phi(e^{tX})$

Let $G$ be a connected matrix Lie group. A universal covering group of G (or just **universal cover**) is a connected, simply connected Lie group $\tilde{G}$, together with a Lie group homomorphism $\phi : \tilde{G} \to G$ (called the projection map) with the following properties:
> $\phi$ maps $\tilde{G}$ onto $G$ (surjective)
> There is a neighborhood $U$ of $I$ in  $\tilde{G}$ which maps homeomorphically under $\phi$ onto a neighborhood $V$ of $I$ in $G$.

If $G$ is any connected matrix Lie group, then a universal covering group $\tilde{G}$ of $G$ exists and is unique up to canonical isomorphism.

$G$ and $\tilde{G}$ have the same Lie algebra:
Let $G$ be a connected matrix Lie group, $\tilde{G}$ its universal cover, and $\phi$ the projection map from $\tilde{G}$ to $G$. Suppose that $\tilde{G}$ is a matrix Lie group with Lie algebra $\tilde{ \mathfrak{g}}$. Then the associated Lie algebra map $\tilde{\phi} :\tilde{ \mathfrak{g}} \to \tilde{G}$ is an isomorphism.

Example: The universal cover of $S_1$ is $\mathbb{R}$, and the projection map is the map $x \to e^{ix}$.

finite-dimensional complex **representation** of $G$ is a Lie group homomorphism $\Pi : G \to GL(n;\mathbb{C})$

finite-dimensional complex representation of $\mathfrak{g}$ is a Lie algebra homomorphism $\pi$ of $\mathfrak{g} \to gl(n;\mathbb{C})$

$\pi(X) = \left . \frac{d}{dx} \right \vert_{t=0} \Pi (e^{tX})$

A representation with no non-trivial invariant subspaces is called **irreducible**.

A finite-dimensional representation of a group or Lie algebra, acting on a space $V$ , is said to be **completely reducible** if the following property is satisfied: Given an invariant subspace $W \subset V$ , and a second invariant subspace $U \subset W \subset V$ , there exists a third invariant subspace $\tilde{U} \subset W$ such that $U \cap \tilde{U} = \emptyset$ and $U \cup \tilde{U} = W$.

Let $\mathfrak{g}$ be a real Lie algebra, and $\mathfrak{g}_C$ its complexification. Then every finite-dimensional complex representation $\pi$ of $\mathfrak{g}$ has a unique extension to a (complex-linear) representation of $\mathfrak{g}_C$, also denoted $\pi$. The representation of $\mathfrak{g}_C$ satisfies: $\pi (X + iY ) = \pi (X) + i \pi (Y )$ for all $X \in \mathfrak{g}$.

Let $G$ be a matrix Lie group, let $H$ be a Hilbert space, and let $U(H)$ denote the group of unitary operators on $H$. Then a homomorphism $\Pi : G \to U(H)$ is called a **unitary representation** of $G$ if $\Pi$ satisfies the following
continuity condition: If $A_,A \in G$ and $A_n \to A$, then $\Pi(A_n)v \to \Pi(A)v$ for all $v \in H$. A unitary representation with no non-trivial closed invariant subspaces
is called irreducible.

A finite-dimensional completely reducible representation of a group or Lie algebra is equivalent to a direct sum of (one or more) irreducible representations.

Let $G$ be a matrix Lie group. Let $\Pi$ be a finite-dimensional unitary representation of $G$, acting on a finite-dimensional real or complex Hilbert space $V$ . Then $\Pi$ is completely reducible.

If $G$ is a finite group, then every finite-dimensional real or complex representation of $G$ is completely reducible.

If $G$ is a compact matrix Lie group, then every finite-dimensional real or complex representation of $G$ is completely reducible.

**Clebsch-Gordan** theory: Suppose $\Pi_1,\Pi_2$ are irreducible representations of a group $G$. If we regard
$\Pi_1 \otimes \Pi_2$ as a representation of $G$, it may no longer be irreducible. If it is not irreducible, one can attempt to decompose it as a direct sum of irreducible representations.

**Schur’s Lemma**:
> Let $V$ and $W$ be irreducible real or complex representations of a group or Lie algebra, and let $\phi : V \to W$ be a
morphism. Then either $\phi = 0$ or $\phi$ is an isomorphism.
> Let $V$ be an irreducible complex representation of a group or Lie algebra, and let $\phi : V \to V$ be a morphism of $V$ with itself. Then $\phi = \lambda I$, for some $\lambda \in \mathbb{C}$.
> Let $V$ and $W$ be an irreducible complex representation of a group or Lie algebra, and let $\phi_1,\phi_2 : V \to W$ be non zero morphisms. Then $\phi_1 = \lambda \phi_2$, for some $\lambda \in \mathbb{C}$.

An irreducible complex representation of a commutative group or Lie algebra is one-dimensional.

## SU(2) and SO(3)

$SU(2)$ : $A = \begin{pmatrix}
\alpha & -\bar{\beta} \\
\beta & \bar{\alpha}
\end{pmatrix}$ where $\vert \alpha \vert ^2 + \vert \beta \vert ^2 = 1$ , that is we have $S^3$ in $\mathbb{R}^4$. compact, path connected and simply connected.

$SO(3)$ can be parametrized with $R(\phi,\vec{n})$, counterclockwise rotation about the axis through $\vec{n}$ by angle $\phi$. If $0 \le \phi \le \pi$, $\vec{n}$ and $\phi$ define exactly a unique rotation: except $R(\pi, \pm n)$ which are identical. We have a solid ball in $\mathbb{R}^3$ with radius $\pi$, Identity is the center of the ball, and antipodal points are glued together. compact, path connected but not simply connected.

[Dirac belt Trick](https://vimeo.com/62228139) (or hand rotation, twice)

Consider the space $V$ of all $2 \times 2$ complex matrices which are self-adjoint and have trace zero. And inner product as $\langle A,B \rangle = trace(AB)$. This is a three dimensional vector space. With orthonormal basis $\{ \sigma_x,\sigma_y,\sigma_z \}$. If $U \in SU(2)$ then $UAU^{-1} \in V$. $\Phi_U(A) \equiv UAU^{-1}$ is a linear map of $V$ into itself. More $\langle \Phi_U(A),\Phi_U(B) \rangle = \langle A,B \rangle$. $U \to \Phi_U$ is homomorphism and continuous.
Thus $U \to \Phi_U$ is a Lie group homomorphism of $SU(2)$ into $SO(3)$ (in fact two to one map: $\Phi_U = \Phi_{-U}$).

## Clifford group

## Links

[An Elementary Introduction to Groups and Representations (Brian C. Hall)](https://arxiv.org/abs/math-ph/0005032)

## To Look

Symplectic group

Orbit-stabilizer theorem and Burnside's lemma

homomorphism theorems

```python
from sympy import *
init_printing(use_latex='mathjax')
from sympy.physics.quantum.dagger import Dagger
a, b, c, d = symbols('a b c d')
m = Matrix([[a,b],[c,d]])
display(m)
Dagger(m)*m, m*Dagger(m)

```

$$\left[\begin{matrix}a & b\\c & d\end{matrix}\right]$$

$$\left ( \left[\begin{matrix}a \overline{a} + c \overline{c} & b \overline{a} + d \overline{c}\\a \overline{b} + c \overline{d} & b \overline{b} + d \overline{d}\end{matrix}\right], \quad \left[\begin{matrix}a \overline{a} + b \overline{b} & a \overline{c} + b \overline{d}\\c \overline{a} + d \overline{b} & c \overline{c} + d \overline{d}\end{matrix}\right]\right )$$

```python
print('aa')
1+1

```

    aa

    2

```python

```


# Taylor problem 1.50

This problem attacks the "oscillating skateboard" problem described in Example 1.2 of Taylor.  A Newton's 2nd law analysis leads to the differential equation for the angle $\phi$ in radians:

$
\begin{align}
  \ddot\phi = -\frac{g}{R}\sin\phi
  \;.
\end{align}
$

This is a 2nd order, *nonlinear* differential equation.  We note it is the same equation describing the motion of a simple (undamped, not driven) pendulum.

Problem 1.50 has us solving this equation numerically for particular initial conditions and comparing the plots to the approximate solution based on the small angle approximation for $\sin\phi$.  We'll build up code to find this solution and plot it in steps to illustrate how a notebook evolves.  We don't create the polished version at once!

**Your goal for problem 1.51: Modify the relevant part of this notebook to produce the required figure, print it out, and turn it in with your homework.**

```python
%matplotlib inline
```

```python
import numpy as np
from scipy.integrate import odeint

import matplotlib.pyplot as plt
#plt.rcParams.update({'font.size': 18})

```

We'll define the right-hand side (rhs) of the ordinary differential equations (ODE) using the standard form from the Python basics notebook:

$$\begin{align}
   \frac{d}{dt}\left(\begin{array}{c}
                          \phi \\
                          \dot\phi
                      \end{array}\right)
               = \left(\begin{array}{c}
                          \dot\phi \\
                          -g \sin(\phi)
                       \end{array}\right)
\end{align}$$

```python
def ode_rhs_exact(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [\phi, \dot\phi] and params = [g, R].  Returns the list of
    d(u_vec)/dt, as prescribed by the differential equation.

    """
    phi, phidot = u_vec  # extract phi and phidot from the passed vector
    g, R = params  # extract g and R from the passed parameters
    return [phidot, -g*np.sin(phi)/R]
```

```python
# parameters
g = 9.8  # in mks units
R = 5    # radius in meters

# absolute and relative tolerances for ode solver
abserr = 1.0e-8
relerr = 1.0e-6

# initial conditions for [phi, phidot]
phi0 = np.pi/180 * 90.  # convert initial phi to radians
u0_vec = [phi0, 0.]

t_max = 15.  # integration time
t_pts = np.arange(0, t_max, 0.01)  # array of time points, spaced 0.01

# Integrate the differential equation and read off phi, phidot (note T!)
phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
fig.tight_layout()  # make the spacing of subplots nicer
```

**Does the plot make sense for $\phi$?  E.g., does it start at the correct angle? Does it have the behavior you expect (e.g., periodic with constant amplitude)?**

Now let's put this into a function:

```python
def solve_for_phi(phi0, phidot0=0, t_min=0., t_max=1., g=9.8, R=5.):
    """
    Solve the differential equation for the skateboard Example 1.2 in Taylor.
    The result for t, \phi(t) and \dot\phi(t) are returned for a grid with
    t_min < t < t_max and a hardwired (for now) spacing of 0.01 seconds.
    The ODE solver is odeint from scipy, with specified tolerances.
    Units are mks and angles are in radians.
    """

    # absolute and relative tolerances for ode solver
    abserr = 1.0e-8
    relerr = 1.0e-6

    # initial conditions for [phi, phidot]
    u0_vec = [phi0, phidot0]

    t_pts = np.arange(t_min, t_max, 0.01)

    # Integrate the differential equation
    phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T

    return t_pts, phi, phidot
```

Check that it works (gives the previous result).

```python
phi0 = np.pi/180 * 90.  # convert initial phi to radians
t_pts, phi, phidot = solve_for_phi(phi0, t_max=15.)
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
fig.tight_layout()  # make the spacing of subplots nicer

```

Ok, now we need an ode function for the small angle approximation.  It's very easy now to copy and modify our other function!

```python
def ode_rhs_small_angle(u_vec, t, *params):
    """
    Right-hand side (rhs) of the differential equation, with
    u_vec = [\phi, \dot\phi] and params = [g, R].  Returns the list of
    d(u_vec)/dt, as prescribed by the differential equation.

    """
    phi, phidot = u_vec  # We don't actually use x or y here, but could!
    g, R = params
    return [phidot, -g*phi/R]
```

And we can put them together into one solver function:

```python
def solve_for_phi_all(phi0, phidot0=0, t_min=0., t_max=1., g=9.8, R=5.):
    """
    Solve the differential equation for the skateboard Example 1.2 in Taylor
    using the exact equation and the small angle approximation.
    The result for t, \phi(t) and \dot\phi(t) are returned for a grid with
    t_min < t < t_max and a hardwired (for now) spacing of 0.01 seconds.
    The ODE solver is odeint from scipy, with specified tolerances.
    Units are mks and angles are in radians.
    """

    # absolute and relative tolerances for ode solver
    abserr = 1.0e-8
    relerr = 1.0e-6

    # initial conditions for [phi, phidot]
    u0_vec = [phi0, phidot0]

    t_pts = np.arange(t_min, t_max, 0.01)

    # Integrate the differential equations
    phi, phidot = odeint(ode_rhs_exact, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T
    phi_sa, phidot_sa = odeint(ode_rhs_small_angle, u0_vec, t_pts, args=(g, R),
                     atol=abserr, rtol=relerr).T

    return t_pts, phi, phidot, phi_sa, phidot_sa
```

Always try it out!

```python
phi0 = np.pi/180 * 90.
t_pts, phi, phidot, phi_sa, phidot_sa = solve_for_phi_all(phi0, t_max=15.)
print(phi0)
```

    1.5707963267948966

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(t_pts, 180./np.pi * phi)
ax.plot(t_pts, 180./np.pi * phi_sa)
fig.tight_layout()  # make the spacing of subplots nicer

```

This is actually the plot that is requested, so we could analyze it at this stage, but instead let's improve the plot and see how to save it.

### Ok, now for some more systematic plotting

Here we see examples of applying limits to the x and y axes as well as labels and a title.

```python
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.set_xlim(0.,15.)
ax.set_ylim(-25.,25.)
ax.set_xlabel('t (sec)')
ax.set_ylabel(r'$\phi$')
ax.set_title(r'$\phi_0 = 20$ degrees')
line_exact, = ax.plot(t_pts, 180./np.pi * phi, label='exact')
line_sa, = ax.plot(t_pts, 180./np.pi * phi_sa, label='small angle')
ax.legend()

# save the figure
fig.savefig('Taylor_prob_1.50.png', bbox_inches='tight')
```

### Bonus: repeat with widgets!

This actually generalizes problems 1.50 and 1.51 so that you can examine any angle in between.  Use it to check your figure for 1.51.

```python
from ipywidgets import interact, fixed
import ipywidgets as widgets

def rad_to_deg(theta_rad):
    """Take as input an angle in radians and return it in degrees."""
    return 180./np.pi * theta_rad

def deg_to_rad(theta_deg):
    """Take as input an angle in degrees and return it in radians."""
    return np.pi/180. * theta_deg

```

```python
def plot_exact_and_small_angle(phi0_deg=0):
    phi0_rad = deg_to_rad(phi0_deg)
    t_pts, phi_rad, phidot, phi_sa_rad, phidot_sa = \
         solve_for_phi_all(phi0_rad, t_max=15.)
    phi_deg = rad_to_deg(phi_rad)
    phi_sa_deg = rad_to_deg(phi_sa_rad)

    fig = plt.figure(figsize=(8,6))
    ax = fig.add_subplot(1,1,1)
    line_exact, = ax.plot(t_pts, phi_deg, label='exact')
    line_sa, = ax.plot(t_pts, phi_sa_deg, label='small angle')
    ax.legend()
    ax.set_xlim(0.,15.)
    #ax.set_ylim(-90.,90.)
    ax.set_xlabel('t (sec)')
    ax.set_ylabel(r'$\phi$')
    ax.set_title(fr'$\phi_0 = {phi0_deg:.0f}$')
    plt.show()

```

```python
interact(plot_exact_and_small_angle, phi0_deg=(0.,90.));
```

    interactive(children=(FloatSlider(value=0.0, description='phi0_deg', max=90.0), Output()), _dom_classes=('widg…

```python
# to avoid the jiggling and do some formatting
phi0_deg_widget = widgets.FloatSlider(min=0., max=120.0, step=0.1, value=0.,
                                     description=r'$\phi_0$ (degrees)',
                                     readout_format='.0f',
                                     continuous_update=False
                                    )
interact(plot_exact_and_small_angle, phi0_deg=phi0_deg_widget);
```

```python

```


# Workshop 7 optional material: Simulating dynamical systems

Physical systems that evolve in time can be easily (but not always quickly) simulated on a computer.   All you need to know is the function that takes the system from its current state to a future state:

$S_2 = f(S_1, dt)$

In the above equation, $S_1$ and $S_2$ represent the current and future states, respectively.  $S_2$ is the state of the system after a short amount of time, $dt$.  The function $f$, which encodes the dynamics of the system, often depends on the current state and the choice of $dt$.  Choosing a smaller time step, $dt$, generally improves the accuracy of any simulation.

For example, the position of a particle, $x(t)$, in one dimension can be written as a function of its velocity $v$:

$\frac{dx}{dt} = v$

You could find $x(t)$ using differential equation solving methods, or you could just simulate the motion of the particle.  To do this, we can express the dynamics as:

\begin{align}
\ x_2 & = f(x_1, dt) \\
\ & = x_1 + v*dt \\
\end{align}

```python
import matplotlib.pyplot as plt

velocity = 1.0
dt = 0.01

def update(x, t):
    '''This is like the function f above; you should
       think of the "state" of the system as its position
       at some time.  We update its state according to the
       dynamics of the system, and I prefer to also update
       the value of time in this same function--but this could
       also be done in the loop below.'''

    # update the state and time
    x = x + velocity*dt
    t = t + dt

    # return the updated values
    return x, t
```

```python
x = 5 # initial x position
t = 0 # initial time

# Some lists to store the positions and times
x_values = [x]  # I like putting the initial values in the list
t_values = [t]

# This is where the actual "simulation" occurs; the update
# function is used in a loop with however many iterations we want
for i in range(100):

    x, t = update(x, t)
    x_values.append(x)
    t_values.append(t)

# Finally, some plotting to visualize the results
plt.plot(x_values, t_values)
plt.ylabel('position')
plt.xlabel('time')
plt.show()
```

Wow! It's a straight line.  You probably could've guessed that or used some fancy, built-in ODE solving libraries.  But the point is, if you can write down an `update( )` function for some system, then you can simulate it--whether or not there's an analytical way to find a solution.

There's a chance that your capstone project is just a variant of this problem, so I don't want to steal away too many project ideas in this notebook.  Once you start simulating a physical system, it's good to consider the accuracy of your simulated "solution".  If you don't have an exact solution to compare against, you can adjust the chosen value of $dt$ and see how it changes the simulation.

## Exercise 1

Change the code above to simulate a particle under constant acceleration in one dimension.  The `update` function should now also update and return the velocity of the particle.  Plot the position of the particle as a function of time using at least two different choices of time step $dt$.  It may be helpful to include $dt$ as an input parameter to the `update` function so you can easily adjust $dt$ in your simulations.

```python

```

Stochastic systems (systems with some randomness) are another great use-case for this method.  [Brownian motion](https://en.wikipedia.org/wiki/Brownian_motion) describes the random motion of a particle (like a piece of dust) in a fluid.  The thermal motion of the fluid molecule bumps the dust particle around in a seemingly random, but well-described, process.  (In theory you could also model this by keeping track of the motion of many tiny fluid particles colliding with each other, the walls of some container, and the large dust particle.)

Below, I've written down an update function for the position of a particle undergoing Brownian motion.

```python
import numpy as np
import matplotlib.pyplot as plt

def update(x, t, dt):

    x = x + np.sqrt(dt)*np.random.randn()
    t = t + dt

    return x, t
```

```python
x = 0
t = 0
x_values = [x]
t_values = [t]

for i in range(100):

    x, t = update(x, t, 0.1)

    x_values.append(x)
    t_values.append(t)

plt.plot(t_values, x_values)
plt.show()
```

## Exercise 2

Repeat the Brownian motion simulation many (~1000) times and plot the distribution of final positions.  The mean of the distribution should be zero, but there is some variance to this distribution.  How does the variance and/or standard deviation of this distribution scale with time?

```python

```

## Viral spreading

(Very relevant in these times of covid...)

For a more complicated example, here's a simulation of viral spreading following the mathematics of [this paper](https://www.pnas.org/content/pnas/111/46/E4911.full.pdf). You can play with the parameter $\mu$ (mu) to see how it controls the nature of the spreading.

```python
import numpy as np

prob = 0.4  # Probability of infection
L = 2000  # Size of grid

# The distribution from which "jumps" are drawn
def jump(y, mu, L, C):
    return np.power((y*(np.power(L, -mu) - np.power(C, -mu)) + np.power(C, -mu)), -1/mu)

# function that updates the state of the population and list of infected individuals
def update(population, infected,  mu, L, C):
    for inf in infected:
        if np.random.random() > prob:
            start_x = inf[0]
            start_y = inf[1]
            angle = np.random.uniform(0, 2*np.pi)
            dist = jump(np.random.random(), mu, L, C)
            end_x = int(start_x + np.cos(angle)*dist) % L
            end_y = int(start_y + np.sin(angle)*dist) % L
            if population[end_x, end_y] == 0:
                population[end_x, end_y] = 1
                infected.append([end_x, end_y])

    return population, infected

```

```python
population = np.zeros((int(L),int(L)))
population[1000,1000] = 1

infected = [[1000,1000]]

mu = 1.8
C = 1.5
pending = []

ts = []
rs = []
populations = np.zeros((30, int(L), int(L)))

# The simulation -- each iteration is more computationally expensive than the last,
# so be careful changing the number of iterations
for i in range(26):
    populations[i] = population
    population, infected = update(population, infected, mu, L, C)
```

```python
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import cm,rc
from matplotlib.colors import ListedColormap, LinearSegmentedColormap

dark = cm.get_cmap('Dark2_r', 256)
newcolors = dark(np.linspace(0, 1, 256))
white = np.array([1,1,1,1])
newcolors[:25, :] = white
newcmp = ListedColormap(newcolors)

fig = plt.figure(figsize=(8,8))

a = populations[25]  # You can change this number to get an earlier/later state of the population

im = plt.imshow(a, interpolation='none', aspect='auto', vmin=0, vmax=1, cmap = newcmp)
plt.show()
```

```python

```