# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Theorems

The theorems of the Laplace transformation relate basic time-domain operations to their equivalents in the Laplace domain. They are of use for the computation of Laplace transforms of signals composed from modified [standard signals](../continuous_signals/standard_signals.ipynb) and for the computation of the response of systems to an input signal. The theorems allow further to predict the consequences of modifying a signal or system by certain operations.

### Temporal Scaling Theorem

A signal $x(t)$ is given for which the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ exists. The Laplace transform of the [temporally scaled signal](../continuous_signals/operations.ipynb#Temporal-Scaling) $x(a t)$ with $a \in \mathbb{R} \setminus \{0\}$ reads

\begin{equation}
\mathcal{L} \{ x(a t) \} = \frac{1}{|a|} \cdot X \left( \frac{s}{a} \right)
\end{equation}

The Laplace transformation of a temporally scaled signal is given by weighting the inversely scaled Laplace transform of the unscaled signal with $\frac{1}{|a|}$. The scaling of the Laplace transform can be interpreted as a scaling of the complex $s$-plane. The region of convergence (ROC) of the temporally scaled signal $x(a t)$ is consequently the inversely scaled ROC of the unscaled signal $x(t)$

\begin{equation}
\text{ROC} \{ x(a t) \} = \left\{ s: \frac{s}{a} \in \text{ROC} \{ x(t) \} \right\}
\end{equation}

Above relation is known as scaling theorem of the Laplace transform. The scaling theorem can be proven by introducing the scaled signal $x(a t)$ into the definition of the Laplace transformation

\begin{equation}
\mathcal{L} \{ x(a t) \} = \int_{-\infty}^{\infty} x(a t) \, e^{- s t} \; dt = \frac{1}{|a|} \int_{-\infty}^{\infty} x(t') \, e^{-\frac{s}{a} t'} \; dt' = \frac{1}{|a|} \cdot X \left( \frac{s}{a} \right)
\end{equation}

where the substitution $t' = a t$ was used. Note that a negative value of $a$ would result in a reversal of the integration limits. In this case a second reversal of the integration limits together with the sign of the integration element $d t'= a \, dt$ was consolidated into the absolute value of $a$.

### Convolution Theorem

The convolution theorem states that the Laplace transform of the convolution of two signals $x(t)$ and $y(t)$ is equal to the scalar multiplication of their Laplace transforms $X(s)$ and $Y(s)$

\begin{equation}
\mathcal{L} \{ x(t) * y(t) \} = X(s) \cdot Y(s)
\end{equation}

under the assumption that both Laplace transforms $X(s) = \mathcal{L} \{ x(t) \}$ and $Y(s) = \mathcal{L} \{ y(t) \}$ exist, respectively. The ROC of the convolution $x(t) * y(t)$ includes at least the intersection of the ROCs of $x(t)$ and $y(t)$

\begin{equation}
\text{ROC} \{ x(t) * y(t) \} \supseteq \text{ROC} \{ x(t) \} \cap \text{ROC} \{ y(t) \}
\end{equation}

The theorem can be proven by introducing the [definition of the convolution](../systems_time_domain/convolution.ipynb) into the [definition of the Laplace transform](definition.ipynb) and changing the order of integration

\begin{align}
\mathcal{L} \{ x(t) * y(t) \} &= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} x(\tau) \cdot y(t-\tau) \; d \tau \right) e^{-s t} \; dt \\
&= \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} y(t-\tau) \, e^{-s t} \; dt \right) x(\tau) \; d\tau \\
&= Y(s) \cdot \int_{-\infty}^{\infty} x(\tau) \, e^{-s \tau} \; d \tau \\
&= Y(s) \cdot X(s)
\end{align}

The convolution theorem is very useful in the context of linear time-invariant (LTI) systems. The output signal $y(t)$ of an LTI system is given as the convolution of the input signal $x(t)$ with the impulse response $h(t)$. The signals can be represented either in the time or Laplace domain. This leads to the following equivalent representations of an LTI system in the time and Laplace domain, respectively

Calculation of the system response by transforming the problem into the Laplace domain can be beneficial since this replaces the evaluation of the convolution integral by a scalar multiplication. In many cases this procedure simplifies the calculation of the system response significantly. A prominent example is the [analysis of a passive electrical network](network_analysis.ipynb). The convolution theorem can also be useful to derive an unknown Laplace transform. The key is here to express the signal as convolution of two other signals for which the Laplace transforms are known. This is illustrated by the following example.

**Example - Transformation of two convolved signals**

The Laplace transform of the convolution of a causal cosine signal $\epsilon(t) \cdot \cos(\omega_0 t)$ with a causal sine signal $\epsilon(t) \cdot \sin(\omega_0 t)$ is derived by the convolution theorem

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot ( \cos(\omega_0 t) * \sin(\omega_0 t) \}
= \frac{s}{s^2 + \omega_0^2} \cdot \frac{\omega_0}{s^2 + \omega_0^2}
= \frac{\omega_0 s}{(s^2 + \omega_0^2)^2}
\end{equation}

where the [Laplace transforms of the causal cosine and sine signals](properties.ipynb#Transformation-of-the-cosine-and-sine-signal) were used. The ROC of the causal cosine and sine signal is $\Re \{ s \} > 0$. The ROC for their convolution is also $\Re \{ s \} > 0$, since no poles and zeros cancel out. Above Laplace transform has one zero $s_{00} = 0$, and two poles of second degree $s_{\infty 0} = s_{\infty 1} = j \omega_0$ and $s_{\infty 2} = s_{\infty 3} = - j \omega_0$.

This example is evaluated numerically in the following. First the convolution of the causal cosine and sine signal is computed

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, tau = sym.symbols('t tau', real=True)
s = sym.symbols('s', complex=True)
w0 = sym.symbols('omega0', positive=True)

x = sym.integrate(sym.cos(w0*tau) * sym.sin(w0*(t-tau)), (tau, 0, t))
x = x.doit()
x
```

For the sake of illustration let's plot the signal for $\omega_0 = 1$

```python
sym.plot(x.subs(w0, 1), (t, 0, 50), xlabel=r'$t$', ylabel=r'$x(t)$');
```

The Laplace transform is computed

```python
X, a, cond = sym.laplace_transform(x, t, s)
X, a
```

which exists for $\Re \{ s \} > 0$. Its zeros are given as

```python
sym.roots(sym.numer(X), s)
```

and its poles as

```python
sym.roots(sym.denom(X), s)
```

### Temporal Shift Theorem

The [temporal shift of a signal](../continuous_signals/operations.ipynb#Temporal-Shift) $x(t - \tau)$ for $\tau \in \mathbb{R}$ can be expressed by the convolution of the signal $x(t)$ with a shifted Dirac impulse

\begin{equation}
x(t - \tau) = x(t) * \delta(t - \tau)
\end{equation}

This follows from the sifting property of the Dirac impulse. Applying a two-sided Laplace transform to the left- and right-hand side and exploiting the convolution theorem yields

\begin{equation}
\mathcal{L} \{ x(t - \tau) \} = X(s) \cdot e^{- s \tau}
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ is assumed to exist. Note that $\mathcal{L} \{ \delta(t - \tau) \} = e^{- s \tau}$ can be derived from the definition of the two-sided Laplace transform together with the sifting property of the Dirac impulse. The Laplace transform of a shifted signal is given by multiplying the Laplace transform of the original signal with $e^{- s \tau}$. The ROC does not change

\begin{equation}
\text{ROC} \{ x(t-\tau) \} = \text{ROC} \{ x(t) \}
\end{equation}

This result is known as shift theorem of the Laplace transform. For a causal signal $x(t)$ and $\tau > 0$ the shift theorem of the one-sided Laplace transform is equal to the shift theorem of the two-sided transform.

#### Transformation of the rectangular signal

The Laplace transform of the [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) $x(t) = \text{rect}(t)$ is derived by expressing it by the Heaviside signal

\begin{equation}
\text{rect}(t) = \epsilon \left(t + \frac{1}{2} \right) - \epsilon \left(t - \frac{1}{2} \right)
\end{equation}

Applying the shift theorem to the [transform of the Heaviside signal](definition.ipynb#Transformation-of-the-Heaviside-Signal) and exploiting the linearity of the Laplace transform yields

\begin{equation}
\mathcal{L} \{ \text{rect}(t) \} = \frac{1}{s} e^{s \frac{1}{2}} - \frac{1}{s} e^{- s \frac{1}{2}} = \frac{\sinh \left( \frac{s}{2} \right) }{\frac{s}{2}}
\end{equation}

where $\sinh(\cdot)$ denotes the [hyperbolic sine function](https://en.wikipedia.org/wiki/Hyperbolic_function#Sinh). The ROC of the Heaviside signal is given as $\Re \{ s \} > 0$. Applying [l'Hopitals rule](https://en.wikipedia.org/wiki/L'H%C3%B4pital's_rule), the pole at $s=0$ can be disregarded leading to

\begin{equation}
\text{ROC} \{ \text{rect}(t) \} = \mathbb{C}
\end{equation}

For illustration, the magnitude of the Laplace transform $|X(s)|$ is plotted in the $s$-plane, as well as $X(\sigma)$ and $X(j \omega)$ for the real and imaginary part of the complex frequency $s = \sigma + j \omega$.

```python
sigma, omega = sym.symbols('sigma omega')
X = sym.sinh(s/2)*2/s

sym.plotting.plot3d(abs(X.subs(s, sigma+sym.I*omega)), (sigma, -5, 5), (omega, -20, 20),
                    xlabel=r'$\Re\{s\}$', ylabel=r'$\Im\{s\}$', title=r'$|X(s)|$')

sym.plot(X.subs(s, sigma) , (sigma, -5, 5), xlabel=r'$\Re\{s\}$', ylabel=r'$X(s)$', ylim=(0, 3))

sym.plot(X.subs(s, sym.I*omega) , (omega, -20, 20), xlabel=r'$\Im\{s\}$', ylabel=r'$X(s)$');
```

**Exercise**

* Derive the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of the causal rectangular signal $x(t) = \text{rect} (a t - \frac{1}{2 a})$
* Derive the Laplace transform of the [triangular signal](../fourier_transform/theorems.ipynb#Transformation-of-the-triangular-signal) $x(t) = \Lambda(a t)$ with $a \in \mathbb{R} \setminus \{0\}$

### Differentiation Theorem

Derivatives of signals are the fundamental operations of differential equations. Ordinary differential equations (ODEs) with constant coefficients play an important role in the theory of linear time-invariant (LTI) systems. Consequently, the representation of the derivative of a signal in the Laplace domain is of special interest.

#### Two-sided transform

A differentiable signal $x(t)$ whose temporal derivative $\frac{d x(t)}{dt}$ exists is given. Using the [derivation property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse), the derivative of the signal can be expressed by the convolution

\begin{equation}
\frac{d x(t)}{dt} = \frac{d \delta(t)}{dt} * x(t)
\end{equation}

Applying a two-sided Laplace transformation to the left- and right-hand side together with the [convolution theorem](#Convolution-Theorem) yields the Laplace transform of the derivative of $x(t)$

\begin{equation}
\mathcal{L} \left\{ \frac{d x(t)}{dt} \right\} = s \cdot X(s)
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$. The two-sided Laplace transform $\mathcal{L} \{ \frac{d \delta(t)}{dt} \} = s$ can be derived by applying the definition of the Laplace transform together with the derivation property of the Dirac impulse. The ROC is given as a superset of the ROC for $x(t)$

\begin{equation}
\text{ROC} \left\{ \frac{d x(t)}{dt} \right\} \supseteq \text{ROC} \{ x(t) \}
\end{equation}

due to the zero at $s=0$ which may cancel out a pole.

Above result is known as differentiation theorem of the two-sided Laplace transform. It states that the differentiation of a signal in the time domain is equivalent to a multiplication of its spectrum by $s$.

#### One-sided transform

Many practical signals and systems are causal, hence $x(t) = 0$ for $t < 0$. A causal signal is potentially discontinuous for $t=0$. The direct application of above result for the two-sided Laplace transform is not possible since it assumes that the signal is differentiable for every time instant $t$. The potential discontinuity at $t=0$ has to be considered explicitly for the derivation of the differentiation theorem for the one-sided Laplace transform [[Girod et al.](index.ipynb#Literature)]

\begin{equation}
\mathcal{L} \left\{ \frac{d x(t)}{dt} \right\} = s \cdot X(s) - x(0+)
\end{equation}

where $x(0+) := \lim_{\epsilon \to 0} x(0+\epsilon)$ denotes the right sided limit value of $x(t)$ for $t=0$. The ROC is given as a superset of the ROC of $x(t)$

\begin{equation}
\text{ROC} \left\{ \frac{d x(t)}{dt} \right\} \supseteq \text{ROC} \{ x(t) \}
\end{equation}

due to the zero at $s=0$ which may cancel out a pole. The one-sided Laplace transform of a causal signal is equal to its two-sided transform. Above result holds therefore also for the two-sided transform of a causal signal.

The main application of the differentiation theorem is the transformation and solution of differential equations under consideration of initial values. Another application area is the derivation of transforms of signals which can be expressed as derivatives of other signals.

### Integration Theorem

An integrable signal $x(t)$ for which the integral $\int_{-\infty}^{t} x(\tau) \; d\tau$ exists is given. The integration can be represented as convolution with the rectangular signal $\epsilon(t)$

\begin{equation}
\int_{-\infty}^{t} x(\tau) \; d\tau = \int_{-\infty}^{\infty} x(\tau) \cdot \epsilon(t - \tau) \; d\tau = \epsilon(t) * x(t)
\end{equation}

as illustrated below

Two-sided Laplace transformation of the left- and right-hand side of above equation, application of the convolution theorem and using the Laplace transform of the Heaviside signal $\epsilon(t)$ yields

\begin{equation}
\mathcal{L} \left\{ \int_{-\infty}^{t} x(\tau) \; d\tau \right\}
= \frac{1}{s} \cdot X(s)
\end{equation}

The ROC is given as a superset of the intersection of the ROC of $x(t)$ and the right $s$-half-plane

\begin{equation}
\text{ROC} \left\{ \int_{-\infty}^{t} x(\tau) \; d\tau \right\} \supseteq \text{ROC} \{ x(t) \} \cap \{s : \Re \{ s \} > 0\}
\end{equation}

due to the pole at $s=0$. This integration theorem holds also for the one-sided Laplace transform.

#### Transformation of the ramp signal

The Laplace transform of the causal [ramp signal](https://en.wikipedia.org/wiki/Ramp_function) $t \cdot \epsilon(t)$ is derived by applying the integration theorem. The ramp signal can be expressed as integration over the Heaviside signal

\begin{equation}
t \cdot \epsilon(t) = \int_{-\infty}^{t} \tau \cdot \epsilon(\tau) \; d \tau
\end{equation}

Laplace transformation of the left- and right-hand side and application of the integration theorem together with the Laplace transform of the Heaviside signal yields

\begin{equation}
\mathcal{L} \{ t \cdot \epsilon(t) \} = \frac{1}{s^2}
\end{equation}

with

\begin{equation}
\text{ROC} \{ t \cdot \epsilon(t) \} = \{s : \Re \{ s \} > 0\}
\end{equation}

**Exercise**

* Derive the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of the signal $x(t) = t^n \cdot \epsilon(t)$ with $n \geq 0$ by repeated application of the integration theorem.
* Compare your result to the numerical result below. Note that $\Gamma(n+1) = n!$ for $n \in \mathbb{N}$.

```python
n = sym.symbols('n', integer=True)

X, a, cond = sym.laplace_transform(t**n, t, s)
X, a, cond
```

### Modulation Theorem

The complex modulation of a signal $x(t)$ is defined as $e^{s_0 t} \cdot x(t)$ with $s_0 \in \mathbb{C}$. The Laplace transform of a modulated signal is derived by introducing it into the definition of the two-sided Laplace transform

\begin{align}
\mathcal{L} \left\{ e^{s_0 t} \cdot x(t) \right\} &=
\int_{-\infty}^{\infty} e^{s_0 t} x(t) \, e^{-s t} \; dt =
\int_{-\infty}^{\infty} x(t) \, e^{- (s - s_0) t} \; dt \\
&= X(s-s_0)
\end{align}

where $X(s) = \mathcal{L} \{ x(t) \}$. Modulation of the signal $x(t)$ leads to a translation of the $s$-plane into the direction given by the complex value $s_0$. Consequently, the ROC is also shifted

\begin{equation}
\text{ROC} \{ e^{s_0 t} \cdot x(t) \} = \{s: s - \Re \{ s_0 \} \in \text{ROC} \{ x(t) \} \}
\end{equation}

This relation is known as modulation theorem.

**Example - Transformation of $t^n e^{-s_0 t} \epsilon(t)$**

The Laplace transform of the signal $t^n \cdot \epsilon(t)$

\begin{equation}
\mathcal{L} \{ t^n \cdot \epsilon(t) \} = \frac{n!}{s^{n+1}}
\end{equation}

for $\Re \{ s \} > 0$ was derived in the previous example. This result can be extended to the class of signals $t^n e^{-s_0 t} \epsilon(t)$ with $s_0 \in \mathbb{C}$ using the modulation theorem

\begin{equation}
\mathcal{L} \{ t^n e^{-s_0 t} \epsilon(t) \} = \frac{n!}{(s + s_0)^{n+1}} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}.
\end{equation}

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


        Lista de Exercícios 5a

         Guilherme Esdras

---

        Imports

```python
import numpy as np
import pandas as pd
from scipy import optimize as opt

import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-poster')

import sympy as sp
sp.init_printing()
```

---

    Exercicio 1: Implemente os algoritmos da iteração de ponto fixo, Newton-Raphson e secante usando somente a biblioteca Numpy.

**Funções de teste:**

```python
g_aula = lambda x: np.exp(-x)
g_aula_x0 = 0

fn_youtube = lambda x: np.cos(x) - np.sin(x)
fD_youtube = lambda x: -np.sin(x) - np.cos(x)
er_youtube = 0.01
x0_youtube = 0
x1_youtube = np.pi/2

f_aula = lambda x: np.sin(x) - x
f_aula_x1 = 0.7
f_aula_x0 = 0.8
```

    Método da Iteração de Ponto Fixo

```python
def ponto_fixo(f, x0, maxiter=1000, xtol=1e-10, verbose=True, r_tb=False, p_tb=False):

    ''' Opcional... '''
    if r_tb or p_tb: tb = []

    i = 0 # contador de iterações

    while i < maxiter:
        # enquanto tanto o erro quanto o número de iterações não extrapolarem os limites estabelecidos...

        y = f(x0) # calcula f(x0) e atribui a y

        # calcula o erro abs
        erro = np.abs(x0 - y)

        ''' Opcional... '''
        if r_tb or p_tb: tb.append([]); tb[i].append(x0); tb[i].append(y); tb[i].append(erro); tb[i].append(xtol);

        if erro < xtol:
            msg = f'Raiz encontrada: {x0} | Em {i} iterações'

            if verbose:
                print('-'*len(msg))
                print(msg)
                print('-'*len(msg))

            ''' Opcional... '''
            if p_tb:
                print(pd.DataFrame(tb, columns=["x0", "f(x0)", "Erro", "xtol"]))
                print('-'*len(msg))
            ''' ----------- '''

            # retorna a raiz e o número de iterações
            if r_tb:
                return x0, i, tb
            else:
                return x0, i

        x0 = y # atualiza o valor de x0 com o atual y
        i += 1 # incrementa contador

    if verbose: print("Número de iterações extrapolou o limite sem encontrar a raíz!")
    if r_tb: return False, False, tb
    return None # saiu do while sem encontrar uma solução
```

```python
x, y, tb = ponto_fixo(g_aula, g_aula_x0, xtol=1e-1, r_tb=True)
pd.DataFrame(tb, columns=["x0", "f(x0)", "Erro", "xtol"])
```

    ----------------------------------------------------
    Raiz encontrada: 0.6062435350855974 | Em 5 iterações
    ----------------------------------------------------

      x0
      f(x0)
      Erro
      xtol

      0
      0.000000
      1.000000
      1.000000
      0.1

      1
      1.000000
      0.367879
      0.632121
      0.1

      2
      0.367879
      0.692201
      0.324321
      0.1

      3
      0.692201
      0.500474
      0.191727
      0.1

      4
      0.500474
      0.606244
      0.105770
      0.1

      5
      0.606244
      0.545396
      0.060848
      0.1

    Método Newton-Raphson

```python
def newton_raphson(f, fD, x, tol=1e-10, maxiter=500, verbose=False, r_tb=False, p_tb=False):

    Xk = x # o x da primeira iteração recebe o x de entrada
    k = 0 # contador de iterações

    ''' Opcional... '''
    if r_tb or p_tb: tb = []

    while k < maxiter:
        # enquanto o número de iterações não extrapolarem o limite estabelecido...

        ''' Opcional... '''
        if r_tb or p_tb: tb.append([]);

        f_Xk = f(Xk) # calcula f(x) do x atual (dessa k-ézima iteração)
        fD_Xk = fD(Xk) # calcula a derivada f'(x) do x atual (dessa k-ézima iteração)

        ''' Opcional... '''
        if r_tb or p_tb: tb[k].append(Xk); tb[k].append(f_Xk); tb[k].append(fD_Xk);

        # se a derivada for 0, não há o que fazer
        if fD_Xk == 0:
            if verbose: print("Derivada é == 0. Divisão por zero. Sem soluções possíveis!")
            return None

        # atualiza o valor de Xk+1 (x da próxima k-ézima iteração)
        newton_div = f_Xk / fD_Xk
        Xk1 = Xk - newton_div

        ''' Opcional... '''
        if r_tb or p_tb: tb[k].append(newton_div); tb[k].append(Xk1);

        erro = np.abs( Xk1 - Xk ) # calcula o erro relativo

        ''' Opcional... '''
        if r_tb or p_tb: tb[k].append(erro);

        # se o erro for menor ou igual a tolerância, retorna o resultado
        if erro <= tol:
            msg = f'Raiz encontrada: {Xk1} | Em {k} iterações'

            if verbose:
                print('-'*len(msg))
                print(msg)
                print('-'*len(msg))

            ''' Opcional... '''
            if p_tb:
                print(pd.DataFrame(tb, columns=["Xk", "f(Xk)", "f(Xk)", "f(Xk)/f'(Xk)", "Xk+1", "Erro"]))
                print('-'*len(msg))
            ''' ----------- '''

            # retorna a raiz e o número de iterações
            if r_tb:
                return Xk1, k, tb
            else:
                return Xk1, k, False

        Xk = Xk1 # atualiza o valor do x para a próxima iteração
        k += 1 # incrementa o contador

    if verbose: print("Número de iterações extrapolou o limite sem encontrar a raíz!")
    if r_tb: return False, False, tb
    return None # saiu do while sem encontrar uma solução
```

```python
x, k, tb = newton_raphson(fn_youtube, fD_youtube, x0_youtube, er_youtube, verbose=True, r_tb=True)
pd.DataFrame(tb, columns=["Xk", "f(Xk)", "f(Xk)", "f(Xk)/f'(Xk)", "Xk+1", "Erro"])
```

    ----------------------------------------------------
    Raiz encontrada: 0.7853981759997019 | Em 2 iterações
    ----------------------------------------------------

      Xk
      f(Xk)
      f(Xk)
      f(Xk)/f'(Xk)
      Xk+1
      Erro

      0
      0.000000
      1.000000
      -1.000000
      -1.000000
      1.000000
      1.000000

      1
      1.000000
      -0.301169
      -1.381773
      0.217958
      0.782042
      0.217958

      2
      0.782042
      0.004746
      -1.414206
      -0.003356
      0.785398
      0.003356

    Método da Secante

```python
def secante(f, x0, x1, tol=1e-10, maxiter=500, verbose=True, r_tb=False, p_tb=False):

    if f(x0) * f(x1) >= 0:
        if verbose: print("Incapaz de prosseguir!")
        return None

    ''' Opcional... '''
    if r_tb or p_tb: tb = []

    erro = None
    k = 0

    while k < maxiter:

        ''' Opcional... '''
#         if r_tb or p_tb: tb.append([]);

        fX1 = f(x1)
        fX0 = f(x0)

        # calcula o valor intermediário
        Xk = (x0 * fX1 - x1 * fX0) / (fX1 - fX0)

        # atualiza valor de x
        x = fX0 * f(Xk);

        # verifica se x é a raiz da equação, em caso positivo, retorna Xk e o número de iterações
        if (x == 0):
            if verbose: print(f"Encontrou a raiz {Xk} em {k} iterações!")
            return Xk, k

        else:
            # atualiza os valores do intervalo
            x0 = x1;
            x1 = Xk;

            Xk1 = Xk

            erro = abs(Xk1 - Xk)

            if erro <= tol:
                if verbose: print(f"Encontrou a raiz {Xk} em {k} iterações!")
                return Xk1, k # retorna a raiz e o número de iterações

            # update number of iteration
            k += 1

    if verbose: print(f"Número de iterações extrapolou o limite sem encontrar a raíz! Valor final foi {Xk1}")
    return None # saiu do while sem encontrar uma soluçãoint("Número de iterações extrapolou o limite sem encontrar a raíz!")
```

```python
secante(fn_youtube, x0_youtube, x1_youtube, 0.01, verbose=True)
```

---

    Exercicio 2: Use a iteração de ponto fixo simples para localizar a raiz de $f(x)=2 sin(\sqrt{x})−x$, tendo $x_0 = 0,5$ e adotando como critério de parada o erro $e_a ≤ 0,001\%$.

```python
ex2_f = lambda x: ( 2 * np.sin(np.sqrt(x)) - x )
ex2_x0 = 0.5
ex2_tol = 0.001
```

```python
ponto_fixo(ex2_f, ex2_x0, xtol=ex2_tol)
```

```python
opt.fixed_point(ex2_f, ex2_x0, xtol=ex2_tol, maxiter=3)
```

    array(0.76864886)

---

    Exercicio 3: Determine a maior raiz real de $f(x)=2x^3 − 11.7x^2 + 17.7x − 5$.

```python
# Definindo f(x)
ex3_f = lambda x: 2 * x**3 - 11.7 * x**2 + 17.7 * x - 5

sp.var('x')
sp.Lambda(x, ex3_f(x)) # 'printando' a função de forma simbólica
```

```python
# ex3_fD = lambda x: 2*x**3 - 11.7*x**2 + 17.7*x - 5
sp.var('x')

# Calculando f'(x)
ex3_fD_sym = lambda x: eval(sp.ccode(sp.diff(ex3_f(x), x)))

sp.Lambda(x, ex3_fD_sym(x)) # 'printando' a função de forma simbólica
```

```python
ex3_fD = lambda x: 6*x**2 - 23.4*x + 17.7
```

    a) Graficamente;

```python
x = np.linspace(0, 4, 50)

y = ex3_f(x)

raiz1 = opt.root(ex3_f, 0)
raiz2 = opt.root(ex3_f, 2)
raiz3 = opt.root(ex3_f, 4)
raizes = np.array([raiz1.x, raiz2.x, raiz3.x])

fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111)

plt.vlines(x=raiz1.x, ymin=ex3_f(0), ymax=0, colors='gray', ls=':', lw=2)
plt.vlines(x=raiz2.x, ymin=ex3_f(0), ymax=0, colors='gray', ls=':', lw=2)
plt.vlines(x=raiz3.x, ymin=ex3_f(0), ymax=0, colors='gray', ls=':', lw=2)

ax.axhline(0, color='b')

ax.plot(x, ex3_f(x), 'r', label="$f(x)$")
ax.plot(raizes, ex3_f(raizes), 'kv', label="$Raizes$")

ax.legend(loc='best')
ax.set_xlabel('$x$')
ax.set_ylabel('$f(x)$')

plt.show()
```

Analisando o gráfico plotado, podemos notar que a maior raíz real está entre 3.5 e 4.0.

    b) Pelo método da iteração de ponto fixo (três iterações, $x_0=3$) (certifique-se de desenvolver uma solução que convirja para a raiz);

```python
ponto_fixo(ex3_f, 3, 3)
```

    Número de iterações extrapolou o limite sem encontrar a raíz!

```python
try: opt.fixed_point(ex3_f, 3, maxiter=3)
except RuntimeError as re: print(str(re))
```

    Failed to converge after 3 iterations, value is 3.7022048269565255

    c) Pelo método de Newton-Raphson (três iterações, $x_0=3$);

```python
opt.root_scalar(ex3_f, fprime=ex3_fD, x0=3, maxiter=3, method='newton')
```

          converged: False
               flag: 'convergence error'
     function_calls: 6
         iterations: 3
               root: 3.7929344806432264

    d) Pelo método da secante (três iterações, $x_{−1}=3$, $x_0=4$).

```python
secante(ex3_f, 3, 4, 3)
```

---

    Exercicio 4: Compare os métodos da bisseção, falsa posição, do ponto fixo, de Newton-Raphson e da secante, localizando a raiz das seguintes equações:

Para as avaliações, deve-se considerar:

o número máximo de iterações de todos os métodos testados não pode ultrapassar 200;

a tolerância deve ser de $10^{-10}$;

para os métodos abertos, escolha os limites do intervalo, respectivamente como $x_{?1}$ e $x_0$.

Para cada método, estamos interessados em comparar:

raiz;

número de iterações até o critério de parada;

se houve erro de convergência;

tempo de cálculo (procure como calcular tempo de execução usando jupyter notebooks, como %timeit).

```python
# Constantes
ex4_maxit = 200
ex4_tol = 1e-10
```

---

Método da Falsa Posição:

```python
def regula_falsi(f, xl, xu, tol=1e-10, maxit=10000):

    if (f(xl) * f(xu) >= 0):
        return -1

    i = 0
    x = xl
    erro, x_ant = 1, x

    while erro > tol and i < maxit:

        x = xu - ( ( f(xu)*(xl-xu) ) / (f(xl)-f(xu)) )

        if f(x) * f(xl) < 0:
            xu = x

        else:
            xl = x

        erro = np.abs((x - x_ant) / np.abs(x))
        x_ant = x
        i += 1

    return ( x, i )
```

---

    a) $f_1(x) = 2x^4 + 4x^3 + 3x^2 - 10x - 15$, com $x^* \in [0, 3]$

**Definindo $f_1(x)$**

```python
sp.var('x')
f1 = lambda x: 2*x**4 + 4*x**3 + 3*x**2 - 10*x - 15
sp.Lambda(x, f1(x)) # 'printando' a função simbolicamente
```

**Calculando $f_1'(x)$**

```python
sp.var('x')
f1D = lambda x: eval(sp.ccode(sp.diff(f1(x), x)))
sp.Lambda(x, f1D(x)) # 'printando' a função simbolicamente
```

```python
f1D_2 = lambda x: 8*x**3 + 12*x**2 + 6*x - 10
```

**Refatorando $f_1'(x)$**

```python
sp.var('x')
f1_ref = lambda x: 15 / (2*(x**3) + 4*(x**2) + 3*(x) - 10)
sp.Lambda(x, f1_ref(x)) # 'printando' a função simbolicamente
```

**Limites do Intervalo**

```python
f1_x0 = 0
f1_x1 = 3
```

**$f_1(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f1, method= 'bisect', bracket=[f1_x0, f1_x1], rtol=ex4_tol)
opt.root_scalar(f1, method= 'bisect', bracket=[f1_x0, f1_x1], rtol=ex4_tol)
```

    28.8 µs ± 5.16 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 37
         iterations: 35
               root: 1.4928787086100783

$f_1(x)$ - Falsa Posição

```python
%timeit regula_falsi(f1, f1_x0, f1_x1, ex4_tol, ex4_maxit)
regula_falsi(f1, f1_x0, f1_x1, ex4_tol, ex4_maxit)
```

$f_1(x)$ - Ponto Fixo

```python
%timeit opt.fixed_point(f1_ref, 1.5, xtol=ex4_tol, maxiter=ex4_maxit)
opt.fixed_point(f1_ref, 1.5, xtol=ex4_tol, maxiter=ex4_maxit)
```

    789 µs ± 126 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    array(1.49287871)

$f_1(x)$ - Newton-Raphson

```python
%timeit opt.root_scalar(f1, fprime=f1D_2, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f1, fprime=f1D_2, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    336 µs ± 23.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 14
         iterations: 7
               root: 1.4928787086636037

$f_1(x)$ - Secante

```python
%timeit opt.root_scalar(f1, x0=1, x1=f1_x1, method='secant')
opt.root_scalar(f1, x0=1, x1=f1_x1, method='secant')
```

    509 µs ± 86.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 10
         iterations: 9
               root: 1.4928787086636037

---

    b) $f_2(x) = (x + 3)(x + 1)(x - 2)^3$, com $x^* \in [0,5]$

**Definindo $f_2(x)$**

```python
sp.var('x')
f2 = lambda x: (x + 3)*(x + 1)*(x - 2)**3
sp.Lambda(x, f2(x)) # 'printando' a função simbolicamente
```

**Refatorando $f_2(x)$**

```python
sp.var('x')
f2R = lambda x: 24 / ( (x**4) - 2*(x**3) - 9*(x**2) + 22*x + 4 )
sp.Lambda(x, f2R(x)) # vendo o resultado simbólico
```

**Calculando $f_2'(x)$**

```python
sp.var('x')
f2D_sym = lambda x: eval(sp.ccode(sp.diff(f2(x), x)))
sp.Lambda(x, f2D_sym(x)) # 'printando' a função simbolicamente
```

```python
f2D = lambda x: (x - 2)**3 * (x + 1) + (x - 2)**3 * (x + 3) + 3*(x - 2)**2 * (x + 1) * (x + 3)
```

**Limites do Intervalo**

```python
f2_x0 = 0
f2_x1 = 5
```

**$f_2(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f2, method= 'bisect', bracket=[f2_x0, f2_x1], rtol=ex4_tol)
opt.root_scalar(f2, method= 'bisect', bracket=[f2_x0, f2_x1], rtol=ex4_tol)
```

    28.4 µs ± 6.51 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 37
         iterations: 35
               root: 1.9999999999708962

**$f_2(x)$ - Falsa Posição**

```python
%timeit regula_falsi(f2, f2_x0, f2_x1, ex4_tol, ex4_maxit)
regula_falsi(f2, f2_x0, f2_x1, ex4_tol, ex4_maxit)
```

**$f_2(x)$ - Ponto Fixo**

```python
%timeit opt.fixed_point(f2R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
opt.fixed_point(f2R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
```

    6.8 ms ± 410 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    array(1.99966007)

**$f_2(x)$ - Newton-Raphson**

```python
%timeit opt.root_scalar(f2, fprime=f2D, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f2, fprime=f2D, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    1.88 ms ± 139 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

          converged: True
               flag: 'converged'
     function_calls: 110
         iterations: 55
               root: 1.9999999998545246

**$f_2(x)$ - Secante**

```python
%timeit opt.root_scalar(f2, x0=f2_x0, x1=f2_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
opt.root_scalar(f2, x0=f2_x0, x1=f2_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
```

    2.48 ms ± 117 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

          converged: True
               flag: 'converged'
     function_calls: 79
         iterations: 78
               root: 2.0000000002448965

---

    c) $f_3(x) = 5x^3 + x^2 - e^{1-2x} + cos(x) + 20$, com $x^* \in [-5, 5]$

**Definindo $f_3(x)$**

```python
sp.var('x')

def f3(x, t=None):
    if t == 'sp':
        return (5*x**3 + x**2 - sp.exp(1-2*x) + sp.cos(x) + 20)
    return (5*x**3 + x**2 - np.exp(1-2*x) + np.cos(x) + 20)

sp.Lambda(x, f3(x, 'sp')) # 'printando' a função simbolicamente
```

**Refatorando $f_3(x)$**

```python
sp.var('x')

def f3R(x, t=None):
    if t == 'sp':
        return ( 5*x**3 + x**2 - sp.exp(1-2*x) + sp.cos(x) + 20 + x )
    return ( 5*x**3 + x**2 - np.exp(1-2*x) + np.cos(x) + 20 + x )

sp.Lambda(x, f3R(x, 'sp')) # vendo o resultado simbólico
```

**Calculando $f_3'(x)$**

```python
sp.var('x')

f3D_sym = lambda x: sp.diff(f3(x, 'sp'), x) # calcula a derivada

sp.Lambda(x, f3D_sym(x)) # vendo o resultado simbólico
```

```python
# aplicando o resultado
def f3D(x, t=None):
    if t == 'sp':
        return (15*x**2 + 2*x + 2*sp.exp(1-2*x) - sp.sin(x))
    return (15*x**2 + 2*x + 2*np.exp(1-2*x) - np.sin(x))

sp.Lambda(x, f3D(x, 'sp')) # 'printando' a função simbolicamente
```

**Limites do Intervalo**

```python
f3_x0 = -5
f3_x1 = 5
```

**$f_3(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f3, method= 'bisect', bracket=[f3_x0, f3_x1], rtol=ex4_tol)
opt.root_scalar(f3, method= 'bisect', bracket=[f3_x0, f3_x1], rtol=ex4_tol)
```

    115 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 39
         iterations: 37
               root: -0.9295604598446516

**$f_3(x)$ - Falsa Posição**

```python
%timeit regula_falsi(f3, f3_x0, f3_x1, ex4_tol, ex4_maxit)
regula_falsi(f3, f3_x0, f3_x1, ex4_tol, ex4_maxit)
```

**$f_3(x)$ - Ponto Fixo**

```python
%timeit opt.fixed_point(f3R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
opt.fixed_point(f3R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
```

    25.6 ms ± 3.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

    array(-0.92956046)

**$f_3(x)$ - Newton-Raphson**

```python
%timeit opt.root_scalar(f3, fprime=f3D, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f3, fprime=f3D, x0=1, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    395 µs ± 90.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 14
         iterations: 7
               root: -0.9295604598378413

**$f_3(x)$ - Secante**

```python
%timeit opt.root_scalar(f3, x0=f3_x0, x1=f3_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
opt.root_scalar(f3, x0=f3_x0, x1=f3_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
```

    974 µs ± 40.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 25
         iterations: 24
               root: -0.9295604598378412

---

    d) $f_4(x) = sin(x)x + 4$, com $x^* \in [1, 5]$

**Definindo $f_4(x)$**

```python
sp.var('x')

def f4(x, t=None):
    if t == 'sp':
        return (sp.sin(x)*x + 4)
    return (np.sin(x)*x + 4)

sp.Lambda(x, f4(x, 'sp')) # 'printando' a função simbolicamente
```

**Refatorando $f_4(x)$**

```python
sp.var('x')

def f4R(x, t=None):
    if t == 'sp':
        return ( (-4) / sp.sin(x) )
    return ( (-4) / np.sin(x) )

sp.Lambda(x, f4R(x, 'sp')) # vendo o resultado simbólico
```

**Calculando $f_4'(x)$**

```python
sp.var('x')

f4D_sym = lambda x: sp.diff(f4(x, 'sp'), x) # calcula a derivada

sp.Lambda(x, f4D_sym(x)) # vendo o resultado simbólico
```

```python
# aplicando o resultado
def f4D(x, t=None):
    if t == 'sp':
        return (x * sp.cos(x) + sp.sin(x))
    return (x * np.cos(x) + np.sin(x))

sp.Lambda(x, f4D(x, 'sp')) # 'printando' a função simbolicamente
```

**Limites do Intervalo**

```python
f4_x1 = 1
f4_x0 = 5
```

**$f_4(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f4, method= 'bisect', bracket=[f4_x0, f4_x1], rtol=ex4_tol)
opt.root_scalar(f4, method= 'bisect', bracket=[f4_x0, f4_x1], rtol=ex4_tol)
```

    55.1 µs ± 2.85 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 36
         iterations: 34
               root: 4.323239543708041

**$f_4(x)$ - Falsa Posição**

```python
%timeit regula_falsi(f4, f4_x0, f4_x1, ex4_tol, ex4_maxit)
regula_falsi(f4, f4_x0, f4_x1, ex4_tol, ex4_maxit)
```

**$f_4(x)$ - Ponto Fixo**

```python
%timeit opt.fixed_point(f4R, x0=f4_x0, xtol=ex4_tol, maxiter=ex4_maxit)
opt.fixed_point(f4R, x0=f4_x0, xtol=ex4_tol, maxiter=ex4_maxit)
```

    687 µs ± 51.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    array(4.32323954)

**$f_4(x)$ - Newton-Raphson**

```python
%timeit opt.root_scalar(f4, fprime=f4D, x0=4, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f4, fprime=f4D, x0=4, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    209 µs ± 3.07 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 10
         iterations: 5
               root: 4.323239543713714

**$f_4(x)$ - Secante**

```python
%timeit opt.root_scalar(f4, x0=f4_x0, x1=f4_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
opt.root_scalar(f4, x0=f4_x0, x1=f4_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
```

    267 µs ± 2.57 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 8
         iterations: 7
               root: 4.323239543713714

---

    e) $f_5(x) = (x - 3)^5 ln(x)$, com $x^* \in [2, 5]$

**Definindo $f_5(x)$**

```python
sp.var('x')

def f5(x, t=None):
    if t == 'sp':
        return ( (x - 3)**5 * sp.ln(x) )
    return ( (x - 3)**5 * np.log(x) )

sp.Lambda(x, f5(x, 'sp')) # 'printando' a função simbolicamente
```

**Refatorando $f_5(x)$**

```python
sp.var('x')

def f5R(x, t=None):
    if t == 'sp':
        return ( (x-3)**5 * sp.log(x) + x )
    return ( (x-3)**5 * np.log(x) + x )

sp.Lambda(x, f5R(x, 'sp')) # vendo o resultado simbólico
```

**Calculando $f_5'(x)$**

```python
sp.var('x')

f5D_sym = lambda x: sp.diff(f5(x, 'sp'), x) # calcula a derivada

sp.Lambda(x, f5D_sym(x)) # vendo o resultado simbólico
```

```python
# aplicando o resultado
def f5D(x, t=None):
    if t == 'sp':
        return ( 5*(x - 3)**4 * sp.log(x) + sp.Pow((x-3),5) / x )
    return ( 5*(x - 3)**4 * np.log(x) + ((x-3)**5 / x) )

sp.Lambda(x, f5D(x, 'sp')) # 'printando' a função simbolicamente
```

**Limites do Intervalo**

```python
f5_x0 = 2
f5_x1 = 5
```

**$f_5(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f5, method= 'bisect', bracket=[f5_x0, f5_x1], rtol=ex4_tol)
opt.root_scalar(f5, method= 'bisect', bracket=[f5_x0, f5_x1], rtol=ex4_tol)
```

    46.2 µs ± 3.06 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 36
         iterations: 34
               root: 2.9999999999417923

**$f_5(x)$ - Falsa Posição**

```python
%timeit regula_falsi(f5, f5_x0, f5_x1, ex4_tol, ex4_maxit)
regula_falsi(f5, f5_x0, f5_x1, ex4_tol, ex4_maxit)
```

**$f_5(x)$ - Ponto Fixo**

```python
%timeit opt.fixed_point(f5R, x0=2, xtol=10**-9.9, maxiter=ex4_maxit)
opt.fixed_point(f5R, x0=2, xtol=10**-9.9, maxiter=ex4_maxit)
```

    27 ms ± 1.89 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

    array(2.98889825)

**$f_5(x)$ - Newton-Raphson**

```python
%timeit opt.root_scalar(f5, fprime=f5D, x0=f5_x0, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f5, fprime=f5D, x0=f5_x0, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    4.73 ms ± 1.09 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

          converged: True
               flag: 'converged'
     function_calls: 194
         iterations: 97
               root: 2.99999999964898

**$f_5(x)$ - Secante**

```python
%timeit opt.root_scalar(f5, x0=f5_x0, x1=f5_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
opt.root_scalar(f5, x0=f5_x0, x1=f5_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
```

    7.79 ms ± 1.62 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

          converged: True
               flag: 'converged'
     function_calls: 139
         iterations: 138
               root: 2.999999999418057

---

    f) $f_6(x) = x^{10} - 1$, com $x^* \in [0.8, 1.2]$

**Definindo $f_6(x)$**

```python
sp.var('x')

f6 = lambda x: x**10 - 1

sp.Lambda(x, f6(x)) # 'printando' a função simbolicamente
```

**Refatorando $f_6(x)$**

```python
sp.var('x')

f6R = lambda x: 1/(x)**9

sp.Lambda(x, f6R(x)) # vendo o resultado simbólico
```

**Calculando $f_6'(x)$**

```python
sp.var('x')

f6D_sym = lambda x: eval(sp.ccode(sp.diff(f6(x), x)))

sp.Lambda(x, f6D_sym(x)) # 'printando' a função simbolicamente
```

```python
f6D = lambda x: 10 * x**9
```

**Limites do Intervalo**

```python
f6_x0 = 0.8
f6_x1 = 1.2
```

**$f_6(x)$ - Bisseção**

```python
%timeit opt.root_scalar(f6, method= 'bisect', bracket=[f6_x0, f6_x1], rtol=ex4_tol)
opt.root_scalar(f6, method= 'bisect', bracket=[f6_x0, f6_x1], rtol=ex4_tol)
```

    9.41 µs ± 1.1 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 3
         iterations: 1
               root: 1.0

**$f_6(x)$ - Falsa Posição**

```python
%timeit regula_falsi(f6, f6_x0, f6_x1, ex4_tol, ex4_maxit)
regula_falsi(f6, f6_x0, f6_x1, ex4_tol, ex4_maxit)
```

**$f_6(x)$ - Ponto Fixo**

```python
%timeit opt.fixed_point(f6R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
opt.fixed_point(f6R, x0=1, xtol=ex4_tol, maxiter=ex4_maxit)
```

    157 µs ± 18.4 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

    array(1.)

**$f_6(x)$ - Newton-Raphson**

```python
%timeit opt.root_scalar(f6, fprime=f6D, x0=f6_x0, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
opt.root_scalar(f6, fprime=f6D, x0=f6_x0, xtol=ex4_tol, maxiter=ex4_maxit, method='newton')
```

    414 µs ± 28.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 20
         iterations: 10
               root: 1.0

**$f_6(x)$ - Secante**

```python
%timeit opt.root_scalar(f6, x0=f6_x0, x1=f6_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
opt.root_scalar(f6, x0=f6_x0, x1=f6_x1, maxiter=ex4_maxit, xtol=ex4_tol, method='secant')
```

    370 µs ± 3.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

          converged: True
               flag: 'converged'
     function_calls: 12
         iterations: 11
               root: 1.0

---

Class 05/Lista_5a - Guilherme Esdras.ipynb


```python
%pylab inline
%config InlineBackend.figure_format = 'retina'
from ipywidgets import interact
import scipy.stats as stats
import pandas as pd
# import pymc3 as pm
import arviz as az
import seaborn as sns

import tensorflow as tf
import tensorflow_probability as tfp
tfd = tfp.distributions
tfb = tfp.bijectors

import warnings # the warning spam is pointless and annoying
warnings.simplefilter(action="ignore", category=FutureWarning)
```

    Populating the interactive namespace from numpy and matplotlib

```python

```

# State space models
Let us continue with our example about tracking polar bears. Suppose we now have data from a tracking collar of one of our bears. The GPS measurements that come from the collar are samples at equal time intervals of 1 hour. Our collar measurements will be given by
$$ z_{k} \sim \text{Normal}(x_{k}, \sigma_{\rm obs}),$$
where $x_{k}$ is the actual position of the bear at times $t_k = k$. We will model the actual position of the bear as a simple random walk
$$ x_k \sim \text{Normal}(x_{k-1}, \sigma)$$

The full generative model is then
\begin{align}
  z_{k} &\sim \text{Normal}(x_{k}, \sigma_{\rm obs}) \\
   x_k &\sim \text{Normal}(x_{k-1}, \sigma) \\
   x_0 & \sim  \text{Normal}(0, 25)\\
   \sigma_{\rm obs} &\sim ??? \\
   \sigma &\sim ???
\end{align}

## State space model in Tensorflow
First, we will sample the posterior
$$\pi(x_0, x_1, \ldots, x_T | z_1, \ldots, z_T; \theta) \propto \prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)$$
The generative model is
$$p(x_0, x_1, \ldots, x_T, z_1, \ldots, z_T | \theta) = \prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)$$

### (See also previous results in the Week 12 notebook)

# Expectation Maximization Algorithm
We want to compute the log likelihood $\mathcal{L}(\theta)$ of the parameters $\theta$ so that we can obtain the maximum likelihood estimator
  $$\theta_{MLE} =  \text{arg max}_{\theta}\mathcal{L}(\theta).$$

$$ \mathcal{L}(\theta) = \log(\int dx_0 \cdots \int dx_T p(x_0, x_1, \ldots, x_T, z_1, \ldots, z_T | \theta) )$$

$$\pi(x_0, x_1, \ldots, x_T | z_1, \ldots, z_T; \theta) = \frac{\prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)}{\int dx_0 \cdots \int dx_T \prod_{k=1}^{T}g(z_k| x_k)f(x_k | x_{k-1})p_0(x_0)}$$

  1. E-step: given fixed parameters $\theta_j$ sample
  $$ X_{1:T} \mid z_{1:T}, \theta_j$$
  Then compute the expectation
  \begin{align}
  Q(\theta | \theta_j) &= E[\log(\prod_{k=1}^{T}g(z_k| X_k, \theta)f(X_k | X_{k-1}, \theta)p_0(X_0|\theta)) \mid \theta_j]\\
  &= E[\sum_{k=1}^{T}\log(g(z_k| X_k, \theta)) + \sum_{k=1}^{T}\log(f(X_k | X_{k-1}, \theta)) + \log(p_0(X_0|\theta)) \mid \theta_j]
  \end{align}
  2. M-step: compute $$\theta_{j+1} = \text{arg max}_{\theta} Q(\theta | \theta_j)  $$

Repeat these steps until convergence is reached.

# Expectation Maximization with Tensorflow gradient descent
Suppose we want to estimate the bear's mobility parameter $\sigma$

```python
Nt = 500 ## number of measured positions from bear's trackinkg collar

X0true = -25. + 50.*rand(1, 2) ## bear's initial position
Xtrue = X0true + normal(0, 2., [Nt, 2]).cumsum(axis=0) ## bear's true path
Z = Xtrue + normal(0, 0.2, [Nt, 2]) ## add observation error to the path
plot(Z[:, 0], Z[:, 1]);
```

```python
Nsteps = 10 ## number of MCMC steps
N_burn_in = 50
Nchains = 8

log_sigma_obs = tf.Variable(log(1.), dtype=tf.float64)
log_sigma = tf.Variable(log(5.), dtype=tf.float64)

@tf.function()
def log_target_density(Xfull):
    ## Xfull must be a three dimensional array
    Sigma_obs = tf.math.exp(log_sigma_obs)*tf.ones([1, 1, 2], dtype=tf.float64)
    Sigma = tf.math.exp(log_sigma)*tf.ones([1, 1, 2], dtype=tf.float64) # tensorflow variable appears here
    Sigma_x0 = 100*tf.ones([1, 2], dtype=tf.float64)
    ## X : [parallel chains, time, space]
    X0 = Xfull[:, 0, :]
    X = Xfull[:, 1:, :] ## same shape as `Z`
    ## observation likelihood: density for Z given X
    G = tfd.MultivariateNormalDiag(
        loc=X,
        scale_diag=Sigma_obs)
    logG = tf.reduce_sum(
        G.unnormalized_log_prob(Z),
        axis=1)
    ## motion model: density for X
    F = tfd.MultivariateNormalDiag(
        loc=Xfull[:, :-1, :],
        scale_diag=Sigma)
    logF = tf.reduce_sum(
        F.unnormalized_log_prob(X),
        axis=1)
    ## initial position prior: density for X0
    F0 = tfd.MultivariateNormalDiag(
        loc=tf.zeros([1, 2], dtype=tf.float64),
        scale_diag=Sigma_x0)
    logF0 = F0.unnormalized_log_prob(X0)
    return logG + logF + logF0

```

```python
@tf.function()
def track_sampler(Xinit):
    kernel = tfp.mcmc.NoUTurnSampler(
        target_log_prob_fn=log_target_density,
        max_tree_depth=8,
        max_energy_diff=1000,
        step_size=tf.cast(0.1, tf.float64))

    adaptive_kernel = tfp.mcmc.DualAveragingStepSizeAdaptation(
        inner_kernel=kernel,
        num_adaptation_steps=int(0.8 * N_burn_in),
        target_accept_prob=tf.cast(0.7, tf.float64))

    Xchain = tfp.mcmc.sample_chain(
        num_results=Nsteps,
        num_burnin_steps=N_burn_in,
        current_state=Xinit,
        #trace_fn=trace_everything,
        trace_fn=None,
        kernel=adaptive_kernel,
        parallel_iterations=4)
    return Xchain
```

```python
## Estep

def Estep():
    Xinit = zeros([Nchains, Nt+1, 2])
    Xinit[:, 0, :] = normal(0, 25., [Nchains, 2])
    Xinit[:, 1:, :] = Z + normal(0, 5., [Nchains, Nt, 2])
    Xchain = track_sampler(Xinit)
    Xchain = array(Xchain)
    return Xchain ## [MCMC steps, parallel chains, time, space]
@tf.function()
def averaged_log_density():
    return tf.reduce_mean(log_target_density(Xchain.reshape(-1, Nt+1, 2)))
```

```python
## M step

N_gd_steps = 10
# learning_rate = 0.001
# optimizer = tf.optimizers.SGD(learning_rate=learning_rate)
learning_rate = 0.01
optimizer = tf.optimizers.Adam(learning_rate=learning_rate)

trainable_variables = [log_sigma, log_sigma_obs]
@tf.function()
def Mstep(Xchain):
    loss = tf.constant(0., dtype=tf.float64)
    for _ in tf.range(N_gd_steps):
        with tf.GradientTape() as tape:
            tape.watch(trainable_variables) ## must tell Tensorflow what trainable variables to use for gradients
            loss = -averaged_log_density() ## we use the negative averaged log likelihood because our optimizer is set up to solve mimimization
        grad = tape.gradient(loss, trainable_variables) ## compute the gradient of the loss
        optimizer.apply_gradients( ## this takes one step of gradient descent to change the trainable prameters
            zip(grad, trainable_variables)) ## the values stored in the trainable variable object changes as a result of this operation
    return loss
```

```python
log_sigma.assign(log(4.)), log_sigma_obs.assign(log(0.1))
print('sigma:', exp(log_sigma), 'sigma_obs:', exp(log_sigma_obs))
for _ in arange(10):
    Xchain = Estep()
    loss = Mstep(Xchain)
    print('loss:', array(loss))
    print('sigma:', exp(log_sigma), 'sigma_obs:', exp(log_sigma_obs))
```

    sigma: 4.0 sigma_obs: 0.10000000000000002
    loss: 4491.916551124594
    sigma: 3.716253648974166 sigma_obs: 0.1188342649331675
    loss: 3539.0156385701775
    sigma: 3.29980151024446 sigma_obs: 0.1438357578033359
    loss: 3020.776776848029
    sigma: 2.922243023958905 sigma_obs: 0.16723689149435864
    loss: 2743.580971032723
    sigma: 2.6197676396234217 sigma_obs: 0.18688944485227846
    loss: 2590.3514533351445
    sigma: 2.3921184464462306 sigma_obs: 0.202848642580912
    loss: 2503.4821482992297
    sigma: 2.230423956534621 sigma_obs: 0.2158529209407091
    loss: 2453.0874754752235
    sigma: 2.123701962063079 sigma_obs: 0.2266435598992937
    loss: 2422.553834934196
    sigma: 2.060074366069413 sigma_obs: 0.23578044381199925
    loss: 2402.760376505583
    sigma: 2.0274529949123816 sigma_obs: 0.24364823396331345
    loss: 2389.0566413753054
    sigma: 2.0146843706493964 sigma_obs: 0.25050585277768694

```python

```

```python

```


> ## Calculadora de funções para cálculo

Criei essa calcuadora com um objetivo simples de criar gráfico através de um
input de função feito através do widgets.

```
import plotly.graph_objects as go
import numpy as np

import sympy
from sympy import *

import math
import cmath
from math import *
init_printing()

import ipywidgets as widgets
from sympy.parsing.sympy_parser import parse_expr
```

```
periodo = widgets.IntSlider()
display(periodo)

funcao = widgets.Text(disabled=False)
display(funcao)
```

    IntSlider(value=0)

    Text(value='')

```
try:
  display(parse_expr(funcao.value))
  def f(x): return eval(funcao.value)

  if type(f(x)) is complex:
    print('O numero é complexo\n\n')
    if f(x).imag < 10**-10:
      display(f(x).real)

except:
  print('Digite a função.')

```


# Introduction to symbols

```python
import warnings
warnings.filterwarnings('ignore')

from lambeq.ccg2discocat import DepCCGParser
from lambeq.tensor import TensorAnsatz
from lambeq.core.types import AtomicType
from discopy import Dim

# Define atomic types
N = AtomicType.NOUN
S = AtomicType.SENTENCE

# Parse a sentence
parser = DepCCGParser()
diagram = parser.sentence2diagram("John walks in the park")

# Apply a tensor ansatz
ansatz = TensorAnsatz({N: Dim(4), S: Dim(2)})
tensor_diagram = ansatz(diagram)
tensor_diagram.draw(figsize=(12,5), fontsize=12)
```

```python
tensor_diagram.free_symbols
```

    {John__n, in__s.r@n.r.r@n.r@s@n.l, park__n, the__n@n.l, walks__n.r@s}

Each symbol is associated with a specific size, which is defined from the applied ansatz.

```python
[(s, s.size) for s in tensor_diagram.free_symbols]
```

    [(the__n@n.l, 16),
     (in__s.r@n.r.r@n.r@s@n.l, 256),
     (park__n, 4),
     (walks__n.r@s, 8),
     (John__n, 4)]

For example, you see that preposition "in" has been assigned 256 dimensions, which is derived by multiplying the dimensions of each individual wire ($2 \cdot 4 \cdot 4 \cdot 2 \cdot 4$), nouns are assigned 4 dimensions, and the determiner 16 dimensions.

## Circuit symbols

```python
from lambeq.circuit import IQPAnsatz

iqp_ansatz = IQPAnsatz({N: 1, S: 1}, n_layers=1)
circuit = iqp_ansatz(diagram)
circuit.draw(figsize=(12,8), fontsize=12)
```

```python
[(s, s.size) for s in circuit.free_symbols]
```

    [(park__n_2, 1),
     (John__n_0, 1),
     (park__n_1, 1),
     (John__n_2, 1),
     (in__s.r@n.r.r@n.r@s@n.l_2, 1),
     (in__s.r@n.r.r@n.r@s@n.l_3, 1),
     (park__n_0, 1),
     (in__s.r@n.r.r@n.r@s@n.l_1, 1),
     (the__n@n.l_0, 1),
     (walks__n.r@s_0, 1),
     (in__s.r@n.r.r@n.r@s@n.l_0, 1),
     (John__n_1, 1)]

## From symbols to tensors

```python
from sympy import default_sort_key

parameters = sorted(tensor_diagram.free_symbols, key=default_sort_key)
```

We will use `numpy` arrays for the tensors, initialised randomly:

```python
import numpy as np

tensors = [np.random.rand(p.size) for p in parameters]
print(tensors[0])
```

    [0.72853877 0.0489024  0.58544848 0.33123311]

```python
tensor_diagram_np = tensor_diagram.lambdify(*parameters)(*tensors)
print("Before lambdify:", tensor_diagram.boxes[0].data)
print("After lambdify:", tensor_diagram_np.boxes[0].data)
```

    Before lambdify: John__n
    After lambdify: [0.72853877 0.0489024  0.58544848 0.33123311]

```python
result = tensor_diagram_np.eval()
print(result)
```

    Tensor(dom=Dim(1), cod=Dim(2), array=[14.40077809, 13.14739668])

```python
result.array
```

    array([14.40077809, 13.14739668])


## State space models - concentrating the scale out of the likelihood function

```python
import numpy as np
import pandas as pd
import statsmodels.api as sm

dta = sm.datasets.macrodata.load_pandas().data
dta.index = pd.date_range(start='1959Q1', end='2009Q4', freq='Q')
```

### Introduction

(much of this is based on Harvey (1989); see especially section 3.4)

State space models can generically be written as follows (here we focus on time-invariant state space models, but similar results apply also to time-varying models):

$$
\begin{align}
y_t & = Z \alpha_t + \varepsilon_t, \quad \varepsilon_t \sim N(0, H) \\
\alpha_{t+1} & = T \alpha_t + R \eta_t \quad \eta_t \sim N(0, Q)
\end{align}
$$

Often, some or all of the values in the matrices $Z, H, T, R, Q$ are unknown and must be estimated; in statsmodels, estimation is often done by finding the parameters that maximize the likelihood function. In particular, if we collect the parameters in a vector $\psi$, then each of these matrices can be thought of as functions of those parameters, for example $Z = Z(\psi)$, etc.

Usually, the likelihood function is maximized numerically, for example by applying quasi-Newton "hill-climbing" algorithms, and this becomes more and more difficult the more parameters there are. It turns out that in many cases we can reparameterize the model as $[\psi_*', \sigma_*^2]'$, where $\sigma_*^2$ is the "scale" of the model (usually, it replaces one of the error variance terms) and it is possible to find the maximum likelihood estimate of $\sigma_*^2$ analytically, by differentiating the likelihood function. This implies that numerical methods are only required to estimate the parameters $\psi_*$, which has dimension one less than that of $\psi$.

### Example: local level model

(see, for example, section 4.2 of Harvey (1989))

As a specific example, consider the local level model, which can be written as:

$$
\begin{align}
y_t & = \alpha_t + \varepsilon_t, \quad \varepsilon_t \sim N(0, \sigma_\varepsilon^2) \\
\alpha_{t+1} & = \alpha_t + \eta_t \quad \eta_t \sim N(0, \sigma_\eta^2)
\end{align}
$$

In this model, $Z, T,$ and $R$ are all fixed to be equal to $1$, and there are two unknown parameters, so that $\psi = [\sigma_\varepsilon^2, \sigma_\eta^2]$.

#### Typical approach

First, we show how to define this model without concentrating out the scale, using statsmodels' state space library:

```python
class LocalLevel(sm.tsa.statespace.MLEModel):
    _start_params = [1., 1.]
    _param_names = ['var.level', 'var.irregular']

    def __init__(self, endog):
        super(LocalLevel, self).__init__(endog, k_states=1, initialization='diffuse')

        self['design', 0, 0] = 1
        self['transition', 0, 0] = 1
        self['selection', 0, 0] = 1

    def transform_params(self, unconstrained):
        return unconstrained**2

    def untransform_params(self, unconstrained):
        return unconstrained**0.5

    def update(self, params, **kwargs):
        params = super(LocalLevel, self).update(params, **kwargs)

        self['state_cov', 0, 0] = params[0]
        self['obs_cov', 0, 0] = params[1]

```

There are two parameters in this model that must be chosen: `var.level` $(\sigma_\eta^2)$ and `var.irregular` $(\sigma_\varepsilon^2)$. We can use the built-in `fit` method to choose them by numerically maximizing the likelihood function.

In our example, we are applying the local level model to consumer price index inflation.

```python
mod = LocalLevel(dta.infl)
res = mod.fit(disp=False)
print(res.summary())
```

We can look at the results from the numerical optimizer in the results attribute `mle_retvals`:

```python
print(res.mle_retvals)
```

#### Concentrating out the scale

Now, there are two ways to reparameterize this model as above:

1. The first way is to set $\sigma_*^2 \equiv \sigma_\varepsilon^2$ so that $\psi_* = \psi / \sigma_\varepsilon^2 = [1, q_\eta]$ where $q_\eta = \sigma_\eta^2 / \sigma_\varepsilon^2$.
2. The second way is to set $\sigma_*^2 \equiv \sigma_\eta^2$ so that $\psi_* = \psi / \sigma_\eta^2 = [h, 1]$ where $h = \sigma_\varepsilon^2 / \sigma_\eta^2$.

In the first case, we only need to numerically maximize the likelihood with respect to $q_\eta$, and in the second case we only need to numerically maximize the likelihood with respect to $h$.

Either approach would work well in most cases, and in the example below we will use the second method.

To reformulate the model to take advantage of the concentrated likelihood function, we need to write the model in terms of the parameter vector $\psi_* = [g, 1]$. Because this parameter vector defines $\sigma_\eta^2 \equiv 1$, we now include a new line `self['state_cov', 0, 0] = 1` and the only unknown parameter is $h$. Because our parameter $h$ is no longer a variance, we renamed it here to be `ratio.irregular`.

The key piece that is required to formulate the model so that the scale can be computed from the Kalman filter recursions (rather than selected numerically) is setting the flag `self.ssm.filter_concentrated = True`.

```python
class LocalLevelConcentrated(sm.tsa.statespace.MLEModel):
    _start_params = [1.]
    _param_names = ['ratio.irregular']

    def __init__(self, endog):
        super(LocalLevelConcentrated, self).__init__(endog, k_states=1, initialization='diffuse')

        self['design', 0, 0] = 1
        self['transition', 0, 0] = 1
        self['selection', 0, 0] = 1
        self['state_cov', 0, 0] = 1

        self.ssm.filter_concentrated = True

    def transform_params(self, unconstrained):
        return unconstrained**2

    def untransform_params(self, unconstrained):
        return unconstrained**0.5

    def update(self, params, **kwargs):
        params = super(LocalLevelConcentrated, self).update(params, **kwargs)
        self['obs_cov', 0, 0] = params[0]

```

Again, we can use the built-in `fit` method to find the maximum likelihood estimate of $h$.

```python
mod_conc = LocalLevelConcentrated(dta.infl)
res_conc = mod_conc.fit(disp=False)
print(res_conc.summary())
```

The estimate of $h$ is provided in the middle table of parameters (`ratio.irregular`), while the estimate of the scale is provided in the upper table. Below, we will show that these estimates are consistent with those from the previous approach.

And we can again look at the results from the numerical optimizer in the results attribute `mle_retvals`. It turns out that two fewer iterations were required in this case, since there was one fewer parameter to select. Moreover, since the numerical maximization problem was easier, the optimizer was able to find a value that made the gradient for this parameter slightly closer to zero than it was above.

```python
print(res_conc.mle_retvals)
```

#### Comparing estimates

Recall that $h = \sigma_\varepsilon^2 / \sigma_\eta^2$ and the scale is $\sigma_*^2 = \sigma_\eta^2$. Using these definitions, we can see that both models produce nearly identical results:

```python
print('Original model')
print('var.level     = %.5f' % res.params[0])
print('var.irregular = %.5f' % res.params[1])

print('\nConcentrated model')
print('scale         = %.5f' % res_conc.scale)
print('h * scale     = %.5f' % (res_conc.params[0] * res_conc.scale))
```

### Example: SARIMAX

By default in SARIMAX models, the variance term is chosen by numerically maximizing the likelihood function, but an option has been added to allow concentrating the scale out.

```python
# Typical approach
mod_ar = sm.tsa.SARIMAX(dta.cpi, order=(1, 0, 0), trend='ct')
res_ar = mod_ar.fit(disp=False)

# Estimating the model with the scale concentrated out
mod_ar_conc = sm.tsa.SARIMAX(dta.cpi, order=(1, 0, 0), trend='ct', concentrate_scale=True)
res_ar_conc = mod_ar_conc.fit(disp=False)
```

These two approaches produce about the same loglikelihood and parameters, although the model with the concentrated scale was able to improve the fit very slightly:

```python
print('Loglikelihood')
print('- Original model:     %.4f' % res_ar.llf)
print('- Concentrated model: %.4f' % res_ar_conc.llf)

print('\nParameters')
print('- Original model:     %.4f, %.4f, %.4f, %.4f' % tuple(res_ar.params))
print('- Concentrated model: %.4f, %.4f, %.4f, %.4f' % (tuple(res_ar_conc.params) + (res_ar_conc.scale,)))
```

This time, about 1/3 fewer iterations of the optimizer are required under the concentrated approach:

```python
print('Optimizer iterations')
print('- Original model:     %d' % res_ar.mle_retvals['iterations'])
print('- Concentrated model: %d' % res_ar_conc.mle_retvals['iterations'])
```


```python
from IPython.display import Image
from IPython.core.display import HTML
from sympy import *; x,h,t,y,z,a,b,c   = symbols("x h t y z a b c", real=True)
Image(url= "https://i.imgur.com/1epEWCc.png")
```

```python
f  = 2*y**3 + y**2 -y**5 - x**4 +2*x**3 -x**2
f
```

$\displaystyle - x^{4} + 2 x^{3} - x^{2} - y^{5} + 2 y^{3} + y^{2}$

```python
solve(f,x)
```

    [1/2 - sqrt(-4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2,
     1/2 - sqrt(4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2,
     sqrt(-4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2 + 1/2,
     sqrt(4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2 + 1/2]

```python
(1/2 - sqrt(-4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2).subs(y,0)
```

$\displaystyle 0$

```python
(1/2 - sqrt(4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2).subs(y,0)
```

$\displaystyle 0$

```python
(sqrt(-4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2 + 1/2).subs(y,0)
```

$\displaystyle 1.0$

```python
(sqrt(4*sqrt(-y**3 + 2*y + 1)*Abs(y) + 1)/2 + 1/2).subs(y,0)
```

$\displaystyle 1.0$

```python
#third value 0.5 was found by guessing integers
```

```python
Image(url= "https://i.imgur.com/F3Dr2h3.png")
```


# Pair Graphs

```python
import sys
sys.path.append("../../src")
%reload_ext autoreload
%autoreload 2
from theory202104 import *
from helpers import graph
```

## Recap of 2021-04-21

Pair Cardinality:
$$
\begin{align}
n_T: & \Sigma^2 \rightarrow \mathbb{N} \cup \{- \infty \} \\
n_T(ab) & = \left\{
  \begin{array}{rcl} n & \text{ if } a \neq b \text{ and } T \cap \{a,b\} = n \times (ab)   \\
  -\infty & \text{ otherwise } \end{array}\right.
\end{align}
$$

* The pair $(a,b)$ is paired in $T$ if $n_T(ab) \ge 0$
* The pair $(a,b)$ is unpaired in $T$ if $n_T(ab) \lt 0$
* The set  of all pairs in $T$ is $\mathcal{P}_T = \{ (a,b) | a,b \in \Sigma(T); n_T( a,b ) \ge 0 \}$

Sequences:

* $\mathcal{S}_T = \{ S \in \Sigma^* | S \text{ is sequence in } T \}$ (all sequences in $T$)
* A sequence $S$ is a **maximal sequence in $T$** if is not a combination of other sequences in $T$: $\nexists R, S' \in \mathcal{S}_T; S \neq R $ such that $\Sigma(S') = \Sigma(S) \cup \Sigma(R)$.
* $\overline{ \mathcal{S}_T } = \{ S \in \mathcal{S}_T | S \text{ is maximal sequence in } T \}$ (maximal sequences in $T$)

**Lemma (consecutive sequence order)**: $s_j$ is the $j$-esim element of a sequence $S$ in $T$ $\iff$ there are exactly $j-1$ pairs $(x, s_j)$ paired in $T$ where $n_T(S) = n_T(x, s_j)$. All such $x$ lies inside $S$.

```python
pandasPair( pairs_in_trace("abcabcabc")  ).T
```

      (b, b)
      (b, a)
      (a, b)
      (b, c)
      (c, b)
      (a, a)
      (a, c)
      (c, a)

      pairs
      -1
      -1
      3
      3
      -1
      -1
      3
      -1

## Equivalent Pair Graph

Let be $G_T=(V,E,n_T)$ a directed weighted graph where:

* $V = \Sigma(T)$
* $E = \Sigma(T) \times \Sigma(T) $
* $n_T( e )= n_T( ab )$ the cardinality of $e=(a,b)$

$G_T$ is called the equivalent pair graph. By construction the non negative edges are exactly the pairs of the trace:

$\mathcal{P}_T = \{ (a,b) | a,b \in \Sigma(T); n_T( a,b ) \ge 0 \} =  \{ e | e \in E; n_T( e ) \ge 0 \}$

```python
G_T = pair_graph("abXYabXaYb")
graph(G_T)
```

## n-layer of a pair graph

An n-layer of $G_T$ is the directed subgraph where $n_T(e) = n$, formally defined as:

$G_T^n = (V_T^n, E_T^n)$ where:

* $E_T^n = \{ e \in E(G_T) | n_T(e) = n \}$
* $V_T^n = \Sigma( E_T^n ) $

Note how the n-layer captures all pairs $(a,b)$ with $n_T(a,b) = n$

```python
G_T = pair_graph("abXYabXaYb")
```

```python
graph( n_layer(G_T, 2) )
```

```python
graph( n_layer(G_T, 3) )
```

## Sequences in Pair Graphs

Any sequence $S=s_1 ... s_m$ in $T$ is totally contained in the n-layer with $n = n_T(S)$, by using the property that all its pairs has the same cardinality $n_T(S)$.

```python
T="abcdabcdXYZXYZ"

G_T = pair_graph(T)
graph( non_negative_graph(G_T) )

print('cardinality of abcd in T = {}'.format( sequence_cardinality('abcd', T) ))
print('cardinality of XYZ in T = {}'.format( sequence_cardinality('XYZ', T) ))
```

## Sequence inference using cliques

By the lemma of consecutive sequence order, (... long way from here...) then the cliques ordered by its inDegree corresponds to maximal sequences with cardinality $n$.

Based on the previous example, the cliques in the 2-layer $G^{n=2}_T$ are the following:

```python
G_T_2 =  n_layer(G_T, 2)
cliques = cliques_from_layer( G_T_2 )
cliques
```

    [['X', 'Z', 'Y'], ['d', 'b', 'a', 'c']]

Let's take the first clique: and inspect how its inDegrees looks like.

```python
degrees=sorted( G_T_2.in_degree() , key=lambda u: u[1], reverse=False)

for clique in cliques:
    print([ (node,deg) for node,deg in degrees if node in clique ])
```

    [('X', 0), ('Y', 1), ('Z', 2)]
    [('a', 0), ('b', 1), ('c', 2), ('d', 3)]

Note that by using the original layer graph, not all cliques can be ordered by its inDegree as $0,1,...,|clique|-1$. For example in $T=aXYbaYXb$ the is the cliques $\{a,Y,B\}$ and $\{a,X,B\}$  which has not correct degree ordering.

```python
G_T=pair_graph('aXYbaYXb')
G_T_2 =  n_layer(G_T, 2)
graph( G_T_2 )
cliques=cliques_from_layer( G_T_2 )
degrees=sorted( G_T_2.in_degree() , key=lambda u: u[1], reverse=False)

for clique in cliques:
    print([ (node,deg) for node,deg in degrees if node in clique ])
```

However, the subgraph induced by the clique actually has correct degree ordering.

```python
indD = induced_graph(G_T_2, ['b', 'a', 'X'])
graph( indD )
```

Then the procedure must be improved as:

```python
for clique in cliques:
    degrees=sorted( induced_graph(G_T_2, clique).in_degree() , key=lambda u: u[1], reverse=False)
    print([ (node,deg) for node,deg in degrees if node in clique ])
```

    [('a', 0), ('X', 1), ('b', 2)]
    [('a', 0), ('Y', 1), ('b', 2)]

The procedure above is included in this function:

```python
sequences_from_pair_graph( G_T )
```

    {2: [['a', 'X', 'b'], ['a', 'Y', 'b']]}

## More sequence detection examples

```python
G_T=pair_graph('abc')
sequences_from_pair_graph( G_T )
```

    {1: [['a', 'b', 'c']]}

```python
G_T=pair_graph('abc123abc')
sequences_from_pair_graph( G_T )
```

    {1: [['1', '2', '3']], 2: [['a', 'b', 'c']]}

```python
G_T=pair_graph('abc123abc123123')
sequences_from_pair_graph( G_T )
```

    {3: [['1', '2', '3']], 2: [['a', 'b', 'c']]}

```python
G_T=pair_graph('1ab2c3a12312b3c')
sequences_from_pair_graph( G_T )
```

    {3: [['1', '2', '3']], 2: [['a', 'b', 'c']]}

```python
G_T=pair_graph('aXYbaYXb')
sequences_from_pair_graph( G_T )
```

    {2: [['a', 'X', 'b'], ['a', 'Y', 'b']]}

## Overhead in Python

The experiments below shows no significant overhead by using pandas or networkx library.

```python
for i in [100,200,300,400,500]:
    print("\n\n computations with sigma={}\n".format(i))
    %time a=pairs_in_trace( range(i) )
    %time a=pandasPair( pairs_in_trace( range(i) )  )
    %time a=pair_graph( range(i) )
```

     computations with sigma=100

    CPU times: user 35.6 ms, sys: 1.28 ms, total: 36.9 ms
    Wall time: 36.1 ms
    CPU times: user 63.9 ms, sys: 1.53 ms, total: 65.4 ms
    Wall time: 65.6 ms
    CPU times: user 72.7 ms, sys: 1.75 ms, total: 74.5 ms
    Wall time: 74.5 ms

     computations with sigma=200

    CPU times: user 248 ms, sys: 2.31 ms, total: 250 ms
    Wall time: 250 ms
    CPU times: user 354 ms, sys: 5.83 ms, total: 360 ms
    Wall time: 360 ms
    CPU times: user 401 ms, sys: 7.93 ms, total: 409 ms
    Wall time: 409 ms

     computations with sigma=300

    CPU times: user 823 ms, sys: 6.25 ms, total: 830 ms
    Wall time: 830 ms
    CPU times: user 1.08 s, sys: 14.3 ms, total: 1.09 s
    Wall time: 1.09 s
    CPU times: user 1.18 s, sys: 16.9 ms, total: 1.19 s
    Wall time: 1.2 s

     computations with sigma=400

    CPU times: user 2.01 s, sys: 7.85 ms, total: 2.02 s
    Wall time: 2.02 s
    CPU times: user 2.47 s, sys: 20.5 ms, total: 2.49 s
    Wall time: 2.5 s
    CPU times: user 2.66 s, sys: 30.6 ms, total: 2.69 s
    Wall time: 2.7 s

     computations with sigma=500

    CPU times: user 4.01 s, sys: 16.7 ms, total: 4.03 s
    Wall time: 4.03 s
    CPU times: user 4.68 s, sys: 41.7 ms, total: 4.72 s
    Wall time: 4.73 s
    CPU times: user 5.05 s, sys: 55.4 ms, total: 5.11 s
    Wall time: 5.11 s

```python

```


We want to generate questions where we ask the student if a polynomial equation $aX^2 + bX +c = 0$ with integer coefficients has a solution or no.

```python
from moodlexport import Category, Question
from sympy.abc import x # allows to define polynomials in x
from sympy import latex # conveniently returns latex strings

coefficients = [-3, -2, -1, 1, 2, 3] # the coefficients we allow
category = Category("Number of roots for a polynomial") # Gather all the questions

for a in coefficients: # we loop over the allowed coefficients
    for b in coefficients:
        for c in coefficients:
            question = Question("multichoice") # a new question
            P = a*x**2 + b*x + c # our sympy polynomial
            question.text("The equation $"+latex(P)+"=0$ has:")
            question.answer("at least a solution", b**2 - 4*a*c > 0 )
            question.answer("no real solution", b**2 - 4*a*c <= 0 )
            question.addto(category) # question goes into the category

category.savepdf() # save the category
category.savexml() # export the category to Moodle
```

```python

```


# Latent Factor DCMM Analysis

This example highlights several advanced modeling strategies.

**Dynamic Count Mixture Model**

A Dynamic Count Mixture Model (DCMM) is the combination of Bernoulli and Poisson DGLMs. The Bernoulli DGLM models the probability of a zero outcome, while the Poisson models the outcome, conditional on it being non-zero (Berry and West, 2019):

\begin{equation} \label{eqn-dcmm}
z_t \sim Ber(\pi_t) \text{ and } y_t \mid z_t =
\begin{cases}
0, & \text{if } z_t = 0,\\
1 + x_t, \quad x_t \sim Po(\mu_t), & \text{if }z_t = 1
\end{cases}
\end{equation}

where $\pi_t$ and $\mu_t$ vary according to the dynamics of independent Bernoulli and Poisson DGLMs respectively:

\begin{equation*}
\text{logit}(\pi_t) = \mathbf{F}_{ber, t}^{'}\boldsymbol{\theta}_{ber, t} \qquad \text{and} \qquad \text{log}(\mu_t) = \mathbf{F}_{Po, t}^{'} \boldsymbol{\theta}_{Po, t}
\end{equation*}

A DCMM is useful to capture a higher prevalence of $0$ outcomes than in a standard Poisson DGLM.

**Latent Factors**

The second concept is the use of *latent factors* for multiscale modeling. This example features simulated data from a retail sales setting. There is data for an item, and also for total sales at the store. The total sales are smoother and more predictable than the item level sales. A model is fit to the total sales which includes a day-of-week seasonal effect. This seasonal effect is extracted from the model on total sales, and used as a predictor in the item level model.

Modeling effects at different levels of a hierarchy - aka multiscale modeling - can improve forecast accuracy, because the sales of an individual can be very noisy, making it difficult to learn accurate patterns.

**Copula Forecasting**

The final concept is Copula forecasting. In this example we are focusing on forecasting sales *1* through *14* days into the future. To do this, we will simulate from the joint forecast distribution *1:14* days into the future. This is accomplished in an efficient manner through the use of a Copula model, which accounts for dependence in the forecasts across days.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
from pybats.analysis import analysis, analysis_dcmm
from pybats.latent_factor import seas_weekly_lf
from pybats.shared import load_dcmm_latent_factor_example
from pybats.plot import plot_data_forecast
from pybats.point_forecast import median
from pybats.loss_functions import MAD
```

We start by loading the simulated retail sales data.

The dataframe 'totaldata' has the total daily sales in a store, along with a predictor, which represents a measure of average price.

The dataframe 'data' has the sales of a single item, and along with a predictor which represents the daily change in price.

```python
data = load_dcmm_latent_factor_example()
totaldata, data = data.values()
totaldata['Y'] = np.log(totaldata['Y'] + 1)
```

```python
totaldata.head()
```

      Y
      X

      2014-06-01
      5.613128
      -0.133124

      2014-06-02
      5.602119
      -0.070102

      2014-06-03
      5.831882
      1.020901

      2014-06-04
      5.393628
      2.219605

      2014-06-05
      5.267858
      -0.970556

```python
data.head()
```

      Y
      X

      2014-06-01
      14.0
      2.413448

      2014-06-02
      13.0
      0.989871

      2014-06-03
      7.0
      -0.489734

      2014-06-04
      34.0
      -1.544868

      2014-06-05
      15.0
      1.639670

Here we define hyper parameters for the analysis:
-  *rho* is a discount factor which calibrates the forecast distribution. A value of *rho* less than $1$ increases the forecast variance.

- *k* is the number of days ahead to forecast.

- *nsamps* is the number of forecast samples to draw.

- *prior_length* is the number of days of data to use when defining the priors for model coefficients.

```python
#Define hyper parameters
rho = .2
k = 14 # Number of days ahead that we will forecast
nsamps = 100
prior_length = 21
```

Next, we define the window of time that we want to forecast over. For each day within this time period, the model will sample from the path (joint) forecast distribution *1:k* days into the future.

```python
# Define forecast range for final year of data
T = len(totaldata)
forecast_end_date = totaldata.index[-k]
forecast_start_date = forecast_end_date - pd.DateOffset(days=365)
```

The first analysis is run on the 'totaldata', which is used to learn a latent factor. In this case, we fit a normal DLM to the log of total sales in order to learn the weekly season effect, which is stored in the latent factor.

```python
# Get multiscale signal (a latent factor) from higher level log-normal model
latent_factor = analysis(totaldata['Y'].values, totaldata['X'].values, k,
                         forecast_start_date, forecast_end_date, dates=totaldata.index,
                         seasPeriods=[7], seasHarmComponents=[[1,2,3]],
                         family="normal", ret=['new_latent_factors'], new_latent_factors= [seas_weekly_lf],
                         prior_length=prior_length)
```

    beginning forecasting

The second analysis fits a DCMM to 'data', which is the simulated sales of a single item. The latent factor we derived above is used as a predictive signal for the sales of this individual item. On each day within the forecasting period, a copula is used to draw joint forecast samples *1:k* days ahead.

```python
# Update and forecast the model
mod, forecast_samples = analysis_dcmm(data['Y'].values, data['X'].values.reshape(-1,1), k,
                                      forecast_start_date, forecast_end_date,
                                      prior_length=prior_length, nsamps=nsamps, rho=rho,
                                      latent_factor=latent_factor, dates=totaldata.index)

forecast = median(forecast_samples)
```

    beginning forecasting

Finally we can examine the results, first by plotting both the *1-* and *14-* step ahead forecasts. The point forecast is the median (blue line), and the forecast samples provide easy access to 95% credible intervals.

```python
# Plotting 1-step ahead forecasts

horizon = 1
plot_length = 50
fig, ax = plt.subplots(figsize=(8,4))
start_date = forecast_end_date + pd.DateOffset(horizon - plot_length)
end_date = forecast_end_date + pd.DateOffset(horizon - 1)
ax = plot_data_forecast(fig, ax, data.loc[start_date:end_date].Y,
                        forecast[-plot_length:,horizon - 1],
                        forecast_samples[:,-plot_length:,horizon - 1],
                        data.loc[start_date:end_date].index,
                        linewidth = 2)
```

```python
# Plotting 14-step ahead forecasts

horizon = 14
plot_length = 50
fig, ax = plt.subplots(figsize=(8,4))
start_date = forecast_end_date + pd.DateOffset(horizon - plot_length)
end_date = forecast_end_date + pd.DateOffset(horizon - 1)
ax = plot_data_forecast(fig, ax, data.loc[start_date:end_date].Y,
                        forecast[-plot_length:,horizon - 1],
                        forecast_samples[:,-plot_length:,horizon - 1],
                        data.loc[start_date:end_date].index,
                        linewidth = 2)
```

To evaluate the point forecasts, we can look at the mean absolute deviation (MAD) between the forecast median and the observations across forecast horizons. Interestingly, there is only a small increase in the MAD for longer forecast horizons.

```python
# Mean absolute deviation at increasing forecast horizons
horizons = list(range(1, k+1))
list(map(lambda k: MAD(data.loc[forecast_start_date + pd.DateOffset(k-1):forecast_end_date + pd.DateOffset(k-1)].Y,
                      forecast[:,k-1]),
         horizons))
```

    [2.8756830601092895,
     2.9139344262295084,
     2.8538251366120218,
     2.8743169398907105,
     2.8442622950819674,
     2.887978142076503,
     2.8647540983606556,
     2.819672131147541,
     2.898907103825137,
     2.8934426229508197,
     2.883879781420765,
     2.819672131147541,
     2.8852459016393444,
     2.8579234972677594]

```python

```

examples/DCMM Latent Factor Example.ipynb