# Modelagem SEIR
Neste notebook está implementado o modelo SEIR de Zhilan Feng. Que inclui quarentena, e hospitalizações

\begin{align}
\frac{dS}{dt}&=-\beta S (I+(1-\rho)H)\\
\frac{dE}{dt}&= \beta S (I+(1-\rho)H)-(\chi+\alpha)E\\
\frac{dQ}{dt}&=\chi E -\alpha Q\\
\frac{dI}{dt}&= \alpha E - (\phi+\delta)I\\
\frac{dH}{dt}&= \alpha Q +\phi I -\delta H\\
\frac{dR}{dt}&= \delta I +\delta H
\end{align}

Denotando o tamanho total da epidemia por $Y_e(t)=E(t)+Q(t)+I(t)+H(t)$, podemos escrever:

\begin{align}
\nonumber \frac{dY_e}{dt}&= \frac{dE}{dt}+\frac{dQ}{dt}+\frac{dI}{dt}+\frac{dH}{dt}\\
\label{ye}&=\beta S (I(1-\rho)H) -\delta(I+H)
\end{align}

```python
from scipy.integrate import odeint
from scipy.integrate import DOP853
import pandas as pd
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import warnings
import humanizer_portugues as hp
import seaborn as sns
from datetime import timedelta
warnings.filterwarnings("ignore")
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
def seqihr(y,t,*params):
    S,E,Q,I,H,R = y
    chi,phi,beta,rho,delta,alpha = params
    return[
        -beta*S*(I+(1-rho)*H), #dS/dt
        beta*S*(I+(1-rho)*H) - (chi+alpha)*E,#dE/dt
        chi*E -alpha*Q,#dQ/dt
        alpha*E - (phi+delta)*I,#dI/dt
        alpha*Q + phi*I -delta*H,#dH/dt
        delta*I + delta*H,#dR/dt
    ]
```

```python
chi=.05  # Taxa de quarentenação
phi=.01 # Taxa de Hospitalização
beta=.2 #taxa de Transmissão
rho=.6 # Atenuação da transmissão quando hospitalizado
delta=1/10 # Taxa de recuperação hospitalar
alpha=1/3 # Taxa de incubaçao
```

```python
inits = [0.9,.1,0,0,0,0]
trange = arange(0,100,.1)
res = odeint(seqihr,inits,trange,args=(chi,phi,beta,rho,delta,alpha))
```

```python

@interact(chi=(0,.1, 0.01),phi=(0,.1,.005),beta=(0,.5,.05),rho=(0,1,.1),delta=(0,1,.05),alpha=(0,1,0.01))
def plota_simulação(chi=0.05,phi=.01,beta=.2,rho=.6,delta=.1,alpha=.33):
    res = odeint(seqihr,inits,trange,args=(chi,phi,beta,rho,delta,alpha))
    fig, ax = subplots(1,1, figsize=(15,10))
    ax.plot(trange,res[:,1:-1])
    ax.set_ylabel('Fração da População')
    ax.set_xlabel('Tempo (dias)')
    ax.grid()
    ax.legend(['Exposto','Quar','Inf','Hosp']);
```

    interactive(children=(FloatSlider(value=0.05, description='chi', max=0.1, step=0.01), FloatSlider(value=0.01, …

## Adicionando Assintomáticos

O modelo de Feng não é apropriado para representar a epidemia da COVID-19. Vamos Modificá-lo para incluir entre outras coisas, os assintomáticos. Para simplificar a notação vamos denotar por $\lambda=\beta(I\color{red}{+A}+(1-\rho)H)$, já incluindo os assintomáticos, $A$.

\begin{align}
\frac{dS}{dt}&=-\lambda[\color{red}{(1-\chi)} S] \\
\frac{dE}{dt}&= \lambda [\color{red}{(1-\chi)} S] -\alpha E\\
\frac{dI}{dt}&= \color{red}{(1-p)}\alpha E - (\phi+\delta)I\\
\color{red}{\frac{dA}{dt}}&= \color{red}{p\alpha E -\delta A}\\
\frac{dH}{dt}&= \phi I -\delta H\\
\frac{dR}{dt}&= \delta I +\delta H \color{red}{+ \delta A}
\end{align}

Neste novo modelo alteramos mais algumas coisinhas ressaltadas em vermelho acima. Em primeiro lugar, fizemos com que a quarentena seja aplicada aos suscetíveis, efetivamente removendo-os da cadeia de transmissão, para simplificar removemos o compartimento $Q$. Além disso, para alimentar a classe dos Assintomáticos, definimos $p$ como a fração de assintomáticos na população. $R_0\approx$

### Adicionando ou removendo a quarentena
Para permitir que a quarentena seja acionada no modelo em uma data definida por um parâmetro $q$, precisamos expressá-la como uma função diferenciável que varie de zero a um rapidamente. Esta função é então multiplicada pelo parâmetro de quarentena $\chi$.
\begin{equation}
    \chi^+(t)=\chi\frac{1+tanh(t-q)}{2}
\end{equation}

Para remover a quarentena podemos usar a função com o sinal trocado na tangente hiperbólica.
\begin{equation}
    \chi^-(t)=\chi\frac{1-tanh(t-q)}{2}
\end{equation}
Abaixo temos uma exemplificação grafica de $\chi^+(t)$ e $\chi^-(t)$

```python
t = arange(-3,10,.01)
d=1
q = 0.3 #Nivel de quarentena desejado
addq = lambda t,d: (1+tanh(t-d))/2
remq = lambda t,d: (1-tanh(t-d))/2
f,(ax1,ax2)=subplots(1,2, figsize=(15,8))
ax1.plot(t,q*addq(t,d), label=f"liga a quarent. no dia {d}")
ax1.plot(t,q*remq(t,d), label=f"desliga a quarent. no dia {d}")
ax2.plot(t,q*addq(t,d)*remq(t,d+5))
ax1.vlines(d,0,q)
ax2.vlines(d,0,q,'g')
ax2.vlines(d+5,0,q, 'r')
ax1.grid()
ax2.grid()
ax1.legend();
```

```python
f,ax=subplots(1,1, figsize=(15,8))
ax.plot(t,q*addq(t,d)*remq(t,d+5))
ax.vlines(d,0,q,'g')
ax.vlines(d+5,0,q, 'r')
ax.set_ylabel('Fraction Quarantined')
ax.set_xlabel('Time(days)')
ax.grid()
savefig('chi_of_t.png')
```

```python
def seqiahr(y,t,*params):
    S,E,I,A,H,R,C,M = y
    chi,phi,beta,rho,delta,alpha,mu,p,q,r = params
    lamb = beta*(I+A)
    chi *= ((1+np.tanh(t-q))/2) * ((1 - np.tanh(t - (q+r))) / 2 )
    return[
        -lamb*((1-chi)*S), #dS/dt
        lamb*((1-chi)*S) - alpha*E,#dE/dt
        (1-p)*alpha*E - delta*I,#dI/dt
        p*alpha*E - delta*A,
        phi*delta*I -(rho+mu)*H,#dH/dt
        (1-phi)*delta*I + rho*H+delta*A ,#dR/dt
        phi*delta*I,#(1-p)*alpha*E+ p*alpha*E # Casos acumulados
        mu*H  #Mortes acumuladas
    ]
```

```python
trange = arange(0, 365, 1)
inits = [0.99, 0, 1.0277e-6, 0.0, 0, 0, 0, 0]
N = 97.3e6
fat = 0.035  # case fatality rate
# sumario = open('cenarios.csv','w')
# sumario.write('R0,Quarentena,tamanho_total,hosp_total,hosp_pico,morte_total\n')

def r0(chi, phi, beta, rho, delta, p, S=1):
    "R0 for seqiahr2"
    return -(S*beta*chi - S*beta)/delta

@interact(χ=(0, 1, 0.05),
          φ=(0, .5, .01),
          β=(0, 1, .02),
          ρ=(0, 1, .1),
          δ=(0, 1, .05),
          α=(0, 10, 0.01),
          mu=(0,1,.01),
          p=(0, 1, .05),
          q=(0, 120, 10),
          r=(0,100,10))
def plota_simulação(χ=0.7, φ=.01, β=.5, ρ=0.05, δ=.1, α=.33, mu=0.03, p=.75, q=30, r=10):
    res = odeint(seqiahr, inits, trange, args=(χ, φ, β, ρ, δ, α, mu, p, q,r))
    rzero = r0(χ, φ, β, ρ, δ, p)
#     et = 1 / rzero
#     idx = np.argwhere(np.abs(res[:, 0] - et) == min(np.abs(res[:, 0] -
#                                                            et)))[0, 0]
    fig, ax = subplots(1, 1, figsize=(15, 10))
    ax.plot(trange, res[:, 1])#E
#     ax.plot(trange, res[:, 2:4])
    ax.plot(trange, res[:, 4]) #H
    ax.plot(trange, res[:,-1]) #M
    ax.set_ylim([0, 0.02])
    Ye = res[:, 1:4].sum(axis=1)
    Imax = (1 - res[-1, 0]) * N  # N-S(inf)
    Hosp_p = res[:, 4].max() * N  # pico de Hospitalizações
    Hosp_tot = (res[-1, -2]) * N
    casos = (res[:, 2] + res[:, 4])  # Casos notificados H+I
    casosT = Hosp_tot + ((1 - p) * α * res[:, 1]).sum() * N  # N*sum(p*α*E)
    M = res[-1,-1]*N
    #     secax = ax.twinx()
    #     secax.plot(trange, Ye,'k:')
    #     secax.set_ylabel('Total prevalence fraction')
    ax.text(0,
            0.005,
            f"$R_0~{rzero:.1f}$\n$p={p}$\n$\\rho={ρ}$\n$\\chi={χ*100}$%",
            fontsize=16)
    ax.text(
        110,
        .01,
        f"Tamanho total: {hp.intword(Imax)} infectados\nHosp. pico: {hp.intword(Hosp_p)}\nHosp. totais: {hp.intword(Hosp_tot)}\nMortes: {hp.intword(M)}",
        fontsize=16)
#     ax.vlines(trange[idx], 0, 0.02,'b')  #res[:,1:-1].max())
    ax.vlines(q, 0, .02)
    ax.vlines(q+r, 0, .02,'r')
    ax.set_ylabel('Fração da População')
    ax.set_xlabel('Tempo (dias)')
    ax.grid()
    #         sumario.write(f'{rzero},{χ*100},{Imax},{Hosp_tot},{Hosp_p},{M}\n')
#     ax.legend(['Exposto', 'Inf', 'Assintomático', 'Hosp', 'Mortes'])
    ax.legend(['Exposto', 'Hospitalizado', 'Mortes'])
#     plt.savefig(f"", dpi=300)
```

    interactive(children=(FloatSlider(value=0.7, description='χ', max=1.0, step=0.05), FloatSlider(value=0.01, des…

```python
# sumario.close()
# suma = pd.read_csv('cenarios.csv')
# suma.drop_duplicates(inplace=True)
# suma
```

```python
def save_data():
    fig,ax = subplots(1,1, figsize=(15,10), constrained_layout=True)
    χ=0.05;φ=.01;β=.25;ρ=.6;δ=.1;α=10; p=.75; q=0
    i=1
    dff = pd.DataFrame(data={'Sem controle':range(365),'Com controle':range(365)})
    for q,R0,c in [(0,3.6,'Sem controle'),(10,1.7,'Com controle')]:
        res = odeint(seqiahr2,inits,trange,args=(q/100,φ,R0/10,ρ,δ,α,p,q))
        df = pd.DataFrame(data=res*N, columns=['S','E','I','A','H','R','C'],index=trange)
        ax = df[['H']].plot(ax=ax, grid=True)
        dff[c] =df.H
    plt.legend(['Sem Controles','Com controles'])
    plt.savefig(f'export/achatando a curva.png', dpi=300)
    dff.to_csv(f'export/achatando a curva.csv')

save_data()
```

## Carregando os dados do Brasil

```python
# Pegando os casos do Brasil.io
cases = pd.read_csv('https://brasil.io/dataset/covid19/caso?format=csv')
cases.date = pd.to_datetime(cases.date)
```

```python
df_states = cases[cases.place_type!='state'].groupby(['date','state']).sum()
df_states.reset_index(inplace=True)
df_states.set_index('date', inplace=True)
```

```python
# df_states.set_index('date', inplace=True)
fig,ax = subplots(1,1,figsize=(15,8))
for uf in ['SP','RJ','MG','CE','RS']:
    df_states[df_states.state==uf].confirmed.plot(style='-o', label=uf)

ax.legend()
plt.savefig('Casos_confirmados_estados.png',dpi=200)
```

```python
# Pegando os dados do Ministério
casos = pd.read_csv('COVID19_20200411.csv',sep=';')
casos.data = pd.to_datetime(casos.data, dayfirst=True)
casos_estados = casos.groupby(['data','estado']).sum()
casos_estados
```

      casosNovos
      casosAcumulados
      obitosNovos
      obitosAcumulados

      data
      estado

      2020-01-30
      AC
      0
      0
      0
      0

      AL
      0
      0
      0
      0

      AM
      0
      0
      0
      0

      AP
      0
      0
      0
      0

      BA
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...
      ...

      2020-04-11
      RS
      4
      640
      1
      15

      SC
      39
      732
      3
      21

      SE
      0
      42
      0
      4

      SP
      203
      8419
      20
      560

      TO
      0
      23
      0
      0

1971 rows × 4 columns

```python
casos.columns
```

    Index(['regiao', 'estado', 'data', 'casosNovos', 'casosAcumulados',
           'obitosNovos', 'obitosAcumulados'],
          dtype='object')

```python
fig,ax = subplots(1,1,figsize=(15,8))
casos_estados.reset_index(inplace=True)
casos_estados.set_index('data', inplace=True)
for uf in ['SP','RJ','MG','CE','RS', 'BA']:
    casos_estados[casos_estados.estado==uf].casosAcumulados.plot(style='-o', label=uf)

ax.legend()
plt.savefig('Casos_Acumulados_estados.png',dpi=200);
```

```python
trange = arange(0,1095,1)
χ=0.0;φ=.1;β=.5;ρ=.6;δ=.1;α=.53; p=.75; q=-2; mu=0.02; r=1000
inits = [0.99,0,1.0277e-8, 0.0,0,0,0,0]
res = odeint(seqiahr,inits,trange,args=(χ,φ,β,ρ,δ,α,mu,p,q,r))
# Com controle
χ=0.1;β=.17
res_c = odeint(seqiahr,inits,trange,args=(χ,φ,β,ρ,δ,α,mu,p,q, r))
plt.plot(trange,res[:,-1],label='no control');
plt.plot(trange,res_c[:,-1],label='with control');
plt.legend();
```

```python
df_brasil = casos_estados.groupby('data').sum()
# df_brasil['casos_simulados']=res[:dias,-1]*N
# dfcp = df_brasil[['confirmed']]
# dfcp.to_csv('export/dados_brasil.csv')
df_brasil
```

      casosNovos
      casosAcumulados
      obitosNovos
      obitosAcumulados

      data

      2020-01-30
      0
      0
      0
      0

      2020-01-31
      0
      0
      0
      0

      2020-02-01
      0
      0
      0
      0

      2020-02-02
      0
      0
      0
      0

      2020-02-03
      0
      0
      0
      0

      ...
      ...
      ...
      ...
      ...

      2020-04-07
      1661
      13717
      114
      667

      2020-04-08
      2210
      15927
      133
      800

      2020-04-09
      1930
      17857
      141
      941

      2020-04-10
      1781
      19638
      115
      1056

      2020-04-11
      1089
      20727
      68
      1124

73 rows × 4 columns

```python
df_brasil.casosAcumulados.plot();
```

```python
dias=400
offset = 35 # quantos dias antes do primeiro caso notificado devemos começar a simulaçao
drange = pd.date_range(df_brasil[df_brasil.casosAcumulados>0].index.min()-timedelta(offset),periods=dias,freq='D')
# df_states.reset_index(inplace=True)
# df_brasil = df_states.groupby('date').sum()
fig,ax = subplots(1,1,figsize=(15,8))
ax.plot(drange,res[:dias,-1]*N,'-v', label='Sem controle')
# ax.plot(drange,res[:dias,-3]*N,'-v', label='Hosp')
# ax.vlines('2020-05-5',0,1e6)
ax.plot(drange,res_c[:dias,-1]*N,'-v', label='Com controle (Simulados)')
# ax.plot(drange,res[:dias,2]*N,'-^', label='Prevalência')
df_brasil[df_brasil.casosAcumulados>0].casosAcumulados.plot(ax=ax, style='-o',
                                                            label='Dados oficiais',
                                                            grid=True,
                                                            logy=True)
# Sem controle
ax.text('2020-05-15',0.6e5, f'Casos Totais: {res[dias,-1]*N:.0f}\nMortes: {res[dias,-1]*N*fat:.0f}',
        fontsize=16)
# com 10% quarentena e R0=1.7
ax.text('2020-11-15',1e3, f'Casos Totais: {res_c[dias,-1]*N:.0f}\nMortes:{res_c[dias,-1]*N*fat:.0f}',
        fontsize=16)
ax.legend();
# plt.savefig('export/Casos_vs_Projeções_log.png',dpi=200)
```

```python
df_sim = pd.DataFrame(data={'sem controle':res[:dias,-1]*N,'com controle':res_c[:dias,-1]*N}, index=drange)
df_sim.to_csv('simulação_brasil_com_e_sem_controle.csv')
```

## Comparando a série do Brasil com a de outros países

```python
confirmed = pd.read_csv('https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')
confirmed = confirmed.groupby('Country/Region').sum()
```

```python
confirmed
```

      Lat
      Long
      1/22/20
      1/23/20
      1/24/20
      1/25/20
      1/26/20
      1/27/20
      1/28/20
      1/29/20
      ...
      3/25/20
      3/26/20
      3/27/20
      3/28/20
      3/29/20
      3/30/20
      3/31/20
      4/1/20
      4/2/20
      4/3/20

      Country/Region

      Afghanistan
      33.0000
      65.0000
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      84
      94
      110
      110
      120
      170
      174
      237
      273
      281

      Albania
      41.1533
      20.1683
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      146
      174
      186
      197
      212
      223
      243
      259
      277
      304

      Algeria
      28.0339
      1.6596
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      302
      367
      409
      454
      511
      584
      716
      847
      986
      1171

      Andorra
      42.5063
      1.5218
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      188
      224
      267
      308
      334
      370
      376
      390
      428
      439

      Angola
      -11.2027
      17.8739
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      3
      4
      4
      5
      7
      7
      7
      8
      8
      8

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      Venezuela
      6.4238
      -66.5897
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      91
      107
      107
      119
      119
      135
      135
      143
      146
      153

      Vietnam
      16.0000
      108.0000
      0
      2
      2
      2
      2
      2
      2
      2
      ...
      141
      153
      163
      174
      188
      203
      212
      218
      233
      237

      West Bank and Gaza
      31.9522
      35.2332
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      59
      84
      91
      98
      109
      116
      119
      134
      161
      194

      Zambia
      -15.4167
      28.2833
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      12
      16
      22
      28
      29
      35
      35
      36
      39
      39

      Zimbabwe
      -20.0000
      30.0000
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      3
      3
      5
      7
      7
      7
      8
      8
      9
      9

181 rows × 75 columns

```python
conf_US = pd.read_csv('https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_US.csv')
```

```python
conf_US
```

      UID
      iso2
      iso3
      code3
      FIPS
      Admin2
      Province_State
      Country_Region
      Lat
      Long_
      ...
      3/25/20
      3/26/20
      3/27/20
      3/28/20
      3/29/20
      3/30/20
      3/31/20
      4/1/20
      4/2/20
      4/3/20

      0
      16
      AS
      ASM
      16
      60.0
      NaN
      American Samoa
      US
      -14.2710
      -170.1320
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1
      316
      GU
      GUM
      316
      66.0
      NaN
      Guam
      US
      13.4443
      144.7937
      ...
      37
      45
      51
      55
      56
      58
      69
      77
      82
      84

      2
      580
      MP
      MNP
      580
      69.0
      NaN
      Northern Mariana Islands
      US
      15.0979
      145.6739
      ...
      0
      0
      0
      0
      0
      0
      2
      6
      6
      6

      3
      630
      PR
      PRI
      630
      72.0
      NaN
      Puerto Rico
      US
      18.2208
      -66.5901
      ...
      51
      64
      79
      100
      127
      174
      239
      286
      316
      316

      4
      850
      VI
      VIR
      850
      78.0
      NaN
      Virgin Islands
      US
      18.3358
      -64.8963
      ...
      17
      17
      19
      22
      0
      0
      30
      30
      30
      37

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      3248
      84090053
      US
      USA
      840
      90053.0
      Unassigned
      Washington
      US
      0.0000
      0.0000
      ...
      51
      69
      67
      0
      125
      274
      274
      303
      344
      501

      3249
      84090054
      US
      USA
      840
      90054.0
      Unassigned
      West Virginia
      US
      0.0000
      0.0000
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3250
      84090055
      US
      USA
      840
      90055.0
      Unassigned
      Wisconsin
      US
      0.0000
      0.0000
      ...
      0
      0
      61
      0
      0
      0
      0
      0
      0
      0

      3251
      84090056
      US
      USA
      840
      90056.0
      Unassigned
      Wyoming
      US
      0.0000
      0.0000
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      3252
      84099999
      US
      USA
      840
      99999.0
      NaN
      Grand Princess
      US
      0.0000
      0.0000
      ...
      28
      28
      28
      103
      103
      103
      103
      103
      103
      103

3253 rows × 84 columns

```python
conf_US.groupby('Country_Region').sum()
```

      UID
      code3
      FIPS
      Lat
      Long_
      1/22/20
      1/23/20
      1/24/20
      1/25/20
      1/26/20
      ...
      3/25/20
      3/26/20
      3/27/20
      3/28/20
      3/29/20
      3/30/20
      3/31/20
      4/1/20
      4/2/20
      4/3/20

      Country_Region

      US
      272936469664
      2730712
      104327612.0
      120958.868944
      -290083.749251
      1
      1
      2
      2
      5
      ...
      65778
      83836
      101657
      121478
      140886
      161807
      188172
      213362
      243453
      275582

1 rows × 78 columns

```python
serie_US = conf_US.groupby('Country_Region').sum().T
serie_US = serie_US.iloc[5:]
serie_US.index  = pd.to_datetime(serie_US.index)
serie_US.index.name = 'data'
# serie_US['acumulados'] = np.cumsum(serie_US.US)
serie_US
```

      Country_Region
      US

      data

      2020-01-22
      1.0

      2020-01-23
      1.0

      2020-01-24
      2.0

      2020-01-25
      2.0

      2020-01-26
      5.0

      ...
      ...

      2020-03-30
      161807.0

      2020-03-31
      188172.0

      2020-04-01
      213362.0

      2020-04-02
      243453.0

      2020-04-03
      275582.0

73 rows × 1 columns

```python
# outros = pd.DataFrame(data=)
suecia = confirmed.loc['Sweden'][2:]
alemanha = confirmed.loc['Germany'][2:]
espanha = confirmed.loc['Spain'][2:]
italia = confirmed.loc['Italy'][2:]
outros = pd.concat([suecia,alemanha,espanha,italia], axis=1)
outros.index = pd.to_datetime(outros.index)
# outros['Sweden_acc'] = np.cumsum(outros.Sweden)
# outros['Germany_acc'] = np.cumsum(outros.Germany)
# outros['Spain_acc'] = np.cumsum(outros.Spain)
# outros['Italy_acc'] = np.cumsum(outros.Italy)
outros
```

      Sweden
      Germany
      Spain
      Italy

      2020-01-22
      0.0
      0.0
      0.0
      0.0

      2020-01-23
      0.0
      0.0
      0.0
      0.0

      2020-01-24
      0.0
      0.0
      0.0
      0.0

      2020-01-25
      0.0
      0.0
      0.0
      0.0

      2020-01-26
      0.0
      0.0
      0.0
      0.0

      ...
      ...
      ...
      ...
      ...

      2020-03-30
      4028.0
      66885.0
      87956.0
      101739.0

      2020-03-31
      4435.0
      71808.0
      95923.0
      105792.0

      2020-04-01
      4947.0
      77872.0
      104118.0
      110574.0

      2020-04-02
      5568.0
      84794.0
      112065.0
      115242.0

      2020-04-03
      6131.0
      91159.0
      119199.0
      119827.0

73 rows × 4 columns

```python
fig,ax = subplots(1,1,figsize=(15,8))
df_brasil[df_brasil.casosAcumulados>0].casosAcumulados.plot(ax=ax, style='-o',
                                                            label='Brasil',
                                                            grid=True,
                                                            logy=True)
serie_US.US.plot(ax=ax, style='-v', grid=True,
                        label='EUA',);
outros.Sweden.plot(ax=ax, style='-v', grid=True,
                        label='Suécia')
outros.Germany.plot(ax=ax, style='-v', grid=True,
                        label='Alemanha')
outros.Spain.plot(ax=ax, style='-v', grid=True,
                        label='Espanha')
outros.Italy.plot(ax=ax, style='-v', grid=True,
                        label='Italia')
ax.legend();
```

```python
fig,ax = subplots(1,1,figsize=(15,8))
alinhados = pd.concat([suecia[suecia>100].reset_index(),
           alemanha[alemanha>100].reset_index(),
           espanha[espanha>100].reset_index(),
           italia[italia>100].reset_index()],
          axis=1)
alinhados['EUA'] = serie_US[serie_US.US>100].reset_index().US
alinhados['Brasil'] = df_brasil[df_brasil.casosAcumulados>100].reset_index().casosAcumulados
alinhados.plot(ax=ax,logy=True, grid=True);
plt.savefig('export/Brasil_vs_outros.png', dpi=300)
alinhados.to_csv('export/Brasil_vs_outros.csv')
```

```python
serie_US[serie_US.acumulados>100].reset_index().acumulados
```

    0         109.0
    1         120.0
    2         131.0
    3         142.0
    4         154.0
    5         166.0
    6         179.0
    7         192.0
    8         205.0
    9         218.0
    10        231.0
    11        244.0
    12        257.0
    13        270.0
    14        285.0
    15        300.0
    16        315.0
    17        330.0
    18        345.0
    19        360.0
    20        376.0
    21        392.0
    22        416.0
    23        446.0
    24        499.0
    25        572.0
    26        676.0
    27        848.0
    28       1065.0
    29       1401.0
    30       1851.0
    31       2365.0
    32       3073.0
    33       4178.0
    34       5735.0
    35       7882.0
    36      10739.0
    37      13657.0
    38      17964.0
    39      24060.0
    40      32933.0
    41      46957.0
    42      66187.0
    43      91814.0
    44     125435.0
    45     169098.0
    46     222834.0
    47     288612.0
    48     372448.0
    49     474105.0
    50     595583.0
    51     736469.0
    52     898276.0
    53    1086448.0
    54    1299810.0
    55    1543263.0
    Name: acumulados, dtype: float64

```python
1-timedelta(32)
```

```python
pd.date_range(df_brasil[df_brasil.casosAcumulados>0].index.min()-timedelta(offset),periods=dias,freq='D')
```

    DatetimeIndex(['2020-01-22', '2020-01-23', '2020-01-24', '2020-01-25',
                   '2020-01-26', '2020-01-27', '2020-01-28', '2020-01-29',
                   '2020-01-30', '2020-01-31',
                   ...
                   '2021-02-15', '2021-02-16', '2021-02-17', '2021-02-18',
                   '2021-02-19', '2021-02-20', '2021-02-21', '2021-02-22',
                   '2021-02-23', '2021-02-24'],
                  dtype='datetime64[ns]', length=400, freq='D')

```python

```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

```python
from __future__ import print_function
from __future__ import absolute_import

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Interpolation

There are times when you have estimates for the values of a function for specific inputs. The values of the function may be obtained in a variety of ways either through experiment or through the use of other approximation techniques.  Our goal in this chapter is to explore techniques that allow us to determine a new function whose values match the known observations at a set of predetermined input values. We first formally define the term we will use to describe the process.

**Definition:**  Given a discrete set of values $y_i$ at locations $x_i$, an __*interpolant*__ is a (piece-wise) continuous function $f(x)$ that passes exactly through the data (*i.e.* $f(x_i) = y_i$).

**Example 0** The linear polynomial

$$
    P_1(x) = 2(x-1)+3
$$

interpolates the coordinates $(1,3)$ and $(3,7)$.

In general a polynomial of degree $N$ can be used to interpolate $N+1$ data points. There are many different kinds of functions to use to interpolate values, but here we focus on polynomials.

## Applications

 - Data filling
 - Function approximation: only have data on discrete set of x, approximate continuous function
 - Fundamental component of other algorithms
   - Root finding (secant method)
   - Optimization, minima/maxima (successive parabolic interpolation)
   - Numerical integration and differentiation
   - The Finite Element Method

## Polynomial Interpolation

**Theorem:**  There is a *unique* polynomial of degree $N$, $P_N(x)$, that passes exactly through $N + 1$ values $y_1, y_2, \ldots, y_N, y_{N+1}$ at *distinct* points $x_1, x_2, \ldots, x_N, x_{N+1}$.

Consequence of the number of unknowns in $P_N(x)$.

#### Example 1: 2 Points

Given points are $(x_0, y_0)$ and $(x_1, y_1)$ which will lead to a line:

Define $P_1(x) = p_1 x + p_0$ and use the two points to find $p_0$ and $p_1$:

We first note that we have two equations and two unknowns. The two equations can be found by assuming the function $P_1(x)$ interpolates the two data points
$$
    \begin{align}
       y_0 &= p_1 x_0 + p_0, \\
       y_1 &= p_1 x_1 + p_0.
    \end{align}
$$
In this example we will solve the first equation for $p_0$, substitute the result into the second equation, and then solve for $p_1$.

$$y_0 = p_1 x_0 + p_0 \quad \Rightarrow \quad p_0 = y_0 - p_1 x_0$$

Or in matrix form:
$$
\begin{bmatrix}
    y_0 \\
    y_1
\end{bmatrix} =
\begin{bmatrix}
    1 & x_0 \\
    1 & x_1
\end{bmatrix}
\begin{bmatrix}
    p_0 \\
    p_1
\end{bmatrix}
$$

$$\begin{aligned}
    y_1 &= p_1 x_1 + p_0 & \Rightarrow \\
    y_1 &= p_1 x_1 + y_0 - p_1 x_0 & \Rightarrow \\
    p_1 &= \frac{y_1 - y_0}{x_1 - x_0} & \Rightarrow \\
    p_0 &= y_0 - \frac{y_1 - y_0}{x_1 - x_0} x_0 &
\end{aligned}$$

$$P_1(x) = \frac{y_1 - y_0}{x_1 - x_0} x + y_0 - \frac{y_1 - y_0}{x_1 - x_0} x_0 = \frac{y_1 - y_0}{x_1 - x_0} (x - x_0) + y_0$$

#### Example 2:  3 Points

Given points are $(x_0, y_0)$, $(x_1, y_1)$, and $(x_2, y_2)$ which will lead to quadratic polynomial:

Define $P_2(x) = p_0 x^2 + p_1 x + p_2$ leading to the equations

$$y_0 = p_2 x_0^2 + p_1 x_0 + p_0$$

$$y_1 = p_2 x_1^2 + p_1 x_1 + p_0$$

$$y_2 = p_2 x_2^2 + p_1 x_2 + p_0$$

This gets complicated quickly! Note, we have three equations and three unknowns, and the previous system is a linear system of three equations.
and in general, the problem will reduce to a linear system
$$
    A(\mathbf{x})\mathbf{p} = \mathbf{y}
$$
A more general approach to solving the system will be explored later, but first it is important to determine whether or not the system even has a solution.

*A must be square and invertible.

### Proof - Uniqueness of Polynomial Interpolants

Let

$$\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n $$
or
$$\mathcal{P}_N(x) = p_0 + p_1 x + \cdots + p_{N - 1} x^{N - 1} + p_{N} x^N$$

and require $\mathcal{P}_N(x_i) = y_i$ for $i=0,1,\ldots,N$ and $x_i \neq x_j ~~~ \forall i,j$.

### Preliminaries: Monomial Basis

We can think of  $\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n$ as a polynomial,  or more fundamentally as a *linear combination* of a set of simpler functions, the monomials

$$1, x, x^2, x^3, \ldots, x^{N-1}, x^N$$

with weights

$$p_0, p_1, p_2, p_3, \ldots, p_{N-1}, \text{and } p_N$$

respectively.

### Linear independence of the Monomials

The monomials, form a *linearly independent* set of functions such that no monomial $x^n$ can be written as a linear combination of any other monomial (i.e. can't form another monomial in the set by multiplying one monomial by a scalar and adding itself).  We can see this graphically, for the first few monomials.

```python
x = numpy.linspace(-1,1,100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
for n in range(4):
    axes.plot(x,x**n,label='$x^{}$'.format(n))
axes.set_xlabel('x')
axes.grid()
axes.legend(loc='best')
axes.set_title('The First 4 Monomials')
plt.show()
```

But more fundamentally. A set of functions is linearly independent if the only linear combination that add to form the zero function, e.g.

$$
    P_N(x) = p_0 1 + p_1 x + p_2 x^2 + \ldots + p_n x^n = 0
$$

is if all the coefficients $p_i = 0$, $\forall i=0,\ldots N$

**Theorem**:  The monomials $x^0,\ldots, x^n$ are linearly independent.

**Proof**: consider $P_N(x) = 0$ for all $x$.  Since the polynomials (and monomials) are differentiable at least $n$ times,  differentiate $n$ times to yield
$$
    P^{(n)}_N(x) = n!p_n = 0
$$
which implies $p_n=0$.

Using  this result and differentiating $n-1$ times shows $p_{n-1}=0$, which by induction gives all $p_i = 0$.

Put another way,  the only $n$th degree polynomial that is zero everywhere is if all coefficients are zero.

#### The Fundamental theorem of algebra

Every $n$th degree polynomial has exactly  $n$ complex roots and at most $n$ real roots, i.e.

$$
    P_N(x) = (x - a_1)(x - a_2)\ldots(x - a_n)
$$
for $a_i\in \mathbb{C}$.  Therefore, a non-trivial $n$th order polynomial can only be zero at $n$ points.

### Proof - Uniqueness of Polynomial Interpolants

Let

$$\mathcal{P}_N(x) = \sum^N_{n=0} p_n x^n $$

**interpolate** the $N+1$ points $y_i$ at $x_i$.

i.e.
$$\mathcal{P}_N(x_i) = y_i,\quad \mathrm{for}\quad i=0,1,\ldots,N
$$
and $x_i \neq x_j ~~~ \forall i,j$.

Assume there exists another polynomial

$$Q_N(x) = \sum^N_{n=0} q_n x^n$$

that passes through the same set of points such that $Q_N(x_i) = y_i$.  For uniqueness, $p_n = q_n$ for all n. Now compute $T_N(x) = \mathcal{P}_N(x) - Q_N(x)$:

Now, by construction,  $T_N(x_i) = 0$ which implies that it is equal to zero at $n+1$ points.  However,

$$T_N(x) = \mathcal{P}_N(x) - Q_N(x) = \sum^N_{n=0} p_n x^n - q_n x^n = \sum^N_{n=0} (p_n - q_n) x^n$$

is a $n$th order polynomial which has at most $n$ real roots.  The only way to reconcile this is if T_n(x) = 0, for all $x$, and therefore $p_n - q_n = 0$ individually and therefore $\mathcal{P}_N(x) = Q_N(x)$.

#### Example 3:  Monomial Basis

Consider $\mathcal{P}_3(x) = p_0 + p_1 x + p_2 x^2 + p_3 x^3$ with the four data points $(x_i, y_i), ~~ i = 0,1,2,3$.  We have four equations and four unknowns as expected:

$$\mathcal{P}_3(x_0) = p_0 + p_1 x_0 + p_2 x_0^2 + p_3 x_0^3 = y_0$$

$$\mathcal{P}_3(x_1) = p_0 + p_1 x_1 + p_2 x_1^2 + p_3 x_1^3 = y_1$$

$$\mathcal{P}_3(x_2) = p_0 + p_1 x_2 + p_2 x_2^2 + p_3 x_2^3 = y_2$$

$$\mathcal{P}_3(x_3) = p_0 + p_1 x_3 + p_2 x_3^2 + p_3 x_3^3 = y_3$$

Lets rewrite these as a matrix equation:

$$\mathbf{x} = \begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} \quad \mathbf{y} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix} \quad \mathbf{p} = \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix}$$

When we write the system in matrix/vector form the matrix that arises is called __*Vandermonde*__ matrix:

$$
V = \begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix}.
$$

We can now write the system of linear equations as $V \mathbf{p} = \mathbf{y}$:

$$\begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix} \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix}.$$

**Note**: the columns  of $V$ are simply the monomial functions sampled at the discrete points $x_i$.  Because the monomials are linearly independent, so are the columns of $V$

$$\begin{bmatrix}
     1 & x_0 & x_0^2 & x_0^3 \\
     1 & x_1 & x_1^2 & x_1^3 \\
     1 & x_2 & x_2^2 & x_2^3 \\
     1 & x_3 & x_3^2 & x_3^3
\end{bmatrix} \begin{bmatrix} p_0 \\ p_1 \\ p_2 \\ p_3 \end{bmatrix} = \begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix}$$

- Uniqueness: $V$ is invertible
- What happens if we have redundant data?  Either $(x_i, y_i)$ is repeated or for one $i$ we have two values of $y$.
    - $V$ is singular
- What if we have more points then the order of polynomial we want?
    - Add rows to $V$: no longer square matrix
    - If extra points are consistent, can interpolate
    - If not, will need to approximate (least squares)
- How does this relate to solving the above linear system of equations?

Vandermonde matrices in general are defined as

$$V = \begin{bmatrix}
    1      & x_0    & x_0^2  & \cdots & x_0^N  \\
    1      & x_1    & x_1^2  & \cdots & x_1^N  \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    1      & x_m    & x_m^2  & \cdots & x_m^N  \\
\end{bmatrix}
$$

where $V$ is a $m \times n$ matrix with points $(x_i, y_i)$ for $i = 0, 1, 2, 3, \ldots m$ and for an order $N$ polynomial $\mathcal{P}_N(x)$.

### Finding $p_i$

Finding the coefficients of $\mathcal{P}_N(x)$ can be done by solving the system outlined above.  There are functions in `numpy` that can do this for us such as:
 - `numpy.polyfit(x, y, x.shape[0] - 1)`
 - `numpy.vander(x, N=None)` to construct the matrix and use a linear solver routine.

We can also use a different **basis** that might be easier to use.

Note: large $V$ matrices are ill-conditioned (columns (points) are very close to each other)

### Basis

**Def:**  A basis for a $N$ dimensional vector space is a set of linearly independent vectors that span the space.

The monomials,  $1,x,\ldots, x^n$, form the usual basis for the vector space of $n$th degree polynomials $P_N(x)$.

**Example** $P_2(x)$ is the space of all quadratic functions. i.e. $P_2(x) = \mathrm{span}< 1,x,x^2>$

$$
    P_2(x) = p_0 + p_1 x + p_2 x^2
$$

i.e for every vector $\mathbf{p}\in\mathbb{R}^3$, there is a unique quadratic function in $P_2(x)$.  (we say $P_2$ is *isomorphic* to $\mathbb{R}^3$ and is a three dimensional function space).

**However**,  the monomials are not the only basis for $P_N$, and want to choose a good basis to work with

### Lagrange Basis

Given $N+1$ points $(x_0,y_0), (x_1,y_1), \ldots, (x_{N},y_{N})$ again assuming the $x_i$ are all unique, the interpolating polynomial $\mathcal{P}_N(x)$ can be written as

$$\mathcal{P}_N(x) = \sum^{N}_{i=0} y_i \ell_i(x)$$

where

$$\ell_i(x) = \prod^{N}_{j=0, j \neq i} \frac{x - x_j}{x_i - x_j} = \frac{x - x_0}{x_i - x_0} \frac{x - x_1}{x_i - x_1} \cdots \frac{x - x_{i-1}}{x_i - x_{i-1}}\frac{x - x_{i+1}}{x_i - x_{i+1}} \cdots \frac{x - x_{N}}{x_i - x_{N}}$$

are the **Lagrange Polynomials**

### Lagrange Polynomials

$$\ell_i(x) = \prod^{N}_{j=0, j \neq i} \frac{x - x_j}{x_i - x_j} $$

A Key property of the Lagrange polynomials is that

$$
    \ell_i(x_j) = \delta_{ij} = \left\{\begin{matrix}
            0 & i\neq j \\
            1 & i=j\\
            \end{matrix}\right.
$$
which is why the weights in $P_N(x)$ are simply the $y$ values of the interpolant

### Solving for the coefficients of $P_N(x)$

In general,   if

$$
    P_N(x) = \sum_{n=0}^N w_j\phi_j(x)
$$
where $\phi_j(x)$ is any basis function for $P_N$ (i.e. monomial, Lagrange, and there are many more). Then finding the unique set of weights for the interpolating polynomial through $N+1$ distinct data points $(x_i, y_i)$, just reduces to solving $N+1$ linear equations $y_i = P_N(x_i)$.

For the monomial basis this reduces to the linear system

$$
    V(\mathbf{x})\mathbf{w} = \mathbf{y}
$$

What is the matrix for the Lagrange Basis?

$$
y_0 = w_0\ell_0(x_0) + w_1\ell_1(x_0) + w_2\ell_2(x_0) \\
y_1 = w_0\ell_0(x_1) + w_1\ell_1(x_1) + w_2\ell_2(x_1) \\
y_2 = w_0\ell_0(x_2) + w_1\ell_1(x_2) + w_2\ell_2(x_2) \\
$$

For $x_i$ and $\ell_j$:
- When $i = j$, $w_j\ell_j(x_i) = 1$
- When $i \neq j$, $w_j\ell_j(x_i) = 0$

__So Lagrange matrix is identity matrix, so $P_n(x) = \sum_{i=0}^{N}y_i\ell_i(x)$__

### Visualizing the Lagrange Polynomials

```python
# ====================================================
# Compute the Lagrange basis (\ell_i(x))
def lagrange_basis(x, data):
    """Compute Lagrange basis at x given data"""
    basis = numpy.ones((data.shape[0], x.shape[0]))
    for i in range(data.shape[0]):
        for j in range(data.shape[0]):
            if i != j:
                basis[i, :] *= (x - data[j, 0]) / (data[i, 0] - data[j, 0])
    return basis

# ====================================================
# Calculate full polynomial
def poly_interpolant(x, data):
    """Compute polynomial interpolant of (x,y) using Lagrange basis"""
    P = numpy.zeros(x.shape[0])
    basis = lagrange_basis(x, data)
    for n in range(data.shape[0]):
        P += basis[n, :] * data[n, 1]
    return P

# ====================================================
```

```python
x_data = numpy.array([0., 1., 2., 3.])
y_data = numpy.ones(x_data.shape)
data = numpy.array([x_data, y_data]).T

x = numpy.linspace(x_data.min(),x_data.max(),100)
```

```python
data
```

    array([[0., 1.],
           [1., 1.],
           [2., 1.],
           [3., 1.]])

```python
# ====================================================
# Plot individual basis functions
fig = plt.figure(figsize=(8, 6))
axes = fig.add_subplot(1, 1, 1)
basis = lagrange_basis(x, data)
for i in range(len(x_data)):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)

axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.grid()
axes.legend(loc='best')
plt.show()
```

### Linear Independence of the Lagrange Polynomials

Because the weights of each basis function in the Lagrange basis is just the $y$ value at the interpolation points,  it is straightforward to show that the Lagrange polynomials are linearly independent.  I.e. the statement

$$
    \sum_{n=0}^N w_j\phi_j(x) =0
$$
is equivalent to interpolating the zero function,  where all the $w_j =0$

**Example 0 Revisited** In example 0 above the linear polynomial that interpolates the coordinates $(1,3)$ and $(3,7)$ was simply stated as
$$
    P_1(x) = 2(x-1)+3.
$$
Another way to look at this example is to first note that when we add two linear polynomials
the result is another linear polynomial. The first polynomial to define interpolates $(1,1)$
and $(3,0)$,
$$
    \ell_0(x) = \frac{x-3}{1-3}.
$$
The second polynomial to define interpolates $(1,0)$ and $(3,1)$,
$$
    \ell_1(x) = \frac{x-1}{3-1}.
$$

A linear combination of these two functions can be defined that will interpolate the points $(1,3)$ and $(3,7)$,
$$
    P_1(x) = 3\cdot\ell_0(x) + 7\cdot\ell_1(x).
$$
The graphs of these functions are shown below.

```python
# =============================================================
# Plot the two example basis functions in the current example
x = numpy.linspace(1.0, 3.0, 2)

fig = plt.figure(figsize=(8, 6))

axes = fig.add_subplot(1, 1, 1)
axes.set_ylim([0,9])
axes.plot(x, (x-3)/(-2), color='r', label="$\ell_{%s}(x)$" % 0)
axes.plot(x, (x-1)/(2),  color='b', label="$\ell_{%s}(x)$" % 1)
axes.plot(x, 3*(x-3)/(-2) + 7*(x-1)/(2),color='g',label='interpolant')
axes.set_title("Interpolant for (1,3) and (3,7)")
axes.set_xlabel("x")
axes.grid()
plt.show()
```

#### Example 4:  $N = 1$ Lagrange Polynomial

Given 2 points $(x_0, y_0)$ and $(x_1, y_1)$ the Lagrange form of $\mathcal{P}_N(x)$ is given by

$$\ell_0(x) = \frac{x - x_1}{x_0 - x_1}$$

and

$$\ell_1(x) = \frac{x - x_0}{x_1 - x_0}$$

so that

$$\mathcal{P}_1(x) = \ell_0(x) \cdot y_0 + \ell_1(x) \cdot y_1 = \frac{x - x_1}{x_0 - x_1} \cdot y_0 + \frac{x - x_0}{x_1 - x_0} \cdot y_1$$

One important aspect of Lagrange polynomials to note is that the $\ell_i(x)$ functions are exactly 1 when $x  = x_i$ and that every other $\ell_j(x)$ where $j \neq i$ is 0.

Partition of unity(?): sum of Lagrange bases always = 1

```python
data = numpy.array([[-1.5, -0.5], [0.0, 0.5]])
```

```python
# data = numpy.array([[-1.5, -0.5], [0.0, 0.5], [-0.5, 1.0]])
N = data.shape[0] - 1
M = data.shape[0]
x = numpy.linspace(-2.0, 2.0, 100)

# Plot individual basis functions
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
basis = lagrange_basis(x, data)
for i in range(N + 1):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)
axes.grid()
axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.legend(loc=8)

# Plot full polynomial P_N(x)
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, poly_interpolant(x, data), label="$P_{%s}(x)$" % N)
for point in data:
    axes.plot(point[0], point[1], 'ko')
axes.set_title("$P_N(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$P_N(x)$")
axes.grid()

plt.show()
```

#### Example 5:  Interpolate four points from $sin(\pi x)$

Use four points to approximate $\sin$ on the interval $x \in [-1, 1]$.  What is the behavior as $N \rightarrow \infty$?  Also plot the error between $f(x)$ and the interpolant $P_N(x)$.

```python
num_points = 21
```

```python
# num_points = 5
# num_points = 6
num_points = 20
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = numpy.sin(2.0 * numpy.pi * data[:, 0])
N = data.shape[0] - 1 # Degree of polynomial
M = data.shape[0]
x = numpy.linspace(-1.0, 1.0, 100)

# ====================================================
# Plot individual basis functions
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
basis = lagrange_basis(x, data)
for i in range(N + 1):
    axes.plot(x, basis[i, :], label="$\ell_{%s}(x)$" % i)

axes.set_title("Lagrange Basis $\ell_i(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$\ell_i(x)$")
axes.legend(loc=1)
axes.grid()

# Plot full polynomial P_N(x)
axes = fig.add_subplot(1, 2, 2)
axes.plot(x, poly_interpolant(x, data), label="$P_{%s}(x)$" % N)
axes.plot(x, numpy.sin(2.0 * numpy.pi * x), 'r--', label="True $f(x)$")
for point in data:
    axes.plot(point[0], point[1], 'ko')
axes.set_title("$P_N(x)$")
axes.set_xlabel("x")
axes.set_ylabel("$P_N(x)$")
axes.legend(loc=1)
axes.grid()

plt.show()
```

#### Example 6: Runge's Function

Interpolate $f(x) = \frac{1}{1 + 25 x^2}$ using 6 points of your choosing on $x \in [-1, 1]$.

Try it with 11 points.

Keep increasing the number of points and see what happens.

Runge's effect:

```python
def f(x):
    return 1.0 / (1.0 + 25.0 * x**2)

x = numpy.linspace(-1, 1, 100)
# x = numpy.linspace(-2, 2, 100)

num_points = 15
```

```python
num_points = 11
# num_points = 20
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = f(data[:, 0])
N = data.shape[0] - 1

# Plot the results
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, poly_interpolant(x, data), 'b', label="$P_6(x)$")
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(data[:, 0], data[:, 1], 'ro', label="data")

axes.set_title("Interpolation of Runge's function")
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)
axes.grid()

plt.show()
```

Adding more points: ends (x = +/-1) are not able to be approximated

#### Example 7: Weierstrass "Monster" Function

Defined as
$$
    f(x) = \sum^\infty_{n=0} a^n \cos(b^n \pi x)
$$
such that
$$
    0 < a < 1 \quad \text{and} \quad a b > 1 + \frac{3\pi}{2}.
$$
This function is continuous everywhere but not differentiable anywhere.

```python
def f(x, a=0.9, N=100):
    summation = 0.0
    b = (1.0 + 3.0 / 2.0 * numpy.pi) / a + 0.01
    print(b)
    for n in range(N + 1):
        summation += a**n * numpy.cos(b**n * numpy.pi * x)
    return summation

x = numpy.linspace(-1, 1, 1000)
# x = numpy.linspace(-2, 2, 100)

num_points = 10
data = numpy.empty((num_points, 2))
data[:, 0] = numpy.linspace(-1, 1, num_points)
data[:, 1] = f(data[:, 0])
N = data.shape[0] - 1

# Plot the results
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, poly_interpolant(x, data), 'b', label="$P_6(x)$")
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(data[:, 0], data[:, 1], 'ro', label="data")

axes.set_title("Interpolation of Runge's function")
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)

plt.show()
```

### Rules of Thumb

 - Avoid __high-order interpolants__ when possible!  Keep increasing the number of points and see what happens.
 - Avoid __extrapolation__ - Increase the range of $x$ in the above example and check how good the approximation is beyond our sampling interval

### Error Analysis: Minimization

**Theorem:** Lagrange Remainder Theorem - Let $f(x) \in C^{N+1}[-1, 1]$, then
$$
    f(x) = \mathcal{P}_N(x) + R_N(x)
$$
where $\mathcal{P}_N(x)$ is the interpolating polynomial and
$$
    R_N(x) = Q(x) \frac{f^{(N+1)}(c)}{(N+1)!} \quad \text{with} \quad c \in [-1,1]
$$
with
$$
    Q(x) = \prod^N_{i=0} (x - x_i) = (x-x_0)(x-x_1)\cdots(x-x_N) .
$$

A few things to note:
 - For Taylor's theorem note that $Q(x) = (x - x_0)^{N+1}$ and the error only vanishes at $x_0$.
 - For Lagrange's theorem the error vanishes at all $x_i$.
 - To minimize $R_N(x)$ requires minimizing $|Q(x)|$ for $x \in [-1, 1]$.

#### Minimizing $R_N(x)$

Minimizing the error $R_N(x)$ in Lagrange's theorem is equivalent to minimizing $|Q(x)|$ for $x \in [-1, 1]$.

Minimizing error $\Leftrightarrow$ picking roots of $Q(x)$ or picking the points where the interpolant data is located.  How do we this?

### Chebyshev Polynomials

*Chebyshev polynomials* $T_N(x)$ are another basis that can be used for interpolation.

First 5 polynomials
$$T_0(x) = 1$$

$$T_1(x) = x$$

$$T_2(x) = 2 x^2 - 1$$

$$T_3(x) = 4 x^3 - 3 x$$

$$T_4(x) = 8x^4 - 8x^2 + 1$$

$$ T_N(x) = 2 x (T_{N-1}) - T_{N-2} $$

Even N: doesn't change sign if x is negative

Odd N: changes sign if x is negative

```python
def cheb_poly(x, N):
    """Compute the *N*th Chebyshev polynomial and evaluate it at *x*"""
    T = numpy.empty((3, x.shape[0]))
    T[0, :] = numpy.ones(x.shape)
    T[1, :] = x
    if N == 0:
        return T[0, :]
    elif N == 1:
        return T[1, :]
    else:
        for k in range(2, N + 1):
            T[2, :] = 2.0 * x * T[1, :] - T[0, :]
            T[0, :] = T[1, :]
            T[1, :] = T[2, :]

        return T[2, :]
```

```python
x = numpy.linspace(-1, 1, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

for n in range(5):
    axes.plot(x, cheb_poly(x, n), label="$T_%s$" % n)

axes.set_ylim((-1.1, 1.1))
axes.set_title("Chebyshev Polynomials")
axes.set_xlabel("x")
axes.set_ylabel("$T_N(x)$")
axes.legend(loc='best')
axes.grid()

plt.show()
```

1. Chebyshev nodes of the 1st kind (roots)
$$
    x_k = \cos \left (\frac{(2 k - 1) \pi}{2 N} \right ) \quad k = 1, \ldots, N
$$
1. Chebyshev nodes of the 2nd kind (extrema)
$$
    x_k = \cos \left( \frac{k \pi}{N} \right) \quad k = 0, \ldots, N
$$

```python
N = 4
x_extrema = numpy.cos(numpy.arange(N + 1) * numpy.pi / N)
x_nodes = numpy.cos((2.0 * numpy.arange(1, N + 1) - 1.0) / (2.0 * N) * numpy.pi)
```

```python
fig = plt.figure()
# fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 1, 1)

# Plot points
axes.plot(x_extrema, numpy.zeros(N+1), 'ro')
axes.plot(x_nodes, numpy.zeros(N), 'bo')

# Plot some helpful lines
axes.plot((-1.0, -1.0), (-1.1, 1.1), 'k--')
axes.plot((1.0, 1.0), (-1.1, 1.1), 'k--')
axes.plot((-1.0, 1.0), (0.0, 0.0), 'k--')
for i in range(x_extrema.shape[0]):
    axes.plot((x_extrema[i], x_extrema[i]), (-1.1, 1.1), 'r--')
    axes.plot(x_extrema[i], cheb_poly(x_extrema, N)[i], 'ro')

print('Nodes   = {}'.format(numpy.sort(x_nodes)))
print('Extrema = {}'.format(numpy.sort(x_extrema)))
#print(numpy.cos(x_extrema))

# Plot Chebyshev polynomial
x_hat = numpy.linspace(-1, 1, 1000)
axes.plot(x_hat, cheb_poly(x_hat, N), 'k')
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))

# Labels
axes.set_title("Chebyshev Nodes and Extrema, N={}".format(N), fontsize="20")
axes.set_xlabel("x", fontsize="15")
axes.set_ylabel("$T_{N+1}(x)$", fontsize="15")

plt.show()
```

```python
# First-kind Nesting (3 x)
fig = plt.figure()
# fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 1, 1)

N = 5
factor = 3
x_1 = numpy.cos((2.0 * numpy.arange(1, N + 1) - 1.0) / (2.0 * N) * numpy.pi)
x_2 = numpy.cos((2.0 * numpy.arange(1, factor * N + 1) - 1.0) / (2.0 * factor * N) * numpy.pi)

axes.plot(x_1, numpy.zeros(N), "o", color="r", markerfacecolor="lightgray", markersize="15")
axes.plot(x_2, numpy.zeros(N * factor), 'kx', markersize="10")

x_hat = numpy.linspace(-1, 1, 1000)
axes.plot(x_hat, cheb_poly(x_hat, N), 'k')
axes.plot(x_hat, cheb_poly(x_hat, factor * N), 'k')
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.set_title("Nesting of 1st and 2nd Kind Chebyshev Polynomials")
axes.set_xlabel("$x$")
axes.set_ylabel("$T_N(x)$")
plt.show()
```

#### Properties of Chebyshev Polynomials
1. Defined by a recurrence relation

   $$T_k(x) = 2 x T_{k-1}(x) - T_{k-2}(x)$$

2. Leading coefficient of $x^N$ in $T_N(x)$ is $2^{N-1}$ for $N \geq 1$ (NOT a monic polynomial ie leading term coefficient is not always 1)

3. Extreme values:

   $$|T_N(x)| \leq 1 \quad \text{for} \quad  -1 \leq x \leq 1$$

#### Properties of Chebyshev Polynomials
4. __Minimax principle__:  The polynomial

   $$T(x) = \frac{T_{N+1}(x)}{2^N}$$

   is a *monic polynomial*, a univariate function with the leading coefficient equal to 1, with the property that

   $$
       \max |T(x)| \leq \max |Q(X)| \quad \text{for} \quad  x \in [-1, 1], \quad \text{and}
   $$
   $$
       \max |T(x)| = \frac{1}{2^N}
   $$

 Recall that the remainder term in the Lagrange Remainder Theorem was
   $$
        R_N(x) = Q(x) \frac{f^{(N+1)}(c)}{(N+1)!} \quad \text{with} \quad c \in [-1,1]
   $$
   with
   $$
        Q(x) = \prod^N_{i=0} (x - x_i) = (x-x_0)(x-x_1)\cdots(x-x_N) .
   $$

#### Error Analysis Redux

Given that the Chebyshev polynomials are a minimum on the interval $[-1, 1]$ we would like $T(x) = Q(x)$.

Since we only know the roots of $Q(x)$ (the points where the interpolant data is located) we require these points to be the roots of the Chebyshev polynomial $T_{N+1}(x)$ therefore enforcing $T(x) = Q(x)$.

The zeros of $T_N(x)$ in the interval $[-1, 1]$ can be shown to satisfy
$$
    x_k = \cos\left( \frac{(2k - 1) \pi}{2 N} \right ) \quad \text{for} \quad k=1, \ldots, N
$$
These __nodal points__ (sampling the function at these points) can be shown to minimize interpolation error.

```python
x = numpy.linspace(0, numpy.pi, 100)
N = 15

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1, aspect="equal")

axes.plot(numpy.cos(x), numpy.sin(x), 'r--')
axes.plot(numpy.linspace(-1.1, 1.1, 100), numpy.zeros(x.shape), 'r')
for k in range(1, N + 1):
    location = [numpy.cos((2.0 * k - 1.0) * numpy.pi / (2.0 * N)),
                numpy.sin((2.0 * k - 1.0) * numpy.pi / (2.0 * N))]
    axes.plot(location[0], location[1], 'ko')
    axes.plot(location[0], 0.0, 'ko')
    axes.plot([location[0], location[0]], [0.0, location[1]], 'k--')

axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-0.1, 1.1))
plt.show()
```

#### Summary

1. Minimizing the error in Lagrange's theorem is equivalent to minimizing
$$
    |Q(x)| \quad \text{for} \quad x \in [-1, 1].
$$
1. We know Chebyshev polynomials are a minimum on the interval $[-1, 1]$ so we would like to have $T(x) = Q(x)$.
1. Since we only know the roots of $Q(x)$ (the points where the interpolant data is located) we require these points to be the roots of the Chebyshev polynomial $T_{N+1}(x)$ therefore enforcing $T(x) = Q(x)$.
1. The zeros of $T_N(x)$ in the interval $[-1, 1]$ can be shown to satisfy
$$
    x_k = \cos\left( \frac{(2k - 1) \pi}{2 N} \right ) \quad \text{for} \quad k=1, \ldots, N
$$
These nodal points (sampling the function at these points) can be shown to minimize interpolation error.

#### Notes
 - The Chebyshev nodes minimize interpolation error for any polynomial basis (due to uniqueness of the interpolating polynomial, any polynomial that interpolates these points are identical regardless of the basis).
 - Chebyshev nodes uniquely define the Chebyshev polynomials.
 - The boundedness properties of Chebyshev polynomials are what lead us to the roots as a minimization but there are other used for these orthogonal polynomials.
 - There are two kinds of Chebyshev nodes and therefore two definitions.
 - Chebyshev polynomial used to determine *nodal points*, but can then use Lagrange to interpolate.

### Return to Runge's function

```python
# Runge's function again
def f(x):
    return 1.0 / (1.0 + 25.0 * x**2)

# Parameters
x = numpy.linspace(-1, 1, 100)
num_points = 21
```

```python
# ============================================================
#  Equidistant nodes
equidistant_data = numpy.empty((num_points, 2))
equidistant_data[:, 0] = numpy.linspace(-1, 1, num_points)
equidistant_data[:, 1] = f(equidistant_data[:, 0])
N = equidistant_data.shape[0] - 1

P_lagrange = poly_interpolant(x, equidistant_data)

# ============================================================
#  Chebyshev nodes
chebyshev_data = numpy.empty((num_points, 2))
chebyshev_data[:, 0] = numpy.cos((2.0 * numpy.arange(1, num_points + 1) - 1.0) * numpy.pi / (2.0 * num_points))
chebyshev_data[:, 1] = f(chebyshev_data[:, 0])

P_cheby1 = poly_interpolant(x, chebyshev_data)

# Fit directly with Chebyshev polynomials
coeff = numpy.polynomial.chebyshev.chebfit(chebyshev_data[:, 0], chebyshev_data[:, 1], N)
P_cheby2 = numpy.polynomial.chebyshev.chebval(x, coeff)

# Check on unique polynomials
#print(numpy.allclose(P_cheby1, P_cheby2))

# calculate errornorms for different interpolants
equidistant_err = numpy.linalg.norm(P_lagrange - f(x))
cheb_err = numpy.linalg.norm(P_cheby1 - f(x))

# ============================================================
# Plot the results
fig = plt.figure(figsize=(16,6))
fig.subplots_adjust(hspace=.5)

axes = fig.add_subplot(1, 2, 1)
axes.plot(x, P_lagrange, 'b', label="$P_%s(x)$" % N)
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(equidistant_data[:, 0], equidistant_data[:, 1], 'ro', label="data")
axes.set_title("Interpolation at Equispaced Points: err = {}".format(equidistant_err))
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=8)
#print('Equispaced error = {}'.format(numpy.linalg.norm(P_lagrange - f(x))))

axes = fig.add_subplot(1, 2, 2)
axes.plot(x, f(x), 'k', label="True $f(x)$")
axes.plot(x, P_cheby1, 'b', label="$P_%s(x)$" % N)
axes.plot(chebyshev_data[:, 0], chebyshev_data[:, 1], 'ro', label="data")

axes.set_title("Interpolation at Chebyshev Points: err = {}".format(cheb_err))
axes.set_xlabel("x")
axes.set_ylabel("y")
axes.legend(loc=1)
#print('Chebyshev error = {}'.format(numpy.linalg.norm(P_cheby1 - f(x))))

plt.show()
```

## Piece-Wise Polynomial Interpolation

Given $N$ points, use lower order polynomial interpolation to fit the function in pieces.  We can choose the order of the polynomials and the continuity.

 - $C^0$: Interpolant is continuous
   - Linear interpolation
   - Quadratic interpolation
 - $C^1$: Interpolation and 1st derivative are continuous
   - Cubic Hermite polynomials (PCHiP)
 - $C^2$: Interpolation, 1st and 2nd derivatives are continuous
   - Cubic splines

### Piece-Wise Linear

Given a segment between point $(x_k, y_k)$a nd $(x_{k+1}, y_{k+1})$ define the segment as

$$\mathcal{P}_k(x) = \frac{y_{k+1} - y_k}{x_{k+1} - x_k} (x - x_k) + y_k$$

The final interpolant $\mathcal{P}(x)$ is then defined on $[x_k, x_{k+1}]$ using this function.

```python
data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)
N = data.shape[0] - 1

# Lagrange Basis
P_lagrange = poly_interpolant(x, data)

# C^0 Piece-wise linear
# P_pw_linear = numpy.interp(x, data[:, 0], data[:, 1])
P_linear = numpy.zeros(x.shape)
for n in range(1, N + 1):
    P_linear += ((data[n, 1] - data[n - 1, 1]) / (data[n, 0] - data[n - 1, 0]) * (x - data[n - 1, 0])
                         + data[n - 1, 1]) * (x > data[n - 1, 0]) * (x <= data[n, 0])

```

```python
# Add end points for continuity
P_linear += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_linear += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# Plot
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko')
axes.plot(x, P_lagrange, 'b--')
axes.plot(x, P_linear, 'r')

axes.set_title("Interpolated Data - $C^0$ Linear")
axes.set_xlabel("x")
axes.set_ylabel("$P_1(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])

plt.show()
```

### Piece-Wise Non-Overlapping Polynomials

In sets of three points $(x_{k+1}, y_{k+1})$, $(x_{k}, y_{k})$, and $(x_{k-1}, y_{k-1})$, find quadratic interpolant and define final interpolant $P(x)$ using the quadratic interpolant $\mathcal{P}_k(x)$ on $[x_{k-1}, x_{k+1}]$.

```python
data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)
N = data.shape[0] - 1

# This isn't overlapping, it's more like C_0 P_2
# C^0 Piece-wise quadratic
P_quadratic = numpy.zeros(x.shape)
for k in range(1, N + 1, 2):
    p = numpy.polyfit(data[k - 1:k + 2, 0], data[k - 1:k + 2, 1], 2)
    P_quadratic += numpy.polyval(p, x) * (x > data[k - 1, 0]) * (x <= data[k + 1, 0])
# Add end points for continuity
P_quadratic += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_quadratic += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# Plot
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko')
axes.plot(x, P_lagrange, 'b--', label = "Lagrange")
axes.plot(x, P_quadratic, 'r', label = "Piecewise quadratic")

axes.set_title("Interpolated Data - $C^0$ Quadratic")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])
axes.legend()

plt.show()
```

### Piece-Wise $C^1$ Cubic Interpolation

For the previous two cases we had discontinous 1st derivatives!  We can make this better by constraining the polynomials to be continuous at the boundaries of the piece-wise intervals.

Given a segment between points $(x_k, y_k)$ and $(x_{k+1}, y_{k+1})$ we want to fit a cubic function between the two points.

$$\mathcal{P}_k(x) = p_0 + p_1 x + p_2 x^2 + p_3 x^3$$

$$\mathcal{P}_k(x_k) = y_k, \quad \mathcal{P}_k(x_{k+1}) = y_{k+1}$$

Now we have 4 unknowns but only two data points!  Constraining the derivative at each interval end will lead to two new equations and therefore we can solve for the interpolant.

$$\frac{\text{d}}{\text{dx}} \mathcal{P}_k(x_k) = d_k, \quad \frac{\text{d}}{\text{dx}} \mathcal{P}_k(x_{k+1}) = d_{k+1}$$

where we need to prescribe the $d_k$s.  Since we know the polynomial we can write these 4 equations as

$$\begin{aligned}
p_0 + p_1 x_k + p_2 x_k^2 + p_3 x_k^3 &= y_k \\
p_0 + p_1 x_{k+1} + p_2 x_{k+1}^2 + p_3 x_{k+1}^3 &= y_{k+1} \\
p_1 +  2p_2 x_k + 3 p_3 x_k^2 &= d_k \\
p_1 + 2 p_2 x_{k+1} + 3 p_3 x_{k+1}^2 &= d_{k+1}
\end{aligned}$$

Rewriting this as a system we get
$$\begin{bmatrix}
    1 & x_k & x_k^2 & x_k^3 \\
    1 & x_{k+1} & x_{k+1}^2 & x_{k+1}^3 \\
    0 & 1 & 2 x_k & 3 x_k^2 \\
    0 & 1 & 2 x_{k+1} & 3 x_{k+1}^2
\end{bmatrix} \begin{bmatrix}
p_0 \\ p_1 \\ p_2 \\ p_3
\end{bmatrix} = \begin{bmatrix}
y_k \\ y_{k+1} \\ d_k \\ d_{k+1}
\end{bmatrix}$$

A common simplification to the problem description re-parameterizes the locations of the points such that $s \in [0, 1]$ and recast the problem with $(0, y_k)$ and $(1, y_{k+1})$.  This simplifies the above system to
$$\begin{bmatrix}
    1 & 0 & 0 & 0 \\
    1 & 1 & 1 & 1 \\
    0 & 1 & 0 & 0 \\
    0 & 1 & 2 & 3
\end{bmatrix} \begin{bmatrix}
p_0 \\ p_1 \\ p_2 \\ p_3
\end{bmatrix} = \begin{bmatrix}
y_k \\ y_{k+1} \\ d_k \\ d_{k+1}
\end{bmatrix}$$

which can be solved to find

$$\begin{aligned}
\mathcal{P}(s) &= (1-s)^2 (1 + 2s) y_k + s^2 (3 - 2 s) y_{k+1} + s (1 - s)^2 d_k - s^2 (1 - s)d_{k+1}\\
\mathcal{P}'(s) &= 6s(s-1) y_k + 6s(1-s) y_{k+1} + (s-1)(3s-1) d_k - s(3s-2) d_{k+1}\\
\mathcal{P}''(s) &= 6 (1-2s)(y_{k+1} - y_k) + (6s - 4) d_k + (6s-2) d_{k+1}
\end{aligned}$$

Now, how to choose $d_k$?

#### PCHIP

Piecewise Cubic Hermite Interpolation Polynomial

 - Picks the slope that preserves monotonicity
 - Also tried to preserve the shape of the data
 - Note that in general this interpolant is $\mathcal{P}_k(x) \in C^1$

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# C^1 Piece-wise PCHIP
P_pchip = interpolate.pchip_interpolate(data[:, 0], data[:, 1], x)
```

```python
# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ro')
axes.plot(x, P_pchip, 'r')

axes.set_title("Interpolated Data - $C^1$ Cubic PCHIP")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])
axes.grid()
plt.show()
```

#### Cubic Splines

Enfores continuity on second derivatives as well:

$$\mathcal{P}''_{k}(x_{k}) = \mathcal{P}''_{k-1}(x_k)$$

From our generalization before we know

$$\mathcal{P}''(s) = 6 (1-2s)(y_{k+1} - y_k) + (6s - 4) d_k + (6s-2) d_{k+1}$$

and our constraint now becomes

$$\mathcal{P}''_{k}(0) = \mathcal{P}''_{k-1}(1)$$

$$\mathcal{P}''_{k-1}(1) = 6 (1-2 \cdot 1)(y_{k} - y_{k-1}) + (6\cdot 1 - 4) d_{k-1} + (6\cdot 1-2) d_{k}$$

$$\mathcal{P}''_{k}(0) = 6 (1-2 \cdot 0)(y_{k+1} - y_k) + (6\cdot 0 - 4) d_k + (6\cdot 0-2) d_{k+1}$$

$$-6(y_{k} - y_{k-1}) + 2 d_{k-1} + 4 d_{k} = 6 (y_{k+1} - y_k) - 4 d_k -2  d_{k+1}$$

We now have constraints on choosing the $d_k$ values.  Note that we still need to prescribe them at the boundaries of the full interval.

This forms a linear set of equations for the $d_k$s based on the $y_k$ values and can be reformulated into a tri-diagonal linear system

$$\begin{bmatrix}
         & \ddots  & \ddots  & \ddots  & &  &\\
         & 0 & 2 & 8 & 2 & 0 & & \\
         & & 0 & 2 & 8 & 2 & 0 & & & \\
         & &   & 0 & 2 & 8 & 2 & 0 & & \\
         & &    & & & \ddots & \ddots & \ddots &
\end{bmatrix}\begin{bmatrix}
    \vdots \\ d_{k-1} \\ d_{k} \\ d_{k+1} \\ \vdots
\end{bmatrix} = \begin{bmatrix}
    \vdots \\ 6 (y_{k} - y_{k-2}) \\ 6 (y_{k+1} - y_{k-1}) \\ 6 (y_{k+2} - y_{k}) \\\vdots
\end{bmatrix}$$

The boundaries are still left unconstrained and we must pick some rule to specify the derivatives there.

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# C^2 Piece-wise Splines
# Note that to get an interpolant we need to set the smoothing
# parameters *s* to 0
P_spline = interpolate.UnivariateSpline(data[:, 0], data[:, 1], s=0)
```

```python
# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ro')
axes.plot(x, P_spline(x), 'r', label = '$C^2$')
axes.plot(x, P_pchip, 'b--', label = 'Pchip')
axes.set_title("Interpolated Data - $C^2$ Cubic Splines")
axes.set_xlabel("x")
axes.set_ylabel("$P_3(x)$")
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])
axes.grid()
axes.legend(loc='best')

plt.show()
```

### Let's compare all of these methods

```python
import scipy.interpolate as interpolate

data = numpy.array([[1.0, 3.0], [2.0, 1.0], [3.5, 4.0], [5.0, 0.0], [6.0, 0.5], [9.0, -2.0], [9.5, -3.0]])
x = numpy.linspace(0.0, 10, 100)

# Lagrange Basis
N = data.shape[0] - 1

lagrange_basis = numpy.ones((N + 1, x.shape[0]))
for i in range(N + 1):
    for j in range(N + 1):
        if i != j:
            lagrange_basis[i, :] *= (x - data[j, 0]) / (data[i, 0] - data[j, 0])

# Calculate full polynomial
P_lagrange = numpy.zeros(x.shape[0])
for n in range(N + 1):
    P_lagrange += lagrange_basis[n, :] * data[n, 1]

# C^0 Piece-wise linear
# P_pw_linear = numpy.interp(x, data[:, 0], data[:, 1])
P_linear = numpy.zeros(x.shape)
for n in range(1, N + 1):
    P_linear += ((data[n, 1] - data[n - 1, 1]) / (data[n, 0] - data[n - 1, 0]) * (x - data[n - 1, 0])
                         + data[n - 1, 1]) * (x > data[n - 1, 0]) * (x <= data[n, 0])

# Add end points for continuity
P_linear += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_linear += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])
# C^0 Piece-wise quadratic
P_quadratic = numpy.zeros(x.shape)
for k in range(1, N + 1, 2):
    p = numpy.polyfit(data[k - 1:k + 2, 0], data[k - 1:k + 2, 1], 2)
    P_quadratic += numpy.polyval(p, x) * (x > data[k - 1, 0]) * (x <= data[k + 1, 0])
# Add end points for continuity
P_quadratic += numpy.ones(x.shape) * data[0, 1] * (x < data[0, 0])
P_quadratic += numpy.ones(x.shape) * data[-1, 1] * (x >= data[-1, 0])

# C^1 Piece-wise PCHIP
P_pchip = interpolate.pchip_interpolate(data[:, 0], data[:, 1], x)

# C^2 Piece-wise Splines
P_spline = interpolate.UnivariateSpline(data[:, 0], data[:, 1], s=0)

# Plot
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(data[:,0], data[:,1], 'ko', label="Data")
axes.plot(x, P_lagrange, 'y', label="Lagrange")
axes.plot(x, P_linear, 'g', label="PW Linear")
axes.plot(x, P_quadratic, 'r', label="PW Quadratic")
axes.plot(x, P_pchip, 'c', label="PW Cubic - PCHIP")
axes.plot(x, P_spline(x), 'b', label="PW Cubic - Spline")
axes.grid()

axes.set_title("Interpolated Data - Method Comparisons")
axes.set_xlabel("x")
axes.set_ylabel("$P(x)$")
axes.legend(loc='best')
axes.set_xlim([0.0, 10.0])
axes.set_ylim([-4.0, 15.0])

plt.show()
```

How do you choose which method to use?

## Relationship to Regression

What if we have more data and want a lower degree polynomial but do not want to use a piece-wise defined interpolant?
Regression techniques are often used to minimize a form of error between the data points $y_i$ at $x_i$ with an approximating function $f(x_i)$.  __Note that this is NOT interpolation anymore!__

### Least-Squares

One way of doing this is to require that we minimize the least-squares error
$$
    E = \left( \sum^m_{i=1} |y_i - f(x_i)|^2 \right )^{1/2}.
$$
where as before we have data $y_i$ at locations $x_i$ and an approximating function $f(x_i)$.

From the beginning of our discussion we know we can write the interpolant as a system of linear equations which we can then solve for the coefficients of a monomial basis.  If we wanted to fit a line
$$
    \mathcal{P}_1(x) = p_0 + p_1 x
$$
to $N$ data points we would have
$$
   \begin{bmatrix}
        1 & x_1 \\
        1 & x_2 \\
        \vdots & \vdots \\
        1 & x_N
   \end{bmatrix} \begin{bmatrix}
       p_0 \\ p_1
   \end{bmatrix} = \begin{bmatrix}
       y_1 \\ y_2 \\ \vdots \\ y_N
   \end{bmatrix}
$$
or
$$
    A p = y
$$
What's wrong with this system?

This leads to the likelihood that there is no solution to the system as
$$
    A \in \mathbb{R}^{N \times 2}, p \in \mathbb{R}^{2 \times 1}, \text{  and  } y \in \mathbb{R}^{N \times 1}.
$$

Instead we can solve the related least-squares system, multiplying both sides by $A^T$, so that $A^T A$ is a square matrix.
$$
    A^T A p = A^T y
$$
whose solution minimizes the least-square error defined before as $E$.

Note:  In general, this is __not__ the most stable way to solve least squares problems,  in general, using an orthogonalization technique like $QR$ factorization is better numerically.

```python
# Linear Least Squares Problem
N = 50
x = numpy.linspace(-1.0, 1.0, N)
y = x + numpy.random.random((N))
```

```python
A = numpy.ones((x.shape[0], 2))
A[:, 1] = x
p = numpy.linalg.solve(numpy.dot(A.transpose(), A), numpy.dot(A.transpose(), y))
#p = numpy.linalg.lstsq(A, y, rcond=None)[0]
f = lambda x: p[0] + p[1] * x
E = numpy.linalg.norm(y - f(x), ord=2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, y, 'ko')
axes.plot(x, f(x), 'r')
axes.set_title("Least Squares Fit to Data, err={}".format(E))
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Themes and variations

You can play all sorts of games, whether they are justified by the data or not,  for example we can fit the same random data with a function like

$$
    f(x) = p_0 + p_1\tanh(x)
$$

which is still a linear problem for the coefficients $p_0$ and $p_1$, however the vandermonde matrix now has columns of $\mathbf{1}$ and $\tanh\mathbf{x}$.

```python
# Linear Least Squares Problem
A = numpy.ones((x.shape[0], 2))
A[:, 1] = numpy.tanh(x)
p = numpy.linalg.solve(numpy.dot(A.transpose(), A), numpy.dot(A.transpose(), y))
# p = numpy.linalg.lstsq(A, y)[0]
f = lambda x: p[0] + p[1] * numpy.tanh(x)
E = numpy.linalg.norm(y - f(x), ord=2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(x, y, 'ko')
axes.plot(x, f(x), 'r')
axes.set_title("Least Squares Fit to Data, err = {}".format(E))
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Let ye be warned...

(Original image can be found at [Curve Fitting](https://xkcd.com/2048/).)


## Getting Started

In the following we like to introduce *pymoo* by presenting an example optimization scenario. This guide goes through the most important steps to get started with our framework. First, we present an example optimization problem to be solved using *pymoo*. Second, we show how to formulate the optimization problem in our framework and how to instantiate an algorithm object to be used for optimization. Then, a termination criterion for the algorithm is defined and the optimization method is called. Finally, we quickly show a possible post-processing step which analyzes the optimization run performance.

### Multi-Objective Optimization

In general, multi-objective optimization has several objective functions with subject to inequality and equality constraints to optimize <cite data-cite="multi_objective_book"></cite>. The goal is to find a set of solutions that do not have any constraint violation and are as good as possible regarding all its objectives values. The problem definition in its general form is given by:

\begin{align}
\begin{split}
\min \quad& f_{m}(x) \quad \quad \quad \quad m = 1,..,M  \\[4pt]
\text{s.t.}   \quad& g_{j}(x) \leq 0  \quad \; \; \,  \quad j = 1,..,J \\[2pt]
\quad& h_{k}(x) = 0        \quad  \; \; \quad k = 1,..,K \\[4pt]
\quad& x_{i}^{L} \leq x_{i} \leq x_{i}^{U}  \quad i = 1,..,N \\[2pt]
\end{split}
\end{align}

The formulation above defines a multi-objective optimization problem with $N$ variables, $M$ objectives, $J$ inequality and $K$ equality constraints. Moreover, for each variable $x_i$ lower and upper variable boundaries ($x_i^L$ and $x_i^U$) are defined.

### Example Optimization Problem

In the following, we investigate exemplarily a bi-objective optimization with two constraints.
The selection of a suitable optimization problem was made based on having enough complexity for the purpose of demonstration, but not being too difficult to lose track of the overall idea. Its definition is given by:

\begin{align}
\begin{split}
\min \;\; & f_1(x) = (x_1^2 + x_2^2) \\
\max \;\; & f_2(x) = -(x_1-1)^2 - x_2^2 \\[1mm]
\text{s.t.} \;\; & g_1(x) = 2 \, (x_1 - 0.1) \, (x_1 - 0.9) \leq 0\\
& g_2(x) = 20 \, (x_1 - 0.4) \, (x_1 - 0.6) \geq 0\\[1mm]
& -2 \leq x_1 \leq 2 \\
& -2 \leq x_2 \leq 2
\end{split}
\end{align}

 It consists of two objectives ($M=2$) where $f_1(x)$ is minimized and $f_2(x)$ maximized. The optimization is with subject to two inequality constraints ($J=2$) where $g_1(x)$ is formulated as a less than and $g_2(x)$ as a greater than constraint. The problem is defined with respect to two variables ($N=2$), $x_1$ and $x_2$, which both are in the range $[-2,2]$. The problem does not contain any equality constraints ($K=0$).

```python
import numpy as np

X1, X2 = np.meshgrid(np.linspace(-2, 2, 500), np.linspace(-2, 2, 500))

F1 = X1**2 + X2**2
F2 = (X1-1)**2 + X2**2
G = X1**2 - X1 + 3/16

G1 = 2 * (X1[0] - 0.1) * (X1[0] - 0.9)
G2 = 20 * (X1[0] - 0.4) * (X1[0] - 0.6)

import matplotlib.pyplot as plt
plt.rc('font', family='serif')

levels = [0.02, 0.1, 0.25, 0.5, 0.8]
plt.figure(figsize=(7, 5))
CS = plt.contour(X1, X2, F1, levels, colors='black', alpha=0.5)
CS.collections[0].set_label("$f_1(x)$")

CS = plt.contour(X1, X2, F2, levels, linestyles="dashed", colors='black', alpha=0.5)
CS.collections[0].set_label("$f_2(x)$")

plt.plot(X1[0], G1, linewidth=2.0, color="green", linestyle='dotted')
plt.plot(X1[0][G1<0], G1[G1<0], label="$g_1(x)$", linewidth=2.0, color="green")

plt.plot(X1[0], G2, linewidth=2.0, color="blue", linestyle='dotted')
plt.plot(X1[0][X1[0]>0.6], G2[X1[0]>0.6], label="$g_2(x)$",linewidth=2.0, color="blue")
plt.plot(X1[0][X1[0]<0.4], G2[X1[0]<0.4], linewidth=2.0, color="blue")

plt.plot(np.linspace(0.1,0.4,100), np.zeros(100),linewidth=3.0, color="orange")
plt.plot(np.linspace(0.6,0.9,100), np.zeros(100),linewidth=3.0, color="orange")

plt.xlim(-0.5, 1.5)
plt.ylim(-0.5, 1)
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")

plt.legend(loc='upper center', bbox_to_anchor=(0.5, 1.12),
          ncol=4, fancybox=True, shadow=False)

plt.tight_layout()
plt.show()
```

The figure above shows the contours of the problem. The contour lines of the objective function $f_1(x)$ is represented by a solid and $f_2(x)$ by a dashed line. The constraints $g_1(x)$ and $g_2(x)$ are parabolas which intersect the $x_1$-axis at $(0.1, 0.9)$ and $(0.4, 0.6)$. The pareto-optimal set is illustrated by a thick orange line. Through the combination of both constraints the pareto-set is split into two parts.
Analytically, the pareto-optimal set is  given by $PS = \{(x_1, x_2) \,|\, (0.1 \leq x_1 \leq 0.4) \lor (0.6 \leq x_1 \leq 0.9) \, \land \, x_2 = 0\}$ and the Pareto-front by $f_2 = (\sqrt{f_1} - 1)^2$ where $f_1$ is defined in $[0.01,0.16]$ and $[0.36,0.81]$.

### Problem Definition

In *pymoo*, we consider pure minimization problems for optimization in all our modules. However, without loss of generality an objective which is supposed to be maximized, can be multiplied by $-1$ and be minimized. Therefore, we minimize $-f_2(x)$ instead of maximizing $f_2(x)$ in our optimization problem. Furthermore, all constraint functions need to be formulated as a $\leq 0$ constraint.
The feasibility of a solution can, therefore, be expressed by:

$$ \begin{cases}
\text{feasible,} \quad \quad \sum_i^n \langle g_i(x)\rangle = 0\\
\text{infeasbile,} \quad \quad \quad \text{otherwise}\\
\end{cases}
$$

$$
\text{where} \quad \langle g_i(x)\rangle =
\begin{cases}
0, \quad \quad \; \text{if} \; g_i(x) \leq 0\\
g_i(x), \quad  \text{otherwise}\\
\end{cases}
$$

For this reason, $g_2(x)$ needs to be multiplied by $-1$ in order to flip the $\geq$ to a $\leq$ relation. We recommend the normalization of constraints to give equal importance to each of them.
For $g_1(x)$, the coefficient results in $2 \cdot (-0.1) \cdot (-0.9) = 0.18$ and for $g_2(x)$ in $20 \cdot (-0.4) \cdot (-0.6) = 4.8$, respectively. We achieve normalization of constraints by dividing $g_1(x)$ and $g_2(x)$ by its corresponding coefficient.

Finally, the optimization problem to be optimized using *pymoo* is defined by:

\begin{align}
\label{eq:getting_started_pymoo}
\begin{split}
\min \;\; & f_1(x) = (x_1^2 + x_2^2) \\
\min \;\; & f_2(x) = (x_1-1)^2 + x_2^2 \\[1mm]
\text{s.t.} \;\; & g_1(x) = 2 \, (x_1 - 0.1) \, (x_1 - 0.9)  \, /  \,  0.18 \leq 0\\
& g_2(x) = - 20 \, (x_1 - 0.4) \, (x_1 - 0.6) \,  /  \,  4.8 \leq 0\\[1mm]
& -2 \leq x_1 \leq 2 \\
& -2 \leq x_2 \leq 2
\end{split}
\end{align}

Next, the derived problem formulation is implemented in Python. Each optimization problem in *pymoo* has to inherit from the *Problem* class. First, by calling the `super()` function the problem properties such as the number of variables `n_var`, objectives `n_obj` and constraints `n_constr` are initialized. Furthermore, lower `xl` and upper variables boundaries `xu` are supplied as a NumPy array. Additionally, the evaluation function _evaluate needs to be overwritten from the superclass.
The method takes a two-dimensional NumPy array x with n rows and m columns as an input. Each row represents an individual and each column an optimization variable. After doing the necessary calculations, the objective values have to be added to the dictionary out with the key `F` and the constraints with key `G`.

```python
import autograd.numpy as anp
import numpy as np
from pymoo.util.misc import stack
from pymoo.model.problem import Problem

class MyProblem(Problem):

    def __init__(self):
        super().__init__(n_var=2,
                         n_obj=2,
                         n_constr=2,
                         xl=anp.array([-2,-2]),
                         xu=anp.array([2,2]))

    def _evaluate(self, x, out, *args, **kwargs):
        f1 = x[:,0]**2 + x[:,1]**2
        f2 = (x[:,0]-1)**2 + x[:,1]**2

        g1 = 2*(x[:, 0]-0.1) * (x[:, 0]-0.9) / 0.18
        g2 = - 20*(x[:, 0]-0.4) * (x[:, 0]-0.6) / 4.8

        out["F"] = anp.column_stack([f1, f2])
        out["G"] = anp.column_stack([g1, g2])

    # --------------------------------------------------
    # Pareto-front - not necessary but used for plotting
    # --------------------------------------------------
    def _calc_pareto_front(self, flatten=True, **kwargs):
        f1_a = np.linspace(0.1**2, 0.4**2, 100)
        f2_a = (np.sqrt(f1_a) - 1)**2

        f1_b = np.linspace(0.6**2, 0.9**2, 100)
        f2_b = (np.sqrt(f1_b) - 1)**2

        a, b = np.column_stack([f1_a, f2_a]), np.column_stack([f1_b, f2_b])
        return stack(a, b, flatten=flatten)

    # --------------------------------------------------
    # Pareto-set - not necessary but used for plotting
    # --------------------------------------------------
    def _calc_pareto_set(self, flatten=True, **kwargs):
        x1_a = np.linspace(0.1, 0.4, 50)
        x1_b = np.linspace(0.6, 0.9, 50)
        x2 = np.zeros(50)

        a, b = np.column_stack([x1_a, x2]), np.column_stack([x1_b, x2])
        return stack(a,b, flatten=flatten)

problem = MyProblem()

```

Because we consider a test problem where the optimal solutions in design and objective space are known, we have implemented the `_calc_pareto_front` and `_calc_pareto_set` functions to observe the convergence of the algorithm later on. For the optimization run itself the methods need not to be overwritten. So, no worries if you are investigating benchmark or real-world optimization problems.

Moreover, we would like to mention that in many test optimization problems implementation already exist. For example, the test problem *ZDT1* can be initiated by:

```python
from pymoo.factory import get_problem

zdt1 = get_problem("zdt1")
```

Our framework has various single- and many-objective optimization test problems already implemented. Furthermore, a more advanced guide for custom problem definitions is available. In case problem functions are computationally expensive, parallelization of the evaluations functions might be an option.

[Optimization Test Problems](problems/index.ipynb) |
[Define a Custom Problem](problems/custom.ipynb) |
[Parallelization](problems/parallelization.ipynb)
[Callback](misc/callback.ipynb)

### Initialize an Algorithm

Moreover, we need to initialize a method to optimize the problem.
In *pymoo* factory methods create an `algorithm` object to be used for optimization. For each of those methods an API documentation is available and through supplying different parameters, algorithms can be customized in a plug-and-play manner.
Depending on the optimization problem different algorithms can be used to optimize the problem. Our framework offers various [Algorithms](algorithms/index.ipynb) which can be used to solve problems with different characteristics.

In general, the choice of a suitable algorithm for optimization problems is a challenge itself. Whenever problem characteristics are known beforehand, we recommended using those through customized operators.
However, in our case the optimization problem is rather simple, but the aspect of having two objectives and two constraints should be considered. For this reason, we decided to use [NSGA-II](algorithms/nsga2.ipynb) with its default configuration with minor modifications. We chose a population size of 40 (`pop_size=40`) and decided instead of generating the same number of offsprings to create only 10 (`n_offsprings=40`). This is a greedier  variant and improves the convergence of rather simple optimization problems without difficulties regarding optimization, such as the existence of local Pareto fronts.
Moreover, we enable a duplicate check (`eliminate_duplicates=True`) which makes sure that the mating produces offsprings which are different with respect to themselves and the existing population regarding their design space values. To illustrate the customization aspect, we listed the other unmodified default operators in the code-snippet below.

```python
from pymoo.algorithms.nsga2 import NSGA2
from pymoo.factory import get_sampling, get_crossover, get_mutation

algorithm = NSGA2(
    pop_size=40,
    n_offsprings=10,
    sampling=get_sampling("real_random"),
    crossover=get_crossover("real_sbx", prob=0.9, eta=15),
    mutation=get_mutation("real_pm", eta=20),
    eliminate_duplicates=True
)

```

The `algorithm` object contains the implementation of NSGA-II  with the custom settings supplied to the factory method.

### Define a Termination Criterion

Furthermore, a termination criterion needs to be defined to finally start the optimization procedure. Different kind of [Termination Criteria](misc/termination_criterion.ipynb) are available. Here, since the problem is rather simple, we run the algorithm for some number of generations.

```python
from pymoo.factory import get_termination

termination = get_termination("n_gen", 40)
```

Instead of number of generations (or iterations) other criteria such as the number of function evaluations or the improvement in design or objective space from the last to the current generation can be used.

### Optimize

Finally, we are solving the problem with the algorithm and termination criterion we have defined.

```python
from pymoo.optimize import minimize

res = minimize(problem,
               algorithm,
               termination,
               seed=1,
               pf=problem.pareto_front(use_cache=False),
               save_history=True,
               verbose=True)

```

    ==========================================================================================
    n_gen |  n_eval |   cv (min)   |   cv (avg)   |     igd      |      gd      |      hv
    ==========================================================================================
        1 |      40 |  0.00000E+00 |  2.36399E+01 |  0.323661577 |  1.114456088 |  0.259340233
        2 |      50 |  0.00000E+00 |  1.17898E+01 |  0.323661577 |  1.335274628 |  0.259340233
        3 |      60 |  0.00000E+00 |  5.657379846 |  0.320526332 |  1.926659946 |  0.259340233
        4 |      70 |  0.00000E+00 |  2.416757355 |  0.300166590 |  1.617753498 |  0.266749155
        5 |      80 |  0.00000E+00 |  0.969701447 |  0.165500768 |  1.535620684 |  0.287348889
        6 |      90 |  0.00000E+00 |  0.183302529 |  0.165500768 |  1.443230755 |  0.287348889
        7 |     100 |  0.00000E+00 |  0.020538438 |  0.156128592 |  1.404304867 |  0.293954680
        8 |     110 |  0.00000E+00 |  0.000279181 |  0.073420213 |  1.248034651 |  0.394476116
        9 |     120 |  0.00000E+00 |  0.00000E+00 |  0.072646964 |  0.793678104 |  0.398800177
       10 |     130 |  0.00000E+00 |  0.00000E+00 |  0.070217746 |  0.571516864 |  0.398800177
       11 |     140 |  0.00000E+00 |  0.00000E+00 |  0.055648038 |  0.313333192 |  0.407781774
       12 |     150 |  0.00000E+00 |  0.00000E+00 |  0.046858363 |  0.165736198 |  0.411202862
       13 |     160 |  0.00000E+00 |  0.00000E+00 |  0.043001686 |  0.075260262 |  0.416874083
       14 |     170 |  0.00000E+00 |  0.00000E+00 |  0.041120015 |  0.056460681 |  0.417527396
       15 |     180 |  0.00000E+00 |  0.00000E+00 |  0.035881270 |  0.045362876 |  0.418684299
       16 |     190 |  0.00000E+00 |  0.00000E+00 |  0.031726648 |  0.031181960 |  0.426721723
       17 |     200 |  0.00000E+00 |  0.00000E+00 |  0.027885895 |  0.018989167 |  0.431163180
       18 |     210 |  0.00000E+00 |  0.00000E+00 |  0.026692335 |  0.009421017 |  0.431945172
       19 |     220 |  0.00000E+00 |  0.00000E+00 |  0.026021835 |  0.007962640 |  0.433105085
       20 |     230 |  0.00000E+00 |  0.00000E+00 |  0.023344056 |  0.008017948 |  0.435418583
       21 |     240 |  0.00000E+00 |  0.00000E+00 |  0.023130354 |  0.007340408 |  0.436471441
       22 |     250 |  0.00000E+00 |  0.00000E+00 |  0.023130616 |  0.007523541 |  0.436474250
       23 |     260 |  0.00000E+00 |  0.00000E+00 |  0.023019215 |  0.008866366 |  0.436699102
       24 |     270 |  0.00000E+00 |  0.00000E+00 |  0.023152933 |  0.008694703 |  0.436703727
       25 |     280 |  0.00000E+00 |  0.00000E+00 |  0.022693903 |  0.009290452 |  0.437347050
       26 |     290 |  0.00000E+00 |  0.00000E+00 |  0.019819708 |  0.009987094 |  0.439780517
       27 |     300 |  0.00000E+00 |  0.00000E+00 |  0.016667584 |  0.006936027 |  0.442909098
       28 |     310 |  0.00000E+00 |  0.00000E+00 |  0.018568007 |  0.004785434 |  0.443178540
       29 |     320 |  0.00000E+00 |  0.00000E+00 |  0.018465458 |  0.003341749 |  0.443266952
       30 |     330 |  0.00000E+00 |  0.00000E+00 |  0.016865096 |  0.002818623 |  0.449364577
       31 |     340 |  0.00000E+00 |  0.00000E+00 |  0.015600645 |  0.002744785 |  0.450427370
       32 |     350 |  0.00000E+00 |  0.00000E+00 |  0.014513849 |  0.002663537 |  0.451657874
       33 |     360 |  0.00000E+00 |  0.00000E+00 |  0.012592618 |  0.001895409 |  0.454111680
       34 |     370 |  0.00000E+00 |  0.00000E+00 |  0.012569395 |  0.001749484 |  0.454202052
       35 |     380 |  0.00000E+00 |  0.00000E+00 |  0.009111864 |  0.001880313 |  0.455049071
       36 |     390 |  0.00000E+00 |  0.00000E+00 |  0.009118849 |  0.001944459 |  0.455025500
       37 |     400 |  0.00000E+00 |  0.00000E+00 |  0.009041905 |  0.002021041 |  0.455062331
       38 |     410 |  0.00000E+00 |  0.00000E+00 |  0.009146514 |  0.001935355 |  0.455157243
       39 |     420 |  0.00000E+00 |  0.00000E+00 |  0.009146514 |  0.001935355 |  0.455157243
       40 |     430 |  0.00000E+00 |  0.00000E+00 |  0.008957385 |  0.001827609 |  0.455298401

The [Result](misc/results.ipynb) object provides the corresponding X and F values and some more information.

### Visualize

The optimization results are illustrated below (design and objective space). The solid line represents the analytically derived Pareto set and front in the corresponding space and the circles solutions found by the algorithm. It can be observed that the algorithm was able to converge, and a set of nearly-optimal solutions was obtained.

```python
from pymoo.visualization.scatter import Scatter

# get the pareto-set and pareto-front for plotting
ps = problem.pareto_set(use_cache=False, flatten=False)
pf = problem.pareto_front(use_cache=False, flatten=False)

# Design Space
plot = Scatter(title = "Design Space", axis_labels="x")
plot.add(res.X, s=30, facecolors='none', edgecolors='r')
plot.add(ps, plot_type="line", color="black", alpha=0.7)
plot.do()
plot.apply(lambda ax: ax.set_xlim(-0.5, 1.5))
plot.apply(lambda ax: ax.set_ylim(-2, 2))
plot.show()

# Objective Space
plot = Scatter(title = "Objective Space")
plot.add(res.F)
plot.add(pf, plot_type="line", color="black", alpha=0.7)
plot.show()
```

Visualization is an important post-processing step in multi-objective optimization. Although it seems to be pretty easy for our example optimization problem, it becomes much more difficult in higher dimensions where trade-offs between solutions are not easily observable. For visualizations in higher dimensions, various more advanced [Visualizations](visualization/index.ipynb) are implemented in our framework.

### Performance Tracking

If the optimization scenario is repetitive it makes sense to track the performance of the algorithm. Because we have stored the history of the optimization run, we can now analyze the convergence over time. To measure the performance, we need to decide what metric to be used. Here, we are using Hypervolume. Of course, other [Performance Indicators](misc/performance_indicator.ipynb) are available as well.

```python
import matplotlib.pyplot as plt
from pymoo.performance_indicator.hv import Hypervolume

# create the performance indicator object with reference point (4,4)
metric = Hypervolume(ref_point=np.array([1.0, 1.0]))

# collect the population in each generation
pop_each_gen = [a.pop for a in res.history]

# receive the population in each generation
obj_and_feasible_each_gen = [pop[pop.get("feasible")[:,0]].get("F") for pop in pop_each_gen]

# calculate for each generation the HV metric
hv = [metric.calc(f) for f in obj_and_feasible_each_gen]

# visualze the convergence curve
plt.plot(np.arange(len(hv)), hv, '-o')
plt.title("Convergence")
plt.xlabel("Generation")
plt.ylabel("Hypervolume")
plt.show()

```

We hope you have enjoyed the getting started guide. For more topics we refer to each section covered by on the [landing page](https://pymoo.org). If you have any question or concern do not hesitate to [contact us](contact.rst).


# Introduction

There are several loads that act on the rigid bodies of the system. There are three gravitaional forces acting on the mass center of each body and there are three joint torques that act between each body causing rotation. Here we will specify vectors for each load and the points or reference frame they act on.

# Setup

Import the solutions from the last notebook:

```python
from __future__ import print_function, division
from solution.inertia import *
```

Enable mathematical printing:

```python
from sympy.physics.vector import init_vprinting
```

```python
init_vprinting(use_latex=True, pretty_print=False)
```

Images!

```python
from IPython.display import Image
```

# Gravity

First we need a gravitaional constant.

```python
g = symbols('g')
```

Forces are bound vectors, i.e. they act on a point. We need a force with a magnitude $mg$ acting in the negative $y$ direction of the inertial reference frame.

```python
lower_leg_grav_force_vector = -lower_leg_mass * g * inertial_frame.y
lower_leg_grav_force_vector
```

Now we make a tuple to represent the bound vector acting on the mass center of the lower leg.

```python
lower_leg_grav_force = (lower_leg_mass_center, lower_leg_grav_force_vector)
```

Note that the bound force vector is compromised of a Point and a Vector.

```python
type(lower_leg_mass_center)
```

```python
type(lower_leg_grav_force_vector)
```

The same is done for the upper leg and torso.

```python
upper_leg_grav_force = (upper_leg_mass_center, -upper_leg_mass * g * inertial_frame.y)
torso_grav_force = (torso_mass_center, -torso_mass * g * inertial_frame.y)
```

# Joint Torques

Joint torques can be used to simpilfy the effect that muscles have on making body segments move relative to each other. We need to specify three torque vectors which represent the total torque acting on each rigid body. First, specify three time varying variables to represent the magnitude of the joint torques: $T_a$, $T_k$, and $T_h$.

```python
ankle_torque, knee_torque, hip_torque = dynamicsymbols('T_a, T_k, T_h')
```

Similar to the bound force vectors we must specify a reference frame and a vector of all the torques acting on that reference frame (rigid body). The external torques acting on the lower leg can be represented as a vector combining the ankle and knee torque contributions. Don't forget [Newton's third law of motion](https://en.wikipedia.org/wiki/Newton%27s_laws_of_motion#Newton.27s_3rd_Law) (i.e. equal and oposite forces).

```python
Image('figures/lower_leg_torque.png')
```

```python
lower_leg_torque_vector = ankle_torque * inertial_frame.z - knee_torque * inertial_frame.z
lower_leg_torque_vector
```

Now store the lower leg's reference frame and the torque vector in a tuple for later use.

```python
lower_leg_torque = (lower_leg_frame, lower_leg_torque_vector)
```

Again, we do the same for the upper leg and torso.

```python
Image('figures/upper_leg_torque.png')
```

```python
upper_leg_torque = (upper_leg_frame, knee_torque * inertial_frame.z - hip_torque * inertial_frame.z)
```

```python
Image('figures/torso_torque.png')
```

```python
torso_torque = (torso_frame, hip_torque * inertial_frame.z)
```