# Tema 3: Aproximación de autovalores

```python
!pip install -r https://raw.githubusercontent.com/alexmascension/ANMI/main/requirements.txt
```

```python
from sympy import *
from sympy.matrices import Matrix as mat
from sympy.matrices import randMatrix
from sympy import symbols
import sympy

import numpy as np

from scipy.linalg import orth
```

```python
from anmi.genericas import norma, print_verbose

from anmi.T2 import factorizacion_QR
from anmi.T3 import matriz_krylov, sucesion_krylov, potencia_iterada, metodo_autovals_QR
```

### Sucesiones de Krylov
Sea $A$ una matriz (aplicación lineal) y $x$ un vector. Si aplicamos la multiplicación de $A$ por $x$ de manera iterada obtenemos una serie de vectores $\{x, Ax, A^2x, A^3x, \cdots\}$. Si $x$ no es un autovector de $A$, entonces esa sucesión tendrá $n$ (dimensión de $A$) vectores independientes. Si $x$ es un autovector, con su autovalor $\lambda$, entonces la sucesión de vectores será, $\{x, \lambda x, \lambda^2x, \lambda^3x, \cdots\}$. Estas sucesiones de vectores se llaman *sucesiones de Krylov*.

Por otra parte, por el teorema de Cayley-Hamilton se tiene que $A^nx$ tiene que ser una combinación lineal de los siguientes elementos de la sucesión, es decir:
$$(-1)^nA^n + a_{n-1}A^{n-1} + \cdots + a_1A + a_0I = 0$$

Luego si tomamos $a = \begin{bmatrix}a_0\\a_1\\ \cdots \\ a_n\end{bmatrix}$ se tiene que
$$(x|Ax|\cdots|A^{n-1}x)a = (-1)^{n+1}A^nx$$

Y si resolvemos $a$, entonces se tienen los coeficientes del polinómio característico $p(\lambda) = a_0 + a_1\lambda + a_2\lambda^2 + \cdots + a_n\lambda^n$$

```python
help(matriz_krylov)
```

```python
# EJERCICIO 26
A = mat([[1, 1, 1], [0, 2, 2], [3, -1, 0]])
x = mat([[1, 0, 0]]).T

m_krylov = matriz_krylov(A, x)
m_krylov
```

```python
sk, a = sucesion_krylov(A, x)
sk
```

```python
# EJEMPLO 15
A = mat([[2, -1, 0], [-1, 2, -1], [0, -1, 2]])
x = mat([[-1, 0, 1]]).T

matriz_krylov(A, x)
```

### Método de la potencia iterada

En el método de la potencia iterada, se aplica la matriz de krylov hasta una potencia determinada, $k$. Entonces, se tiene que
$$\lim_{k \to \infty} \frac{A^kw}{A^{k-1}w} = |\lambda_1|$$
Es decir, el mayor autovalor.

Además, si tomamos $ B= A^{-1} $, tenemos que
$$\lim_{k \to \infty} \frac{B^kw}{B^{k-1}w} = \frac{1}{|\lambda_n|}$$
Donde $\lambda_n$ es el menor autovalor.

```python
help(potencia_iterada)
```

```python
A.eigenvals()
```

```python
x = mat([[-2, 0, 1]]).T

matriz_krylov(A, x, 17)
```

```python
x = mat([[-1, 0, 0]]).T

np.array(potencia_iterada(A, x, 30, devolver_ultimo=False)[:, -3:], dtype=float)
```

```python
x = mat([[-1, 0, 0]]).T

np.array(potencia_iterada(A, x, 300, devolver_ultimo=True), dtype=float)
```

```python
N(2+sqrt(2))  # Vemos que converge al mayor autovalor
```

```python
np.array(potencia_iterada(A**-1, x, 300, devolver_ultimo=True), dtype=float)
```

```python
1/N(2-sqrt(2))  # Y lo mismo con el menor
```

```python
# Si tomamos una matriz ortogonal, el metodo de la potencia no tiene validez
# porque se requiere que haya autovalores dominantes, y en este caso los
# autovalores tienen módulo 1.

dict_QR = factorizacion_QR(A)
Q = dict_QR['Q']

Q
```

```python
Q * Q.T
```

```python
Q.eigenvals()
```

```python
N(-1/2 + 3*sqrt(70)/70 + 3*sqrt(14)/28 + sqrt(5)/5 + sqrt(70)*I*sqrt(6*sqrt(14) + 20*sqrt(5) + 73)/140)
```

```python
matriz_krylov(Q, x, 5)
N(matriz_krylov(N(Q), x, 30), 4)
```

```python
potencia_iterada(N(Q), x, 100, devolver_ultimo=False)[:, -5:]  # No hay una convergencia
```

### Método QR

El método QR consiste en aplicar los siguientes pasos:

$$A^{(1)} = A$$

De ahí sacamos que
$$A^{(1)}  = Q^{(1)}R^{(1)}$$

De ahí construimos:
$$A^{(2)}  = R^{(1)}Q^{(1)}$$

Y se cumple que $A^{(1)}$ y $A^{(2)}$ son semejantes, luego tienen los mismos
autovalores.

Con ello se reitera el proceso, y se cumple que las matrices equivalentes
construidas convergen a una matriz triangular superior. Los la diagonal de $A^{(k)}$ converge a los autovalores de $A$.

```python
help(metodo_autovals_QR)
```

```python
dict_QR = metodo_autovals_QR(A, n_iters=10)
```

```python
N(dict_QR['A'][-2], 3)
```

```python
N(dict_QR['A'][-1], 3)
```

```python
N(2- sqrt(2), 3), 2, N(2 + sqrt(2), 3),
```

```python
A = mat([[1, 1, 1], [0, 0, 1], [0, 1, 1]])
dict_QR = metodo_autovals_QR(A, n_iters=30, verbose=False)
```

```python
N(dict_QR['A'][-15], 3)
```

```python
N(dict_QR['A'][-1], 3)
```

```python
[N(i, 3) for i in list(A.eigenvals().keys())]
```

## Ejercicios

### Ejercicio 27

Determinar las primera iteraciones generadas por el método de la potencia con normalización con norma infinito cuando se aplica a la matriz con vector inicial.

```python
A = Matrix([[0, -1, 1], [0, 1, -1], [-1, -1, 2]])
A
```

```python
x0 = Matrix([1, -1, 2])
```

```python
n_iters = 15
m_krylov_x, m_krylov_w = zeros(A.shape[0], n_iters), zeros(A.shape[0], n_iters)
m_krylov_x[:, 0] = x0
m_krylov_w[:, 0] = x0/max(x0)

for i in range(1, n_iters):
    kriv_i_x = A * m_krylov_w[:, i - 1]
    m_krylov_x[:, i] = kriv_i_x
    m_krylov_w[:, i] = kriv_i_x / max(kriv_i_x)

```

```python
m_krylov_x
```

```python
m_krylov_w
```

```python
[N(i) for i in (np.array(m_krylov_x[0, 1:]) / np.array(m_krylov_w[0, :-1])).tolist()[0]]
```

```python
# Probamos sin normalizar

n_iters = 15
m_krylov_x  = zeros(A.shape[0], n_iters)
m_krylov_x[:, 0] = x0

for i in range(1, n_iters):
    kriv_i_x = A * m_krylov_x[:, i - 1]
    m_krylov_x[:, i] = kriv_i_x

m_krylov_x
```

```python
[N(i) for i in (np.array(m_krylov_x[0, 1:]) / np.array(m_krylov_x[0, :-1])).tolist()[0]]
```

### Ejercicio 28

Realizar una iteración con el método QR para el cálculo de autovalores de la matriz $A$

```python
A = Matrix([[1, 1, 0], [2, 1, 0], [2, 0, 1]])
```

```python
metodo_autovals_QR(A, n_iters=1, verbose=True)
```

```python
dict_QR = metodo_autovals_QR(A, n_iters=15, verbose=False)
N(dict_QR['A'][-1])
```

```python
N(dict_QR['A'][-2])
```

```python
np.sort(solve(det(A - eye(3) * Symbol('lambda')), Symbol('lambda')))[::-1]
```

### Ejercicio 29
Sea $A$ con $\theta$ dado. Determinar el n-ésimo términa de la sucesión de Krylov asociada a $A$ y al vector $x$. Por medio de la sucesión de Krylov determia el polinomio característico de $A$.

```python
t = Symbol('theta')
A = Matrix([[1, 0, 0], [0, cos(t), -sin(t)], [0, sin(t), cos(t)]])
x0 = Matrix([1, cos(t), -sin(t)])
```

```python
ss = simplify(matriz_krylov(A, x0, n_iters=8))
ss
```

Así a priori tiene pintas de que $A^nx = (1, \cos(n-1)\theta, sin(n-1)\theta)^t$.

Como $n=3$ solo necesitamos los primeros 4 términos de la sucesión de krylov para resolver el sistema y crear el polinomio característico.

```python
M = ss[:3, :3]
am = Matrix([Symbol('a0'), Symbol('a1'), Symbol('a2')])
rhs = ss[:3, 3]
```

```python
sol = simplify(M.inv() * rhs)
sol
```

```python
l = Symbol('lambda')
simplify(sol[0] + sol[1] * l + sol[2] * l ** 2 - l ** 3)
```

### Ejercicio 31

Si $\alpha$ es una constante, aproximar el autovalore de módulo máximo de la matriz $A$ usando el método de la potencia (sin normalización) como vector inicial $v$, probando las diferentes componentes.

```python
a = Symbol('alpha')
n = Symbol('n')
A = Matrix([[1, 0, 0], [0, -2*cos(a)**2, 2*cos(a)*sin(a)], [0, 2*cos(a)*sin(a), -2*sin(a)**2]])
A
```

```python
v = Matrix([1, 1, 1])
v
```

```python
pol, vecfinal = sucesion_krylov(A, v)
pol
```

```python
solve(pol, l)
```

```python
vecfinal
```

```python
# Vamos a ir desglosando
mk = matriz_krylov(A, v, n_iters=7)
mk
```

```python
simplify(Matrix(np.array(mk[:, 1:]) / np.array(mk[:, :-1])))
```


*This notebook contains course material from [CBE30338](https://jckantor.github.io/CBE30338)
by Jeffrey Kantor (jeff at nd.edu).
Text is released under the [CC-BY-NC-ND-4.0 license](https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode). Code is released under the [MIT license](https://opensource.org/licenses/MIT).*

# Modeling and Control of a Campus COVID-19 Outbreak

The following cell simulates a COVID-19 outbreak in campus population of size N starting with one infectious case.  The following parameter values for a standard SEIR model were selected from values given in the recent survey by Boldog, et al. (2020).

| Parameter |  Symbol  | Typical |
| :------- | :-- | :-----------: |
| Reproduction number | $R_0$ | 2.4 |
| Incubation period (days)| $\tau_{incubation}$ | 5.1
| Infectious period (days) | $\tau_{infectious}$ | 3.3
| Campus population | $N$ | 20,000
| Initial number exposed | $n$ | 10
| Mitigation by social distancing | $u$ | 0.2
| Start of social distancing following exposure (weeks) | $t_{sd}$ | 2

To run the model, adjust the sliders below, then click the run arrow that appears in the left marging. The charts will automatically update when the sliders are readjusted.

```python
#@title SEIR Model with Social Distancing { run: "auto" }
#@markdown
#@markdown Reproduction number
R0 = 2.4 #@param {type:"slider", min:0.9, max:5, step:0.1}
#@markdown Disease periods in days
t_incubation = 5.1 #@param {type:"slider", min:1, max:14, step:0.1}
t_infective = 3.3 #@param {type:"slider", min:1, max:14, step:0.1}
#@markdown Population Size
N = 14000 #@param {type:"slider", min:1000, max:350000, step: 1000}
#@markdown Initial number exposed
n = 10 #@param {type:"slider", min:0, max:100, step:1}
#@markdown Start of social distancing following exposure (weeks)
t_social_distancing = 2 #@param {type:"slider", min:0, max:30, step:0.1}
#@markdown Effectiveness of social distancing (0 to 100%)
u_social_distancing = 40 #@param {type:"slider", min:0, max:100, step:1}

import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
import seaborn as sns

# initial number of infected and recovered individuals
e_initial = n/N
i_initial = 0.00
r_initial = 0.00
s_initial = 1 - e_initial - i_initial - r_initial

alpha = 1/t_incubation
gamma = 1/t_infective
beta = R0*gamma

def step(t):
    return 1 if t >= 7*t_social_distancing else 0

# SEIR model differential equations.
def deriv(x, t, u, alpha, beta, gamma):
    s, e, i, r = x
    dsdt = -(1-u*step(t)/100)*beta * s * i
    dedt =  (1-u*step(t)/100)*beta * s * i - alpha * e
    didt = alpha * e - gamma * i
    drdt =  gamma * i
    return [dsdt, dedt, didt, drdt]

t = np.linspace(0, 210, 210)
x_initial = s_initial, e_initial, i_initial, r_initial
s, e, i, r = odeint(deriv, x_initial, t, args=(u_social_distancing, alpha, beta, gamma)).T
s0, e0, i0, r0 = odeint(deriv, x_initial, t, args=(0, alpha, beta, gamma)).T

# plot the data
fig = plt.figure(figsize=(12, 10))
ax = [fig.add_subplot(311, axisbelow=True),
      fig.add_subplot(312)]

pal = sns.color_palette()

ax[0].stackplot(t/7, N*s, N*e, N*i, N*r, colors=pal, alpha=0.6)
ax[0].set_title('Susceptible and Recovered Populations with {0:3.0f}% Effective Social Distancing'.format(u_social_distancing))
ax[0].set_xlabel('Weeks following Initial Campus Exposure')
ax[0].set_xlim(0, t[-1]/7)
ax[0].set_ylim(0, N)
ax[0].legend([
    'Susceptible',
    'Exposed/no symptoms',
    'Infectious/symptomatic',
    'Recovered'],
    loc='best')
ax[0].plot(np.array([t_social_distancing, t_social_distancing]), ax[0].get_ylim(), 'r', lw=3)
ax[0].plot(np.array([0, t[-1]])/7, [N/R0, N/R0], lw=3, label='herd immunity')
ax[0].annotate("Start of social distancing",
    (t_social_distancing, 0), (t_social_distancing + 1.5, N/10),
    arrowprops=dict(arrowstyle='->'))
ax[0].annotate("Herd Immunity without social distancing",
    (t[-1]/7, N/R0), (t[-1]/7 - 8, N/R0 - N/5),
    arrowprops=dict(arrowstyle='->'))

ax[1].stackplot(t/7, N*i0,N*e0, colors=pal[2:0:-1], alpha=0.5)
ax[1].stackplot(t/7, N*i, N*e, colors=pal[2:0:-1], alpha=0.5)
ax[1].set_title('Infected Population with no Social Distancing and with {0:3.0f}% Effective Social Distancing'.format(u_social_distancing))
ax[1].set_xlim(0, t[-1]/7)
ax[1].set_ylim(0, max(0.3*N, 1.05*max(N*(e + i))))
ax[1].set_xlabel('Weeks following Initial Campus Exposure')
ax[1].legend([
    'Infective/Symptomatic',
    'Exposed/Not Sympotomatic'],
    loc='upper right')
ax[1].plot(np.array([t_social_distancing, t_social_distancing]), ax[0].get_ylim(), 'r', lw=3)

y0 = N*(e0 + i0)
k0 = np.argmax(y0)
ax[1].annotate("No social distancing", (t[k0]/7, y0[k0] + 100))

y = N*(e + i)
k = np.argmax(y)
ax[1].annotate("With {0:3.0f}% effective social distancing ".format(u_social_distancing), (t[k]/7, y[k] + 100))

for a in ax:
    a.xaxis.set_major_locator(plt.MultipleLocator(5))
    a.xaxis.set_minor_locator(plt.MultipleLocator(1))
    a.xaxis.set_major_formatter(plt.FormatStrFormatter('%d'))
    a.grid(True)

plt.tight_layout()
```

This Jupyter/Python notebook presents a standard epidimiological model to simulate the outbreak of an infectious disease into a susceptible population. Model parameters are taken from a rapidly evolving scientific literature documenting the global COVID-19 outbreak. Mitigation strategies based on 'social distancing' are incorporated in the model as a control parameter.

The notebook is organized as follows:

1. Background on COVID-19 and the SARS-CoV-2 Coronavirus
1. Sources of Information
1. Introduction to the SEIR Model
1. Social Distancing Strategies
1. Ideas for student projects

## 1. COVID-19 and the SARS-CoV-2 Coronavirus

The purpose of this notebook is to demonstrate the modeling of an infectious epidemic using the latest available data for [COVID-19](https://www.who.int/health-topics/coronavirus), and to provide a framework for evaluating the performance of 'social distancing' and other mitigation strategies. COVID-19 is caused by the human coronavirus SARS-CoV-2. First identified in the 1960's, there are currently four human coronaviruses endemic to populations around the world:

* 229E (alpha coronavirus)
* NL63 (alpha coronavirus)
* OC43 (beta coronavirus)
* HKU1 (beta coronavirus)

These four common coronaviruses cause an upper respiratory disease that can progress to pneumonia. These endemic viruses cause about a quarter of all common colds. Most people will suffer from at least one during their lifetimes.

In recent decades, three additional coronaviruses that normally infect animals have evolved to infect humans. These include:

* MERS-CoV (the beta coronavirus that causes Middle East Respiratory Syndrome, or MERS)
* SARS-CoV (the beta coronavirus that causes severe acute respiratory syndrome, or SARS)
* SARS-CoV-2 (the novel coronavirus that causes coronavirus disease COVID-19)

The last of these, now called SARS-CoV-2, first appeared in December, 2019, at a seafood market in Wuhan (Hubei, China). The rapid spread of SARS-CoV-2 in Wuha and subsequent appearance in other locations around the globe led to declaration of a [global health emergency by the World Health Organization (WHO)](https://www.who.int/emergencies/diseases/novel-coronavirus-2019).

Countries are mobilizing to track the virus and control new outbreaks. At this stage it is too early to know if efforts to contain the mitigate transmission of the virus will be successful in preventing COVID-19 from becoming a pandemic or, later, evolving into an endemic disease with a global footprint.

## 2. Sources of Information

General resources on COVID-19

* [Harvard Health](https://www.health.harvard.edu/diseases-and-conditions/coronavirus-resource-center)
* [U.S. Center for Disease Control (CDC)](https://www.cdc.gov/coronavirus/2019-ncov/index.html)

The current status on the global outbreak of COVID-19 can be found at the following links:

* [Coronavirus COVID-19 Global Cases by Johns Hopkins CSSE](https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6)
* [WHO Novel Coronavirus (COVID-19) Situation](https://experience.arcgis.com/experience/685d0ace521648f8a5beeeee1b9125cd)

The models and data used in this notebook have been extracted from a rapidly emerging and changing literature. Recent papers on COVID-19 can be found at the following links.

* [Cell Press Coronavirus Resource Hub](https://www.cell.com/2019-nCOV)
* [Lancet COVID-19 Resource Centre](https://www.thelancet.com/coronavirus)
* [medRxiv preprint server search on COVID-19](https://www.medrxiv.org/search/COVID-19%20numresults%3A50%20sort%3Apublication-date%20direction%3Adescending)
* [New England Journal of Medicine Coverage on Coronavirus (COVID-19)](https://www.nejm.org/coronavirus)
* [WHO Database of publications on coronavirus disease (COVID-19)](https://www.who.int/emergencies/diseases/novel-coronavirus-2019/global-research-on-novel-coronavirus-2019-ncov)

## 3. Introduction to the SEIR model for epidemics

### Readings

* Keeling, Matt J., and Pejman Rohani. Modeling Infectious Diseases in Humans and Animals. Princeton University Press, 2008. JSTOR, www.jstor.org/stable/j.ctvcm4gk0. Accessed 25 Feb. 2020.
* Pan, Jinhua, et al. "Effectiveness of control strategies for Coronavirus Disease 2019: a SEIR dynamic modeling study." medRxiv (2020). https://www.medrxiv.org/content/10.1101/2020.02.19.20025387v3.full.pdf
* Peng, Liangrong, et al. "Epidemic analysis of COVID-19 in China by dynamical modeling." arXiv preprint arXiv:2002.06563 (2020). https://arxiv.org/abs/2002.06563
* Delamater PL, Street EJ, Leslie TF, Yang YT, Jacobsen KH. Complexity of the basic reproduction number (R0). Emerg Infect Dis. 2019 Jan. Accessed 9 Mar. 2020. https://doi.org/10.3201/eid2501.171901

### SEIR Basics

The SEIR model is deterministic compartment model for the spread of an infectious disease that describes key phenomena encountered in epidemiology. In the SIR model, a population is broken into foir non-overlapping groups corresponding to stages of the disease:

* **Susceptible (S).** The subpopulation susceptible to acquire the disease.
* **Exposed (E).** The subpopulation that has been been infected with the virus, but not yet in an infective state capable of transmitting the virus to others.
* **Infectious (I).** The subpopulation that has acquired the virus and can infect others.
* **Recovered (R).** The subpopulation that has recovered from infection and presumed to be no longer susceiptible to the disease.

Neglecting demographic processes of birth and death from other causes, and assuming a negligible death rate due to infectious disease at issue, the progression of an epidemic can be modeled by rate processesl

$$\text{Susceptible}
\xrightarrow{(1-u)\frac{\beta S I}{N}}
\text{Exposed}
\xrightarrow{\alpha E}
\text{Infectious}
\xrightarrow{\gamma I}
\text{Recovered} $$

The rate processes are modeled as follows.

* $(1-u)\frac{\beta S I}{N}$ is the rate at which susecptible population encounters the infected population resulting in trasmission of the disease. $S$ is the size of the susceptible population. $\beta$ is a the model parameters with units of 1/day.
* $u$ describes the effectiveness on any public health interventions to control transmission of the disease. $u=0$ corresponds to no effective public health interventions, $u=1$ implies total elimination of disease transmission.
* $\alpha E$ is the rate at which exposed population becomes infective, where $E$ is the size of the exposed population. The average period of time in the exposed state is the incubation period of the disease, and equal to $\frac{1}{\alpha}$.
* $\gamma I$ is the rate at which infected population recovers and becomes resistent to further infection. $I$ is the size of the infective population. The average period of infectious state is $\frac{1}{\gamma}$.

A model for the spread of an infectious disease in a uniform population is given by the deterministic SIR equations

\begin{align*}
\frac{dS}{dt} & = -(1-u)\frac{\beta S I}{N} \\
\frac{dE}{dt} & = (1-u)\frac{\beta S I}{N} - \alpha E \\
\frac{dI}{dt} & = \alpha E - \gamma I \\
\frac{dR}{dt} & = \gamma I
\end{align*}

The model becomes more generic by working with population fractions rather than raw population counts. To this end, define

\begin{align}
s = \frac{S}{N} \qquad
e = \frac{E}{N} \qquad
i = \frac{I}{N} \qquad
r = \frac{R}{N}
\end{align}

After substitution, this results in a system of four equations.

\begin{align*}
\frac{ds}{dt} & = -(1-u)\beta s i \\
\frac{de}{dt} & = (1-u)\beta s i - \alpha e \\
\frac{di}{dt} & = \alpha e  - \gamma i \\
\frac{dr}{dt} & = \gamma i
\end{align*}

where $s + e + i + r = 1$ is an invariant.

### Properties of the SEIR Model

The SEIR model describes key epidemiological phenemena. Here is a brief summary of the key parameters appearing in the SEIR model.

* The parameters $\alpha$, $\beta$ and $\gamma$ have units of inverse time.
* $\beta$ is the rate constant associated with transmission of the virus.
* $\alpha$ is rate at which individuals convert from the exposed state to the infectious state. The average period of time is $\tau_{incubate} = \frac{1}{\alpha}$.
* $\gamma$ is the rate of recovery from infections. The associated time constant $\tau_{recovery}=\frac{1}{\gamma}$ is average time to recover from an infection.

The relationships of rate constants to time constants can be summaried as

\begin{align}
\tau_{incubate} = \frac{1}{\alpha} \qquad
\tau_{recovery} = \frac{1}{\gamma}
\end{align}

The SEIR model makes key predictions concerning the outbreak and eventual recovery from an epidemic. These are summarized as follows:

* The infectious population grows only if $\beta s > \gamma$, that is the rate of infection is greater than the rate of recovery.
* The ratio $R_0 = \frac{\beta}{\gamma}$ is the "Basic Reproduction Number". $R_0$ describes the transmissability or contagiousness of an infectious disease.
* $R_0$ is the average number of people acquiring the virus from an infected individual in an otherwise completely susceptible population.
* The infectious population can grow only if $R_0 s > 1$. If $s=1$, then $R_0 > 1$ is sufficient for growth of the infectious population.
* The infected population, that is number of people in the subpopulations E and I, decreases if $s R_0 < 1$ or, equivalently, $s < \frac{1}{R_0}$.
* The population has 'herd immunity' when a sufficient number of individuals are immune to the virus. Under this condition, an outbreak may lead to new cases, but the size of the infected population always decreases. This condition is achieved when the fraction of susceptibles is less than $\frac{1}{R_0}$.

## 4. Social Distancing Strategies

The lack of a vaccine reduces the options for controlling the COVID-19 outbreak. Current efforts are focused on '[social distancing](https://www.cdc.gov/coronavirus/2019-ncov/community/guidance-ihe-response.html)' designed to reduce transmission of the virus from individuals in the infective state to susceptible individuals. Social distancing has the following goals:
* Reduce the cumulative number of individuals infected with the virus.
* Slow progress of any outbreak enough to introduce other mitigation strategies.
* Reduce strain on health care services.

Campus strategies for social distancing might include:

* Replace large section courses with streaming lectures. Close the largest lecture halls for the remainder of the term.
* Bring food to the dorms rather than the dorms to food.
* Locate an area to isolate infectious students, and quarantine students, faculty, and staff with potential exposure.
* Promoting rigorous public hygiene.

* The addition of an exposed population compartment slows the outbreak and reduces the impact on health services, but doesn't reduce the number of people ultimately infected by the disease.

* What are the campus policy implications of these results?

Social distancing has several beneficial effects:

* Slows down the progress of the epidemic.
* Seduces the fraction of the population infected at any point in time, thereby reducing strain on health care resources.
* Reduces the number of individuals ultimately experiencing the disease. For a disease with a non-zero mortaility, this saves lives.

## 5. Additional Modeling Opportunities.

This notebook was originally prepared as a lecture for [CBE 30338 Chemical Process Control](http://jckantor.github.io/CBE30338/) taught at the University of Notre Dame. Students in the course are encouraged to develop their own models to test alternative hypotheses. The following papers are suggested starting points.

### Power Law Kinetics

The SEIR model is a well-known and widely used deterministic model for epidemiology. An underlying assumption of the SEIR model is a rate law proportional to the product of susceptible and infectious sub-populations that predicts exponential growth in the early phases of an epidemic. A network model would leads to power law kinetics that may be a better fit to observed data.

* Ziff, Robert M., and Anna L. Ziff. "Fractal kinetics of COVID-19 pandemic." medRxiv (2020). https://www.medrxiv.org/content/10.1101/2020.02.16.20023820v1

### Transportation

The simulation model presented in this notebook assumes single, homegeneous sub-populations of susceptible, exposed, infectious, and recovered individuals. The notebook model could be extended to geographically distributed populations by adding the transportation terms outlined, for example, in the following papers.

* Peng, Liangrong, et al. "Epidemic analysis of COVID-19 in China by dynamical modeling." arXiv preprint arXiv:2002.06563 (2020).  https://arxiv.org/abs/2002.06563

* Wu, Joseph T., Kathy Leung, and Gabriel M. Leung. "Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study." The Lancet (2020). https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30260-9/fulltext

* Chinazzi, Matteo, et al. "The effect of travel restrictions on the spread of the 2019 novel coronavirus (2019-nCoV) outbreak." medRxiv (2020). https://science.sciencemag.org/content/early/2020/03/05/science.aba9757

### Now-casting with Limited Testing Data

Reporting only cases that have been confirmed by postive test results undercounts the number of exposed and infected individuals. Is it possible to construct a procedure to better estimate the actual count of exposed and infected individuals from limited test data?

* Bedford, Trevor. Cryptic transmission of novel coronavirus revealed by genomic epidemiology. Accessed 4 Mar 2020. https://bedford.io/blog/ncov-cryptic-transmission/

* Tuite, Ashleigh R., and David N. Fisman. "Reporting, Epidemic Growth, and Reproduction Numbers for the 2019 Novel Coronavirus (2019-nCoV) Epidemic." Annals of Internal Medicine (2020). https://annals.org/aim/fullarticle/2760912

* Wood, Graeme. "Iran Has Far More Coronavirus Cases Than It Is Letting On." The Atlantic, 9 March 2020. https://www.theatlantic.com/ideas/archive/2020/03/irans-coronavirus-problem-lot-worse-it-seems/607663/

```python

```


# <center> ЛАБОРАТОРНАЯ  РАБОТА 2 </center>
# <center> РЕШЕНИЕ  НЕЛИНЕЙНЫХ  УРАВНЕНИЙ </center>

Теоретический материал к данной теме содержится в [1, глава 4].

```python
from sympy.solvers import solve
from sympy import exp, sin, cos, sqrt, log, ln, pi
from sympy import Rational as syR
from sympy import Symbol, diff
from scipy.misc import derivative

import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
```

## Задание 1
Даны два уравнения  $f(x)=0$ и $g(x)=0$. Найти с точностью $10^{-10}$  все корни уравнений, содержащиеся на отрезке [a, b]. Для решения  задачи использовать  метод бисекции.  Найти корни с помощью встроенной функции.
## ПОРЯДОК  РЕШЕНИЯ  ЗАДАЧИ:
1. Найти аналитическое решение уравнения  $f(x)=0$.
2. Локализовать корни  $f(x)=0$   графически.
3. Найти корни уравнения $f(x)=0$  с точностью    с помощью метода бисекции.
4. Используя  встроенную функцию найти корни уравнения $f(x)=0$ с точностью  .
5. Аналогично п. 1-4 попытаться найти корни уравнения  $g(x)=0$. Объяснить полученные результаты.

Ваши функции:
$f(x) = {Task1[TASK_VARIANT][f_latex]}$
$g(x) = {Task1[TASK_VARIANT][f_latex]}$

Пределы:
$[a;b] = [Task1[TASK_VARIANT][a];Task1[TASK_VARIANT][b]] $

```python
epsilon1 = 10**(-10)
# Функции
f1 = {Task1[TASK_VARIANT][f_latex]}
g1 = {Task1[TASK_VARIANT][f_latex]}
# Пределы:
a1,b1 = {Task1[TASK_VARIANT][a]}, {Task1[TASK_VARIANT][b]}
```

```python
Локализуйте корни уравнения графически, используя пакет matplotlib
```

```python
## ЛОКАЛИЗУЙТЕ КОРНИ ГРАФИЧЕСКИ ##
```

Допишите функцию BisecMethod, которая реализует метод бисекции.

```python
def BisecMethod():
    return 0
```

```python
## Тут решение ##
```

```python
## Тут тест ##
```

```python
## Решение встроенной функцией ##
```

```python
## Тест ##
```

```python
## Простор для функции g ##
```

```python
## Простор для функции g ##
```

```python
## Простор для функции g ##
```

## Задание 2
Найти указанный в варианте  корень уравнения $f(x)=0$ с точностью $10^{-6}$ ,  двумя  способами.

   а) Использовать метод бисекции. Предварительно определить отрезок локализации  [a, b].

   b) Использовать метод Ньютона. В качестве начального приближения для метода Ньютона взять  середину отрезка локализации из п. а).

   Сравнить число итераций в п. a), b).

Ваша функция $f(x) = {Task2[TASK_VARIANT][f_latex]}$
Корень, который Вам нужен: {Task2[TASK_VARIANT][which]}

```python
epselon2 = 10**(-6)
f2 = {Task2[TASK_VARIANT][f]}
```

Определите отрезок локализации

```python
## Определите отрезок локализации ##
```

Решите задачу используя метод бисекции

```python
## Решение метод бисекции ##
```

```python
## ТЕСТ ##
```

Определите метод Ньютона решения нелинейных уравнений

```python
def NewtoonMethod():
    return 0
```

```python
## Тут тест ##
```

Сравните число итераций в двух методах

```python
## Тут сравнение числа итераций ##
```

```python
## ТЕСТ ДЛЯ ЧИСЛА ИТЕРАЦИЙ ##
```

## Задание 3

Локализовать корни уравнения $f(x)=0$ и найти их с точностью $10^{-5}$ , используя метод простой итерации. К виду $x=\phi(x)$, удобному для итераций, уравнение $ f(x)=0$ привести двумя способами.

a) Преобразовать уравнение к виду  $x=x-\alpha f(x)$,    где $\alpha=\frac{2}{M+m}$, $0 < m \leq f'(x) \leq M$, а $x$ принадлежит отрезку    локализации [a, b].

b) Любым  другим  преобразованием  уравнения. Проверить достаточное условие сходимости метода.

Использовать критерий окончания итерационного процесса вида  , где  в  п. a)  $q=\frac{(M-m)}{(M+m)}$, в п. b) $q = \max\limits_{x \in [a,b]}{\phi'(x)}$

 Сравнить число итераций и значения величины q  в  п. a),  b).

Ваша функция:
$ f(x) = {Task3[TASK_VARIANT][f_latex]} $

```python
epselon3 = 10**(-5)
f3 = {Task3[TASK_VARIANT][f]}
```

Реализуйте метод простой итерации

```python
def SimpleIteration():
    return 0
```

```python
## Первое решение ##
```

```python
## Второе решение ##
```

#### ЛИТЕРАТУРА:
1. Амосов А.А., Дубинский Ю.А., Копченова Н.В. Вычислительные методы для инженеров. М.: Высшая школа, 1994.


```
def check_goldbach_for_num(n,primes_set) :
    '''gets an even integer- n, and a set of primes- primes_set. Returns whether there're two primes which their sum is n'''

    relevant_primes_set={p for p in primes_set if p<n}

    for prime in primes_set :
        if (n-prime) in  relevant_primes_set:
            return True
    return False

```

```
from sympy.ntheory.generate import primerange
primes = set(primerange(50,1000))
check_goldbach_for_num(116, primes)
```


# Using physics informed neural networks (PINNs) to solve parabolic PDEs

In this notebook, we illustrate physics informed neural networks (PINNs) to solve partial differential equations (PDEs) as proposed in

- Maziar Raissi, Paris Perdikaris, George Em Karniadakis. *Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations*. [arXiv 1711.10561](https://arxiv.org/abs/1711.10561)
- Maziar Raissi, Paris Perdikaris, George Em Karniadakis. *Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations*. [arXiv 1711.10566](https://arxiv.org/abs/1711.10566)
- Maziar Raissi, Paris Perdikaris, George Em Karniadakis. *Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations*. J. Comp. Phys. 378 pp. 686-707 [DOI: 10.1016/j.jcp.2018.10.045](https://www.sciencedirect.com/science/article/pii/S0021999118307125)

This notebook is partially based on another implementation of the PINN approach published on [GitHub by pierremtb](https://github.com/pierremtb/PINNs-TF2.0) as well as the original code, see [Maziar Raissi on GitHub](https://github.com/maziarraissi/PINNs).

## Introduction

We describe the PINN approach for approximating the solution $u:[0,T] \times \mathcal{D} \to \mathbb{R}$ of an evolution equation

$$
\begin{align}
    \partial_t u (t,x) + \mathcal{N}[u](t,x) &= 0, && (t,x) \in (0,T] \times \mathcal{D},\\
    u(0,x) &= u_0(x) \quad && x \in \mathcal{D},
\end{align}
$$

where $\mathcal{N}$ is a nonlinear differential operator acting on $u$,
$\mathcal{D} \subset \mathbb{R}^d$ a bounded domain,
$T$ denotes the final time and
$u_0: \mathcal{D} \to \mathbb{R}$ the prescribed initial data.
Although the methodology allows for different types of boundary conditions, we restrict our discussion to the inhomogeneous Dirichlet case and prescribe

$$
\begin{align}
        \hspace{7em} u(t,x) &= u_b(t,x)  && \quad (t,x) \in (0,T] \times \partial \mathcal{D},
\end{align}
$$

where $\partial \mathcal{D}$ denotes the boundary of the domain $\mathcal{D}$ and $u_b: (0,T] \times \partial \mathcal{D} \to \mathbb{R}$ the given boundary data.

## Methodology

The method constructs a neural network approximation

$$
u_\theta(t,x) \approx u(t,x)
$$

of the solution of nonlinear PDE, where $u_\theta :[0,T] \times \mathcal{D} \to \mathbb{R}$ denotes a function realized by a neural network with parameters $\theta$.

The continuous time approach for the parabolic PDE as described in ([Raissi et al., 2017 (Part I)](https://arxiv.org/abs/1711.10561)) is based on the (strong) residual of a given neural network approximation $u_\theta \colon [0,T] \times \mathcal{D} \to \mathbb{R} $ of the solution $u$, i.e.,

$$
\begin{align}
    r_\theta (t,x) := \partial_t u_\theta (t,x) + \mathcal{N}[u_\theta] (t,x).
\end{align}
$$

To incorporate this PDE residual $r_\theta$ into a loss function to be minimized, PINNs require a further differentiation to evaluate the differential operators $\partial_t u_\theta$ and $\mathcal{N}[u_\theta]$.
Thus the PINN term $r_\theta$ shares the same parameters as the original network $u_\theta(t,x)$, but respects the underlying "physics" of the nonlinear PDE.
Both types of derivatives can be easily determined through automatic differentiation with current state-of-the-art machine learning libraries, e.g., TensorFlow or PyTorch.

The PINN approach for the solution of the initial and boundary value problem now proceeds by minimization of the loss functional

$$
\begin{align}
    \phi_\theta(X) := \phi_\theta^r(X^r) + \phi_\theta^0(X^0) + \phi_\theta^b(X^b),
\end{align}
$$

where $X$ denotes the collection of training data and the loss function $\phi_\theta$ contains the following terms:

  - the mean squared residual
$$
  \begin{align*}
   \phi_\theta^r(X^r) := \frac{1}{N_r}\sum_{i=1}^{N_r} \left|r_\theta\left(t_i^r, x_i^r\right)\right|^2
\end{align*}
$$
in a number of collocation points $X^r:=\{(t_i^r, x_i^r)\}_{i=1}^{N_r} \subset (0,T] \times \mathcal{D}$, where $r_\theta$ is the physics-informed neural network,
  - the mean squared misfit with respect to the initial and boundary conditions
$$
    \begin{align*}
   \phi_\theta^0(X^0)
   :=
   \frac{1}{N_0}
   \sum_{i=1}^{N_0} \left|u_\theta\left(t_i^0, x_i^0\right) - u_0\left(x_i^0\right)\right|^2
   \quad \text{ and } \quad
   \phi_\theta^b(X^b)
   :=
   \frac{1}{N_b}
   \sum_{i=1}^{N_b} \left|u_\theta\left(t_i^b, x_i^b\right) - u_b\left(t_i^b, x_i^b\right)\right|^2
    \end{align*}
$$
in a number of points $X^0:=\{(t^0_i,x^0_i)\}_{i=1}^{N_0} \subset \{0\} \times \mathcal{D}$ and $X^b:=\{(t^b_i,x^b_i)\}_{i=1}^{N_b} \subset (0,T] \times \partial \mathcal{D}$, where $u_\theta$ is the neural network approximation of the solution $u\colon[0,T] \times \mathcal{D} \to \mathbb{R}$.

Note that the training data $X$ consists entirely of time-space coordinates.

## Example: Burgers equation

To illustrate the PINN approach we consider the one-dimensional Burgers equation on the spatial domain $\mathcal{D} = [-1,1]$

$$
\begin{aligned}
    \partial_t u + u \, \partial_x u - (0.01/\pi) \, \partial_{xx} u &= 0, \quad &&\quad (t,x) \in (0,1] \times (-1,1),\\
   u(0,x) &= - \sin(\pi \, x),                 \quad &&\quad x \in [-1,1],\\
   u(t,-1) = u(t,1) &= 0,                      \quad &&\quad t \in (0,1].
\end{aligned}
$$

This PDE arises in various disciplines such as traffic flow, fluid mechanics and gas dynamics, and can be derived from the Navier-Stokes equations, see
([Basdevant et al., 1986](https://www.researchgate.net/publication/222935980_Spectral_and_finite_difference_solutions_of_Burgers_equation)).

### 1. Import necessary packages and set problem specific data
This code runs with TensorFlow version `2.3.0`.
The implementation relies mainly on the scientific computing library [NumPy](https://numpy.org/doc/stable/user/whatisnumpy.html) and the machine learning library [TensorFlow](https://www.tensorflow.org/).

All computations were performed on an Intel i7 CPU (8th Gen) with 16 GByte DDR3 RAM (2133 MHz) within a couple of minutes.

```python
# Import TensorFlow and NumPy
import tensorflow as tf
import numpy as np

# Set data type
DTYPE='float32'
tf.keras.backend.set_floatx(DTYPE)

# Set constants
pi = tf.constant(np.pi, dtype=DTYPE)
viscosity = .01/pi

# Define initial condition
def fun_u_0(x):
    return -tf.sin(pi * x)

# Define boundary condition
def fun_u_b(t, x):
    n = x.shape[0]
    return tf.zeros((n,1), dtype=DTYPE)

# Define residual of the PDE
def fun_r(t, x, u, u_t, u_x, u_xx):
    return u_t + u * u_x - viscosity * u_xx
```

### 2. Generate a set of collocation points

We assume that the collocation points $X_r$ as well as the points for the initial time and boundary data $X_0$ and $X_b$ are generated by random sampling from a uniform distribution.
Although uniformly distributed data are sufficient in our experiments, the authors of
([Raissi et al., 2017 (Part I)](https://arxiv.org/abs/1711.10561))
employed a space-filling Latin hypercube sampling strategy ([Stein, 1987](https://www.tandfonline.com/doi/abs/10.1080/00401706.1987.10488205)).
Our numerical experiments indicate that this strategy slightly improves the observed convergence rate, but for simplicity the code examples accompanying this paper employ uniform sampling throughout.
We choose training data of size $N_0 = N_b =50$ and $N_f=10000$.

```python
# Set number of data points
N_0 = 50
N_b = 50
N_r = 10000

# Set boundary
tmin = 0.
tmax = 1.
xmin = -1.
xmax = 1.

# Lower bounds
lb = tf.constant([tmin, xmin], dtype=DTYPE)
# Upper bounds
ub = tf.constant([tmax, xmax], dtype=DTYPE)

# Set random seed for reproducible results
tf.random.set_seed(0)

# Draw uniform sample points for initial boundary data
t_0 = tf.ones((N_0,1), dtype=DTYPE)*lb[0]
x_0 = tf.random.uniform((N_0,1), lb[1], ub[1], dtype=DTYPE)
X_0 = tf.concat([t_0, x_0], axis=1)

# Evaluate intitial condition at x_0
u_0 = fun_u_0(x_0)

# Boundary data
t_b = tf.random.uniform((N_b,1), lb[0], ub[0], dtype=DTYPE)
x_b = lb[1] + (ub[1] - lb[1]) * tf.keras.backend.random_bernoulli((N_b,1), 0.5, dtype=DTYPE)
X_b = tf.concat([t_b, x_b], axis=1)

# Evaluate boundary condition at (t_b,x_b)
u_b = fun_u_b(t_b, x_b)

# Draw uniformly sampled collocation points
t_r = tf.random.uniform((N_r,1), lb[0], ub[0], dtype=DTYPE)
x_r = tf.random.uniform((N_r,1), lb[1], ub[1], dtype=DTYPE)
X_r = tf.concat([t_r, x_r], axis=1)

# Collect boundary and inital data in lists
X_data = [X_0, X_b]
u_data = [u_0, u_b]
```

```python
t_r.shape
```

    TensorShape([10000, 1])

```python
x_r.shape
```

    TensorShape([10000, 1])

```python
plt.plot(x_r)
```

Next, we illustrate the collocation points (red circles) and the positions where the boundary and initial conditions will be enforced (cross marks, color indicates value).

```python
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(9,6))
#plt.scatter(t_0, x_0, c=u_0, marker='X', vmin=-1, vmax=1)
plt.scatter(t_b, x_b, c=u_b, marker='X', vmin=-1, vmax=1)
#plt.scatter(t_r, x_r, c='r', marker='.', alpha=0.1)
plt.xlabel('$t$')
plt.ylabel('$x$')

plt.title('Positions of collocation points and boundary data');
#plt.savefig('Xdata_Burgers.pdf', bbox_inches='tight', dpi=300)
```

### 3. Set up network architecture

In this example, adopted from
([Raissi et al., 2017 (Part I)](https://arxiv.org/abs/1711.10561)), we assume a feedforward neural network of the following structure:
- the input is scaled elementwise to lie in the interval $[-1,1]$,
- followed by 8 fully connected layers each containing 20 neurons and each followed by a hyperbolic tangent activation function,
- one fully connected output layer.

This setting results in a network with $3021$ trainable parameters (first hidden layer: $2 \cdot 20 + 20 = 60$; seven intermediate layers: each $20 \cdot 20 + 20 = 420$; output layer: $20 \cdot 1 + 1 = 21$).

```python
def init_model(num_hidden_layers=8, num_neurons_per_layer=20):
    # Initialize a feedforward neural network
    model = tf.keras.Sequential()

    # Input is two-dimensional (time + one spatial dimension)
    model.add(tf.keras.Input(2))

    # Introduce a scaling layer to map input to [lb, ub]
    scaling_layer = tf.keras.layers.Lambda(
                lambda x: 2.0*(x - lb)/(ub - lb) - 1.0)
    model.add(scaling_layer)

    # Append hidden layers
    for _ in range(num_hidden_layers):
        model.add(tf.keras.layers.Dense(num_neurons_per_layer,
            activation=tf.keras.activations.get('tanh'),
            kernel_initializer='glorot_normal'))

    # Output is one-dimensional
    model.add(tf.keras.layers.Dense(1))

    return model
```

### 4. Define routines to determine loss and gradient

In the following code cell, we define a function which evaluates the residual

$$
\begin{align}
    r_\theta (t,x) := \partial_t u_\theta (t,x) + \mathcal{N}[u_\theta] (t,x).
\end{align}
$$

of the nonlinear PDE in the points $X_r = \{(t^r_i,x^r_i)\}_{i=1}^{N_r}$.
To compute the necessary partial derivatives we use the automatic differentiation capabilities of TensorFlow.

For the Burgers equation, this entails computing $\partial_t u_\theta$, $\partial_x u_\theta$ and $\partial_{xx} u_\theta$.
In TensorFlow, this is done via a `GradientTape`, see also the [documentation](https://www.tensorflow.org/api_docs/python/tf/GradientTape), which keeps track of the `watched` variables, in our case `t` and `x`, in order to compute the derivatives.

```python
def get_r(model, X_r):

    # A tf.GradientTape is used to compute derivatives in TensorFlow
    with tf.GradientTape(persistent=True) as tape:
        # Split t and x to compute partial derivatives
        t, x = X_r[:, 0:1], X_r[:,1:2]

        # Variables t and x are watched during tape
        # to compute derivatives u_t and u_x
        tape.watch(t)
        tape.watch(x)

        # Determine residual
        u = model(tf.stack([t[:,0], x[:,0]], axis=1))

        # Compute gradient u_x within the GradientTape
        # since we need second derivatives
        u_x = tape.gradient(u, x)

    u_t = tape.gradient(u, t)
    u_xx = tape.gradient(u_x, x)

    del tape

    return fun_r(t, x, u, u_t, u_x, u_xx)
```

The next function computes the loss for our model

$$
\begin{align}
    \phi_\theta(X) := \phi_\theta^r(X^r) + \phi_\theta^0(X^0) + \phi_\theta^b(X^b),
\end{align}
$$

as a function of our the training data.
The collocation points are given by `X_r`, the initial and boundary data is contained in `X_data = [X_0, X_b]` and `u_data = [u_0, u_b]`.

```python
def compute_loss(model, X_r, X_data, u_data):

    # Compute phi^r
    r = get_r(model, X_r)
    phi_r = tf.reduce_mean(tf.square(r))

    # Initialize loss
    loss = phi_r

    # Add phi^0 and phi^b to the loss
    for i in range(len(X_data)):
        u_pred = model(X_data[i])
        loss += tf.reduce_mean(tf.square(u_data[i] - u_pred))

    return loss
```

The next function computes the gradient of the loss function $\phi_\theta$ with respect to the unknown variables in the model, also called `trainable_variables` in TensorFlow, i.e. $\nabla_\theta \phi_\theta$.
This is also done via a `GradientTape`, but now it keeps track of the parameters $\theta$ in our model, which can be accessed by `model.trainable_variables`.

```python
def get_grad(model, X_r, X_data, u_data):

    with tf.GradientTape(persistent=True) as tape:
        # This tape is for derivatives with
        # respect to trainable variables
        tape.watch(model.trainable_variables)
        loss = compute_loss(model, X_r, X_data, u_data)

    g = tape.gradient(loss, model.trainable_variables)
    del tape

    return loss, g
```

### 5. Set up optimizer and train model

Next we initialize the model, set the learning rate to the step function

$$
\delta(n) = 0.01 \, \textbf{1}_{\{n < 1000\}} + 0.001 \, \textbf{1}_{\{1000 \le n < 3000\}} + 0.0005 \, \textbf{1}_{\{3000 \le n\}}
$$

which decays in a piecewise constant fashion, and set up a `tf.keras.optimizer` to train the model.

```python
# Initialize model aka u_\theta
model = init_model()

# We choose a piecewise decay of the learning rate, i.e., the
# step size in the gradient descent type algorithm
# the first 1000 steps use a learning rate of 0.01
# from 1000 - 3000: learning rate = 0.001
# from 3000 onwards: learning rate = 0.0005

lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([1000,3000],[1e-2,1e-3,5e-4])

# Choose the optimizer
optim = tf.keras.optimizers.Adam(learning_rate=lr)
```

    WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.

Train the model for $N=5000$ epochs (takes approximately 3 minutes).
Here, we set up a function `train_step()` which performs one training step.

*Note*: The `@tf.function` is a so-called `Decorator` within Python. This particular decorator redefines the function that follows, in our case `train_step`, as a TensorFlow graph which may speed up the training significantly.

```python
from time import time

# Define one training step as a TensorFlow function to increase speed of training
@tf.function
def train_step():
    # Compute current loss and gradient w.r.t. parameters
    loss, grad_theta = get_grad(model, X_r, X_data, u_data)

    # Perform gradient descent step
    optim.apply_gradients(zip(grad_theta, model.trainable_variables))

    return loss

# Number of training epochs
N = 5000
hist = []

# Start timer
t0 = time()

for i in range(N+1):

    loss = train_step()

    # Append current loss to hist
    hist.append(loss.numpy())

    # Output current loss after 50 iterates
    if i%50 == 0:
        print('It {:05d}: loss = {:10.8e}'.format(i,loss))

# Print computation time
print('\nComputation time: {} seconds'.format(time()-t0))
```

    WARNING:tensorflow:Calling GradientTape.gradient on a persistent tape inside its context is significantly less efficient than calling it outside the context (it causes the gradient ops to be recorded on the tape, leading to increased CPU and memory usage). Only call GradientTape.gradient inside the context if you actually want to trace the gradient in order to compute higher order derivatives.
    It 00000: loss = 7.01582313e-01
    It 00050: loss = 2.27828562e-01
    It 00100: loss = 1.40766904e-01
    It 00150: loss = 1.01825498e-01
    It 00200: loss = 9.56485569e-02
    It 00250: loss = 8.21928009e-02
    It 00300: loss = 1.01965994e-01
    It 00350: loss = 1.76442951e-01
    It 00400: loss = 4.73851413e-02
    It 00450: loss = 1.03412114e-01
    It 00500: loss = 8.36287588e-02
    It 00550: loss = 7.31263235e-02
    It 00600: loss = 7.06463158e-02
    It 00650: loss = 6.60299212e-02
    It 00700: loss = 5.40955216e-02
    It 00750: loss = 7.12597817e-02
    It 00800: loss = 2.20131204e-02
    It 00850: loss = 2.52600610e-01
    It 00900: loss = 1.13268271e-01
    It 00950: loss = 9.52311382e-02
    It 01000: loss = 8.63729045e-02
    It 01050: loss = 8.54246914e-02
    It 01100: loss = 8.45917836e-02
    It 01150: loss = 8.37247372e-02
    It 01200: loss = 8.28168094e-02
    It 01250: loss = 8.18559974e-02
    It 01300: loss = 8.08199793e-02
    It 01350: loss = 7.96679482e-02
    It 01400: loss = 7.83230811e-02
    It 01450: loss = 7.66276494e-02
    It 01500: loss = 7.42337555e-02
    It 01550: loss = 7.07529187e-02
    It 01600: loss = 6.73880428e-02
    It 01650: loss = 6.50626719e-02
    It 01700: loss = 6.30353242e-02
    It 01750: loss = 6.09757155e-02
    It 01800: loss = 5.87439612e-02
    It 01850: loss = 5.62031418e-02
    It 01900: loss = 5.32821864e-02
    It 01950: loss = 4.96916100e-02
    It 02000: loss = 4.47017848e-02
    It 02050: loss = 3.89369242e-02
    It 02100: loss = 3.26713584e-02
    It 02150: loss = 2.71941163e-02
    It 02200: loss = 2.35194862e-02
    It 02250: loss = 2.10947841e-02
    It 02300: loss = 1.93291735e-02
    It 02350: loss = 1.78986024e-02
    It 02400: loss = 1.66746061e-02
    It 02450: loss = 1.55441277e-02
    It 02500: loss = 1.44903362e-02
    It 02550: loss = 1.35757318e-02
    It 02600: loss = 1.26500335e-02
    It 02650: loss = 1.18541094e-02
    It 02700: loss = 1.26679018e-02
    It 02750: loss = 1.05052842e-02
    It 02800: loss = 9.89938900e-03
    It 02850: loss = 9.60997026e-03
    It 02900: loss = 8.89403373e-03
    It 02950: loss = 8.40344094e-03
    It 03000: loss = 8.54582805e-03
    It 03050: loss = 7.77806249e-03
    It 03100: loss = 7.57182948e-03
    It 03150: loss = 7.37109734e-03
    It 03200: loss = 7.17304461e-03
    It 03250: loss = 6.97774114e-03
    It 03300: loss = 6.78541791e-03
    It 03350: loss = 6.59627188e-03
    It 03400: loss = 6.41041528e-03
    It 03450: loss = 6.22784998e-03
    It 03500: loss = 6.04829984e-03
    It 03550: loss = 5.87130943e-03
    It 03600: loss = 5.69654070e-03
    It 03650: loss = 5.52431541e-03
    It 03700: loss = 5.35512716e-03
    It 03750: loss = 5.17611066e-03
    It 03800: loss = 4.72522667e-03
    It 03850: loss = 4.42015287e-03
    It 03900: loss = 3.92150506e-03
    It 03950: loss = 3.69463302e-03
    It 04000: loss = 3.53755732e-03
    It 04050: loss = 3.40292393e-03
    It 04100: loss = 3.28305108e-03
    It 04150: loss = 3.17391031e-03
    It 04200: loss = 3.07263248e-03
    It 04250: loss = 2.96917488e-03
    It 04300: loss = 2.51863780e-03
    It 04350: loss = 2.38363189e-03
    It 04400: loss = 2.13802606e-03
    It 04450: loss = 1.79789425e-03
    It 04500: loss = 1.64110155e-03
    It 04550: loss = 1.57481653e-03
    It 04600: loss = 1.52362534e-03
    It 04650: loss = 1.47993152e-03
    It 04700: loss = 1.44081819e-03
    It 04750: loss = 1.40490406e-03
    It 04800: loss = 1.37143361e-03
    It 04850: loss = 1.33994559e-03
    It 04900: loss = 1.31011917e-03
    It 04950: loss = 1.28172617e-03
    It 05000: loss = 1.25459174e-03

    Computation time: 382.5827832221985 seconds

### Plot solution

```python
from mpl_toolkits.mplot3d import Axes3D

# Set up meshgrid
N = 600
tspace = np.linspace(lb[0], ub[0], N + 1)
xspace = np.linspace(lb[1], ub[1], N + 1)
T, X = np.meshgrid(tspace, xspace)
Xgrid = np.vstack([T.flatten(),X.flatten()]).T

# Determine predictions of u(t, x)
upred = model(tf.cast(Xgrid,DTYPE))

# Reshape upred
U = upred.numpy().reshape(N+1,N+1)

# Surface plot of solution u(t,x)
fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(T, X, U, cmap='viridis');
ax.view_init(35,35)
ax.set_xlabel('$t$')
ax.set_ylabel('$x$')
ax.set_zlabel('$u_\\theta(t,x)$')
ax.set_title('Solution of Burgers equation');
#plt.savefig('Burgers_Solution.pdf', bbox_inches='tight', dpi=300);
```

### Plot the evolution of loss

```python
fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)
ax.semilogy(range(len(hist)), hist,'k-')
ax.set_xlabel('$n_{epoch}$')
ax.set_ylabel('$\\phi_{n_{epoch}}$');
```

## Class implementation of PINNs

In this section, we implement PINNs as a class which can be used for further testing. Here, we derive the class `PINN_NeuralNet` from `tf.keras.Model`.

Required arguments are the lower bound `lb` and upper bound `ub`.

```python
# Define model architecture
class PINN_NeuralNet(tf.keras.Model):
    """ Set basic architecture of the PINN model."""

    def __init__(self, lb, ub,
            output_dim=1,
            num_hidden_layers=8,
            num_neurons_per_layer=20,
            activation='tanh',
            kernel_initializer='glorot_normal',
            **kwargs):
        super().__init__(**kwargs)

        self.num_hidden_layers = num_hidden_layers
        self.output_dim = output_dim
        self.lb = lb
        self.ub = ub

        # Define NN architecture
        self.scale = tf.keras.layers.Lambda(
            lambda x: 2.0*(x - lb)/(ub - lb) - 1.0)
        self.hidden = [tf.keras.layers.Dense(num_neurons_per_layer,
                             activation=tf.keras.activations.get(activation),
                             kernel_initializer=kernel_initializer)
                           for _ in range(self.num_hidden_layers)]
        self.out = tf.keras.layers.Dense(output_dim)

    def call(self, X):
        """Forward-pass through neural network."""
        Z = self.scale(X)
        for i in range(self.num_hidden_layers):
            Z = self.hidden[i](Z)
        return self.out(Z)

```

Next, we derive a class `PINNSolver` which can be used as a base class.
It possesses two methods to solve the PDE:
  1. the method `solve_with_TFoptimizer` uses a `TensorFlow` optimizer object as input, e.g., the `AdamOptimizer` above;
  2. the method `solve_with_LBFGS` resembles the LBFGS method proposed in the original paper using an LBFGS method provided by [`SciPy`](https://www.scipy.org/).

```python
import scipy.optimize

class PINNSolver():
    def __init__(self, model, X_r):
        self.model = model

        # Store collocation points
        self.t = X_r[:,0:1]
        self.x = X_r[:,1:2]

        # Initialize history of losses and global iteration counter
        self.hist = []
        self.iter = 0

    def get_r(self):

        with tf.GradientTape(persistent=True) as tape:
            # Watch variables representing t and x during this GradientTape
            tape.watch(self.t)
            tape.watch(self.x)

            # Compute current values u(t,x)
            u = self.model(tf.stack([self.t[:,0], self.x[:,0]], axis=1))

            u_x = tape.gradient(u, self.x)

        u_t = tape.gradient(u, self.t)
        u_xx = tape.gradient(u_x, self.x)

        del tape

        return self.fun_r(self.t, self.x, u, u_t, u_x, u_xx)

    def loss_fn(self, X, u):

        # Compute phi_r
        r = self.get_r()
        phi_r = tf.reduce_mean(tf.square(r))

        # Initialize loss
        loss = phi_r

        # Add phi_0 and phi_b to the loss
        for i in range(len(X)):
            u_pred = self.model(X[i])
            loss += tf.reduce_mean(tf.square(u[i] - u_pred))

        return loss

    def get_grad(self, X, u):
        with tf.GradientTape(persistent=True) as tape:
            # This tape is for derivatives with
            # respect to trainable variables
            tape.watch(self.model.trainable_variables)
            loss = self.loss_fn(X, u)

        g = tape.gradient(loss, self.model.trainable_variables)
        del tape

        return loss, g

    def fun_r(self, t, x, u, u_t, u_x, u_xx):
        """Residual of the PDE"""
        return u_t + u * u_x - viscosity * u_xx

    def solve_with_TFoptimizer(self, optimizer, X, u, N=1001):
        """This method performs a gradient descent type optimization."""

        @tf.function
        def train_step():
            loss, grad_theta = self.get_grad(X, u)

            # Perform gradient descent step
            optimizer.apply_gradients(zip(grad_theta, self.model.trainable_variables))
            return loss

        for i in range(N):

            loss = train_step()

            self.current_loss = loss.numpy()
            self.callback()

    def solve_with_ScipyOptimizer(self, X, u, method='L-BFGS-B', **kwargs):
        """This method provides an interface to solve the learning problem
        using a routine from scipy.optimize.minimize.
        (Tensorflow 1.xx had an interface implemented, which is not longer
        supported in Tensorflow 2.xx.)
        Type conversion is necessary since scipy-routines are written in Fortran
        which requires 64-bit floats instead of 32-bit floats."""

        def get_weight_tensor():
            """Function to return current variables of the model
            as 1d tensor as well as corresponding shapes as lists."""

            weight_list = []
            shape_list = []

            # Loop over all variables, i.e. weight matrices, bias vectors and unknown parameters
            for v in self.model.variables:
                shape_list.append(v.shape)
                weight_list.extend(v.numpy().flatten())

            weight_list = tf.convert_to_tensor(weight_list)
            return weight_list, shape_list

        x0, shape_list = get_weight_tensor()

        def set_weight_tensor(weight_list):
            """Function which sets list of weights
            to variables in the model."""
            idx = 0
            for v in self.model.variables:
                vs = v.shape

                # Weight matrices
                if len(vs) == 2:
                    sw = vs[0]*vs[1]
                    new_val = tf.reshape(weight_list[idx:idx+sw],(vs[0],vs[1]))
                    idx += sw

                # Bias vectors
                elif len(vs) == 1:
                    new_val = weight_list[idx:idx+vs[0]]
                    idx += vs[0]

                # Variables (in case of parameter identification setting)
                elif len(vs) == 0:
                    new_val = weight_list[idx]
                    idx += 1

                # Assign variables (Casting necessary since scipy requires float64 type)
                v.assign(tf.cast(new_val, DTYPE))

        def get_loss_and_grad(w):
            """Function that provides current loss and gradient
            w.r.t the trainable variables as vector. This is mandatory
            for the LBFGS minimizer from scipy."""

            # Update weights in model
            set_weight_tensor(w)
            # Determine value of \phi and gradient w.r.t. \theta at w
            loss, grad = self.get_grad(X, u)

            # Store current loss for callback function
            loss = loss.numpy().astype(np.float64)
            self.current_loss = loss

            # Flatten gradient
            grad_flat = []
            for g in grad:
                grad_flat.extend(g.numpy().flatten())

            # Gradient list to array
            grad_flat = np.array(grad_flat,dtype=np.float64)

            # Return value and gradient of \phi as tuple
            return loss, grad_flat

        return scipy.optimize.minimize(fun=get_loss_and_grad,
                                       x0=x0,
                                       jac=True,
                                       method=method,
                                       callback=self.callback,
                                       **kwargs)

    def callback(self, xr=None):
        if self.iter % 50 == 0:
            print('It {:05d}: loss = {:10.8e}'.format(self.iter,self.current_loss))
        self.hist.append(self.current_loss)
        self.iter+=1

    def plot_solution(self, **kwargs):
        N = 600
        tspace = np.linspace(self.model.lb[0], self.model.ub[0], N+1)
        xspace = np.linspace(self.model.lb[1], self.model.ub[1], N+1)
        T, X = np.meshgrid(tspace, xspace)
        Xgrid = np.vstack([T.flatten(),X.flatten()]).T
        upred = self.model(tf.cast(Xgrid,DTYPE))
        U = upred.numpy().reshape(N+1,N+1)
        fig = plt.figure(figsize=(9,6))
        ax = fig.add_subplot(111, projection='3d')
        ax.plot_surface(T, X, U, cmap='viridis', **kwargs)
        ax.set_xlabel('$t$')
        ax.set_ylabel('$x$')
        ax.set_zlabel('$u_\\theta(t,x)$')
        ax.view_init(35,35)
        return ax

    def plot_loss_history(self, ax=None):
        if not ax:
            fig = plt.figure(figsize=(7,5))
            ax = fig.add_subplot(111)
        ax.semilogy(range(len(self.hist)), self.hist,'k-')
        ax.set_xlabel('$n_{epoch}$')
        ax.set_ylabel('$\\phi^{n_{epoch}}$')
        return ax
```

### Burgers equation with L-BFGS

The following code cell shows how the new classes `PINN_NeuralNet` and `PINNSolver` can be used to solve the Burgers equation, this time using the `SciPy` implementation of L-BFGS (takes around 3 minutes).

```python
# Initialize model
model = PINN_NeuralNet(lb, ub)
model.build(input_shape=(None,2))

# Initilize PINN solver
solver = PINNSolver(model, X_r)

# Decide which optimizer should be used
mode = 'TFoptimizer'
#mode = 'ScipyOptimizer'

# Start timer
t0 = time()

if mode == 'TFoptimizer':
    # Choose optimizer
    lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([1000,3000],[1e-2,1e-3,5e-4])
    optim = tf.keras.optimizers.Adam(learning_rate=lr)
    solver.solve_with_TFoptimizer(optim, X_data, u_data, N=4001)

elif mode == 'ScipyOptimizer':
    solver.solve_with_ScipyOptimizer(X_data, u_data,
                            method='L-BFGS-B',
                            options={'maxiter': 50000,
                                     'maxfun': 50000,
                                     'maxcor': 50,
                                     'maxls': 50,
                                     'ftol': 1.0*np.finfo(float).eps})

# Print computation time
print('\nComputation time: {} seconds'.format(time()-t0))
```

    WARNING:tensorflow:Calling GradientTape.gradient on a persistent tape inside its context is significantly less efficient than calling it outside the context (it causes the gradient ops to be recorded on the tape, leading to increased CPU and memory usage). Only call GradientTape.gradient inside the context if you actually want to trace the gradient in order to compute higher order derivatives.
    It 00000: loss = 7.47430623e-01
    It 00050: loss = 1.01619229e-01
    It 00100: loss = 6.35943040e-02
    It 00150: loss = 3.25269625e-02
    It 00200: loss = 1.50044905e-02
    It 00250: loss = 6.29830034e-03
    It 00300: loss = 3.50301410e-03
    It 00350: loss = 2.31543742e-03
    It 00400: loss = 1.50433823e-03
    It 00450: loss = 1.15917006e-03
    It 00500: loss = 8.52243335e-04
    It 00550: loss = 6.76931406e-04
    It 00600: loss = 5.38023887e-04
    It 00650: loss = 4.38482384e-04
    It 00700: loss = 3.78300232e-04
    It 00750: loss = 3.11853073e-04
    It 00800: loss = 2.73802900e-04
    It 00850: loss = 2.42481561e-04
    It 00900: loss = 2.08921920e-04
    It 00950: loss = 1.90496008e-04
    It 01000: loss = 1.71222870e-04
    It 01050: loss = 1.53139626e-04
    It 01100: loss = 1.33037611e-04
    It 01150: loss = 1.18380267e-04
    It 01200: loss = 1.04971958e-04
    It 01250: loss = 9.33278861e-05
    It 01300: loss = 8.51595032e-05
    It 01350: loss = 7.84658405e-05
    It 01400: loss = 7.28691230e-05
    It 01450: loss = 6.92488684e-05
    It 01500: loss = 6.52952585e-05
    It 01550: loss = 6.07191650e-05
    It 01600: loss = 5.65865848e-05
    It 01650: loss = 5.33541388e-05
    It 01700: loss = 5.02513103e-05
    It 01750: loss = 4.72823231e-05
    It 01800: loss = 4.42495875e-05
    It 01850: loss = 4.22273333e-05
    It 01900: loss = 4.03602608e-05
    It 01950: loss = 3.85904059e-05
    It 02000: loss = 3.67677567e-05
    It 02050: loss = 3.50074588e-05
    It 02100: loss = 3.34919496e-05
    It 02150: loss = 3.20918116e-05
    It 02200: loss = 3.03973266e-05
    It 02250: loss = 2.88925130e-05
    It 02300: loss = 2.77497475e-05
    It 02350: loss = 2.64277405e-05
    It 02400: loss = 2.53934286e-05
    It 02450: loss = 2.43336144e-05
    It 02500: loss = 2.32927632e-05
    It 02550: loss = 2.22925682e-05
    It 02600: loss = 2.15615419e-05
    It 02650: loss = 2.05937722e-05
    It 02700: loss = 1.96819710e-05
    It 02750: loss = 1.88990452e-05
    It 02800: loss = 1.81996675e-05
    It 02850: loss = 1.75351397e-05
    It 02900: loss = 1.70338080e-05
    It 02950: loss = 1.65064066e-05
    It 03000: loss = 1.57791910e-05
    It 03050: loss = 1.49651023e-05
    It 03100: loss = 1.44398709e-05
    It 03150: loss = 1.39299464e-05
    It 03200: loss = 1.34347547e-05
    It 03250: loss = 1.30736789e-05
    It 03300: loss = 1.27100866e-05
    It 03350: loss = 1.23950567e-05
    It 03400: loss = 1.19544120e-05
    It 03450: loss = 1.14893719e-05
    It 03500: loss = 1.09732200e-05
    It 03550: loss = 1.05684785e-05
    It 03600: loss = 1.03753928e-05
    It 03650: loss = 1.00760144e-05
    It 03700: loss = 9.84483904e-06
    It 03750: loss = 9.60725993e-06
    It 03800: loss = 9.30466376e-06
    It 03850: loss = 9.13700387e-06
    It 03900: loss = 8.98185863e-06

    Computation time: 1036.4056777954102 seconds

```python
# Initialize model
model = PINN_NeuralNet(lb, ub)
model.build(input_shape=(None,2))

# Initilize PINN solver
solver = PINNSolver(model, X_r)

# Decide which optimizer should be used
mode = 'TFoptimizer'
#mode = 'ScipyOptimizer'

# Start timer
t0 = time()

if mode == 'TFoptimizer':
    # Choose optimizer
    lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([1000,3000],[1e-2,1e-3,5e-4])
    optim = tf.keras.optimizers.Adam(learning_rate=lr)
    solver.solve_with_TFoptimizer(optim, X_data, u_data, N=4001)

elif mode == 'ScipyOptimizer':
    solver.solve_with_ScipyOptimizer(X_data, u_data,
                            method='L-BFGS-B',
                            options={'maxiter': 50000,
                                     'maxfun': 50000,
                                     'maxcor': 50,
                                     'maxls': 50,
                                     'ftol': 1.0*np.finfo(float).eps})
    #tfp.optimizer.bfgs_minimize(
    #value_and_gradients_function, initial_position, tolerance=1e-08, x_tolerance=0,
    #f_relative_tolerance=0, initial_inverse_hessian_estimate=None,
    #max_iterations=50, parallel_iterations=1, stopping_condition=None,
    #validate_args=True, max_line_search_iterations=50, name=None)

# Print computation time
print('\nComputation time: {} seconds'.format(time()-t0))
```

    It 00000: loss = 5.71737766e-01
    It 00050: loss = 1.36384860e-01
    It 00100: loss = 9.68331248e-02
    It 00150: loss = 8.57249051e-02
    It 00200: loss = 8.33153948e-02
    It 00250: loss = 8.21514726e-02
    It 00300: loss = 7.17309713e-02
    It 00350: loss = 1.07278109e-01
    It 00400: loss = 1.01103321e-01
    It 00450: loss = 1.02324359e-01
    It 00500: loss = 2.33296603e-02
    It 00550: loss = 4.61507402e-02
    It 00600: loss = 9.27970111e-02
    It 00650: loss = 1.58080440e-02
    It 00700: loss = 5.42153418e-01
    It 00750: loss = 2.01510355e-01
    It 00800: loss = 1.27206072e-01
    It 00850: loss = 9.58015695e-02
    It 00900: loss = 6.10700846e-02
    It 00950: loss = 7.21340254e-02
    It 01000: loss = 1.56814873e-01
    It 01050: loss = 8.61870795e-02
    It 01100: loss = 7.78558552e-02
    It 01150: loss = 6.90250620e-02
    It 01200: loss = 5.76238781e-02
    It 01250: loss = 4.64916043e-02
    It 01300: loss = 3.70992236e-02
    It 01350: loss = 2.93608196e-02
    It 01400: loss = 2.30197441e-02
    It 01450: loss = 1.88027676e-02
    It 01500: loss = 1.60066914e-02
    It 01550: loss = 1.37239695e-02
    It 01600: loss = 1.17948446e-02
    It 01650: loss = 1.02468366e-02
    It 01700: loss = 9.06473584e-03
    It 01750: loss = 8.17063171e-03
    It 01800: loss = 7.46778212e-03
    It 01850: loss = 6.87789312e-03
    It 01900: loss = 6.33415952e-03
    It 01950: loss = 5.87853743e-03
    It 02000: loss = 5.49683115e-03
    It 02050: loss = 5.16445888e-03
    It 02100: loss = 4.87180473e-03
    It 02150: loss = 4.61262278e-03
    It 02200: loss = 4.38214606e-03
    It 02250: loss = 4.17643040e-03
    It 02300: loss = 3.99200339e-03
    It 02350: loss = 3.82575952e-03
    It 02400: loss = 3.67475976e-03
    It 02450: loss = 3.53593379e-03
    It 02500: loss = 3.39972042e-03
    It 02550: loss = 3.19943600e-03
    It 02600: loss = 3.08870221e-03
    It 02650: loss = 2.99258274e-03
    It 02700: loss = 2.90414458e-03
    It 02750: loss = 2.82111834e-03
    It 02800: loss = 2.74231471e-03
    It 02850: loss = 2.67916918e-03
    It 02900: loss = 2.60274624e-03
    It 02950: loss = 2.53411010e-03
    It 03000: loss = 2.46724114e-03
    It 03050: loss = 2.43335264e-03
    It 03100: loss = 2.39986624e-03
    It 03150: loss = 2.36602314e-03
    It 03200: loss = 2.33172486e-03
    It 03250: loss = 2.29680026e-03
    It 03300: loss = 2.26098252e-03
    It 03350: loss = 2.22362764e-03
    It 03400: loss = 2.18219846e-03
    It 03450: loss = 1.98601582e-03
    It 03500: loss = 1.76724407e-03
    It 03550: loss = 1.57118111e-03
    It 03600: loss = 1.41963828e-03
    It 03650: loss = 1.34334981e-03
    It 03700: loss = 1.29525317e-03
    It 03750: loss = 1.25310849e-03
    It 03800: loss = 1.20363152e-03
    It 03850: loss = 1.14060810e-03
    It 03900: loss = 1.09924306e-03
    It 03950: loss = 1.06592360e-03
    It 04000: loss = 1.03628205e-03

    Computation time: 341.7781763076782 seconds

Plot solution and loss history.

```python
solver.plot_solution();
solver.plot_loss_history();
```

### Solution of a time-dependent Eikonal equation

As a second example we consider the one-dimensional Eikonal equation backward in time on the domain $\mathcal{D}=[-1,1]$

$$
\begin{align}
   -\partial_t u(t,x) + |\nabla u|(t,x) &= 1,
   		\quad & &(t,x) \in [0,T) \times [-1,1],\\
   u(T,x) &= 0, \quad & &x \in [-1,1],\\
   u(t,-1) = u(t, 1) &= 0, \quad & & t \in [0,T).
\end{align}
$$

Note that the partial differential equation can be equally written as a Hamilton-Jacobi-Bellman equation, viz

$$
    -\partial_t u(t,x) + \sup_{|c| \le 1} \{c \, \nabla u(t,x)\} = 1 \quad (t,x) \in [0,T) \times [-1,1],
$$

which characterizes the solution of an optimal control problem seeking to minimize the distance from a point $(t,x)$ to the boundary $[0,T] \times \partial \mathcal{D} \cup \{T\} \times \mathcal{D}$.
As is easily verified, the solution is given by $u(t,x) = \min\{ 1 - t, 1 - |x| \}$.
The fact that the Eikonal equation runs backward in time is in accordance with its interpretation as the optimality condition of a control problem.
Note that this is equation can be transformed into a forward evolution problem by the change of variables $\hat t = T - t$.

#### Problem specific definitions

```python
N_0 = 50
N_b = 50
N_r = 10000

# Specify boundaries
lb = tf.constant([0., -1.], dtype=DTYPE)
ub = tf.constant([1., 1.], dtype=DTYPE)

def Eikonal_u_0(x):
    n = x.shape[0]
    return tf.zeros((n,1), dtype=DTYPE)

def Eikonal_u_b(t, x):
    n = x.shape[0]
    return tf.zeros((n,1), dtype=DTYPE)
```

#### Generate data
This code snippet is almost identical to the code from above.
We choose $N_b = 50$ and $N_0 = 50$ uniformly distributed initial value and boundary points and sample $N_r = 10000$ collocation points uniformly within the domain boundaries.
We derive a new solver with `PINNSolver` as base class.

```python
tf.random.set_seed(0)

# Final time data
t_0 = tf.ones((N_0,1), dtype=DTYPE) * ub[0]
x_0 = tf.random.uniform((N_0,1), lb[1], ub[1], dtype=DTYPE)
X_0 = tf.concat([t_0, x_0], axis=1)
u_0 = Eikonal_u_0(x_0)

# Boundary data
t_b = tf.random.uniform((N_b,1), lb[0], ub[0], dtype=DTYPE)
x_b = lb[1] + (ub[1] - lb[1]) * tf.keras.backend.random_bernoulli((N_b,1), 0.5, dtype=DTYPE)
X_b = tf.concat([t_b, x_b], axis=1)
u_b = Eikonal_u_b(t_b, x_b)

# Collocation points
t_r = tf.random.uniform((N_r,1), lb[0], ub[0], dtype=DTYPE)
x_r = tf.random.uniform((N_r,1), lb[1], ub[1], dtype=DTYPE)
X_r = tf.concat([t_r, x_r], axis=1)

# Collect boundary and inital data in lists
X_data = [X_0,X_b]
u_data = [u_0,u_b]
```

#### Derive Eikonal solver class
Now, we derive a solver for the Eikonal equation from the `PINNSolver` class. Since the Eikonal equation does not depend on second-order derivatives, we implement a new method `get_r` which avoids the computation of second derivatives.

```python
class EikonalPINNSolver(PINNSolver):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def fun_r(self, t, x, u, u_t, u_x, u_xx):
        """Residual of the PDE"""
        return -u_t + tf.abs(u_x) - 1.

    def get_r(self):
        """We update get_r since the Eikonal equation is a first-order equation.
        Therefore, it is not necessary to compute second derivatives."""

        with tf.GradientTape(persistent=True) as tape:
            # Watch variables representing t and x during this GradientTape
            tape.watch(self.t)
            tape.watch(self.x)

            # Compute current values u(t,x)
            u = self.model(tf.stack([self.t[:,0], self.x[:,0]], axis=1))

        u_x = tape.gradient(u, self.x)

        u_t = tape.gradient(u, self.t)

        del tape

        return self.fun_r(self.t, self.x, u, u_t, u_x, None)
```

#### Setting up the neural network architecture

The neural network model chosen for this particular problem can be simpler.
We decided to use only two hidden layers with 20 neurons in each, resulting in $501$ unknown parameters (first hidden layer: $2 \cdot 20 + 20 = 60$; one intermediate layer: $20 \cdot 20 + 20 = 420$; output layer: $20 \cdot 1 + 1 = 21$).
To account for the lack of smoothness of the solution, we choose a non-differentiable activation function, although the hyperbolic tangent function seems to be able to approximate the kinks in the solution sufficiently well.
Here, we decided to use the \emph{leaky rectified linear unit (leaky ReLU)} activation function

$$
\begin{align*}
    \sigma(z) = \begin{cases}
        z &\text{ if } z \ge 0,\\
        0.1 \, z &\text{ otherwise},
    \end{cases}
\end{align*}
$$

which displays a non-vanishing gradient when the unit is not active, i.e., when $z < 0$.

```python
# Initialize model
model = PINN_NeuralNet(lb, ub, num_hidden_layers=2,
                       activation=tf.keras.layers.LeakyReLU(alpha=0.1),
                                       kernel_initializer='he_normal')
model.build(input_shape=(None,2))

# Initilize PINN solver
eikonalSolver = EikonalPINNSolver(model, X_r)
```

Start training (take approximately 40 seconds).

```python
# Choose step sizes aka learning rate
lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([3000,7000],[1e-1,1e-2,1e-3])

# Solve with Adam optimizer
optim = tf.keras.optimizers.Adam(learning_rate=lr)

# Start timer
t0 = time()
eikonalSolver.solve_with_TFoptimizer(optim, X_data, u_data, N=10001)

# Print computation time
print('\nComputation time: {} seconds'.format(time()-t0))
```

Plot the results.

```python
eikonalSolver.plot_solution();
#plt.savefig('Eikonal_Solution.pdf', bbox_inches='tight', dpi=300)
eikonalSolver.plot_loss_history();
```

## Parameter identification setting

In this section, we want to demonstrate how the PINN approach could be used to solve partial differential equations with unknown parameters $\lambda$.
To be more precise, we consider the parabolic Eikonal equation

$$
\begin{aligned}
- \partial_t u + \sup_{|c|\le1} c \cdot \nabla u = -\partial_t u + |\nabla u| &= \lambda^{-1}\\
u(T,x) &= 0\\
u(t,-1) = u(t,1) &= 0
\end{aligned}
$$

with unknown parameter $\lambda$.
The explicit solution is $u^*(t,x) = \lambda^{-1} \min\{1-t, 1-|x|\}$.

```python
lambd_star = 3.

def u_expl(t, x, lambd_star):
    """Explicit solution of the parametric Eikonal equation."""
    y = 1./lambd_star
    return y * tf.math.minimum(1-t, 1-tf.abs(x))
```

Next, we draw $N_d = 500$ uniformly distributed measurements of the exact solution.

```python
N_d = 500
noise = 0.0

# Draw points with measurements randomly
t_d = tf.random.uniform((N_d,1), lb[0], ub[0], dtype=DTYPE)
x_d = tf.random.uniform((N_d,1), lb[1], ub[1], dtype=DTYPE)
X_d = tf.concat([t_d, x_d], axis=1)
u_d = u_expl(t_d, x_d, lambd_star)
u_d += noise * tf.random.normal(u_d.shape, dtype=DTYPE)

# Copy original data
X_param = X_data
u_param = u_data
```

Since both the boundary and initial time data are of Dirichlet type, we may handle the measured data exactly like $X_0$ and $X_b$.
Thus, we can simply append $X_d$ and $u_d$ to `Xdata` and `udata`.

Note that the approach illustrated here is slightly different from the one introduced in ([Raissi et al., 2017 (Part II)](https://arxiv.org/abs/1711.10566)) which takes only measurement data into account.

```python
X_param.append(X_d)
u_param.append(u_d)
```

Next, we derive a new network class which takes the additional parameter $\lambda$ into account.
Note that this parameter has to be part of the model in order to be learnt during training.

```python
class PINNIdentificationNet(PINN_NeuralNet):
    def __init__(self, *args, **kwargs):

        # Call init of base class
        super().__init__(*args,**kwargs)

        # Initialize variable for lambda
        self.lambd = tf.Variable(1.0, trainable=True, dtype=DTYPE)
        self.lambd_list = []
```

Now, we derive a new solver class which only updates the evaluation of the residual `fun_r` which now incorporates the $\lambda$-dependency.
In addition, we modify the `callback` function to store the iterates of $\lambda$ in a list `lambd_list` as well.

```python
class EikonalPINNIdentification(EikonalPINNSolver):

    def fun_r(self, t, x, u, u_t, u_x, u_xx):
        """Residual of the PDE"""
        return -u_t + tf.abs(u_x) - 1./self.model.lambd

    def callback(self, xr=None):
        lambd = self.model.lambd.numpy()
        self.model.lambd_list.append(lambd)

        if self.iter % 50 == 0:
            print('It {:05d}: loss = {:10.8e} lambda = {:10.8e}'.format(self.iter, self.current_loss, lambd))

        self.hist.append(self.current_loss)
        self.iter += 1

    def plot_loss_and_param(self, axs=None):
        if axs:
            ax1, ax2 = axs
            self.plot_loss_history(ax1)
        else:
            ax1 = self.plot_loss_history()
            ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis

        color = 'tab:blue'
        ax2.tick_params(axis='y', labelcolor=color)
        ax2.plot(range(len(self.hist)), self.model.lambd_list,'-',color=color)
        ax2.set_ylabel('$\\lambda^{n_{epoch}}$', color=color)
        return (ax1,ax2)
```

Finally, we set up the model consisting of only two hidden layers employing the Leaky ReLU function with slope parameter $\alpha = 0.1$, i.e.,

$$
\sigma(z) = \begin{cases} x & \text{ if } x \ge 0\\ \alpha \, x & \text{ otherwise.} \end{cases}
$$

The training for $n_{epochs} = 10000$ epochs takes around 45 seconds.

```python
# Initialize model
model = PINNIdentificationNet(lb, ub, num_hidden_layers=2,
                                        activation=tf.keras.layers.LeakyReLU(alpha=0.1),
                                        kernel_initializer='he_normal')

model.build(input_shape=(None,2))

# Initilize solver
eikonalIdentification = EikonalPINNIdentification(model, X_r)

# Choose step sizes aka learning rate
lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([3000,7000],[1e-1,1e-2,1e-3])

# Solve with Adam optimizer
optim = tf.keras.optimizers.Adam(learning_rate=lr)

# Start timer
t0 = time()
eikonalIdentification.solve_with_TFoptimizer(optim, X_param, u_param, N=10001)

# Print computation time
print('\nComputation time: {} seconds'.format(time()-t0))
```

Plot solution $u_\theta(t,x)$ and evolution of loss values $\phi^{n_\text{epoch}}$ and estimated parameters $\lambda^{n_\text{epoch}}$..

```python
ax = eikonalIdentification.plot_solution()
#plt.savefig('Eikonal_PI_Solution.pdf', bbox_inches='tight', dpi=300)
axs = eikonalIdentification.plot_loss_and_param()
#plt.savefig('Eikonal_PI_LossEvolution.pdf', bbox_inches='tight', dpi=300)
```

Finally, we compute the relative error of the identified parameter $\lambda$.

```python
lambd_rel_error = np.abs((eikonalIdentification.model.lambd.numpy()-lambd_star)/lambd_star)
print('Relative error of lambda ', lambd_rel_error)
```

The next code cell performs the previous training 5 times in order to give a more reliable picture of the convergence since the weight matrices are initialized randomly at each run (takes about 4 minutes).

```python
lambd_hist = []
loss_hist = []
for i in range(5):
    print('{:s}\nStart of iteration {:d}\n{:s}'.format(50*'-',i,50*'-'))
    # Initialize model
    model = PINNIdentificationNet(lb, ub, num_hidden_layers=2,
                                        activation=tf.keras.layers.LeakyReLU(alpha=0.1),
                                        kernel_initializer='he_normal')

    model.build(input_shape=(None,2))

    # Initilize solver
    eikonalIdentification = EikonalPINNIdentification(model, X_r)

    # Choose step sizes aka learning rate
    lr = tf.keras.optimizers.schedules.PiecewiseConstantDecay([3000,7000],[1e-1,1e-2,1e-3])
    N=10001

    # Solve with Adam optimizer
    optim = tf.keras.optimizers.Adam(learning_rate=lr)

    eikonalIdentification.solve_with_TFoptimizer(optim, X_param, u_param, N=N)

    # Store evolution of lambdas
    lambd_hist.append(model.lambd_list)

    # Store evolution of losses
    loss_hist.append(eikonalIdentification.hist)
```

Next, we generate a table printing the mean and standard deviations of the identified parameter $\lambda$ obtained for the previous runs.

```python
print('  i     Mean of lambda   Std. of lambda')
for i in [1000,2000,3000,4000,5000,6000,7000,8000,9000,10000]:
    xi = np.array([ x[i] for x in lambd_hist])
    print('{:05d}     {:6.4e}       {:6.4e}'.format(i, xi.mean(), xi.std()))
```

Next, we plot the five evolutions of $\lambda$ (dark gray), its mean (solid blue) and one standard deviation (shaded blue) together with the true value of $\lambda$ (dashed blue).

```python
fig = plt.figure()
ax = fig.add_subplot(111)
color = 'tab:blue'

Lambd = np.stack(lambd_hist)
lmean = Lambd.mean(axis=0)
lstd = Lambd.std(axis=0)
Lambd_RelError = np.abs((Lambd-lambd_star)/lambd_star)
lrange=range(len(lmean))

for i in range(len(lambd_hist)):
    ax.plot(lrange, lambd_hist[i],'-',color='black', alpha=0.5)
ax.plot(lrange, lmean,'-',color=color)
ax.plot(lrange, lambd_star*np.ones_like(lmean),'--',color=color)
ax.fill_between(lrange,lmean-lstd,lmean+lstd, alpha=0.2)
ax.set_ylabel('$\\lambda^{n_{epoch}}$')
ax.set_xlabel('$n_{epoch}$')
ax.set_ylim([2.8,3.2])
#plt.savefig('Eikonal_PI_Evolution.pdf', bbox_inches='tight', dpi=300)
```

Finally, we plot the mean relative error of $\lambda$.

```python
fig = plt.figure()
ax = fig.add_subplot(111)
ax.semilogy(lrange,Lambd_RelError.mean(axis=0))
ax.fill_between(lrange,Lambd_RelError.mean(axis=0)-Lambd_RelError.std(axis=0),
                Lambd_RelError.mean(axis=0)+Lambd_RelError.std(axis=0), alpha=0.2)
ax.set_xlabel('$n_{epoch}$')
ax.set_ylabel('$e_{\\lambda}^{rel}$')
ax.set_title('Mean relative error of $\\lambda$');
```


# 3.5 Methods Using Derived Input Directions

In many situations, we have a large number of inputs, ofter very correlated. This methods in this section produce a small number of linear combinations **Z** of the original inputs **X**.

### 3.5.1 Principal Components Regression

Princinal component regression (PCR) forms the derived input columns $z_m=\mathbf{X}v_m$ and then regresses $\mathbf{y} \text{ on } \mathbf{z}_1,...,\mathbf{z}_M$ for some M <= p. Since the **Z** are orthogonal (3.61):
$$
\hat{\mathbf{y}}_{(M)}^{pcr} = \overline{y}\mathbf{1} + \sum_{m=1}^M \hat{\theta}_m\mathbf{z}_m
$$

where $\hat{\theta}_m = \langle \mathbf{z}_m, \mathbf{y} \rangle / \langle \mathbf{z}_m, \mathbf{z}_m\rangle$. Since the $\mathbf{z}_m$ are linear combinations of the original $\mathbf{x}_j$, we can express in terms of coefficients of the $\mathbf{x}_j$ (3.62):

$$
\hat{\beta}^{pcr}(M) = \sum_{m=1}^M \hat{\theta}_m v_m
$$

**TODO**: proof

- we first standardize the inputs.

- Note that if M = p, we would get least squares estimates, since **Z**=**UD** span the column space of **X**.

- PCR is very similar to ridge regression.

# 3.5.2 Partial Least Squares

- Partial Least Squares (PLS) begins by computing $\hat{\varphi}_{1j} = \langle \mathbf{x}_j, \mathbf{y} \rangle$ for each j.

- Then we construct the derived input $\mathbf{z}_1=\sum_j \hat{\varphi}_{1j} \mathbf{x}_j$, which the 1st partial least squares direction.

- The outcome $\mathbf{y}$ is regressed on $\mathbf{z}_1$ giving coefficient $\hat{\theta}_1$

- And then we orthogonalize $\mathbf{x}_1,..., \mathbf{x}_p$ w.r.t $\mathbf{z}_1$.

- We continue this process until M <= p directions have been obtained.

Note: PLS produces orthogonal inputs(or directions) $\mathbf{z}_1,...,\mathbf{z}_M$ and if M = p, we would get least squares estimates.

```python
# Algorithm 3.3 Partial Least Squares.
import numpy as np
import pandas as pd
from scipy import stats

df = pd.read_csv('../data/prostate/prostate.data', delimiter='\t', index_col=0)
mask_train = df.pop('train')
df_y = df.pop('lpsa')

train_x = df[mask_train == 'T']
train_y = df_y[mask_train == 'T']

train_x_centered = train_x - train_x.mean(axis = 0)
train_x_centered /= np.linalg.norm(train_x_centered, axis=0)
train_y_centered = train_y - train_y.mean()
```

```python
def PLS(X, y):
    X = X.copy()
    y = y.copy()

    n, p = X.shape
    y_fit = y.mean() * np.ones(n)
    Z = np.zeros_like(X)
    theta = np.zeros(p)

    for m in range(p):
        for j in range(p):
            phi = np.dot(X[:, j], y)
            Z[:, m] += phi * X[:, j]

        theta[m] = np.dot(Z[:, m], y) / np.dot(Z[:, m], Z[:, m])
        y_fit += theta[m] * Z[:, m]

        for j in range(p):
            X[:, j] -= np.dot(Z[:, m], X[:, j]) / np.dot(Z[:, m], Z[:, m]) * Z[:, m]
    return y_fit
```

```python
y_fit = PLS(train_x_centered.values, train_y_centered.values)
train_error = np.mean((y_fit - train_y_centered) ** 2)
print('Train error: ', train_error)
```

    Train error:  0.43919976805833433

It can be shown that PLS seeks directions that have high variance and have high correlation with the response.
The mth PLS direction $\hat{\varphi}_m$ solves:

$$
\begin{equation}
max_\alpha Corr^2(\mathbf{y}, \mathbf{X}\alpha)Var(\mathbf{X}\alpha)\\
\text{ subject to } \| \alpha \| = 1, \alpha^T\mathbf{S}v_l = 0, l = 1, ..., m - 1
\end{equation}
$$


# Лаба 2

# Решение краевых задач. Методы коллокаций, наименьших квадратов и Гелеркина

## Задание

Получить численное решение кревой задачи

$$y''+(1 + x^2)y=-1, -1 \le x \le 1$$

$$y'' + ( \dfrac {1} {sin(7)} + \dfrac {x^2} {tg(7)})y = \dfrac {-1} {sin(7)}, -1 \le x \le 1$$

## Краевые условия

$$y(-1)=0$$

$$y(1) = 0$$

## Необходимые импорты

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plot
from time import time
from math import sin, tan
```

## Решение
Рассматриваем дифференциальные уравнения второго порядка

$$y''(x) + p(x)y'(x) + q(x)y(x) = f(x)$$

В нашем случае имеем

$$p(x)=0, q(x)=1 + x^2, f(x)=-1, -1 \leq x \leq 1$$

$$p(x)=0, q(x)=\dfrac {1} {sin(7)} + \dfrac {x^2} {tg(7)}, f(x)=\dfrac {-1} {sin(7)}, -1 \leq x \leq 1$$

```python
k = 7
a = -1
b = 1

i = sp.Symbol('i')
x = sp.Symbol('x')

def q1():
    return 1 + x**2

def q2():
    return 1 / sin(k) + x**2 / tan(k)

def f1():
    return -1

def f2():
    return -1 / sin(k)

def basis():
    return x**i - x**(2 + i)

def basis_second_diff(x, i):
    if i == 0:
        return -2
    elif i == 1:
        return -6 * x
    else:
        return i * (i - 1) * x**(i - 2) - (2 + i) * (i + 1) * x**i
```

```python
def show_solution(solution, amount_of_basis_func):
    phi = sp.lambdify(i, basis(), "numpy")
    return sum([solution[i] * phi(i) for i in range(amount_of_basis_func)])

def show_solution_colocation(solution, amount_of_basis_func, amount_of_points):
    phi = sp.lambdify(i, basis(), "numpy")

    result = sum(a * poly for (a, poly) in zip(solution, [phi(j) for j in range(amount_of_basis_func)]))
#     for k in range(amount_of_basis_func):
#         for j in range(amount_of_points):
#             result += solution[k] * phi(j)
    return result
```

```python
implementation = "numpy"
phi = sp.lambdify([x, i], basis(), implementation)
q1_l = sp.lambdify(x, q1(), implementation)
q2_l = sp.lambdify(x, q2(), implementation)
```

### Метод коллокаций

```python
def colocation_method(phi, q, f, x, n):
    matrix = np.array([
        [
            basis_second_diff(col_point, j) + q(col_point) * phi(col_point, j) for j in range(n)
        ]
        for col_point in x
    ])
    vector = [f()] * n
    solution = np.linalg.solve(matrix, vector)
    return solution
```

#### Задание 1

```python
amount_of_basis_func = 5
points = np.linspace(a, b, amount_of_basis_func)
```

```python
# %%timeit
# colocation_method(phi, q1_l, f1, points, amount_of_points)
```

```python
result = colocation_method(phi, q1_l, f1, points, amount_of_basis_func)
new_points = np.linspace(a, b, 100)
beautiful_solution = show_solution_colocation(result, amount_of_basis_func, len(new_points))
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in new_points]
plot.plot(new_points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 0.932695592757089$

#### Задание 2

```python
amount_of_points = 100
points = np.linspace(a, b, amount_of_points)
```

```python
%%timeit
colocation_method(phi, q2_l, f2, points, amount_of_points)
```

    57 ms ± 6.7 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
result = colocation_method(phi, q2_l, f2, points, amount_of_points)
beautiful_solution = show_solution(result, amount_of_points)
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in points]
plot.plot(points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 2.40199204232917$

### Метод наименьших квадратов

```python
def least_square_method(phi, q, f, x, n):
    a_xj = [[basis_second_diff(j, i) + q(j) * phi(j, i) for j in x] for i in range(n)]
    matrix = []
    for a_i in a_xj:
        dsum_ai = []
        for a_k in a_xj:
            dsum_ai.append(sum(i * k for i, k in zip(a_i, a_k)))
        matrix.append(dsum_ai)
    vector = [f() * sum(a_i) for a_i in a_xj]
    solution = np.linalg.solve(matrix, vector)
    return solution
```

#### Задание 1

```python
amount_of_points = 300
amount_of_basis_func = 100
points = np.linspace(-1, 1, amount_of_points)
```

```python
%%timeit
least_square_method(phi, q1_l, f1, points, amount_of_basis_func)
```

    1.17 s ± 10.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
result = least_square_method(phi, q1_l, f1, points, amount_of_basis_func)
beautiful_solution = show_solution(result, amount_of_basis_func)
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in points]
plot.plot(points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 0.932053718486132$

#### Задание 2

```python
amount_of_points = 300
amount_of_basis_func = 100
points = np.linspace(-1, 1, amount_of_points)
```

```python
%%timeit
least_square_method(phi, q2_l, f2, points, amount_of_basis_func)
```

    1.2 s ± 14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
result = least_square_method(phi, q2_l, f2, points, amount_of_basis_func)
beautiful_solution = show_solution(result, amount_of_basis_func)
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in points]
plot.plot(points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 2.40199204287984$

### Метод Галеркина

```python
def galerkin_method(phi, q, f, n):
    x_amount = n * n
    dx = (b - a) / x_amount
    x = np.linspace(a, b, x_amount)
    basis_second_diffs = np.array([
        basis_second_diff(x, i) for i in range(n)
    ])
    qs = q(x)
    phis = np.array([
        phi(x, i) for i in range(n)
    ])
    matrix = np.array([
        np.array([np.trapz(phis[j] * (basis_second_diffs[i] + qs * phis[i]), dx=dx) for i in range(n)])
        for j in range(n)
    ])
    vector = np.array([np.trapz(f() * phis[i], dx=dx) for i in range(n)])
    solution = np.linalg.solve(matrix, vector)
    return solution
```

#### Задание 1

```python
amount_of_basis_func = 100
```

```python
%%timeit
galerkin_method(phi, q1_l, f1, amount_of_basis_func)
```

    1.15 s ± 36.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
result = galerkin_method(phi, q1_l, f1, amount_of_basis_func)
beautiful_solution = show_solution(result, amount_of_basis_func)
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in points]
plot.plot(points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 0.932053718342423$

#### Задание 2

```python
amount_of_basis_func = 100
```

```python
%%timeit
galerkin_method(phi, q2_l, f2, amount_of_basis_func)
```

    1.1 s ± 12.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
result = galerkin_method(phi, q2_l, f2, amount_of_basis_func)
beautiful_solution = show_solution(result, amount_of_basis_func)
# print(beautiful_solution)
y = [beautiful_solution.subs(x, point) for point in points]
plot.plot(points, y)
```

```python
beautiful_solution.subs(x, 0)
```

$\displaystyle 2.40199204246966$