# Modeling Diffusion in the Calcium Release Unit

This tutorial we setup and solve for the diffusion of molecules in the electron microscopy derived calcium release unit geometry generated in the [BlendGAMer tutorial](../tutorials/blendertutorial.html).
In order to run this notebook you will need the following packages:

- Python environment
- dolfin
- tqdm

```python
from tqdm import tqdm      # Utility to generate progress bars
import dolfin as d         # import dolfin

# Suppress some of the text that FFC dumps to the notebook
import logging
logging.getLogger('FFC').setLevel(logging.WARNING)
```

An easy way to get everything configure a suitable environment is using [conda](https://docs.continuum.io/anaconda/install/).
Using `conda`, the following steps should configure everything you need.

    conda create -n fenicsproject -c conda-forge fenics
    conda activate fenicsproject
    pip install jupyter tqdm

If you have not completed the [BlendGAMer tutorial](../tutorials/blendertutorial.html), the tetrahedral mesh of the CRU can be [downloaded directly](https://github.com/ctlee/gamer_tutorials/blob/master/data/CRUTetmesh.xml).

## Model of Diffusion With Decay (Strong Formulation)

We will model the dynamics of a molecule with concentration $u$.
The dynamics of $u$ are modeled as

\begin{equation*}
\frac{\partial u}{\partial t} = D\nabla^2 u - \frac{u}{\tau} \enspace \text{in} \enspace \Omega,
\end{equation*}

where $D$ is the diffusion constant, $\tau$ is a decay constant, $\Omega$ is the cytosolic domain, and $t$ is time.

We define boundary conditions

\begin{align}
D(\mathbf{n} \cdot \nabla u) &= J_\text{in} \enspace\text{on}\enspace \partial\Omega_\text{t-tubule}, \\
D(\mathbf{n} \cdot \nabla u) &= 0 \enspace\text{on}\enspace \partial\Omega_\text{other}, \\
\partial\Omega &= \partial\Omega_\text{t-tubule} \cup \partial\Omega_\text{other},
\end{align}

where $J_\text{in}$ is the inward flux on the t-tubule membrane ($\partial \Omega_\text{t-tubule}$).
No flux boundary conditions are applied to all other boundaries ($\partial \Omega_\text{other}$).

Finally we set an initial condition that there is no $u$ in the system at time $t=0$,

\begin{equation*}
u(\mathbf{x},t=0) = 0.
\end{equation*}

We define the values for several physical parameters, initial condition, simulation length, and timestep next.

```python
# Model parameters:
## Physical parameters
D    = 30              # Diffusion coefficient [nm^2/us]
jin  = 0.5             # Inward flux rate [uM*nm/us]
tau  = 500             # Decay timescale [us]

## Initial condition
u0   = d.Constant(0.0) # Initial concentration of u at t=0 [uM]

## Simulation time control
T    = 5000.0          # Total simulation length [us]
dt   =  100.0          # time step size [us]
t    =    0.0          # Initial time
numsteps = round(T/dt)

print(f"Simulation will run {numsteps} steps")
```

    Simulation will run 50 steps

## Model of Diffusion With Decay (Weak/Variational Formulation)

The finite element method uses a variational formulation of the PDE which allows us to search for solutions from a larger function space. Equations input into FEniCS are done so through their variational formulation; we will now show how to convert our problem into a valid FEniCS input.

First we multiply the governing PDE by a test function, $v$, coming from a function space $V$, integrate over the domain.

\begin{equation*}
\int_\Omega \frac{\partial u}{\partial t}v \,dx = \int_\Omega D\nabla^2uv \,dx - \int_\Omega \frac{u}{\tau}v \,dx
\end{equation*}

We then apply the divergence theorem,

\begin{equation*}
\int D\nabla^2uv \,dx = -\int_\Omega D\nabla u \cdot \nabla v \,dx + \int_{\partial\Omega} D(\mathbf{n}\cdot\nabla u)v \,ds,
\end{equation*}

to obtain the following variational formulation:

\begin{equation*}
\boxed{\int_\Omega \frac{\partial u}{\partial t}v \,dx = -\int_\Omega D\nabla u \cdot \nabla v \,dx + \int_{\partial\Omega} D(\mathbf{n}\cdot\nabla u)v \,ds -\int_\Omega \frac{u}{\tau}v \,dx}
\end{equation*}

We discretize the time-derivative with a backward Euler scheme due to its simplicity and unconditional stability properties,

\begin{equation*}
\frac{\partial u}{\partial t} \approx \frac{u - u_n}{\Delta t},
\end{equation*}

where $u_n$ represents the (known) solution computed at the previous timestep.
We further simplify the variational formulation by inserting the Neumann boundary conditions and using the shorthand notation where $\langle \cdot , \cdot \rangle$ represents the inner-product over $\Omega$ and $\langle \cdot , \cdot \rangle_{\partial\Omega_\text{in}}$ represents the inner-product on the boundary $\partial\Omega_\text{in}$.
Terms are separated onto the left or right hand sides by their dependence on $u$:

\begin{equation*}
\boxed{\langle u,v \rangle + D\Delta t \langle \nabla u,\nabla v \rangle + \frac{\Delta t}{\tau}\langle u,v \rangle =  \Delta t \langle J_\text{in},v \rangle_{\partial\Omega_\text{in}} + \langle u_n,v \rangle}
\end{equation*}

Notice that since $u_n$ is a known value it is not dependent on the unknown, $u$, and therefore is placed on the right hand side. Also, since $D(\mathbf{n}\cdot \nabla u)=0$ on the no-flux boundary those terms drop out of the variational formulation.

In the abstract form this is written as,

\begin{align}
a(u,v) &= \langle u,v \rangle  + D\Delta t \langle \nabla u,\nabla v \rangle + \frac{\Delta t}{\tau} \langle u,v \rangle, \\
L(v) &= \Delta t \langle J_\text{in},v \rangle_{\partial\Omega_\text{in}} + \langle u_n,v \rangle,
\end{align}

where $a(u,v)$ is a bilinear form and $L(v)$ is a linear functional.

## Solving the Linear Problem Using Dolfin/FEniCS

```python
# Import the mesh and construct a linear Lagrange function space over the mesh
dolfin_mesh = d.Mesh('../data/CRUTetmesh.xml')
V = d.FunctionSpace(dolfin_mesh,'P',1)
```

```python
# Define trial and test functions
u = d.TrialFunction(V)
v = d.TestFunction(V)
# Known solution corresponding to previous timestep. Initialize with initial condition
un = d.interpolate(u0, V)
```

```python
# Recall that in the CRU tutorial we applied the following boundary markings
# Cytosol = 10
# Mitochondria = 20
# T-tubule = 30
# SR = 40

# define integration domains
face_dimension = 2 # we marked faces which have a topological dimension of 2 (triangles)
# Pull out a subdomain of all faces
meshBoundary = d.MeshFunction("size_t", dolfin_mesh, face_dimension, dolfin_mesh.domains())

# Setup "measures" to refer to volume and boundary elements
# Create a measure for the volume
dx = d.Measure('dx', domain = dolfin_mesh)
# Create a measure for boundary
ds = d.Measure('ds', domain = dolfin_mesh, subdomain_data = meshBoundary)
```

Recalling the abstract form:
\begin{align}
a(u,v) &= \left(1+\frac{\Delta t}{\tau}\right) \langle u,v \rangle  + D\Delta t \langle \nabla u,\nabla v \rangle, \\
L(v) &= \Delta t \langle J_\text{in},v \rangle_{\partial\Omega_\text{t-tubule}} + \langle u_n,v \rangle,
\end{align}

note that the no-flux boundary conditions on $\partial\Omega_\text{other}$ nullify those terms.

```python
# define bilinear and linear forms
a = (1+dt/tau)*u*v*dx + D*dt*d.inner(d.grad(u),d.grad(v))*dx
# ds(30) is the surface of the t-tubule
L = dt*jin*v*ds(30) + un*v*dx
```

### Solve and Save the Simulation

```python
filename = 'solution/dolfinOut.pvd'
vtkfile = d.File(filename)

# store the initial condition
u = d.Function(V)
un.rename("u", "solution")
u.rename("u", "solution")
vtkfile << (un,t)

# Main loop
for idx in tqdm(range(numsteps)):
    # step forward in time
    t += dt
    # Solve the linear problem
    d.solve(a==L, u)
    # Write the solution for the current timestep to file
    vtkfile << (u,t)
    # Copy solution to un
    un.assign(u)
```

    100%|██████████| 50/50 [00:51<00:00,  1.03s/it]

Solutions at each timestep are saved in the $\texttt{.vtk}$ format (the collection of snapshots is stored in the $\texttt{dolfinOut.pvd}$ file) and can be visualized using software such as Paraview.


# Bayesian Statistics

## Chapter Zero Review

Omission

## Chapter One Introduction

### Conditional Probability

As we have mentioned in `Using Python in Advanced Statistics`, **probability** is a number within in 0 to 1(included 0 and 1),and the meaning of probability is that the credibility(可信度) of a event or a prediction. According to this idea, **COnditional Probability** is the probability under some kinds of conditions. For example, if there is a rain tomorrow in Zhuhai. I check the data, found that in tomorrow's date, there are about 30% that will rain. **This is the probability**. If I want to know the possibility of rain tomorrow if it is already raining today. **This is the conditional probability**, which is a probability of an event happen based on a happened fact.

The usual notation for conditional probability is $p(A\mid B)$, which means under B event's condition, the probability of A.

*PS: if the probability of an event is 0, we says it is **false**; in the opposite, if the probability is 1, it is **true**.*

### Conjoint Probability

After known what is **conditional probability**, we can easily know what is **conjoint probability**, which is another way to say the probability that two things are true. This time, the notation is: $p(A B)$ or $P(A\ and \ B)$.

```python
import matplotlib.pyplot as plt
import numpy as np
import sympy
from matplotlib_venn import venn2, venn3_circles

plt.figure(figsize=(4,4))
v = venn2(subsets=(2,2,1), set_labels = ('A', 'B'))

plt.title("Sample Venn diagram - plot one")
plt.annotate('P(AB)', xy=v.get_label_by_id('11').get_position() - np.array([0, 0.05]), xytext=(-70,-70),
             ha='center', textcoords='offset points', bbox=dict(boxstyle='round,pad=0.5', fc='gray', alpha=0.1),
             arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.5',color='gray'))
plt.show()
```

The conjointed part can represent the conjoint probability that $A$ and $B$ are both true. For instance, if $A$ and $B$ are independent, we know that the $P(A\ and \ B) = P(A)P(B) \Leftrightarrow P(A \mid B) = p(A)$, for whether A is true or not will not affect B; but if $A$ and $B$ are not independent, just like the weather, if we know that it raind today, it is more likely that it will rain tomorrow, so $p(A|B) > p(A)$.

In general, the probability of a conjunction is : $\mathbf  p(A\ and \ B) = p(A)p(B \mid A)$

### Bayes's theorem

Known what is conditional probability and cojoint probability, we can move on to the Headlight: **Bayes's theorem**. First, we should make sure we are in the same channel:
   1. conjoint probability: $p(A\ and \ B) = p(B\ and \ A)$
   2. probability of a conjunction: $p(A\ and \ B) = p(A)p(B \mid A)$
   3. interchange $A$ and $B$: $p(B\ and \ A) = p(B)p(A \mid B)$

In the light of point 2 and point 3, we can combine the two equations to one euqation: $\Rightarrow p(A) \ p(B \mid A) = p(B) \ p(A \mid B)$, divided by $p(B)$ in the meantime and we can get the **Bayes's theorem !** : $\Rightarrow \frac{p(A) \ p(B \mid A)}{p(B)} =  p(A \mid B)$

We can have a example, Allen B.Downey called it **The Cookie Problem**: Suppose there are two bowls of cookies. Bowl-1 contains $30$ vanilla cookies and $10$ chocolate cookies. Bowl-2 contains $20$ of each.
Now suppose you choose one of the bowls at random and, without looking, select a cookie at random. The cookie is vanilla. What is the probability that it came from Bowl-1?
This is a conditional probability; we want $p(Bowl-1|vanilla)$, but it is not obvious how to compute it. If I asked a different question — the probability of a vanilla cookie given Bowl-1 — it would be easy:

$$p(vanilla\mid Bowl-1) = \frac{3}{4}$$

Sadly, $p(A|B)$ is not the same as $p(B|A)$, but there is a way to get from one
to the other: Bayes’s theorem.

Using **Bayes's Theorem**, we can easily solve the cookie problem:

First, assume that $B_1$ represent the probility that cookie from Bowl-1 and $p(V)$ is the probability of taking out a Vanilla.

$$P(B_1\mid V) = \frac{p(B_1)p(V\mid B_1)}{p(V)}$$

There is another way to think of Bayes's theorem: it gives us a way to update the probability of a hypothesis, $H$, in light of some body of data,$D$.

This way of thinking about Bayes's theorem is called the **diachronic interpretation**. "Diachronic" means that something is happening over time; in this case the probability of the hypotheses changes, over time, as we see new data.

Rewriting Bayes's theorem with $H$ and $D$ yields:

$$P(H|D) = \frac{p(H)p(D|H)}{p(D)}$$

In this interpretation, each term has a name:

* $p(H)$ is the probability of the hypothesis before we see the data, called the prior probability, or just **prior**;
* $p(D)$ is the probability of the data under any hypothesis, called the normalizing constant(total probability);
* $P(H|D)$ is waht we want to compute, the probability of the hypothesis after we see the data, called the **posterior**;
* $P(D|H)$ is the probability of the data under the hypothesis, called the likelihood

In the classical Statistics theory, the data is the random variable. The the Cookie Problem for example, if we want to know the probability of taking out vanilla cookies from Bowl-1, the random variable is $1,2,3,\dots$, the data or the number. However, if we want to know the probability of which Bowls does the vanilla cookie come from, as we got a vanilla cookies already, this time, the random varaible is the probability, not the data. This is the biggest difference between Bayesian and Frequentist.

## Chapter Two Computational Statistics

In statistics, a **distribution** is a set of values and thier correponding probabilites.

You can get more information about Probabilities and Statistics here

Here is a example of rolling a dice, as we all know that the probability of each side is $\frac{1}{6}$, then, this is a Discrete Uniform Distribution.

```python
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import matplotlib

fig, ax = plt.subplots(1, 1)
a = [1,2,3,4,5,6]
sample = np.random.randint(len(a), size = 10000)

dict = {}
for key in sample:
    dict[key] = dict.get(key, 0) + 1
len(dict)

x = [dict[i] for i in dict] # change dict to list
pmf = [dict[i]/len(sample) for i in dict] #calculate pmf of each element in list

ax.plot(a,pmf,'bo')
ax.vlines(a, 0, pmf, colors='b', lw=5, alpha=0.5)

line1 = [(0, 0.1666667), (10, 0.1666667)]
(line1_xs, line1_ys) = zip(*line1)
ax.add_line(matplotlib.lines.Line2D(line1_xs, line1_ys, linewidth=1, color='red') )
plt.show()
```

## Chapter Three Naive Bayes

Naive Bayes methods is the basic idea in both Naive Bayes Classification and Bayes Belief Network. It is totally an extension of the Bayes Theorm that we consider every pair of features given the value of the class variable are conditional independent.  Given classs vairable $y$ and dependent feature vector $x_1$ through $x_2$:

$$P(y|x_1, \dots, x_n) = \frac{P(y)P(x_1, \dots, x_n|y)}{P(x_1, \dots, x_n)}$$

Under this assumption we can rewrite the Bayes Theorm:

$$P(y|x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^n P(x_i |y)}{P(x_1, \dots, x_n)}$$

Since $P(x_1, \dots, x_n)$ is constant given the input, we can use the following classification rule:

$$P(y|x_1, \dots, x_n) \propto P(y) \prod_{i=1}^n P(x_i | y) \Rightarrow \hat{y} = arg \max \limits_{y} P(y) \prod_{i = 1}^n P(x_i | y)$$

and we can use Maximum A posterior estimation to estimate $P(y)$ and P(x_i | y)$; the former is then the relative frequency of class $y$ in the training set.

### How to build a basic model using scikit-learn

#### Gaussian

It is used in classificaiton and it assumes that feature follow a normal distribution (the likelihood function is assumed to be Normal)

$$P(x_i | y) = \frac{1}{\sqrt{2\pi}\sigma_y} exp( - \frac{(x_i - \mu_y)^2}{2 \sigma_y^2})$$

THe parameter $\sigma_y$ and $\mu_y$ are estimated using MLE.

```python
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import matplotlib.colors as colors
import seaborn as sns
import itertools
from scipy.stats import norm
import scipy.stats
from sklearn.naive_bayes import GaussianNB
%matplotlib inline
sns.set()

# iris is a famous data set  and used in every data mining's reference book, including the sklearn's official doc

iris = sns.load_dataset("iris")
iris = iris.rename(index = str, columns = {'sepal_length':'1_sepal_length','sepal_width':'2_sepal_width', 'petal_length':'3_petal_length', 'petal_width':'4_petal_width'})

#Plot the scatter of sepal length vs sepal width
sns.FacetGrid(iris, hue="species", height=7) .map(plt.scatter,"1_sepal_length", "2_sepal_width", )  .add_legend()
plt.title('Scatter plot')

```

```python
from sklearn.naive_bayes import GaussianNB

#Setup X and y data
X_data = df1.iloc[:,0:2]
y_labels = df1.iloc[:,2].replace({'setosa':0,'versicolor':1,'virginica':2}).copy()

#Fit model
model_sk = GaussianNB(priors = None)
model_sk.fit(X_data,y_labels)

# Our 2-dimensional classifier will be over variables X and Y
N = 100
X = np.linspace(4, 8, N)
Y = np.linspace(1.5, 5, N)
X, Y = np.meshgrid(X, Y)

#fig = plt.figure(figsize = (10,10))
#ax = fig.gca()
color_list = ['Blues','Greens','Reds']
my_norm = colors.Normalize(vmin=-1.,vmax=1.)

g = sns.FacetGrid(iris, hue="species", height=10, palette = 'colorblind') .map(plt.scatter, "1_sepal_length", "2_sepal_width",)  .add_legend()
my_ax = g.ax

#Computing the predicted class function for each value on the grid
zz = np.array(  [model_sk.predict( [[xx,yy]])[0] for xx, yy in zip(np.ravel(X), np.ravel(Y)) ] )

#Reshaping the predicted class into the meshgrid shape
Z = zz.reshape(X.shape)

#Plot the filled and boundary contours
my_ax.contourf( X, Y, Z, 2, alpha = .1, colors = ('blue','green','red'))
my_ax.contour( X, Y, Z, 2, alpha = 1, colors = ('blue','green','red'))

# Addd axis and title
my_ax.set_xlabel('Sepal length')
my_ax.set_ylabel('Sepal width')
my_ax.set_title('Gaussian Naive Bayes decision boundaries')

plt.show()
```

```python

```


## Métodos de diferencias finitas para problemas de valores a la frontera

Es claro que el método de shooting es **muy lento**, pues el tener que resolver toda la ecuación diferencial para evaluar $F$ y poder resolver $F(z^*) = 0$ puede ser muy tardado, además de que la solución obtenida siempre es aproximada, por lo que puede haber errores ahí que nos alejen de la condición verdadera y que nos den una mala solución.

Así resulta conveniente explorar otro tipo de métodos de solución. Supongamos que en nuetro problema de valores a la frontera estamos trabajando con una  **ecuación diferencial lineal de orden dos**, es decir, una problema de la forma

$$
\begin{align}
a(t) \ddot{y}(t) + b(t) \dot{y}(t) &+  c(t) y(t) = f(t) \\
& \\
y(t_0) &= \alpha \\
y(t_f) &= \beta \\
\end{align}
$$

Con $a(t) \neq 0$, $b(t)$, $c
(t)$ y $f(t)$ funciones conocidas, $\alpha$ y $\beta$ constantes.

Una alternativa para resolver problemas de valores a la frontera es utilizar diferencias finitas para aproximar las derivadas $\ddot{y}$ y $\dot{y}$. Utilizando una diferencia centrada para ambas derivadas, llegamos a la siguiente aproximación:

$$
\begin{align}
a(t) \frac{y(t+h) + y(t-h) - 2y(t)}{h^2} &+ b(t) \frac{y(t+h) - y(t-h)}{2h} +  c(t) y(t) = f(t) \\
& \\
y(t_0) &= \alpha \\
y(t_f) &= \beta \\
\end{align}
$$

Así, para un a partición regular de orden $n+1$ (i.e con $n$ subintervalos) del intervalo $[t_0,t_f]$, obtenemos los tiempos $t_0,\ldots,t_n$ que corresponden a un paso  $h = (t_f - t_0)/n$. Utilizando la notación de índices para el tiempo (i.e. $g(t_i) = g_i$), el problema se convierte en:

$$
\begin{align}
a_i \frac{y_{i+1} + y_{i-1} - 2y_i}{h^2} &+ b_i \frac{y_{i+1} - y_{i-1}}{2h} +  c_i y_i = f_i \\
& \\
y_0 &= \alpha \\
y_n &= \beta \\
\end{align}
$$

Con $a_i = a(t_i)$, $b_i=b(t_i)$, $c_i = c(t_i)$, $f_i = f(t_i)$, $y_{i} = y(t_i)$ y $1\leq i \leq n-1$

Podemos reordenar las ecuaciones para que quede de la siguiente forma:

$$
\begin{align}
\left(\frac{a_i}{h^2} - \frac{b_i}{2h}\right) y_{i-1} + \left(c_i - \frac{2a_i}{h^2} \right)& y_{i} +  \left(\frac{a_i}{h^2} + \frac{b_i}{2h}\right) y_{i+1} = f_i \\
& \\
y_0 &= \alpha \\
y_n &= \beta \\
\end{align}
$$

Si consideramos que la primera ecuación solo se cumple para $1 \leq i \leq n-1$, y ya que conocemos quienes son las funciones $a(t)$,$b(t)$, $c(t)$ y $f(t)$ así como los valores $h$, $\alpha$ y $\beta$, entonces **las únicas incógnitas de las ecuaciones son las $y_0,y_1,\ldots,y_n$**

Como todos los demás términos son conocidos, entonces el conjunto de ecuaciones representa un **sistema de $n+1$ ecuaciones lineales** de la forma $\mathbf{Ay} = \mathbf{b}$, para el que la solución $\mathbf{y} = (y_0,y_1,..,y_n)$ es la discretización de la función solución $y(t)$ a nuestra ecuación diferencial.

Dicho sistema puede resolverse utilizando los métodos  que vimos en las clases de Álgebra Lineal:  eliminación Gaussiana, Jacobi, Gauss-Siedel. A esta forma de resolver ecuacioned diferenciales de segundo orden aproximando las derivadas con diferencias finitas y obteniendo un problema de ecuaciones linealesse les conoce como **métodos de diferencias finitas**

### Ejercicio 1

Implementa una función `matriz(a,b,c,f,alpha,beta,t_0,t_f,n)`. Los argumentos de la función deben de ser

* `a`,`b`,`c`,`f` las funciones **escalares** $a(t)$,$b(t)$,$c(t)$ y $f(t)$

* `alpha`, `beta`: las condiciones a la frontera $\alpha$, $\beta$ del problema

* `t_0`,`t_f` los tiempos inicial y final, `n` $>2$ un natural correspondiente el número de subintervalos que se van a tomar. Eso define el paso de tiempo $h$ del sistema a resolver como

$$
h = \frac{t_f - t_0}{n}
$$

La función debe de regresar la matriz $A$, de $n+1 \times n+1$ y un vector $b$ de longitud $n+1$ que correspondan a la discretización del problema de valores a la frontera expuesto.

**Sugerencia:** la matriz $A$ debe de ser **tridiagonal**, es decir, los únicos elementos distintos de cero son elementos de la forma $A_{i,i}$, $A_{i+1,i}$ y $A_{i-1,i}$

### Ejercicio 2

Utiliza tu función `matriz` y alguno de los métodos de las clases de Álgebra lineal para resolver las ecuaciones del ejercicio 3 de la clase 17.2. Compara tus soluciones con la solución analítica y con las soluciones obtenidas por el método de Shooting. ¿Es mejor este método o el método de Shooting?

**Sugerencia** si tienes dudas sobre si tu método de álgebra lineal converge para esa matriz, usa el operador `\` para apoyarte.

## Problemas tipo Sturm-Liouville

Un problema parecido pero fundamentalmente **distinto** a los problemas tradicionales de valores a la frontera son los problemas tipo **Sturm-Liouville**. En dichos problemas, tenemos una ecuación diferencial de forma similar a la ecuación diferencial lineal de orden dos:

$$
\begin{align}
a(t) \ddot{y}(t) + b(t) \dot{y}(t) &+  c(t) y(t) = \lambda y(t) \\
\end{align}
$$

Con $\lambda \in \mathbb{R}$ una constante **desconocida**. Con condiciones iniciales que son de una forma más complicada:

$$
\gamma_1 y(t_0) + \gamma_2 \dot{y}(t_0)  = 0 \\
\delta_1 y(t_f) + \delta_2 \dot{y}(t_f)  = 0 \\
$$

La [teoría de Sturm-Liouville](https://en.wikipedia.org/wiki/Sturm%E2%80%93Liouville_theory) nos garantiza que, bajo ciertas condiciones, existe una infinidad de funciones $y_{(k)}$ y  constantes $\lambda_k$ asociadas a cada una, con $k \in \mathbb{N}$, que satisfacen la ecuación diferencial mostrada, es decir

$$
\begin{align}
a(t) \ddot{y}_{(k)}(t) + b(t) \dot{y}_{(k)}(t) &+  c(t) y_{(k)}(t) = \lambda_k y_{(k)}(t) \\
\end{align}
$$

Por simplicidad, supongamos que las condiciones de frontera son

$$
y(t_0) = 0 \\
y(t_f) = 0 \\
$$

Si volvemos a utilizar diferencias finitas para discretizar las derivadas, de forma totalmente análoga a la sección anterior, podremos entonces expresar la ecuación diferencial de la siguiente forma:

$$
\begin{align}
\left(\frac{a_i}{h^2} - \frac{b_i}{2h}\right) y_{i-1} + \left(c_i - \frac{2a_i}{h^2} \right)& y_{i} +  \left(\frac{a_i}{h^2} + \frac{b_i}{2h}\right) y_{i+1} = \lambda y_i \\
& \\
y_0 &= 0\\
y_n &= 0 \\
\end{align}
$$

Utilizando los valores ya conocidos de $y_0 = 0$ y de $y_n = 0$, podemos reescribir las ecuaciones para $i=1,n$. Así, las ecuaciones para cada $y_i$ son

$$
\begin{gather}
\left(c_1 - \frac{2a_1}{h^2} \right) y_{1} +  \left(\frac{a_1}{h^2} + \frac{b_1}{2h}\right) y_{2} = \lambda y_1 \\
 \\
\left(\frac{a_i}{h^2} - \frac{b_i}{2h}\right) y_{i-1} + \left(c_i - \frac{2a_i}{h^2} \right) y_{i} +  \left(\frac{a_i}{h^2} + \frac{b_i}{2h}\right) y_{i+1} = \lambda y_i \quad i = 2,\ldots,n-2 \\
 \\
\left(\frac{a_{n-1}}{h^2} - \frac{b_{n-1}}{2h}\right) y_{n-2} + \left(c_{n-1} - \frac{2a_{n-1}}{h^2} \right) y_{n-1} = \lambda y_{n-1} \\
\end{gather}
$$

Estas ecuaciones se pueden escribir de forma compacta como un conjunto de **$n-1$ ecuaciones lineales** de la forma

$$
\mathbf{M y } = \lambda \mathbf{y}
$$

Con $\mathbf{M}$ otra matriz tridiagonal. Es claro que los vectores $\mathbf{y}$ y las constantes $\lambda$ deben de ser los **valores y vectores propios** de la matriz $\mathbf{M}$. Así, al aproximar las derivadas con diferencias finitas convertimos el problema de Sturm-Liouville en un problema de valores propios para la matriz $\mathbf{M}$. Los vectores propios de la matriz corresponderán a la discretización de las funcions $y_{(k)}$ mientras que sus valores propios asociados serán los $\lambda_k$

### Ejercicio 3

Escribe una función `sturmLiouville(a,b,c,t_0,t_f,n)`. Los argumentos de la función deben de ser

* `a`,`b`,`c` las funciones **escalares** $a(t)$, $b(t)$ y $c(t)$ del problema de Sturm-Liouville

* `t_0`,`t_f` los tiempos inicial y final, `n` $>2$ un natural correspondiente el número de subintervalos que se van a tomar. Eso define el paso de tiempo $h$ del sistema a resolver como

$$
\frac{t_f - t_0}{n} = h
$$

La función debe de regresar la matriz $\mathbf{M}$, de $n-1 \times n-1$ que correspondan a la discretización del problema de Sturm-Liouville expuesto anteriormente.

**Sugerencia:** puedes ver a la matriz $\mathbf{M}$ como un caso particular de la matriz $\mathbf{A}$. Así, podrías simplemente usar la función `matriz` del ejercicio 1 como base.

### Ejercicio 4

Utiliza la función `sturmLiouville` y las funciones `eigvals` y  `eigvecs` de la librería LinearAlgebra para resolver los siguientes problemas:

* $$\ddot{y} + 3\dot{y} + 2y = -\lambda y \\ y(0) = y(1) = 0 $$

Solución: $y_{(k)}(t) = e^{-3t/2}\sin{(k \pi t)}$, $\lambda_k = (1 + 4k^2 \pi^2)/4$

* $$t^2 \ddot{y} + t \dot{y} = -\lambda y \\ y(1) = y(2) = 0 $$

Solución: $y_{(k)}(t) = \sin{\left(\frac{k \pi}{\ln{2}} \ln{(x)} \right)}$, $\lambda_k = \left(\frac{k \pi}{\ln{2}} \right)^2$

Compara, en la misma figura, la función obtenida numéricamente y la solución analítica para valores bajos de $|\lambda_k|$ (i.e., grafica las funciones asociadas a los valores propios **más pequeños en valor absoluto** ). ¿Qué ocurre para valores altos de $|\lambda_k|$? ¿Los valores propios numéricos y analaíticos se parecen?

**Nota:** toma en cuenta que los signos de los valores propios $\lambda$ aquí mostrados son **negativos**, por lo que deberás tener cuidado a la hora de comparar.

## Aplicación en mecánica cuántica: la ecuación de Schrödinger independiente del tiempo

Un problema de valores a la frontera que surge comúnmente en física es la solución de la ecuación de Schrödinger dependiente del tiempo

La evolución en el tiempo de un sistema cuántico está definida por la ecuación de Schrödinger dependiente del tiempo:

$$
i \hbar \frac{\partial \psi}{\partial t} = H \psi
$$

Con $i$ la unidad imaginaria, $\hbar$ la constante reducida de Planck, $\psi$ la amplitud de probabildad del sistema y $H$ el hamiltoniano. La solución general a la ecuación es

$$
\psi = \sum_{k=1}^{\infty}A_k e^{-E_k t} \psi_k
$$

Con $\psi_k$ y $E_k$ las funciones propias y valores propios, respectivamente, del operador hamiltoniano, es decir, las que cumplen con la propiedad

$$
H \psi_k = E_k \psi_k
$$

A esa ecuación se le llama la **ecuación de Schrödinger independiente del tiempo**. Trabajando un sistema de una partícula en una dimensión, cuya energía potencial es un pozo infinito de la forma

$$
V(x) = \begin{cases}
0 & \text{si } 0 \leq x \leq 1 \\
\infty & \text{en otro caso}
\end{cases}
$$

Escribiendo el hamiltoniano de dicho sistema cuántico y haciendo una adimensionalización, la ecuación de Schrödinger independiente del tiempo se reduce al siguiente problema de valores iniciales

$$
\frac{1}{2} \frac{d^2 \psi_k}{dx^2}(x) + \epsilon_k \psi_k (x) = 0
 \\
\psi_{k} (0) = 0 \\
\psi_{k} (1) = 0 \\
$$

La solución analítica está dada por
$$
\psi_k(x) = \sqrt{2} \sin{(k\pi x)} \\
\epsilon_k = \frac{k^2 \pi^2}{2} \\
$$

### Ejercicio 6

Resuelve el problema tipo Sturm-Liouville asociado a la particula en el pozo infinito de potencial. Compara las soluciones obtenidas con las soluciones analíticas. ¿Las funciones obtenidas numéricamente se pueden comparar bien con la solución analítica? Justifica tu respuesta

**Nota:** las soluciones analíticas de este problema **están normalizadas**, es decir, cumplen que

$$
\int_{0}^1 | \psi_k (x) |^2 \: dx = 1
$$

Lo más probable es que la soluciones que obtenas numéricamente **no estén normalizadas**, por lo que tendrás que encontrar una manera equivalente de normalizarlas para que se puedan comparar.

## La visión general

Resolver ecuaciones diferenciales ordinarias en la computadora es un problema fundamental de la física computacional por su relación íntima con mecánica clásica. No todos los métodos pueden resolver cualquier sistema, por lo que es importante buscar métodos con alta precisión para no tener errores como el del péndulo simple con el método de Euler

Existe un area dedicada a modelar los átomos de manera clásica. Así, resolver un sistema **termodinámico** (moleculas de aire en un cuarto, un fluido ideal) o **biológicos** (átomos en una proteína, moléculas en una membrana,etc) se convierte en resolver las ecuaciones de movimiento de todos los átomos del sistema. En 3 dimensiones, si tenemos $n$ átomos, habrá un sistema de $6n$ ecuaciones difernciales de primer orden que debemos de resolver para saber la evolución del sistema.

A esas simulaciones se les llama **dinámica molecular**. La dinámica molecular es una herramienta **revolucionaria** de la ciencia, pues permite resolver muchisísimos problemas. Hoy en día, se utlizan simulaciones de dinámica molecular para probar la eficiencia de fármacos y segurmante se están utilizando dichas simulaciones para probar y crear vacunas para el COVID19

```julia

```


# 位相キックバック

ユニタリゲート $U$ と、その固有状態 $\lvert\psi\rangle$ を考えます。
ユニタリゲートの性質より、固有値はある位相角 $\theta$ を用いて、$e^{i\theta}$ と表されます。

$$
U\lvert\psi\rangle = e^{i \theta} \lvert\psi\rangle
$$

この $\theta$ を求めるとしましょう。グローバル位相として現れているため、直接測定することは難しいです。

位相キックバックは、$\lvert\psi\rangle$ とは別の量子ビットに $\theta$ の情報を取り出して測定するテクニックです。
そのためにはユニタリゲート $U$ を制御ゲート化した $CU$ を用います。

簡単な例を考えてみましょう。

まず、ユニタリゲートを次のように設定します。
$$
U(\theta) = \begin{pmatrix}
1 & 0 \\
0 & e^{i\theta} \\
\end{pmatrix}
$$

このとき、$\lvert 1 \rangle$ が固有状態になります。固有値は $e^{i\theta}$ です。

$$
U(\theta)\lvert 1 \rangle = e^{i\theta}\lvert 1 \rangle
$$

制御ユニタリゲートは次のようになります。
$$
CU(\theta) = CR(\theta) = \begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & e^{i\theta} \\
\end{pmatrix}
$$

位相キックバックでは次のような量子回路を用います。

まず、補助量子ビットに $H$ゲートを作用させて重ね合わせ状態を作ります。

$$
\lvert\psi_1\rangle = \frac{1}{\sqrt{2}}(\lvert0\rangle + \lvert1\rangle) \otimes \lvert1\rangle
$$

次に、補助量子ビットを制御量子ビットとして固有状態に $U(\theta)$ を作用させます。
これにより、固有値が係数として取り出せます。

$$
\begin{align}
\lvert\psi_2\rangle &= \frac{1}{\sqrt{2}}(\lvert0\rangle \otimes \lvert1\rangle + \lvert1\rangle \otimes U(\theta)\lvert1\rangle) \\
&= \frac{1}{\sqrt{2}}(\lvert0\rangle \otimes \lvert1\rangle + \lvert1\rangle \otimes e^{i\theta}\lvert1\rangle) \\
&= \frac{1}{\sqrt{2}}(\lvert0\rangle \otimes \lvert1\rangle + e^{i\theta}\lvert1\rangle \otimes \lvert1\rangle) \\
&= \frac{1}{\sqrt{2}}(\lvert0\rangle + e^{i\theta} \lvert1\rangle) \otimes \lvert1\rangle)
\end{align}
$$

以上のように、$\theta$ を補助量子ビットの重ね合わせ状態の係数として取り出せました。

ここで補助量子ビットだけに注目します。
このまま補助量子ビットを測定しても $\lvert0\rangle$, $\lvert1\rangle$ が等確率なので、再度 $H$ゲートを作用させます。

$$
\begin{align}
H\cdot \frac{1}{\sqrt{2}}(\lvert0\rangle + e^{i\theta} \lvert1\rangle) &= \frac{1}{\sqrt{2}}
\begin{pmatrix} 1 & 1
\\ 1 & -1 \\
\end{pmatrix}
\cdot
\frac{1}{\sqrt{2}}
\begin{pmatrix}
1 \\ e^{i\theta} \\
\end{pmatrix} \\
&= \frac{1}{2}
\begin{pmatrix}
1 + e^{i\theta} \\ 1 - e^{i\theta} \\
\end{pmatrix}
\end{align}
$$

$H$ ゲートを用いて相対位相の差を振幅の差へ変換することができました。
最後に補助量子ビットを繰り返し測定すると、$\lvert0\rangle$ を測定する確率 $\rm{Pr}(\lvert0\rangle)$ は

$$
\begin{align}
\mathrm{Pr}(\lvert0\rangle) &= \biggl| \frac{1+e^{i\theta}}{2} \biggr|^2 \\
&= \frac{1}{2}(1 + \cos\theta)
\end{align}
$$

以上より、固有値の実部を推定することができます。

以上をblueqatで実装してみましょう。

```python
from blueqat import Circuit
import numpy as np
```

上記の量子回路を実装します。
$\theta$ は乱数で決めておきます。

```python
theta = np.random.rand() * np.pi
shots = 1024

c = Circuit(2)

c.x[1].h[0].cr(theta)[0, 1].h[0].m[0]
res = c.run(shots = shots)
```

実行結果から、補助量子ビットで$\lvert 0 \rangle$を測定した割合を求めます。
それより計算した推定値 $\theta_{est}$ と、事前に乱数で決めた $\theta$ を比較します。

```python
p0 = res['00'] / shots
theta_est = np.arccos(2 * p0 - 1)
```

```python
print("θ推定値　 :", theta_est)
print("実際の値 :", theta)
```

    θ推定値　 : 0.3718944104475685
    実際の値 : 0.36989050621867126

以上より、位相キックバックで $\theta$ を推定できました。
推定の精度は、ショット数に依存します。

```python

```


# Recocido Simulado

La librería **Pyristic** incluye una clase llamada *SimulatedAnnealing* que facilita la implementación de la metaheurística basada en *Recocido Simulado*. Para poder utilizar esta clase, se debe realizar lo siguiente:

1. Definir:
    * La función objetivo $f$.
    * La lista de restricciones.

2. Crear una clase que herede de *SimulatedAnnealing*.

3. Sobreescribir las siguientes funciones de la clase SimulatedAnnealing:
    * get_neighbor (requerido)
    * update_temperature (opcional)

A continuación se muestran los elementos que se deben importar. Posteriormente, se resolverán dos problemas de optimización combinatoria usando la clase *SimulatedAnnealing*.

```python
import sys
import os

#library_path is the path where the Optimpy library is located.
library_path = "/home/dell/Documentos/Git_proejcts/pyristic/"
#library_path = "/Users/adrianamenchacamendez/Documentos/enes_morelia/papime/optimizacion-con-metaheuristicas/"
sys.path.append(os.path.abspath(library_path))
```

```python
from pyristic.heuristic.SimulatedAnnealing_search import SimulatedAnnealing
from pyristic.utils.helpers import *
from pprint import pprint
import numpy as np
import copy
```

## Problema del agente viajero

\begin{equation}
    \label{eq:TSP}
    \begin{array}{rll}
    \text{minimizar:} & f(x) = d(x_n, x_1) + \sum_{i=1}^{n-1} d(x_i, x_{i+1}) &  \\
    \text{tal que: } & x_i \in \{1,2,\cdots,n\} & \\
    \end{array}
\end{equation}

Donde:
* $d(x_i,x_j)$ es la distancia de la ciudad $x_i$ a la ciudad $x_j$.
* $n$ es el número de ciudades.
* $x$ es una permutación de las $n$ ciudades.

A continuación vamos a definir una instancia de este problema utilizando 10 ciudades.

```python
import random
import math

num_cities = 10
dist_matrix = \
[\
[0,49,30,53,72,19,76,87,45,48],\
[49,0,19,38,32,31,75,69,61,25],\
[30,19,0,41,98,56,6,6,45,53],\
[53,38,41,0,52,29,46,90,23,98],\
[72,32,98,52,0,63,90,69,50,82],\
[19,31,56,29,63,0,60,88,41,95],\
[76,75,6,46,90,60,0,61,92,10],\
[87,69,6,90,69,88,61,0,82,73],\
[45,61,45,23,50,41,92,82,0,5],\
[48,25,53,98,82,95,10,73,5,0],\
]

```

### Función objetivo

```python
def f_salesman(x : (list,np.ndarray)) -> float:
    global dist_matrix
    total_dist = dist_matrix[x[-1]][0]
    for i in range(1,len(x)):
        u,v = x[i], x[i-1]
        total_dist+= dist_matrix[u][v]

    return float(total_dist)
```

### Restricciones
Las restricciones se definen como una lista de funciones que retornan valores *booleanos*. Estos valores permitirán verificar si una solución es factible o no.

En el caso del problema del agente viajero queremos comprobar que estamos visitando todas las ciudades exactamente una vez.

```python
def g_salesman(x : np.ndarray) -> bool:

    size = len(x)
    size_ = len(np.unique(x))
    return size == size_

constraints_list= [g_salesman]
```

### Solución inicial

La estrategia utilizada para generar la solución inicial es la siguiente:

1. Introducir la ciudad $1$ en la primera posición de nuestra permutación y crear un arreglo con las ciudades restantes.

2. Seleccionar la ciudad más cercana desde nuestra ubicación actual.

3. Retirar del arreglo la ciudad seleccionada y asignar la como nuestra ubicación actual; Posterior, repetir el punto 2.

```python
def getInitialSolutionTS(distance_matrix, total_cities) -> np.ndarray:
    Solution = [0]
    remaining_cities  = list(range(1,total_cities))

    while len(remaining_cities) != 0:
        from_ = Solution[-1]
        to_ = remaining_cities[0]
        dist = distance_matrix[from_][to_]

        for i in range(1, len(remaining_cities)):
            distance = distance_matrix[from_][remaining_cities[i]]
            if distance < dist:
                to_ = remaining_cities[i]
                dist = distance
        Solution.append(to_)
        ind = remaining_cities.index(to_)
        remaining_cities.pop(ind)

    return np.array(Solution)
```

```python
Path = getInitialSolutionTS(dist_matrix,num_cities)
print(Path)
```

    [0 5 3 8 9 6 2 7 1 4]

### Declaración de  `SimulatedAnnealing`

Para implementar una metaheurística basada en recocido simulado, utilizando la librería **Pyristic**, es necesario crear una clase que herede de la clase `SimulatedAnnealing`. En este ejemplo, la nueva clase es llamada *TravellingSalesman_solver*.

La nueva clase debe sobreescribir la función `get_neighbor`, de lo contrario el algoritmo no va a funcionar.

```python
class TravellingSalesman_solver(SimulatedAnnealing):

    def __init__(self, f_ : function_type , constraints_: list):
        super().__init__(f_,constraints_)

    def get_neighbor(self, x : np.ndarray) -> np.ndarray:

        x_ = x.copy()
        N = len(x_)
        index1 = random.randint(1, N-1)
        index2 = random.randint(1, N-1)

        while index2 == index1:
            index2 = random.randint(1, N-1)

        v = x[index1]
        x_ = list(x_[v != x_])
        x_ = x_[:index2] + [v] + x_[index2:]
        return np.array(x_)

```

La función `get_neigbor` debe regresar una solución *vecina* de la solución actual, es decir, una variación de la solución actual. Para nuestro ejemplo, una solución vecina de la solución $x$, la vamos a definir como sigue:

Se seleccionan dos indices distintos de manera aleatoria llamados $i$ y $j$, donde, tomaremos el elemento $x_i$ y desplazaremos las otras posiciones de la solución $x$ de modo que $x_i$ se encuentre en la posición $j$ y esta nueva solución será retornada.

### Ejecución de la metaheurística

Una vez definida la clase *TravellingSalesman_solver*, se crea una instancia indicando en los parámetros la función objetivo y las restricciones del problema. En este caso llamamos *TravellingSalesman* a la instancia creada.

```python
TravellingSalesman =\
TravellingSalesman_solver( f_salesman, constraints_list)
```

Finalmente, se llama la función `optimize` (esta función es la misma para todas las clases en la librería). La función `optimize` recibe tres parámetros:

* Solución inicial o función generadora de soluciones iniciales.
* La temperatura inicial.
* La temperatura final.

Vamos a utilizar los siguientes parámetros:
* Emplearemos la solución obtenida por la función *getInitialSolutionTS*.
* $1000$ de temperatura inicial.
* $0.1$ de temperatura final.

```python
TravellingSalesman.optimize(Path, 1000.0 , 0.1)
print(TravellingSalesman)
```

    Simulated Annealing:
     f(X) = 248.0
     X = [0 5 3 8 9 6 2 7 4 1]

Para revisar el comportamiento de la metaheurística en determinado problema, la librería **Pyristic** cuenta con una función llamada `get_stats`. Esta función se encuentra en **utils.helpers** y recibe como parámetros:

* El objeto creado para ejecutar la metaheurística.
* El número de veces que se quiere ejecutar la metaheurística.
* Los argumentos que recibe la función `optimize` deben estar contenidos en una tupla.

La función **get_stats** retorna un diccionario con algunas estadísticas de las ejecuciones.

```python
#Ejecutamos get_stats 30 veces.
args = (Path, 1000.0, 0.01)
statistics = get_stats(TravellingSalesman, 30, args)
```

```python
pprint(statistics)
```

    {'Best solution': {'f': 248.0, 'x': array([0, 5, 3, 8, 9, 6, 2, 7, 4, 1])},
     'Mean': 263.9,
     'Median': 271.0,
     'Standard deviation': 10.425449630591478,
     'Worst solution': {'f': 271.0, 'x': array([0, 5, 3, 8, 9, 6, 2, 7, 1, 4])},
     'averageTime': 0.02728691101074219,
     'objectiveFunction': [271.0,
                           271.0,
                           248.0,
                           271.0,
                           271.0,
                           248.0,
                           271.0,
                           269.0,
                           271.0,
                           271.0,
                           271.0,
                           248.0,
                           248.0,
                           271.0,
                           271.0,
                           271.0,
                           269.0,
                           271.0,
                           271.0,
                           248.0,
                           271.0,
                           248.0,
                           271.0,
                           248.0,
                           271.0,
                           269.0,
                           248.0,
                           271.0,
                           271.0,
                           248.0]}

## Problema de la mochila

\begin{equation}
  \label{eq:KP}
  \begin{array}{rll}
  \text{maximizar:} & f(\vec{x}) = \sum_{i=1}^{n} p_i \cdot x_{i} &  \\
  \text{donde: } & g_1(\vec{x}) = \sum_{i=1}^{n} w_i \cdot x_{i}  \leq c &  \\
          &  x_i \in \{0,1\} & i\in\{1,\ldots,n\}\\
  \end{array}
\end{equation}

Para este problema vamos a crear una instancia de $1000$ objetos, donde, cada objeto estará definido:
* $p_{i} \in [50,100]$
* $w_{i} \in [5,100]$
* $C  = 9786$

```python
n = 1000
p = np.random.randint(50,101,n)
w = np.random.randint(5,101,n)
c = 9786
```

A continuación mostraremos dos algorimos basados en recocido simulado. El primero de ellos es un diseño sencillo y el segundo es un diseño más elaborado que obtiene mejores resultados.

### Función objetivo

Dado que la clase `SimulatedAnnealing` considera problemas de minimización, es necesario convertir el problema de la mochila a un problema de minimización. Para esto se multiplica el valor de la función objetivo por -1.

```python
def f(x : np.ndarray) -> float:
    global p
    return -1.0* np.dot(x,p)

```

### Restricciones
La restricción del problema de la mochila es seleccionar un número de objetos sin exceder la capacidad de la mochila.

```python
def g1(x : np.ndarray) -> bool:
    global w,c
    result = np.dot(x,w)
    g1.__doc__="{} <= {}".format(result,c)
    return result <= c

constraints_list= [g1]
```

### Solución inicial
Nuestra solución inicial es creada introduciendo objetos de manera aleatoria, mientras no se exceda el peso de la mochila.

```python
def getInitialSolution(NumObjects=15):
    global n,p,w,c
    #Empty backpack
    x = [0 for i in range(n)]
    weight_x = 0

    #Random order to insert objects.
    objects = list(range(n))
    np.random.shuffle(objects)

    for o in  objects[:NumObjects]:
        #Check the constraint about capacity.
        if weight_x + w[o] <= c:
            x[o] = 1
            weight_x += w[o]

    return np.array(x)
```

### Ejecución de la metaheurística

```python
class Knapsack_solver(SimulatedAnnealing):

    def __init__(self, f_ : function_type , constraints_: list):
        super().__init__(f_,constraints_)

    def get_neighbor(self, x : np.ndarray, **kwargs) -> np.ndarray:

        x_ = x.copy()
        N = len(x_)
        while(True):
            ind = random.randint(0, N-1)
            x_[ind] ^= 1

            if(self.is_valid(x_)):
                break

            x_[ind] ^= 1

        return np.array(x_)

    def update_temperature(self, **kwargs) -> float:
        return self.logger['temperature'] * 0.99
```

Parte clave de este algoritmo es la temperatura. La temperatura es una función que varía de acuerdo al tiempo $t$, donde, al inicio de la búsqueda permite aceptar con mayor probabilidad soluciones peores.

La clase `SimulatedAnnealing` define esta función como `update_temperature` que está implementada por defecto. En este ejemplo vamos a definir nuestra función de cambio lineal $T(t + 1) = \sigma T(t)$

La función `get_neighbor` tomará la solución $x$ (arreglo de 0's y 1's), donde, el valor de retorno será la solución $x$ con una posición aleatoria $i$ que será reemplazada por el valor 0 si la posición $x_i$ es 1, sino, será 1.

```python
Knapsack_ = Knapsack_solver(f, [g1])
Knapsack_.optimize(getInitialSolution,1000.0,0.1)
```

```python
print(Knapsack_)
```

    Simulated Annealing:
     f(X) = -16579.0
     X = [0 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0
     0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0
     0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0
     0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0
     0 0 0 1 0 1 0 0 0 0 1 0 0 0 1 1 0 1 1 0 0 0 1 0 1 0 1 0 0 0 0 0 0 1 1 0 0
     0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0
     1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0 0 1 0
     0 0 0 0 1 0 0 0 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0
     0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 0 0 1 0 0 0 0 0 1
     0 1 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0
     1 0 0 0 1 0 0 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0
     0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 1
     0 1 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0
     0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0
     0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 0 0 0 0 0
     0 0 0 0 1 1 1 1 0 1 0 0 0 1 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 1 1 0
     1 0 1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0
     0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1
     1 1 0 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 1 0
     0 0 1 0 0 0 1 0 0 1 0 0 0 1 1 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0
     0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0
     0 0 0 0 0 1 1 0 0 0 1 0 1 0 1 0 1 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 1 0
     1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 1
     1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 1 0 0 1 1 0 0 0 1 0 0 1 0 0 1
     0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0
     0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0
     0]
     Constraints:
     9785 <= 9786

Para realizar un estudio estadístico del comportamiento de la metaheurística, la librería Pyristic cuenta con una función llamada get_stats. Esta función se encuentra en utils.helpers y recibe como parámetros:

    Objeto que realiza la ejecución de la metaheurística.
    El número de veces que se quiere ejecutar la metaheurística.
    Tupla con los argumentos que recibe la función optimize.
    Argumentos adicionales a la búsqueda (opcional).

La función get_stats considera las solución regresada por la metaheurística en cada ejecución y retorna un diccionario con la mejor y peor solución encontrada, la media y desviación estándar del valor de la función objetivo.

```python
args = (getInitialSolution,1000.0,0.01)
statistics = get_stats(Knapsack_, 30, args)
```

```python
pprint(statistics)
```

    {'Best solution': {'f': -17760.0,
                       'x': array([0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0,
           0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0,
           1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0,
           0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0,
           0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0,
           0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
           0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0,
           1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0,
           1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
           0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1,
           1, 0, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0,
           0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1,
           0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0,
           0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
           1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0,
           0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0,
           0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0,
           1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0,
           0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
           0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0,
           0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1,
           0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
           0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1,
           0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0,
           0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1,
           0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0,
           1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0,
           1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1,
           0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0,
           0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 1, 0, 0])},
     'Mean': -16051.933333333332,
     'Median': -16045.0,
     'Standard deviation': 632.8604866442173,
     'Worst solution': {'f': -15000.0,
                        'x': array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1,
           1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0,
           0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0,
           1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1,
           0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1,
           0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
           1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0,
           0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
           1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
           0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0,
           0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0,
           0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0,
           0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
           0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,
           0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0,
           1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
           1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
           0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0,
           0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0,
           0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0,
           0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0,
           0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
           1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0,
           0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0,
           0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1,
           0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
           0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0,
           1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
           0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1,
           0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
           1, 0, 1, 0, 0, 0, 0, 0, 0, 1])},
     'averageTime': 0.038174986839294434,
     'objectiveFunction': [-15538.0,
                           -15342.0,
                           -15438.0,
                           -16765.0,
                           -16003.0,
                           -16903.0,
                           -15939.0,
                           -16588.0,
                           -15772.0,
                           -15795.0,
                           -16196.0,
                           -15438.0,
                           -17760.0,
                           -16736.0,
                           -15766.0,
                           -16294.0,
                           -16892.0,
                           -16508.0,
                           -15000.0,
                           -16388.0,
                           -15179.0,
                           -15987.0,
                           -16529.0,
                           -15322.0,
                           -15515.0,
                           -16163.0,
                           -16087.0,
                           -15020.0,
                           -16529.0,
                           -16166.0]}

Se puede observar que los resultados obtenidos no son tan buenos como los obtenidos con la metaheurística basada en Búsqueda Tabú. A continuación se harán algunas modificaciones en el diseño, con el objetivo de  mejorar los resultados.

### Solución inicial
En el diseño anterior, no se consideró el beneficio de introducir un objeto determinado a la mochila según su valor y peso. Una forma de hacerlo es la siguiente:

1. Asignar a cada objeto su valor por unidad de peso: $\frac{p_{i}}{w_{i}}$.

2. Ordenar todos los objetos con respecto a este indicador.

3. Seleccionar los objetos que tenga mayor valor en este indicador sin exceder la capacidad.

```python

def Init_GISP():
    global n,p,w,c
    Arr_=[]
    for i in range(n):
        Arr_.append((p[i]/w[i],i))

    Arr_.sort(reverse=True)
    return Arr_
GISP_Arr = Init_GISP()

def getInitialSolution2():
    global n,p,w,c, GISP_Arr
    X = [0 for i in range(n)]
    current_weight = 0
    for i in range(n):
        ind = GISP_Arr[i][1]
        if current_weight+ w[ind] <= c:
            current_weight+=w[ind]
            X[ind] = 1
    return np.array(X)
```

Para crear un nuevo vecino, seguiremos la misma estrategia que en el diseño anterior. Sin embargo, si la nueva solución no es válida vamos a realizar lo siguiente:

1. Fase de reparación.
2. Fase de mejoramiento.

**Fase de reparación:** En esta fase retiramos objetos que se encuentran en la mochila mientras la capacidad actual exceda el límite definido $c$. Los objetos son retirados de acuerdo a los valores más pequeños en el indicador definido ($\frac{p_{i}}{w_{i}}$).

**Fase de mejoramiento:** En esta fase se introducen objetos a la mochila siempre y cuando no excedan la capacidad. Estos objetos son  introducidos priorizando aquellos con los valores más grandes en el indicador definido ($\frac{p_{i}}{w_{i}}$).

```python
class Knapsack_solver2(SimulatedAnnealing):

    def __init__(self, f_ : function_type , constraints_: list):
        super().__init__(f_,constraints_)

    def generate_neighbor(self, x: np.ndarray) -> np.ndarray:
        x_ = x.copy()
        N = len(x_)
        ind = random.randint(0, N-1)
        x_[ind] ^= 1

        return x_

    def repair_neighbor(self, x: np.ndarray) -> np.ndarray:
        global GISP_Arr,w,c,n

        #Get the total weight
        total_weight=0
        for i in range(n):
            if x[i]:
                total_weight += w[i]

        for i in range(n):
            ind = GISP_Arr[n - (1+i)][1] #Lowest

            if x[ind]:
                total_weight -= w[ind]
                x[ind] = 0

            if total_weight <= c:
                break

    def improve_neighbor(self, x: np.ndarray) -> np.ndarray:
        global GISP_Arr,w,c,n

        #Get the total weight
        total_weight=0
        for i in range(n):
            if x[i]:
                total_weight += w[i]

        for i in range(n):
            ind = GISP_Arr[i][1]
            if total_weight + w[ind] > c:
                continue

            if x[ind] == 0 :
                total_weight += w[ind]
                x[ind] = 1

    def get_neighbor(self, x : np.ndarray) -> np.ndarray:

        neighbor_ = self.generate_neighbor(x)

        if(self.is_valid(neighbor_)):
            return neighbor_

        #RI strategy
        self.repair_neighbor(neighbor_)
        self.improve_neighbor(neighbor_)

        return neighbor_
```

Ejecutamos nuestra metaheurística:

```python
f(getInitialSolution2())
```

    -31626.0

```python
Knapsack_2 = Knapsack_solver2(f, [g1])
```

```python
Knapsack_2.optimize(getInitialSolution2,1000.0,0.1)
print(Knapsack_2)
```

    Simulated Annealing:
     f(X) = -31626.0
     X = [1 0 1 0 0 1 0 0 1 1 1 1 0 0 0 1 0 1 1 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0
     1 0 0 0 1 0 1 0 0 0 1 1 0 0 1 1 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 1 0 0 0 1
     0 0 1 1 0 1 1 0 0 1 1 1 1 1 1 0 1 0 0 1 1 1 1 0 1 0 0 1 0 1 1 1 0 0 0 0 1
     0 1 0 1 1 0 0 1 0 0 0 0 0 1 0 1 0 0 0 1 0 0 0 0 1 1 0 1 1 0 1 1 0 1 0 1 0
     1 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 0 1 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 1
     1 1 0 1 0 0 0 1 1 0 1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0
     0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 1 0 1 1 0 0 1 0 0
     0 1 0 0 1 0 0 0 0 1 1 1 0 1 0 0 0 0 0 0 0 1 1 1 0 0 1 0 0 1 0 0 1 0 0 1 1
     1 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0 1 1 0 1 0 1 1 1 0 1 1 1 0 0 1 0 0 1 1 0 0
     0 0 1 1 0 1 0 0 0 1 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 0 0 1 1 1 0 0 1 0 0 0 1
     0 0 1 1 0 1 1 1 1 1 0 1 0 1 1 1 0 0 0 0 0 0 0 0 1 0 0 1 1 0 0 1 1 1 0 1 1
     0 0 1 1 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 0 0 1 1 0 0 1 0 0 1 0 1 1 1 0
     0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 0 0
     1 0 1 1 0 0 0 1 1 1 0 0 1 0 1 0 1 1 0 1 1 0 1 1 0 1 1 0 0 1 1 0 0 1 0 1 0
     0 0 0 0 0 0 0 0 1 1 0 0 0 1 1 0 1 1 0 0 0 0 1 0 1 1 0 0 0 1 1 1 0 0 0 1 0
     0 0 1 0 0 0 1 1 0 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 0 0 1 1 0 0 1 0 0 0
     1 0 1 0 0 0 1 1 0 0 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1
     0 0 0 0 0 0 0 1 1 1 1 0 1 1 1 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0
     1 1 0 0 0 0 1 0 0 0 1 1 1 0 0 1 0 1 0 0 0 1 1 1 0 0 1 1 0 0 1 1 1 0 0 1 0
     1 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 1 0 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0
     1 1 1 1 1 0 1 0 1 0 1 0 1 1 1 1 1 0 0 1 0 1 0 0 1 0 0 1 1 0 0 1 0 0 0 0 1
     1 0 1 1 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 1 0 1 0
     0 1 1 0 1 1 0 1 0 1 1 0 1 0 1 0 1 0 1 0 0 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0
     1 1 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 0 1 0 1 1 0 1 0 1
     1 0 0 1 0 1 0 1 0 1 0 0 1 1 0 0 1 1 1 1 1 0 1 0 0 1 1 0 0 1 1 0 0 1 0 1 0
     1 0 1 0 0 1 1 0 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 0 0 0 1 0 1 0 1 0 0 0 0 1 0
     1 1 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 1 0 0 0 0 0 0 1 1 0 0 1 0 0 1 0 1 0 1 0
     1]
     Constraints:
     9774 <= 9786

## Resultados
Vamos a comparar las distintas combinaciones entre la forma de generar la solución inicial y la estrategia de generar un nuevo vecino.

1. Solución por Indicador y RI estrategia.
2. Solución por Indicador y estrategia ingenua.
3. Solución ingenua y RI estrategia.
4. Solución ingenua y estrategia ingenua.

### Solución por Indicador y RI estrategia.

```python
args = (getInitialSolution2,1000.0,0.01)
statistics = get_stats(Knapsack_2, 30, args)
```

```python
print("f(x*) = {} \nfunción objetivo promedio: {} \nfunción objetivo de la peor solución: {}".format(statistics['Best solution']['f'],statistics['Mean'],statistics['Worst solution']['f']))
```

    f(x*) = -31644.0
    función objetivo promedio: -31630.5
    función objetivo de la peor solución: -31626.0

### Solución por Indicador y estrategia ingenua.

```python
args = (getInitialSolution2,1000.0,0.01)
statistics = get_stats(Knapsack_, 30, args)
```

```python
print("f(x*) = {} \nfunción objetivo promedio: {} \nfunción objetivo de la peor solución: {}".format(statistics['Best solution']['f'],statistics['Mean'],statistics['Worst solution']['f']))
```

    f(x*) = -31626.0
    función objetivo promedio: -31626.0
    función objetivo de la peor solución: -31626.0

### Solución ingenua y RI estrategia.

```python
args = (getInitialSolution,1000.0,0.01)
statistics = get_stats(Knapsack_2, 30, args)
```

```python
print("f(x*) = {} \nfunción objetivo promedio: {} \nfunción objetivo de la peor solución: {}".format(statistics['Best solution']['f'],statistics['Mean'],statistics['Worst solution']['f']))
```

    f(x*) = -31401.0
    función objetivo promedio: -31269.3
    función objetivo de la peor solución: -31143.0

### Solución ingenua y estrategia ingenua.

```python
args = (getInitialSolution,1000.0,0.01)
statistics = get_stats(Knapsack_, 30, args)
```

```python
print("f(x*) = {} \nfunción objetivo promedio: {} \nfunción objetivo de la peor solución: {}".format(statistics['Best solution']['f'],statistics['Mean'],statistics['Worst solution']['f']))
```

    f(x*) = -17258.0
    función objetivo promedio: -16229.833333333334
    función objetivo de la peor solución: -15228.0


# Regression revisited

# Purpose
Does the equation need to be decoupled?

# Setup

```python
# %load imports.py
## Local packages:

%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)

import numpy as np
import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl

from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.plot import track_plot
from src.equation import Equation

## Load models:
# (Uncomment these for faster loading):
#import src.models.vmm_simple  as vmm
import src.models.vmm_martin  as vmm
#import src.models.vmm_abkowitz as vmm

```

```python
u1d,v1d,r1d = sp.symbols('u1d, v1d, r1d')
u_,v_,r_ = sp.symbols('u, v, r')

X_qs_, Y_qs_, N_qs_ = sp.symbols('X_qs, Y_qs, N_qs')

X_eq = vmm.X_eom.subs([
    (X_force,sp.solve(vmm.fx_eq,X_force)[0]),
])

Y_eq = vmm.Y_eom.subs([
    (Y_force,sp.solve(vmm.fy_eq,Y_force)[0]),
])

N_eq = vmm.N_eom.subs([
    (N_force,sp.solve(vmm.mz_eq,N_force)[0]),
])

subs = [
    #(X_qs,X_qs_),
    #(Y_qs,Y_qs_),
    #(N_qs,N_qs_),

    (u1d,u1d),
    (v1d,v1d),
    (r1d,r1d),

    (u,u_),
    (v,v_),
    (r,r_),

]
eq_X_ = X_eq.subs(subs)
eq_Y_ = Y_eq.subs(subs)
eq_N_ = N_eq.subs(subs)

A,b = sp.linear_eq_to_matrix([eq_X_,eq_Y_,eq_N_],[u1d,v1d,r1d])
```

```python
A
```

```python
b
```

Simulation model can be expressed as:

```python
acceleration = sp.matrices.MutableDenseMatrix([u1d,v1d,r1d])
sp.Eq(sp.UnevaluatedExpr(A)*sp.UnevaluatedExpr(acceleration),sp.UnevaluatedExpr(b))
```

The acceleartion can be calculated by calculating the inverse of the mass matrix:

```python
sp.Eq(sp.UnevaluatedExpr(acceleration),sp.UnevaluatedExpr(sp.simplify(A.inv()))*sp.UnevaluatedExpr(b))
```

This expression can be formulated in a more clean way by introducing the following term:

```python
A_inv = A.inv()
S = sp.symbols('S')
eq_S=sp.Eq(S,-sp.fraction(A_inv[1,1])[1])
eq_S
```

```python
A_inv_S = A_inv.subs(eq_S.rhs,S)
sp.Eq(sp.UnevaluatedExpr(acceleration),sp.UnevaluatedExpr(A_inv_S)*sp.UnevaluatedExpr(b))
```

Time integration of this expression will simulate the system.

This expression can also be formulated as a regression problem to identify the hydrodynamic coefficients within $X_{qs}$, $Y_{qs}$, $N_{qs}$ which are the quasi static force model. This is usually the part that differs between various simulation models, such as the Abkowitz model, the SeaMan model etc.

The regression problem is expressed as:

```python
coeff_matrix = sp.matrices.MutableDenseMatrix([A_coeff*X_X,B_coeff*X_Y,C_coeff*X_N])
eq_regression = sp.Eq(acceleration, coeff_matrix)
eq_regression
```

where $A_{coeff}$, $B_{coeff}$ and $C_{coeff}$ are coefficient vectors that should be regressed based on the feature vectors: $X_X$, $X_Y$ and $X_N$. These coefficient vectors cannot be used directly in the simulation model above. In order to derive how these coefficient vectors should be transformed, the acceleration predicted with this regression can be inserted into the equation of motion:

```python
Math(vlatex(sp.Eq(sp.UnevaluatedExpr(A)*sp.UnevaluatedExpr(coeff_matrix),sp.UnevaluatedExpr(b))))
```

The right hand side of this equation is instead expressed with matrices:

```python
eq_X_qs = sp.Eq(X_qs_*X_X, b[0])
Math(vlatex(eq_X_qs))
```

```python
eq_Y_qs = sp.Eq(X_qs_*X_Y, b[1])
Math(vlatex(eq_Y_qs))
```

```python
eq_N_qs = sp.Eq(N_qs_*X_N, b[2])
Math(vlatex(eq_N_qs))
```

```python
subs=[
    (eq_X_qs.rhs,eq_X_qs.lhs),
    (eq_Y_qs.rhs,eq_Y_qs.lhs),
    (eq_N_qs.rhs,eq_N_qs.lhs),

]
Math(vlatex(sp.Eq(sp.UnevaluatedExpr(A)*sp.UnevaluatedExpr(coeff_matrix),sp.UnevaluatedExpr(b.subs(subs)))))
```

So that the parameter vectors $X_{qs}$, $Y_{qs}$, $N_{qs}$ can be calculated from $A_{coeff}$, $B_{coeff}$ and $C_{coeff}$ in the following way.

```python
Math(vlatex(sp.Eq(sp.UnevaluatedExpr(A)*sp.UnevaluatedExpr(sp.matrices.MutableDenseMatrix([A_coeff,B_coeff,C_coeff])),
                  sp.UnevaluatedExpr(sp.matrices.MutableDenseMatrix([X_qs_,Y_qs_,N_qs_])))))
```

Please not that the transformed coefficient vectors: $X_{qs}$, $Y_{qs}$, $N_{qs}$ now also contain inertial forces such as as centrigal force etc. (I think).

```python
subs = {value:key for key,value in p.items()}
A_ = A*sp.matrices.MutableDenseMatrix([A_coeff,B_coeff,C_coeff])
A_lambda=lambdify(A_.subs(subs))
```

## Load test

```python
#id=22773
#id=22616
id=22774
#id=22770

df, units, meta_data = mdl.load(id=id, dir_path='../data/processed/kalman')
df.index = df.index.total_seconds()
df = df.iloc[0:-100].copy()
df.index-=df.index[0]
df['t'] = df.index
df.sort_index(inplace=True)
df['-delta'] = -df['delta']
df['V'] = np.sqrt(df['u']**2 + df['v']**2)

df['thrust'] = df['Prop/PS/Thrust'] + df['Prop/SB/Thrust']
df['U'] = df['V']
df['beta'] = -np.arctan2(df['v'],df['u'])
```

```python
meta_data['rho']=1000
meta_data['mass'] = meta_data['Volume']*meta_data['rho']
```

# Ship parameters

```python
T_ = (meta_data.TA + meta_data.TF)/2
L_ = meta_data.lpp
m_ = meta_data.mass
rho_ = meta_data.rho
B_ = meta_data.beam
CB_ = m_/(T_*B_*L_*rho_)
I_z_ = m_*meta_data.KZZ**2
#I_z_ = 900

ship_parameters = {
        'T' : T_,
        'L' : L_,
        'CB' :CB_,
        'B' : B_,
        'rho' : rho_,
        'x_G' : 0,  # motions are expressed at CG
        'm' : m_,
        'I_z': I_z_,
        'volume':meta_data.Volume,
    }

ps = prime_system.PrimeSystem(**ship_parameters)  # model

scale_factor = meta_data.scale_factor
ps_ship = prime_system.PrimeSystem(L=ship_parameters['L']*scale_factor, rho=meta_data['rho'])  # ship
```

# Brix parameters

```python
def calculate_prime(row, ship_parameters):
    return run(function=row['brix_lambda'], inputs=ship_parameters)

mask = df_parameters['brix_lambda'].notnull()
df_parameters.loc[mask,'brix_prime'] = df_parameters.loc[mask].apply(calculate_prime, ship_parameters=ship_parameters, axis=1)

df_parameters['brix_prime'].fillna(0, inplace=True)
#df_parameters['brix_SI'].fillna(0, inplace=True)

```

```python
df_parameters['prime'] = df_parameters['brix_prime']

def to_SI(row):

    prime = row['prime']
    denominator = row['denominator']
    lamda = lambdify(denominator)
    try:
        denominator_value = run(lamda, inputs=ship_parameters)
        SI = prime*denominator_value
    except:
        return None
    else:
        return SI

df_parameters['brix_SI'] = df_parameters.apply(to_SI, axis=1)

```

# Regression

```python
from statsmodels.sandbox.regression.predstd import wls_prediction_std
def show_pred(X,y,results, label):

    display(results.summary())

    X_ = X
    y_ = y
    y_pred = results.predict(X_)

    prstd, iv_l, iv_u = wls_prediction_std(results, exog=X_, alpha=0.05)
    #iv_l*=-1
    #iv_u*=-1

    fig,ax=plt.subplots()
    ax.plot(X_.index,y_, label='Numerical gradient from model test')
    ax.plot(X_.index,y_pred, '--', label='OLS')
    ax.set_ylabel(label)

    ax.fill_between(X_.index, y1=iv_l, y2=iv_u, zorder=-10, color='grey', alpha=0.5, label=r'5\% confidence')
    ax.legend();
```

## N

```python
N_ = sp.symbols('N_')

diff_eq_N = regression.DiffEqToMatrix(ode=vmm.N_qs_eq.subs(N_qs,N_),
                                      label=N_, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_N.acceleration_equation))
```

```python
X = diff_eq_N.calculate_features(data=df)
y = diff_eq_N.calculate_label(y=df['r1d'])

model_N = sm.OLS(y,X)
results_N = model_N.fit()

show_pred(X=X,y=y,results=results_N, label=r'$\dot{r}$')
```

## Y

```python
Y_ = sp.symbols('Y_')
diff_eq_Y = regression.DiffEqToMatrix(ode=vmm.Y_qs_eq.subs(Y_qs,Y_),
                                      label=Y_, base_features=[delta,u,v,r])
```

```python
Math(vlatex(diff_eq_Y.acceleration_equation))
```

```python
X = diff_eq_Y.calculate_features(data=df)
y = diff_eq_Y.calculate_label(y=df['v1d'])

model_Y = sm.OLS(y,X)
results_Y = model_Y.fit()

show_pred(X=X,y=y,results=results_Y, label=r'$\dot{v}$')
```

## X

```python
X_ = sp.symbols('X_')
diff_eq_X = regression.DiffEqToMatrix(ode=vmm.X_qs_eq.subs(X_qs,X_),
                                      label=X_, base_features=[delta,u,v,r,thrust])
```

```python
diff_eq_X.acceleration_equation
```

```python
X = diff_eq_X.calculate_features(data=df)
y = diff_eq_X.calculate_label(y=df['u1d'])

model_X = sm.OLS(y,X)
results_X = model_X.fit()

show_pred(X=X,y=y,results=results_X, label=r'$\dot{u}}$')
```

```python
results_summary_X = regression.results_summary_to_dataframe(results_X)
results_summary_Y = regression.results_summary_to_dataframe(results_Y)
results_summary_N = regression.results_summary_to_dataframe(results_N)
```

```python
A_coeff_ = results_summary_X['coeff']
B_coeff_ = results_summary_Y['coeff']
C_coeff_ = results_summary_N['coeff']

coeffs = run(A_lambda,A_coeff=A_coeff_.values, B_coeff=B_coeff_.values, C_coeff=C_coeff_.values,
    **df_parameters['brix_SI'], **ship_parameters)
```

```python
results_summary_X['decoupled'] = coeffs[0][0]
results_summary_Y['decoupled'] = coeffs[1][0]
results_summary_N['decoupled'] = coeffs[2][0]

```

```python
A_coeff_*(m_-df_parameters.loc['Xudot','brix_SI'])
```

```python
A_coeff_
```

## Add the regressed parameters
Hydrodynamic derivatives that depend on acceleration cannot be obtained from the VCT regression. They are however essential if a time simulation should be conducted. These values have then been taken from Brix semi empirical formulas for the simulations below.

```python
df_parameters_all = df_parameters.copy()
for other in [results_summary_X, results_summary_Y, results_summary_N]:
    df_parameters_all = df_parameters_all.combine_first(other)

df_parameters_all.rename(columns={'decoupled':'regressed'}, inplace=True)
df_parameters_all.drop(columns=['brix_lambda'], inplace=True)

df_parameters_all['SI'] = df_parameters_all['regressed'].combine_first(df_parameters_all['brix_SI'])  # prefer regressed
df_parameters_all['SI'].fillna(0,inplace=True)
```

# Simulate forces

```python
df['X_qs_pred'] = run(function=vmm.simulator.X_qs_lambda, inputs=df, **df_parameters_all['SI'])
df['Y_qs_pred'] = run(function=vmm.simulator.Y_qs_lambda, inputs=df, **df_parameters_all['SI'])
df['N_qs_pred'] = run(function=vmm.simulator.N_qs_lambda, inputs=df, **df_parameters_all['SI'])

```

```python
X = diff_eq_X.calculate_features(data=df)
X_forces = X*results_summary_X['coeff']
X_forces.index = df.index

X = diff_eq_Y.calculate_features(data=df)
Y_forces = X*results_summary_Y['coeff']
Y_forces.index = df.index

X = diff_eq_N.calculate_features(data=df)
N_forces = X*results_summary_N['coeff']
N_forces.index = df.index

```

```python
display(px.line(X_forces, y=X_forces.columns, width=1000, height=400))
display(px.line(Y_forces, y=Y_forces.columns, width=1000, height=400))
display(px.line(N_forces, y=N_forces.columns, width=1000, height=400))
```

# Simulation

The way that the regression is formulated, inertial forces, such as centrifugal force will be included into the derivatives (I think) which means that centrifugal force : $-m \cdot r \cdot u$ will be included into $Y_{ur}$ coefficient. This coefficient is therefore not pure hydrodynamic, and can potentially be counted twice..?
The coefficients are recalculated below to avooid this:

```python
parameters=df_parameters_all['SI'].copy()
x_G_ = ship_parameters['x_G']
parameters['Xrr']+=(-m_*x_G_)
parameters['Xvr']+=(-m_)
parameters['Yur']+=m_
parameters['Nur']+=m_*x_G_

```

```python
result_vmm = vmm.simulator.simulate(df_=df, parameters=parameters, ship_parameters=ship_parameters)
```

```python
result_vmm.plot_compare()
```

```python

```


# Realization of Non-Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Fast Convolution

The straightforward convolution of two finite-length signals $x[k]$ and $h[k]$ is a numerical complex task. This has led to the development of various techniques with considerably lower complexity. The basic concept of the *fast convolution* is to exploit the correspondence between the convolution and the scalar multiplication in the frequency domain.

### Convolution of Finite-Length Signals

The convolution of a causal signal $x_L[k]$ of length $L$ with a causal impulse response $h_N[k]$ of length $N$ is given as

\begin{equation}
y[k] = x_L[k] * h_N[k] = \sum_{\kappa = 0}^{L-1} x_L[\kappa] \; h_N[k - \kappa] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \; x_L[k - \kappa]
\end{equation}

The resulting signal $y[k]$ is of finite length $M = N+L-1$. The computation of $y[k]$ for $k=0,1, \dots, M-1$ requires $M \cdot N$ multiplications and $M \cdot (N-1)$ additions. The computational complexity of the convolution is consequently [in the order of](https://en.wikipedia.org/wiki/Big_O_notation) $\mathcal{O}(M \cdot N)$. Discrete-time Fourier transformation (DTFT) of above relation yields

\begin{equation}
Y(e^{j \Omega}) = X_L(e^{j \Omega}) \cdot H_N(e^{j \Omega})
\end{equation}

Discarding the effort of transformation, the computationally complex convolution is replaced by a scalar multiplication with respect to the frequency $\Omega$. However, $\Omega$ is a continuous frequency variable which limits the numerical evaluation of this scalar multiplication. In practice, the DTFT is replaced by the discrete Fourier transformation (DFT). Two aspects have to be considered before a straightforward application of the DFT

1. The DFTs $X_L[\mu]$ and $H_N[\mu]$ are of length $L$ and $N$ respectively and cannot be multiplied straightforward

2. For $N = L$, the multiplication of the two spectra $X_L[\mu]$ and $H_L[\mu]$ would result in the [periodic/circular convolution](https://en.wikipedia.org/wiki/Circular_convolution) $x_L[k] \circledast h_L[k]$ due to the periodicity of the DFT. Since we aim at realizing the linear convolution $x_L[k] * h_N[k]$ with the DFT, special care has to be taken to avoid cyclic effects.

### Linear Convolution by Periodic Convolution

The periodic convolution of the two signals $x_L[k]$ and $h_N[k]$ is defined as

\begin{equation}
x_L[k] \circledast h_N[k] = \sum_{\kappa=0}^{M-1} \tilde{x}_M[k - \kappa] \; h_N[\kappa]
\end{equation}

where without loss of generality it is assumed that $L \geq N$ and $M \geq N$. The periodic continuation $\tilde{x}_M[k]$ of $x[k]$ with period $M$ is given as

\begin{equation}
\tilde{x}_M[k] = \sum_{m = -\infty}^{\infty} x_L[m \cdot M + k]
\end{equation}

The result of the circular convolution has a periodicity of $M$.

To compute the linear convolution by the periodic convolution one has to take care that the result of the linear convolution fits into one period of the periodic convolution. Hence, the periodicity has to be chosen as $M \geq N+L-1$. This can be achieved by zero-padding of $x_L[k]$ and $h_N[k]$ to a total length of $M$

\begin{align}
x_M[k] &= \begin{cases}
x_L[k] & \mathrm{for} \; k=0, 1, \dots, L-1 \\
0 & \mathrm{for} \; k=L, L+1, \dots, M-1
\end{cases}
\\
h_M[k] &= \begin{cases}
h_N[k] & \mathrm{for} \; k=0, 1, \dots, N-1 \\
0 & \mathrm{for} \; k=N, N+1, \dots, M-1
\end{cases}
\end{align}

This results in the desired equality of linear and periodic convolution

\begin{equation}
x_L[k] * h_N[k] = x_M[k] \circledast h_M[k]
\end{equation}

for $k = 0,1,\dots, M-1$ with $M = N+L-1$.

#### Example

The following example computes the linear, periodic and linear by periodic convolution of a rectangular signal $x[k] = \text{rect}_L[k]$ of length $L$ with a triangular signal $h[k] = \Lambda_N[k]$ of length $N$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

L = 32  # length of signal x[k]
N = 16  # length of signal h[k]
M = 16  # periodicity of periodic convolution

def cconv(x, y, P):
    # Periodic convolution with period P of two signals x and y
    x = _wrap(x, P)
    h = _wrap(y, P)

    return np.fromiter([np.dot(np.roll(x[::-1], k+1), h) for k in np.arange(P)], float)

def _wrap(x, N):
    # Zero-padding to length N or periodic summation with period N
    M = len(x)
    rows = int(np.ceil(M/N))

    if (M < int(N*rows)):
        x = np.pad(x, (0, int(N*rows-M)), 'constant')

    x = np.reshape(x, (rows, N))

    return np.sum(x, axis=0)

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# periodic convolution
y2 = cconv(x, h, M)
# linear convolution via periodic convolution
xp = np.append(x, np.zeros(N-1))
hp = np.append(h, np.zeros(L-1))
y3 = cconv(xp, hp, L+N-1)

# plot results
def plot_signal(x):
    plt.figure(figsize = (10, 3))
    plt.stem(x)
    plt.xlabel(r'$k$')
    plt.ylabel(r'$y[k]$')
    plt.axis([0, N+L, 0, 1.1*x.max()])

plot_signal(x)
plt.title('Signal $x[k]$')

plot_signal(y1)
plt.title('Linear convolution')

plot_signal(y2)
plt.title('Periodic convolution with period M = %d' %M)

plot_signal(y3)
plt.title('Linear convolution by periodic convolution');
```

**Exercise**

* Change the lengths `L`, `N` and `M` within the constraints given above and check how the results for the different convolutions change

### The Fast Convolution

Using the above derived equality of the linear and periodic convolution one can express the linear convolution $y[k] = x_L[k] * h_N[k]$ by the DFT as

\begin{equation}
y[k] = \text{IDFT}_M \{ \; \text{DFT}_M\{ x_M[k] \} \cdot \text{DFT}_M\{ h_M[k] \} \; \}
\end{equation}

This operation requires three DFTs of length $M$ and $M$ complex multiplications. On first sight this does not seem to be an improvement, since one DFT/IDFT requires $M^2$ complex multiplications and $M \cdot (M-1)$ complex additions. The overall numerical complexity is hence in the order of $\mathcal{O}(M^2)$. The DFT can be realized efficiently by the [fast Fourier transformation](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (FFT), which lowers the computational complexity to $\mathcal{O}(M \log_2 M)$. The resulting algorithm is known as *fast convolution* due to its computational efficiency.

The fast convolution algorithm is composed of the following steps

1. Zero-padding of the two input signals $x_L[k]$ and $h_N[k]$ to at least a total length of $M \geq N+L-1$

2. Computation of the DFTs $X[\mu]$ and $H[\mu]$ using a FFT of length $M$

3. Multiplication of the spectra $Y[\mu] = X[\mu] \cdot H[\mu]$

4. Inverse DFT of $Y[\mu]$ using an inverse FFT of length $M$

The overall complexity depends on the particular implementation of the FFT. Many FFTs are most efficient for lengths which are a power of two. It therefore can make sense, in terms of computational complexity, to choose $M$ as a power of two instead of the shortest possible length $N+L-1$. For real valued signals $x[k] \in \mathbb{R}$ and $h[k] \in \mathbb{R}$ the computational complexity can be reduced significantly by using a real valued FFT.

#### Example

The implementation of the fast convolution algorithm is straightforward. Most implementations of the FFT include the zero-padding to a given length $M$, e.g in `numpy` by `numpy.fft.fft(x, M)`. In the following example an implementation of the fast convolution in `Python` is shown. The output of the fast convolution is compared to a straightforward implementation by means of the absolute difference $|e[k]|$. The observed differences are due to numerical effects in the convolution and the FFT. The differences can be neglected in most applications.

```python
L = 16  # length of signal x[k]
N = 16  # length of signal h[k]
M = N+L-1

# generate signals
x = np.ones(L)
h = sig.triang(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# fast convolution
y2 = np.fft.ifft(np.fft.fft(x, M)*np.fft.fft(h, M))

plt.figure(figsize=(10, 3))
plt.stem(np.abs(y1-y2))
plt.xlabel(r'k')
plt.ylabel(r'|e[k]|');
```

#### Numerical Complexity

It was already argued that the numerical complexity of the fast convolution is considerably lower due to the usage of the FFT. The gain with respect to the convolution is evaluated in the following. In order to measure the execution times for both algorithms the `timeit` module is used. The algorithms are evaluated for the convolution of two signals $x_L[k]$ and $h_N[k]$ of length $L=N=2^n$ for $n=0, 1, \dots, 16$.

```python
import timeit

n = np.arange(17)  # lengths = 2**n to evaluate
reps = 20  # number of repetitions for timeit

gain = np.zeros(len(n))
for N in n:
    length = 2**N
    # setup environment for timeit
    tsetup = 'import numpy as np; from numpy.fft import rfft, irfft; \
            x=np.random.randn(%d); h=np.random.randn(%d)' % (length, length)
    # direct convolution
    tc = timeit.timeit('np.convolve(x, x, "full")', setup=tsetup, number=reps)
    # fast convolution
    tf = timeit.timeit('irfft(rfft(x, %d) * rfft(h, %d))' % (2*length, 2*length), setup=tsetup, number=reps)
    # speedup by using the fast convolution
    gain[N] = tc/tf

# show the results
plt.figure(figsize = (15, 10))
plt.barh(n-.5, gain, log=True)
plt.plot([1, 1], [-1, n[-1]+1], 'r-')
plt.yticks(n, 2**n)
plt.xlabel('Gain of fast convolution')
plt.ylabel('Length of signals')
plt.title('Comparison between direct/fast convolution')
plt.grid()
```

**Exercise**

* When is the fast convolution more efficient/faster than a direct convolution?
* Why is it slower below a given signal length?
* Is the trend of the gain as expected by the numerical complexity of the FFT?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Exponentialfunktionen lösen

```python
from sympy import *
```

```python
x = symbols('x')
l = 3*5**(x-2)
r = 7**(4-x)
g = Eq(l,r)
g

```

$\displaystyle 3 \cdot 5^{x - 2} = 7^{4 - x}$

```python
e = solveset(g)
e
```

$\displaystyle \left\{- \frac{- 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)}}{\log{\left(5 \right)} + \log{\left(7 \right)}}\right\}$

```python
N(e)
```

$\displaystyle \left\{2.78563560054719\right\}$

### Manuelle, schrittweise Lösung

```python
g
```

$\displaystyle 3 \cdot 5^{x - 2} = 7^{4 - x}$

 1 Schritt

Auf beiden Seiten der Gleichung wird der Log angewendet.

```python
l = log(l)
r = log(r)
Eq(l,r)
```

$\displaystyle \log{\left(3 \cdot 5^{x - 2} \right)} = \log{\left(7^{4 - x} \right)}$

 2 Schritt

Auf der linken Seite der Gleichung die 1. Logarithmus-Regel anwenden.

$log(a \cdot b) = log(a) + log(b)$

```python
l = log(3) + log(5**(x-2))
Eq(l,r)
```

$\displaystyle \log{\left(5^{x - 2} \right)} + \log{\left(3 \right)} = \log{\left(7^{4 - x} \right)}$

 3 Schritt

Auf beiden Seiten der Gleichung die 3. Logarithmus-Regel anwenden.

$log(a^b) = b \cdot log(a)$

```python
l = (x-2)*log(5)+log(3)
r = (4-x)*log(7)
Eq(l,r)
```

$\displaystyle \left(x - 2\right) \log{\left(5 \right)} + \log{\left(3 \right)} = \left(4 - x\right) \log{\left(7 \right)}$

 4 Schritt

Klammern durch Distributionsregel auflösen.

$a \cdot (b + c) = ab + ac$

```python
l = factor(l)
r = 4*log(7)-x*log(7)
Eq(l,r)
```

$\displaystyle x \log{\left(5 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)} = - x \log{\left(7 \right)} + 4 \log{\left(7 \right)}$

 5 Schritt

Alle $xlog$ auf eine Seite bringen, ...

```python
l -= x*log(5)
r -= x*log(5)
Eq(l,r)

```

$\displaystyle - 2 \log{\left(5 \right)} + \log{\left(3 \right)} = - x \log{\left(7 \right)} - x \log{\left(5 \right)} + 4 \log{\left(7 \right)}$

... alle $log$ auf die andere Seite.

```python
l -= 4*log(7)
r -= 4*log(7)
Eq(l,r)
```

$\displaystyle - 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)} = - x \log{\left(7 \right)} - x \log{\left(5 \right)}$

 6 Schritt

Klammer für $xlog$ durch umgekehrte Distributionsregel setzen.

```python
r = factor(r)
Eq(l,r)
```

$\displaystyle - 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)} = - x \left(\log{\left(5 \right)} + \log{\left(7 \right)}\right)$

 7 Schritt

x rechts isolieren, indem beide Seiten durch die rechte Klammer geteilt werden.

```python
r /= (log(5) + log(7))
l /= (log(5) + log(7))
Eq(l,r)
```

$\displaystyle \frac{- 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)}}{\log{\left(5 \right)} + \log{\left(7 \right)}} = - x$

 8 Schritt

Beiden Seiten mit - 1 multipizieren, um aus $-x$ ein $x$ zu machen.

```python
l *= -1
r *= -1
Eq(l,r)
```

$\displaystyle - \frac{- 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)}}{\log{\left(5 \right)} + \log{\left(7 \right)}} = x$

 9 Schritt

Ergebnis im Taschenrechner ausrechnen

```python
N(Eq(l,r))
```

$\displaystyle 2.78563560054719 = x$

```python
e
```

$\displaystyle \left\{- \frac{- 4 \log{\left(7 \right)} - 2 \log{\left(5 \right)} + \log{\left(3 \right)}}{\log{\left(5 \right)} + \log{\left(7 \right)}}\right\}$


```python
import numpy as np
import numpy.linalg as la
import sympy as sp
```

```python
def gradient(formula, symbols, values=None):
    '''
    Given a SymPy formula and variables
    Find its analytic gradient without substituion
    as a list of SymPy formulae or numerical gradient
    if values specified
    '''
    size = len(symbols)
    gradient = []
    for i in range(size):
        gradient.append(formula.diff(symbols[i]))

    if values == None:
        return gradient

    # Make sure you don't mess up the analytical gradient
    g_copy = gradient.copy()
    gradient_at = []
    for i in range(len(g_copy)):
        for j in range(len(symbols)):
            g_copy[i] = g_copy[i].subs(symbols[j], values[j])
        gradient_at.append(float(g_copy[i].evalf()))
    return gradient_at
```

```python
def subs_all(formula, variables, values):
    '''
    You know what, it's getting to the point where this function
    is necessary
    '''
    result = formula
    for i in range(len(values)):
        result = result.subs(variables[i], values[i])
    return float(result.evalf())
def hessian(gradient, symbols, values=None):
    '''
    Given an analytic gradient and variables
    Calculate its analytic Hessian
    or numerical Hessian if values are specified
    '''
    size = len(symbols)
    hessian = []
    for i in range(size):
        hessian.append([0]*size)
    for i in range(size):
        for j in range(size):
            hessian[i][j] = gradient[i].diff(symbols[j])

    if values == None:
        return hessian

    hessian_at = hessian.copy()
    size = len(hessian)
    for i in range(size):
        for j in range(size):
            hessian_at[i][j] = subs_all(hessian_at[i][j], symbols, [float(v) for v in values])
#             for k in range(len(symbols)):
#                 hessian_at[i][j] = hessian_at[i][j].subs(symbols[k], values[k])
#             hessian_at[i][j] = float(hessian_at[i][j].evalf())
    return hessian_at
```

```python
def newton_nd_optimization_crude_step(formula, symbols, x_prev):
    x_prev = list(x_prev)
    grad = gradient(formula, symbols)
    neg_gradient_at = -1 * np.array(gradient(formula, symbols, values=x_prev))
    hes_at = np.array(hessian(grad, symbols, values=x_prev))
    return np.array(x_prev) + la.solve(hes_at, neg_gradient_at)
```

```python
def newton_nd_optimization_crude(f_str, s_str, start, tolerance, actual_solution):
    '''
    A crude version of Newton ND Optimization algorithm
    Maybe you can help out implementing an analytic solution
    Returning the numerical solution as well as the number of
    iterations it took
    '''
    formula = sp.sympify(f_str)
    symbols = sp.symbols(s_str)
    curr = np.copy(start)
    iterations = 0
    while (la.norm(curr - actual_solution, 2) > tolerance):
        curr = newton_nd_optimization_crude_step(formula, symbols, curr)
        iterations += 1
    return curr, iterations
```

```python
# Test case set-up
form = sp.sympify('12*x**2+10*x*y+12*y**2+8*E**(9*x*y)+8*(sin(y)**2)+9*cos(x*y)') # 记住，大E才是natural number
x, y = sp.symbols('x y')
tolerance = 10**-7
# Test Case: Gradient
grad_at = gradient(form, (x, y), values=[0, 1])
expected = np.array([
    float(form.diff(x).subs(x, 0).subs(y, 1).evalf()),
    float(form.diff(y).subs(x, 0).subs(y, 1).evalf())
])
actual = np.array(grad_at)
assert la.norm(expected - actual, 2) < tolerance
```

```python
# Test Case: Hessian
hes = hessian(gradient(form, (x, y)), (x, y), values=[0, 1])
expected = np.array([
    [
        float(form.diff(x).diff(x).subs(x, 0).subs(y, 1).evalf()),
        float(form.diff(x).diff(y).subs(x, 0).subs(y, 1).evalf())
    ], [
        float(form.diff(y).diff(x).subs(x, 0).subs(y, 1).evalf()),
        float(form.diff(y).diff(y).subs(x, 0).subs(y, 1).evalf())
    ]
])
actual = np.array(hes)
assert la.norm(expected - actual, 2) < tolerance
```

```python
# Test Case: Newton ND step
H = np.array([
    [663, 82],
    [82, -16*np.sin(1)**2 + 16*np.cos(1)**2 + 24]
])
v = np.array([82, 16*np.sin(1)*np.cos(1) + 24])
expected = la.solve(H, v) * -1 + np.array([0, 1])
actual = newton_nd_optimization_crude_step(form, (x, y), np.array([0, 1]))
assert la.norm(expected - actual, 2) < tolerance
```

```python
form = sp.sympify('11*x**2+13*x*y+11*y**2+6*E**(2*x*y)+5*(sin(y)**2)+7*cos(x*y)')
symbols = sp.symbols('x y')
print(newton_nd_optimization_crude_step(form, symbols, [0, 1]))
print(gradient(form, symbols, values=[0, 1]))
print(hessian(gradient(form, symbols), symbols, values=[0, 1]))
# Works, great!
```

    [ 3.07907313 -4.80335408]
    [25.0, 26.54648713412841]
    [[39.0, 25.0], [25.0, 17.83853163452858]]

```python
# Workspace
form = sp.sympify('4*x**2+6*y**4')
symbols = sp.symbols('x y')
values = [2, 1]
hes = hessian(gradient(form, symbols), symbols, values)
x_1 = newton_nd_optimization_crude_step(form, symbols, values)
step = np.array(x_1) - np.array(values)
print(hes)
print(x_1)
print(step)
```

    [[8.0, 0.0], [0.0, 72.0]]
    [0.         0.66666667]
    [-2.         -0.33333333]

```python

```