```python
from sympy import *
from sympy import init_printing; init_printing(use_latex = 'mathjax')
```

```python
var('l m hbar W pi') #Variables a utilizar

```

$$\left ( l, \quad m, \quad \hbar, \quad W, \quad \pi\right )$$

```python
M=zeros(2,2)
M[0,0]=sympify('hbar**2/(2*m*l**2)*pi**2+(1/2+1/pi)*hbar**2/(m*l**2)-W')
M[0,1]=sympify('(-1/pi)*hbar**2/(m*l**2)')
M[1,0]=sympify('(-1/pi)*hbar**2/(m*l**2)')
M[1,1]=sympify('hbar**2/(2*m*l**2)*(3*pi)**2+(1/2-1/(3*pi))*hbar**2/(m*l**2)-W')
```

```python
M
```

$$\left[\begin{matrix}- W + \frac{\hbar^{2} \left(\frac{1}{\pi} + \frac{1}{2}\right)}{l^{2} m} + \frac{\pi^{2} \hbar^{2}}{2 l^{2} m} & - \frac{\hbar^{2}}{\pi l^{2} m}\\- \frac{\hbar^{2}}{\pi l^{2} m} & - W + \frac{\hbar^{2}}{l^{2} m} \left(- \frac{1}{3 \pi} + \frac{1}{2}\right) + \frac{9 \pi^{2} \hbar^{2}}{2 l^{2} m}\end{matrix}\right]$$

```python
Wn=solve(M.det(),W)
```

```python
(Matrix(Wn).evalf())
```

$$\left[\begin{matrix}\frac{44.8097107237294 \hbar^{2}}{l^{2} m}\\\frac{5.75051787250659 \hbar^{2}}{l^{2} m}\end{matrix}\right]$$

```python
w=[]
w.append(N(Wn[1]))
w.append(N(Wn[0]))
c=[]
c.append(sympify('c1'))
c.append(sympify('c2'))
```

```python
w
```

$$\left [ \frac{5.75051787250659 \hbar^{2}}{l^{2} m}, \quad \frac{44.8097107237294 \hbar^{2}}{l^{2} m}\right ]$$

```python
E=[]
E.append(5.750345*hbar**2/(m*l**2))
E.append(44.808373*hbar**2/(m*l**2))
E
```

$$\left [ \frac{5.750345 \hbar^{2}}{l^{2} m}, \quad \frac{44.808373 \hbar^{2}}{l^{2} m}\right ]$$

```python
err=[]
for i in range(0,len(w)):
    err.append((w[i]-E[i])/E[i]*100)
err
```

$$\left [ 0.00300629799759328, \quad 0.00298543249805642\right ]$$


# Customer Assignment Problem

## Objective and Prerequisites

Sharpen your mathematical optimization modeling skills with this example, in which you will learn how to select the location of facilities based on their proximity to customers. We’ll demonstrate how you can construct a mixed-integer programming (MIP) model of this facility location problem, implement this model in the Gurobi Python API, and generate an optimal solution using the Gurobi Optimizer.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-LOG-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_customer-assignment_COM_EVAL_GITHUB_&utm_term=customer-assignment-problem&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-LOG-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_customer-assignment_ACADEMIC_EVAL_GITHUB_&utm_term=customer-assignment-problem&utm_content=C_JPM) as an *academic user*.

---
## Motivation

Many companies in various industries must, at some point, make strategic decisions about where to build facilities to support their operations. For example:

- Producers of goods need to decide how to design their supply chains – which encompass factories, distribution centers, warehouses, and retail stores.
- Healthcare providers need to determine where to build hospitals to maximize their population coverage.

These are strategic decisions that are difficult to implement and costly to change because they entail long-term investments. Furthermore, these decisions have a significant impact, both in terms of customer satisfaction and cost management. One of the critical factors to consider in this process is the location of the customers that the company is planning to serve.

---
## Problem Description

The Customer Assignment Problem is closely related to the Facility Location Problem, which is concerned with the optimal placement of facilities (from a set of candidate locations) in order to minimize the distance between the company's facilities and the customers. When the facilities have unlimited capacity, customers are assumed to be served by the closest facility.

In cases where the number of customers considered is too big, the customers can be grouped into clusters. Then, the cluster centers can be used in lieu of the individual customer locations. This pre-processing makes the assumption that all customers belonging to a given cluster will be served by the facility assigned to that cluster. The k-means algorithm can be used for this task, which aims to partition $n$ objects into $k$ distinct and non-overlapping clusters.

---
## Solution Approach

Mathematical optimization is a declarative approach where the modeler formulates a mathematical optimization model that captures the key aspects of a complex decision problem. The Gurobi Optimizer solves such models using state-of-the-art mathematics and computer science.

A mathematical optimization model has five components, namely:

- Sets and indices.
- Parameters.
- Decision variables.
- Objective function(s).
- Constraints.

We now present a Binary Integer Programming (BIP) formulation:

### Sets and Indices
$i \in I$: Set of customer clusters.

$j \in J$: Set of potential facility locations.

$\text{Pairings}= \{(i,j) \in I \times J: \text{dist}_{i,j} \leq \text{threshold}\}$: Set of allowed pairings

### Parameters
$\text{threshold} \in \mathbb{R}^+$: Maximum distance for a cluster-facility pairing to be considered.

$\text{max_facilities} \in \mathbb{N}$: Maximum number of facilities to be opened.

$\text{weight}_i \in \mathbb{N}$: Number of customers in cluster $i$.

$\text{dist}_{i,j} \in \mathbb{R}^+$: Distance from cluster $i$ to facility location $j$.

### Decision Variables
$\text{select}_j \in \{0,1\}$: 1 if facility location $j$ is selected; 0 otherwise.

$\text{assign}_{i,j} \in \{0,1\}$: 1 if cluster $i$ is assigned to facility location $j$; 0 otherwise.

### Objective Function

- **Total distance**: Minimize the total distance from clusters to their assigned facility:

\begin{equation}
\text{Min} \quad Z = \sum_{(i,j) \in \text{Pairings}}\text{weight}_i \cdot \text{dist}_{i,j} \cdot \text{assign}_{i,j}
\tag{0}
\end{equation}

### Constraints

- **Facility limit**: The number of facilities built cannot exceed the limit:

\begin{equation}
\sum_{j}\text{select}_j \leq \text{max_facilities}
\tag{1}
\end{equation}

- **Open to assign**: Cluster $i$ can only be assigned to facility $j$ if we decide to build that facility:

\begin{equation}
\text{assign}_{i,j} \leq \text{select}_{j} \quad \forall (i,j) \in \text{Pairings}
\tag{2}
\end{equation}

- **Closest store**: Cluster $i$ must be assigned to exactly one facility:

\begin{equation}
\sum_{j:(i,j) \in \text{Pairings}}\text{assign}_{i,j} = 1 \quad \forall i \in I
\tag{3}
\end{equation}

---
## Python Implementation

### Dataset Generation

In this simple example, we choose random locations for customers and facility candidates. Customers are distributed using Gaussian distributions around a few randomly chosen population centers, whereas facility locations are uniformly distributed.

```python
%matplotlib inline
import random

import gurobipy as gp
from gurobipy import GRB

import matplotlib.pyplot as plt
import numpy as np
from sklearn.cluster import MiniBatchKMeans

# Tested with Gurobi v9.0.0 and Python 3.7.0

seed = 10101
num_customers = 50000
num_candidates = 50
max_facilities = 8
num_clusters = 1000
num_gaussians = 10
threshold = 0.99

random.seed(seed)
customers_per_gaussian = np.random.multinomial(num_customers,
                                               [1/num_gaussians]*num_gaussians)
customer_locs = []
for i in range(num_gaussians):
    # each center coordinate in [-0.5, 0.5]
    center = (random.random()-0.5, random.random()-0.5)
    customer_locs += [(random.gauss(0,.1) + center[0], random.gauss(0,.1) + center[1])
                  for i in range(customers_per_gaussian[i])]
# each candidate coordinate in [-0.5, 0.5]
facility_locs = [(random.random()-0.5, random.random()-0.5)
              for i in range(num_candidates)]
print('First customer location:', customer_locs[0])
```

    First customer location: (0.33164437091949245, -0.2809884943538464)

### Preprocessing
**Clustering**

To limit the size of the optimization model, we group individual customers into clusters and optimize on these clusters. Clusters are computed using the K-means algorithm, as implemented in the scikit-learn package.

```python
kmeans = MiniBatchKMeans(n_clusters=num_clusters, init_size=3*num_clusters,
                         random_state=seed).fit(customer_locs)
memberships = list(kmeans.labels_)
centroids = list(kmeans.cluster_centers_) # Center point for each cluster
weights = list(np.histogram(memberships, bins=num_clusters)[0]) # Number of customers in each cluster
print('First cluster center:', centroids[0])
print('Weights for first 10 clusters:', weights[:10])
```

    First cluster center: [ 0.41682515 -0.38402675]
    Weights for first 10 clusters: [56, 56, 78, 104, 84, 34, 55, 24, 54, 44]

***Viable Customer-Store Pairings***

Some facilities are just too far away from a cluster center to be relevant, so let's heuristically filter all distances that exceed a given `threshold`:

```python
def dist(loc1, loc2):
    return np.linalg.norm(loc1-loc2, ord=2) # Euclidean distance

pairings = {(facility, cluster): dist(facility_locs[facility], centroids[cluster])
            for facility in range(num_candidates)
            for cluster in range(num_clusters)
            if  dist(facility_locs[facility], centroids[cluster]) < threshold}
print("Number of viable pairings: {0}".format(len(pairings.keys())))
```

    Number of viable pairings: 48897

###  Model Deployment

Build facilities from among candidate locations to minimize total distance to cluster centers:

```python
m = gp.Model("Facility location")

# Decision variables: select facility locations
select = m.addVars(range(num_candidates), vtype=GRB.BINARY, name='select')
# Decision variables: assign customer clusters to a facility location
assign = m.addVars(pairings.keys(), vtype=GRB.BINARY, name='assign')

# Deploy Objective Function
# 0. Total distance
obj = gp.quicksum(weights[cluster]
               *pairings[facility, cluster]
               *assign[facility, cluster]
               for facility, cluster in pairings.keys())
m.setObjective(obj, GRB.MINIMIZE)

# 1. Facility limit
m.addConstr(select.sum() <= max_facilities, name="Facility_limit")

# 2. Open to assign
m.addConstrs((assign[facility, cluster] <= select[facility]
             for facility, cluster in pairings.keys()),
            name="Open2assign")

# 3. Closest store
m.addConstrs((assign.sum('*', cluster) == 1
             for cluster in range(num_clusters)),
            name="Closest_store")

# Find the optimal solution
m.optimize()
```

    Using license file c:\gurobi\gurobi.lic
    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 49898 rows, 48947 columns and 146741 nonzeros
    Model fingerprint: 0xe7ec775c
    Variable types: 0 continuous, 48947 integer (48947 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [1e-01, 1e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 8e+00]
    Presolve time: 0.21s
    Presolved: 49898 rows, 48947 columns, 146741 nonzeros
    Variable types: 0 continuous, 48947 integer (48947 binary)
    Found heuristic solution: objective 8666.5999303

    Deterministic concurrent LP optimizer: primal and dual simplex
    Showing first log only...

    Concurrent spin time: 0.00s

    Solved with dual simplex

    Root relaxation: objective 5.922779e+03, 6113 iterations, 1.18 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

    *    0     0               0    5922.7794819 5922.77948  0.00%     -    1s

    Explored 0 nodes (6113 simplex iterations) in 1.74 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 2: 5922.78 8666.6

    Optimal solution found (tolerance 1.00e-04)
    Best objective 5.922779481868e+03, best bound 5.922779481868e+03, gap 0.0000%

## Analysis

Let's plot a map with:

- Customer locations represented as small pink dots.
- Customer cluster centroids represented as large red dots.
- Facility location candidates represented as green dots. Notice that selected locations have black lines emanating from them towards each cluster that is likely to be served by that facility.

```python
plt.figure(figsize=(8,8), dpi=150)
plt.scatter(*zip(*customer_locs), c='Pink', s=0.5)
plt.scatter(*zip(*centroids), c='Red', s=10)
plt.scatter(*zip(*facility_locs), c='Green', s=10)
assignments = [p for p in pairings if assign[p].x > 0.5]
for p in assignments:
    pts = [facility_locs[p[0]], centroids[p[1]]]
    plt.plot(*zip(*pts), c='Black', linewidth=0.1)
```

---
## Conclusions

We learned how mathematical optimization can be used to solve the Customer Assignment Problem. Moreover, it has been shown how machine learning can be used in the pre-processing so as to reduce the computational burden of big datasets. Of course, this comes at a cost, as using fewer clusters will result in coarser approximations to the global optimal solution.

---
## References

1. Drezner, Z., & Hamacher, H. W. (Eds.). (2001). Facility location: applications and theory. Springer Science & Business Media.
2. James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. New York: springer.
3. Klose, A., & Drexl, A. (2005). Facility location models for distribution system design. European journal of operational research, 162(1), 4-29.

Copyright © 2020 Gurobi Optimization, LLC


# Ecuaciones diferenciales ordinarias

Erwin Renzo Franco Diaz

Las ecuaciones diferenciales son parte fundamental de la física. Muchas leyes físicas como la segunda ley de Newton o las ecuaciones de Maxwell se expresan de esta manera, por lo que aprender resolverlas es de suma importancia. En esta sesión aprenderemos algunos métodos simbólicos y numéricos que ofrecen ciertos paquetes en Python para poder obtener la solución a ecuaciones diferenciales ordinarias. Primero veremos como resolverlas de manera simbólica usando el paquete $\texttt{sympy}$.

## Decaimiento radioactivo

Los núcleos radioactivos decaen de acuerdo a la ley

$$
\frac{dN}{dt} = - \lambda N
$$

donde $N$ es la concentración de los nucleidos en el tiempo $t$.

La solución a esta ecuación es una exponencial que decae. Si en $t = 0$ se tiene una concentración inicial $N(0) = N_0$

$$
N = N_0 e^{-\lambda t}.
$$

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
t = sp.symbols('t', real = True)
N = sp.Function('N', real = True, positive = True)(t)
l = sp.symbols('lambda', real = True, positive = True)

#sin condiciones iniciales
eq = sp.Eq(sp.diff(N),  -l*N)
eq
```

$\displaystyle \frac{d}{d t} N{\left(t \right)} = - \lambda N{\left(t \right)}$

```python
sol = sp.dsolve(eq, N)
sol
```

$\displaystyle N{\left(t \right)} = C_{1} e^{- \lambda t}$

```python
#con condiciones iniciales
N_0 = sp.symbols('N_0', positive = True)
sol_ci = sp.dsolve(eq, N, ics = {N.subs(t, 0): N_0})
sol_ci
```

$\displaystyle N{\left(t \right)} = N_{0} e^{- \lambda t}$

```python
#hacemos que N sea una funcion numerica para graficarla
l_num = 0.2
N0_num = 1.0e5

N_num = sp.lambdify(t, sol_ci.subs([(l, l_num), (N_0, N0_num)]).rhs, 'numpy')
```

```python
t_plot = np.linspace(0, 50, 1000)
N_plot = N_num(t_plot)/N0_num

plt.plot(t_plot, N_plot)
```

# Oscilador armónico

## Oscilador armónico simple

Una de los sistemas más importantes de la física, si es que no es el más importante, es el oscilador armónico cuya realización más simple es una partícula en un resorte. De la segunda ley de Newton y la ley de Hooke, la ecuación de movimiento que describe el oscilador armónico está dada por

$$
\frac{d^2 x}{dt^2} + \omega^2 x = 0
$$

donde $x$ es la posición de la partícula y $\omega$ es su frecuencia. La solución general a esta ecuación está dada por

$$
x(t) = C_1 \cos(\omega t) + C_2 \sin(\omega t) = A\cos(\omega t + \phi)
$$

donde $C_1$ y $C_2$, u $\omega$ y $\phi$ dependen de las condiciones iniciales $x(0) = x_0$ y $\dot{x} (0) = \dot{x}_0$

```python
x = sp.Function('x', real = True)(t)
w = sp.symbols('omega', real = True, positive = True)

eq_simple = sp.Eq(sp.diff(x, (t, 2)) + w**2*x, 0)
sol_simple = sp.dsolve(eq_simple, x)
sol_simple
```

$\displaystyle x{\left(t \right)} = C_{1} \sin{\left(\omega t \right)} + C_{2} \cos{\left(\omega t \right)}$

```python
x0 = sp.symbols('x_0', real = True)
x0dot = sp.symbols('\dot{x_0}', real = True)

sol_simple_ci = sp.dsolve(eq_simple, x, ics = {x.subs(t, 0): x0, sp.diff(x, t).subs(t, 0): x0dot})
sol_simple_ci
```

$\displaystyle x{\left(t \right)} = \frac{\dot{x_0} \sin{\left(\omega t \right)}}{\omega} + x_{0} \cos{\left(\omega t \right)}$

```python
w_num = 0.5
x0_num = 5
x0dot_num = 10

x_simple_num = sp.lambdify(t, sol_simple_ci.subs([(w, w_num), (x0, x0_num), (x0dot, x0dot_num)]).rhs, 'numpy')

x_simple_plot = x_simple_num(t_plot)
plt.plot(t_plot, x_simple_plot)
plt.title('un clasico')
plt.show()
```

## Oscilador armónico amortiguado

Consideremos el caso en el que además se tiene una fuerza de resistencia proporcional a la velocidad de la partícula. La ecuación de movimiento será ahora

$$
\frac{d^2 x}{dt^2} + \gamma \dot{x}+ \omega^2 x = 0
$$

cuya solución depende de los valores de $\gamma$ y $\omega$.

```python
g = sp.symbols('gamma', real = True, positive = True)

eq_damped = sp.Eq(sp.diff(x, (t, 2)) + g*sp.diff(x, t) + w**2*x,  0)
eq_damped
```

$\displaystyle \gamma \frac{d}{d t} x{\left(t \right)} + \omega^{2} x{\left(t \right)} + \frac{d^{2}}{d t^{2}} x{\left(t \right)} = 0$

### Sobreamortiguado

```python
#sobreamortiguado
g_sobre = 2.5*2*w

CI = {x.subs(t, 0): x0, sp.diff(x, t).subs(t, 0): x0dot}

eq_damped_sobre = eq_damped.subs(g, g_sobre)
sol_sobre = sp.simplify(sp.dsolve(eq_damped_sobre, x, ics = CI))

sol_sobre
```

$\displaystyle x{\left(t \right)} = \frac{\left(- \left(0.218217890235992 \dot{x_0} + 0.0455447255899809 \omega x_{0}\right) e^{0.20871215252208 \omega t} + \left(0.218217890235992 \dot{x_0} + 1.04554472558998 \omega x_{0}\right) e^{4.79128784747792 \omega t}\right) e^{- 5.0 \omega t}}{\omega}$

```python
sol_sobre = sp.dsolve(eq_damped_sobre, x, ics = CI)

x_sobre_num = sp.lambdify(t, sol_sobre.subs([(w, w_num), (x0, x0_num), (x0dot, x0dot_num)]).rhs, 'numpy')

x_sobre_plot = x_sobre_num(t_plot)
plt.plot(t_plot, x_sobre_plot)
```

### Amortiguamiento crítico

```python
#amortiguamiento critico
g_crit = 2*w

eq_damped_crit = eq_damped.subs(g, g_crit)
sol_crit = sp.dsolve(eq_damped_crit, x, ics = CI)
sol_crit
```

$\displaystyle x{\left(t \right)} = \left(t \left(\dot{x_0} + \omega x_{0}\right) + x_{0}\right) e^{- \omega t}$

```python
x_crit_num = sp.lambdify(t, sol_crit.subs([(w, w_num), (x0, x0_num), (x0dot, x0dot_num)]).rhs, 'numpy')

x_crit_plot = x_crit_num(t_plot)
plt.plot(t_plot, x_sobre_plot, t_plot, x_crit_plot)
plt.legend(['sobreamortiguado', 'critico'])
```

### Subamortiguamiento

```python
#subamortiguado
g_sub = 0.2*2*w

eq_damped_sub = eq_damped_crit.subs(g_crit, g_sub)
sol_sub = sp.dsolve(eq_damped_sub, x, ics=CI)

sol_sub = sol_sub
sol_sub
```

$\displaystyle x{\left(t \right)} = \left(x_{0} \cos{\left(0.979795897113271 \omega t \right)} + \frac{0.204124145231932 \left(5.0 \dot{x_0} + \omega x_{0}\right) \sin{\left(0.979795897113271 \omega t \right)}}{\omega}\right) e^{- 0.2 \omega t}$

```python
#ploteando
x_sub_num = sp.lambdify(t, sol_sub.subs([(w, w_num), (x0, x0_num), (x0dot, x0dot_num)]).rhs, 'numpy')

x_sub_plot = x_sub_num(t_plot)
plt.plot(t_plot, x_sobre_plot, t_plot, x_crit_plot, t_plot, x_sub_plot)
plt.legend(['sobreamortiguado', 'critico', 'subamortiguado'])
```

# Sistema de ecuaciones diferenciales

En una serie radioactiva con dos nucleidos diferentes con concentraciones $N_1(t)$ y $N_2(t)$, se tiene

\begin{align}
\frac{dN_1}{dt} &= -\lambda_1 N_1 \\
\frac{dN_2}{dt} &= \lambda_1 N_1 - \lambda_2 N_2\\
\end{align}

En $t = 0$, $N_1(0) = N_0$ y $N_2(0) = 0$

```python
N1= sp.Function('N_1', real = True)(t)
N2= sp.Function('N_2', real = True)(t)
#l1, l2 = sp.symbols('lambda_1, lambda_2', real = True, positive = True)

l1_num = 0.5
l2_num = 0.7

eq1 = sp.Eq(sp.diff(N1), -l1_num*N1)
eq2 = sp.Eq(sp.diff(N2), l1_num*N1 - l2_num*N2)

eqs = [eq1, eq2]

N1_sol, N2_sol = sp.dsolve(eqs, [N1, N2], ics = {N1.subs(t, 0): N_0, N2.subs(t, 0): 0})

N2_sol
```

$\displaystyle \operatorname{N_{2}}{\left(t \right)} = - 2.5 N_{0} e^{- 0.7 t} + 2.5 N_{0} e^{- 0.5 t}$

```python
N1_num = sp.lambdify(t, N1_sol.subs(N_0, N0_num).rhs, 'numpy')
N2_num = sp.lambdify(t, N2_sol.subs(N_0, N0_num).rhs, 'numpy')

N1_plot = N1_num(t_plot)
N2_plot = N2_num(t_plot)

plt.plot(t_plot, N1_plot, t_plot, N2_plot)
plt.legend(['$N_1$', '$N_2$'])
plt.yscale('log')
plt.show()
```


# Introduction to reproducibility and power issues

## Some Definitions

* $H_0$ : null hypothesis: The hypotheis that the effect we are testing for is null

* $H_A$ : alternative hypothesis : Not $H_0$, so there is some signal

* $T$ : The random variable that takes value "significant" or "not significant"

* $T_S$ : Value of T when test is significant (eg $T = T_S$)

* $T_N$ : Value of T when test is not significant (eg $T = T_N$)

* $\alpha$ : false positive rate - probability to reject $H_0$ when $H_0$ is true (therefore $H_A$ is false)

* $\beta$ : false negative rate - probability to accept $H_0$ when $H_A$ is true (i.e. $H_0$ is false)

power = $1-\beta$

where $\beta$ is the risk of *false negative*

So, to compute power, *we need to know what is the risk of false negative*, ie, the risk to not show a significant effect while we have some signal (null is false).

```python
from __future__ import division
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import scipy.stats as sst
```

```python
from sympy import symbols, Eq, solve, simplify, lambdify, init_printing, latex
init_printing(use_latex=True, order='old')
```

```python
from IPython.display import HTML
# Code to make HTML for a probability table
def association_table(assocs, title):
    latexed = {'title': title}
    for key, value in assocs.items():
        latexed[key] = latex(value)
    latexed['s_total'] = latex(assocs['t_s'] + assocs['f_s'])
    latexed['ns_total'] = latex(assocs['t_ns'] + assocs['f_ns'])
    return """<h3>{title}</h3>
              <TABLE><TR><TH>$H/T$<TH>$T_S$<TH>$T_N$
              <TR><TH>$H_A$<TD>${t_s}$<TD>${t_ns}$
              <TR><TH>$H_0$<TD>${f_s}$<TD>${f_ns}$
              <TR><TH>Total<TD>${s_total}$<TD>${ns_total}$
              </TABLE>""".format(**latexed)
```

```python
from sympy.abc import alpha, beta # get alpha, beta symbolic variables
assoc = dict(t_s = 1 - beta, # H_A true, test significant = true positives
             t_ns = beta, # true, not significant = false negatives
             f_s = alpha, # false, significant = false positives
             f_ns = 1 - alpha) # false, not sigificant = true negatives
HTML(association_table(assoc, 'Not considering prior'))
```

## How do we compute power ?

### What is the effect ?

$$\hspace{3cm}\mu = \mu_1 - \mu_2$$

### What is the standardized effect ? (eg Cohen's d)

$$\hspace{3cm}d = \frac{\mu_1 - \mu_2}{\sigma} = \frac{\mu}{\sigma}$$

### "Z" : Effect accounting for the sample size

$$\hspace{3cm}Z = \frac{\mu}{\sigma / \sqrt{n}}$$

### Cohen's d value:

```python
# print some cohen values
# %pylab inline
muse = (.05, .1,.2,.3,.4,.5);
sigmas = np.linspace(1.,.5,len(muse))
cohenstr = ["For sigma = %3.2f and m = %3.2f Cohen d = %3.2f" %(sig,mu,coh)
       for (sig,mu,coh) in zip(sigmas,muse, np.asarray(muse)/sigmas)]
for s in cohenstr:
    print(s)
```

We have to estimate the effect $\mu$, say under some normal noise. Our statistic will be:

$$
t = \frac{\hat{\mu}}{\hat{\sigma_{\mu}}} = \frac{\hat{\mu}}{\hat{{SE}_{\mu}}}
$$

Power is the probability that the observed t is greater than $t_{.05}$, computing $t_{.05}$ by assuming that we are under the null.

So, we compute $t_{.05}$, and want to compute $P(t > t_{.05})$.

To compute this, __we need the distribution of our measured t - therefore we need to know the signal / effect size !__

Let's assume we know this and call it $t_{nc}$, and $F_{nc}$ for the cumulative distribution (more on this in the appendix).

$\mbox{Power} = 1 - \beta =  P(t > t_{.05}) = 1 - F_{nc}(t_{.05})$

__This power will depend on 4 parameters :__

$$ \mbox{The non standardized effect : } \mu$$

$$\mbox{The standard deviation of the data : } \sigma$$

$$\mbox{The number of subjects : } n$$

$$\mbox{The type I risk of error : } \alpha$$

And on the distribution of the statistic under the alternative hypothesis. Here, we assume our original data are normals, and the $t = \frac{\hat{\mu}}{\hat{{SE}_{\mu}}}$ statistics follows a non central t distribution with non centrality parameter

$$\theta = \mu \sqrt{n}/\sigma$$

and $n-1$ degrees of freedom.

### What is the effect size of APOE on the hippocampal volume ?

Authors find p value of 6.63e-10

They had 733 subjects

```python
n01 = sst.norm(0,1.)
z = n01.isf(6.6311e-10)
d = n01.isf(6.6311e-10)/np.sqrt(733)
print("z = %4.3f d = %4.3f " %(z,d))
```

```python
import scipy.stats as sst
import numpy as np
import matplotlib.pyplot as plt

```

```python
# plot power as a function of n : define a little function that
# takes n, mu, sigma, alpha, and report n.
# Optionally plot power as a function of nfrom matplotlib.patches import Polygon

def stat_power(n=16, mu=1., sigma=1., alpha=0.05):
    """
    This function computes the statistical power of an analysis assuming a normal
    distribution of the data with a one sample t-test

    Parameters:
    -----------
    n: int,
        The number of sample in the experiment
    mu: float
        The mean of the alternative
    sigma: float
        The standard deviation of the alternative
    plot: bool
        Plot something
    alpha: float
        The risk of error (type I)
    xlen: int
        Number of points for the display

    Returns:
    --------
    float
        The statistical power for this number of sample, mu, sigma, alpha
    """
    df = n-1
    theta = np.sqrt(n)*mu/sigma
    t_alph_null = sst.t.isf(alpha, df)
    ncrv = sst.nct(df, theta)
    spow = 1 - ncrv.cdf(t_alph_null)

    return spow

```

```python
from matplotlib.patches import Polygon

def plot_power(n=16, mu=1., sigma=1., alpha=0.05, xlen=500):

    df = n-1
    theta = np.sqrt(n)*mu/sigma
    ncrv = sst.nct(df, theta)
    t_alph_null = sst.t.isf(alpha, df)
    spow = 1 - ncrv.cdf(t_alph_null)

    # define the domain of the plot
    norv = sst.norm(0, 1.)
    bornesnc = ncrv.isf([0.001, .999])
    bornesn = norv.isf([0.001, .999])

    # because the nc t will have higher max borne, and the H0 normal will be on the left
    x = np.linspace(np.min(bornesn), np.max(bornesnc), xlen)
    t_line = np.zeros_like(x)

    # define the line
    x_t_line = np.argmin((x-t_alph_null)**2)
    y_t_line = np.max(np.hstack((ncrv.pdf(x), norv.pdf(x))))
    t_line[x_t_line] = y_t_line

    fig, ax = plt.subplots()
    plt.plot(x, ncrv.pdf(x), 'g', x, norv.pdf(x), 'b', x, t_line, 'r')

    # Make the shaded region
    # http://matplotlib.org/xkcd/examples/showcase/integral_demo.html
    a = x[x_t_line]; b =  np.max(bornesnc);
    ix = np.linspace(a,b)
    iy = ncrv.pdf(ix)
    verts = [(a, 0)] + list(zip(ix, iy)) + [(b, 0)]
    poly = Polygon(verts, facecolor='0.9', edgecolor='0.5')
    ax.add_patch(poly)

    ax.set_xlabel("t-value - H1 centred on " + r"$\theta $" + " = %4.2f;  " %theta
                      + r"$\mu$" + " = %4.2f" %mu);
    ax.set_ylabel("Probability(t)");
    ax.set_title('H0 and H1 sampling densities '
                     + r'$\beta$' + '= %3.2f' %spow + ' n = %d' %n)
    plt.show()

```

```python
n = 17
mu = .5
sigma = 2.
pwr = stat_power(n, mu, sigma, alpha=0.05)
plot_power(n, mu, sigma, alpha=0.05, xlen=500)

print ("Power = ", pwr, "      Z effect (Non centrality parameter) = ", mu*np.sqrt(n)/sigma)
```

```python
n = 50
mu = .5
sigma = 1.
pwr = stat_power(n, mu, sigma, alpha=0.05)
plot_power(n, mu, sigma, alpha=0.05, xlen=500)

print("Power = ", pwr, "      Z effect (Non centrality parameter): ", mu*np.sqrt(n)/sigma)
```

### Plot power as a function of the number of subject in the study

```python
#

def pwr_funcofsubj(muse, nses, alpha=.05, sigma=1):
    """
    Function to plot power as a function of the number of subjects (the nses)
    for an array of effect size (muses)

    muse: array of float
          The effect sizes : mu
    nses: array of integer
          The number of subjects
    alpha: float,
          type I risk
    sigma: float,
          data sigma
    """
    mstr = [ 'd='+str(m) for m in np.asarray(muse)/sigma]
    lines=[]
    for mu in (muse):
        pw = [stat_power(n, mu, sigma, alpha=alpha) for n in nses]
        (pl,) = plt.plot(nses, pw)
        lines.append(pl)
    plt.legend( lines, mstr, loc='upper right', shadow=True)
    plt.xlabel(" Number of subjects ")
    plt.ylabel(" Power ");

    return None

```

### **** For a small number of subjects ****

```python
mus = (.05, .1,.2,.3,.4,.5, .6);
#nse = range(70, 770, 20)
nse = range(7, 77, 2)
alph = 1.e-3

pwr_funcofsubj(mus, nse, alph)
```

### **** For a larger number of subjects ****

```python
mus = (.05,.1,.2,.3,.4,.5,.6);
nse = range(10, 330, 20)
#nse = range(7, 77, 2)
alph = 0.001

pwr_funcofsubj(mus, nse, alph)
```

### Here - stricter $\alpha$

```python
mus = (.05, .1,.2,.3,.4,.5, .6);
#nse = range(10, 770, 20)
nse = range(7, 77, 2)
alph = 0.05/30000
alph = 0.05

pwr_funcofsubj(mus, nse, alph)
```

```python

```

```python
#Example:
n = 259; mu= 230; sigma = 1487
stat_power(n, mu, sigma, alpha=0.05)
```


# 07 Practise Problems
# Review Exercises - Solutions

### Lesson Goal

This lesson is a series of practise problems to test your understanding before we move onto the __Applications of Programming__ section of the course.

### Fundamental programming concepts

To solve the problems you must apply what you have learnt knowledge from the __Fundamentals of Programming__ section of the course.

# 1. Plotting Data

Import the data from the file `temperature_data.csv`.

The file contains the mean temperature (°C) of London, Philadelphia, and Hong Kong recorded during some months of the year.

Plot the temperature for each city against the number of the month (starting with 1 for January) as a single plot.

Label the axes.

Add a figure legend showing which line on the grpah represents London, Philadelphia, and Hong Kong respectively.

Interpolate to estimate the temperature data for the missing months for each city.

Plot the interpolated data.

*Extension : Change the x-axis tick labels from the number of the month to the name of the month (e.g. change 1 to January)*

```python
# Exercise 1 : Plotting Data
# Example Solution

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.interpolate import interp1d

# import data
data = np.loadtxt('sample_data/temperature_data.csv', delimiter=",", dtype=str)
print(data)

# we can see from the result that we should skip row one and import columns 1-12
data = np.loadtxt('sample_data/temperature_data.csv',
                  delimiter=",",
                  skiprows=1,
                  usecols=(tuple(range(1,7))))

# months for which we have data
months = [1, 3, 7, 9, 11, 12]

# plot data
plt.plot(months, data[0], label="London");
plt.plot(months, data[1], label="Philadelphia");
plt.plot(months, data[2], label="Hong Kong");
#plt.xlim(months[0], months[-1])

# label axes and add legend
plt.xlabel('Month')
plt.ylabel('Temperature °C')
plt.xticks(range(1,13), range(1,13))
plt.legend()

all_months = np.arange(1, 13, 1)

# interpolate
for i in range(3):
    interp = interp1d(months, data[i], 'cubic') # type = ‘linear’, ‘nearest’, ‘zero’, ‘cubic’...
    plt.plot(all_months, interp(all_months), 'k--');

# relabel x ticks
plt.xticks(range(1,13), ('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'));
```

# 2. Curve Fitting
Import the data from the file `sample_data/air_temperature.dat`.

The file contains the air temperature (°C) recorded every 2 hours during a 24 hour period.

Plot the data as a scatter plot and label the axes.

Look at the shape of the plot. Based on your observation, approximate a continuous function that describes the data by curve fitting.

Plot the function as a line on the same graph as the original data.

Calculate the root mean sqaured error between your fitted function and the original data.

```python
# Exercise 2 : Curve Fitting
# Example Solution

# import data
data = np.loadtxt('sample_data/air_temperature.dat', delimiter=",")

time = np.array(range(0, 24, 2))

# plot data as scatter plot
plt.plot(time, data, 'o')
plt.xlim(time[0], time[-1])

# label axes
plt.xlabel('time (hours)')
plt.ylabel('Temperature (°C)')

b = np.polyfit(time, data, 3) # third order polynolmial

data_fit = np.poly1d(b)(time)

plt.plot(time, data_fit)
```

# 3. Functions and Libraries
Write a Python function for the following function:

$f(x)=e^{\frac{-K x}{50}}\sin(x)$

The function should:
- take `x` and `K` as input arguments
- return the value of the function
- have a dcoumentation string

Store your function in a seperate file called `my_functions` and import it to your main program.

In your main program, import the data from the file `sample_data/air_temperature.dat`.

For each data point, make a plot of `f` vs. `x` where:

`K` = the value of the imported data point

`x` = integers in the range [0 , 15]

Add a legend.

```python
# Exercise 3 : Functions and Libraries
# Example Solution

from sample_data.my_functions.my_functions import *

data = np.loadtxt('sample_data/air_temperature.dat', delimiter=",")

x = np.array(range(15))

def test(x, K):
    return np.exp(-K * x /50) * np.sin(x)

for d in data:
    plt.plot(x, test(x, d), label=f'K={d}')

plt.legend(bbox_to_anchor=(1,1), loc="upper left")
```

# 4. Systems of Equations

The following four measurements of the quantity $g$ were made at time $t_0, t_1, t_2, t_3$ :

$(t_0,g_0)=(0,3)$

$(t_1,g1)=(0.25,1)$

$(t_2,g_2)=(0.5,-3)$

$(t_3,g_3)=(0.75,1)$.

The measurements lie on a wave function that may be expresssed as:

$g = a\cos(\pi t) + b\cos(2\pi t) + c\cos(3\pi t) + d\cos(4\pi t)$

where $a$, $b$, $c$, and $d$ are constants.

Solve for the four constants by arranging them as a system of four linear equations.

Plot of the wave for $t$ in the range [0, 1].

Indicate the four measurements by ploting them as dots.

```python
# Exercise 4 : Systems of Equations
# Example Solution

#data
t = np.array([0, 0.25, 0.5, 0.75])
g = np.array([ 3, 1, -3, 1])

# array to hold data
lhs = np.zeros((4, 4))
rhs = np.zeros(4)

# populate arrays
for i in range(4):
    lhs[i] = np.cos(1 * np.pi * t[i]), \
             np.cos(2 * np.pi * t[i]), \
             np.cos(3 * np.pi * t[i]), \
             np.cos(4 * np.pi * t[i])  # Store one row at a time

    rhs[i] = g[i]

# solve system of equations
sol = np.linalg.solve(lhs, rhs)

# print value of coefficients
print('a,b,c,d: ',sol)

# range of values for t
t_all = np.linspace(0, 1, 100)

# g for all t
g_all = sol[0] * np.cos(1 * np.pi * t_all) + \
        sol[1] * np.cos(2 * np.pi * t_all) + \
        sol[2] * np.cos(3 * np.pi * t_all) + \
        sol[3] * np.cos(4 * np.pi * t_all)

# plot wave function
plt.plot(t_all, g_all, 'b', label='wave')
# plot data points
plt.plot(t, g, 'ro', label='data')
plt.legend(loc='best');
```

# 5. Numerical Integration

__(a)__ *Analytically* show that the integral of the function
 $f(x)=\text{e}^{-x}$
  for $x$ in the range [1, 5] is equal to $-\text{e}^{-5} + \text{e}^{-1}$.

__(b)__ Show that the following integral is equal to 0.218236 (6 s.f.):

$$\int_1^5 \frac{\text{e}^{-x}}{x}\text{d}x$$

Perform the integration numerically.

```python
# Exercise 5 : Numerical Integration
# Example Solution

# a) Analytical solution

import sympy as sp
from sympy import symbols, integrate
from scipy.integrate import quad

x = symbols('x')

f = sp.exp(-x)

print(integrate(f, (x, 1, 5))) # definite integral from x=0..1

# b) numerical solution

# first create a function to describe the integrand
def integrand(x):
    return np.exp(-x) / x

# create two variables to store the integral and the error
ans, err = quad(integrand, 1, 5)

print(ans)

```

    -exp(-5) + exp(-1)
    0.21823563880424607

# 6. Predator Prey
Also known as Lotka-Volterra equations, the predator-prey equations are a pair of first-order non-linear ordinary differential equations.

They represent a simplified model of the change in populations of two or more species which interact via predation.

Let $x$ represent the population of predators and  $y$  represent the population of prey.

The rate of change of each population can be represented by two first order differential equations.

$$\frac{dx}{dt} = x (a - by)$$

$$\frac{dy}{dt} = y (dx - c)$$

$a ,  b ,  c$  and  $d$  are parameters, which are assumed to be positive.

$a$ : growth of prey population

$b$ : decay of prey population due to predation

$c$ : growth of predator population

$d$ : decay of predator population due to natural death

Let $a = b = c = d = 1$

__(a)__ Find the numerical solution to the system of equations for $x$ and $y$, for time, $t$, in the range [0, 12].

__(b)__ Plot $x$ and $y$, against $t$ =  range [0, 12].

```python
# Exercise 6 : Predator Prey
# Example Solution

# Part A

# Import the required modules
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from scipy.integrate import odeint

# constants
a,b,c,d = 1,1,1,1

def dP_dt(P, t):
    """
    P is a vector such that P[0] = x and P[1] = y.
    Returns [x', y']
    """
    return [ P[0]*(a - b*P[1]),
            -P[1]*(c - d*P[0])]

ts = np.linspace(0, 12, 100) # the value(s) of t at which to evaluate P
P0 = [1.5, 1.0]              # the initial value of each population

# odeint returns solution for x,y at each value of t
Ps = odeint(dP_dt, P0, ts)

prey = Ps[:,0]       # column 0 = solution for x
predators = Ps[:,1]  # column 1 = solution for y
```

```python
# Part B
plt.plot(ts, prey, label="prey")
plt.plot(ts, predators, label="predators")
plt.xlabel("Time")
plt.ylabel("Population")
plt.legend();
```


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams['font.size'] = 13
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

## Background theory

A maximum entropy model is one that makes as little assumptions as possible while still fulfilling some constraints. These constraints are sometimes interpreted as features, but in practice they usually correspond to fixing [moments](https://en.wikipedia.org/wiki/Moment_(mathematics). That is, we seek a model of a distribution that has maximal entropy:

$$
\begin{equation}
 H = - \sum_i p(x_i) \log \left( p(x_i) \right)
\end{equation}
$$

but that also has the same moments as our sample. This is achieved through the method of Lagrange multipliers, which maximizes a functional that depends on the constraints as:

$$
\begin{equation}
 \tilde{H} = - \sum_i p(x_i) \log ( p(x_i) ) + \lambda_0 ( ( \sum_i p(x_i) ) - 1 ) + \sum_k \lambda_k ( ( \sum_i p(x_i) f_k(x_i) ) - \sum_i p_\mathrm{emp.}(x_i) f_k(x_i) ).
\end{equation}
$$

where $( \sum_i p(x_i) ) - 1$ derives from all probabilities summing up to one and $\sum_i p(x_i) f_k(x_i) - \sum_i p_\mathrm{emp.}(x_i) f_k(x_i)$ from matching moments. The actual moment is given by $f$, and matching, for example, the mean thus corresponds to having $f(x)=x$. A maximum entropy model (distribution) can therefore be found by maximizing $\tilde{H}$ with respect to both $p$ and the $\lambda$s. This might seem complicated at first, but as the entropy function is convex and the constraints affine (in $p$), this is actually a convex optimization problem. The global optima can thus be found by setting the gradient to zero. For most cases, however, there is no closed form solution and numerical optimization routines are needed, but it is, nonetheless, possible to analytically derive the shape of $p$.

The partial derivatives for each $p(x_i)$ is given by:

$$
\begin{equation}
 \frac{\partial \tilde{H}}{\partial p(x_i)} = - \log(p(x_i)) - 1 + \lambda_0 + \sum_k \lambda_k f_k(x_i),
\end{equation}
$$

and by setting these to zero, we learn that each $p(x_i)$ must have the form:

$$
\begin{align}
 0 &= - \log(p(x_i)) - 1 + \lambda_0 + \sum_k \lambda_k f_k(x_i), \\
 \log(p(x_i)) &= -1 + \lambda_0 + \sum_k \lambda_k f_x(x_i), \\
 p(x_i) &= \frac{\exp(\sum_k \lambda_k f_x(x_i))}{\exp(1-\lambda_0)}. \\
\end{align}
$$

The normalization constraint further corresponds to setting the partial derivative for $\lambda_0$ to zero:

$$
\begin{align}
 \frac{\partial \tilde{H}}{\partial \lambda_0} &= ( \sum_i p(x_i) ) - 1, \\
 0 &= ( \sum_i p(x_i) ) - 1, \\
 1 &= \sum_i p(x_i), \\
 1 &= \sum_i \frac{\exp(\sum_k \lambda_k f_x(x_i))}{\exp(1-\lambda_0)}, \\
 \exp(1-\lambda_0) &= \sum_i \exp(\sum_k \lambda_k f_x(x_i)).
\end{align}
$$

Thus, it follows that $p(x)$ can be described by:
$$
\begin{align}
 p(x_i) &= \frac{\exp(\sum_k \lambda_k f_x(x_i))}{Z}, \\
 Z &= \sum_i \exp \left( \sum_k \lambda_k f_x(x_i) \right),
\end{align}
$$

and that it belongs to the exponential family. The only parameters left to determine are then $\lambda_1, \ldots, \lambda_K$, which can be found by maximizing the constrained optimization problem $\tilde{H}$ directly or by maximizing a related unconstrained maximum likelihood problem (Berger et al., 1996). Given the empirical distribution $p_\mathrm{emp.}(x_i)$, the **mean** likelihood $L$ and mean log-likelihood $ll$ functions become:

$$
\begin{align}
 L &= \prod_i p(x_i|\lambda_1,\ldots,\lambda_K)^{p_\mathrm{emp.}(x_i)}, \\
 ll &= \sum_i p_\mathrm{emp.}(x_i) \log(p(x_i|\lambda_1,\ldots,\lambda_K)), \\
    &= \sum_i p_\mathrm{emp.}(x_i) \log(\exp(\sum_k \lambda_k f_x(x_i))) - \sum_i p_\mathrm{emp.}(x_i) \log(Z).
\end{align}
$$

The maximum of this log-likelihood function can be found using any gradient based method, and the expressions for the first and second partial derivative, respectively, are:

$$
\begin{align}
 \frac{\partial ll}{\partial \lambda_k} &= \sum_i p_\mathrm{emp.}(x_i) f_k(x_i) - \sum_i p_\mathrm{emp.}(x_i) \sum_j \frac{ \exp( \sum_k \lambda_k f_k(x_j))}{Z} f_k(x_j), \\
 &= \sum_i p_\mathrm{emp.}(x_i) f_k(x_i) - \sum_i p(x_i) f_k(x_i), \\
 &= \mathbb{E}_\mathrm{emp.}[f_k(x)] - \mathbb{E}[f_k(x)],\\
 \frac{\partial^2 ll}{\partial \lambda_k \partial \lambda_l} &= - (\mathbb{E}[f_k(x)f_l(x)] - \mathbb{E}[f_k(x)]\mathbb{E}[f_l(x)]).
\end{align}
$$

The expression for the first derivative neatly illustrates that the maximum likelihood approach yields $\lambda$s that constitute a solution to the original optimization problem $\tilde{H}$: the gradient will be zero when all the equality conditions are fulfilled. Similarly, the second derivatives extends upon this and further tells us that we do not just obtain a solution, but the optimal solution: Each second partial derivative occupies one element in the Hessian. Thus, we can conclude that the Hessian is the negative of a covariance matrix, and as covariance matrices are always positive semi-definite, the maximum found is guaranteed to be optimal and unique.

In the implementation below, we find maximum entropy models by solving the maximum likelihood problem using [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization). This method converges a lot faster than naive gradient descent, but it can become slow for large number of constraints as the inverse of the Hessian needs to be calculated at each step.

## Practical examples

#### Some functions

```python
def generateResponsePatterns(outcomes, n_dims):
    """ Generate all possible response patterns """
    pattern_grid = np.meshgrid(*[outcomes for i in range(n_dims)], indexing='ij')
    patterns = np.vstack([grid_tmp.ravel() for grid_tmp in pattern_grid]).T

    return patterns

def generateModelTerms(patterns, order=1, var=True):
    """ Generate model terms from the response patterns

        Indexes denote the response dimension (e.g., different cells)
        First order: f(x1,...,xk) = x1,...,xk
        Second order: f(x1,...,xk) = x1x1, x1x2,...,xkxk

    """
    n_dims = patterns.shape[1]

    # First order terms
    model_terms = patterns.copy()
    term_tags = []
    for i in range(n_dims):
        term_tags.append('x%1d' % (i+1))

    # Second order terms
    if order > 1:
        for i in range(n_dims):
            if var:
                start_idx = i
            else:
                start_idx = i+1
            for j in range(start_idx, n_dims):
                tmp_term = patterns[:, i] * patterns[:, j]
                model_terms = np.append(model_terms, tmp_term[:, np.newaxis], axis=1)
                term_tags.append('x%1dx%1d' % (i+1, j+1))

    return model_terms, term_tags

def sampleDistribution(p, n_observations):
    """ Sample a 1 or 2 dimensional distribution """

    sample_idx = np.empty([n_observations, len(p.shape)], dtype=np.int64)

    if len(p.shape) == 1:
        p_cum = np.cumsum(p)
        r_sample = np.random.rand(n_observations)
        sample_idx[:, 0] = np.int64(np.array([np.argmax(r < p_cum) for r in r_sample]))

    elif len(p.shape) == 2:
        # Cumulative distributions
        p_v1 = np.sum(p, axis=1)
        p_cum_v1 = np.cumsum(p_v1)
        p_v2_given_v1 = p / p_v1[:, np.newaxis]
        p_cum_v2_given_v1 = np.cumsum(p_v2_given_v1, axis=1)

        # Sample along each dimension separtely
        r_v1_sample = np.random.rand(n_observations)
        sample_idx[:, 0] = np.array([np.argmax(r < p_cum_v1) for r in r_v1_sample])
        r_v2_sample = np.random.rand(n_observations)
        sample_idx[:, 1] = np.array([np.argmax(r < p_cum_v2_given_v1[v1,:]) for r, v1 in zip(r_v2_sample, sample_idx[:,0])])

    return sample_idx

def getEmpiricalDistribution(sample, patterns):
    """ Determine the empirical probability distribution"""
    n_patterns = patterns.shape[0]
    pattern_counts = np.zeros(n_patterns)
    # Count how often each pattern occurs in the sample and normalize at the end
    for i in range(n_patterns):
        matches = np.where((sample == patterns[i, :]).all(axis=1))[0]
        pattern_counts[i] = matches.size
    p_emp = pattern_counts / pattern_counts.sum()

    return p_emp

def getModelDistribution(lambdas, model_terms):
    """ Determine the model's probability distribution"""
    p_model = np.exp(np.dot(model_terms, lambdas))
    p_model /= p_model.sum()

    return p_model

def logLikelihood(P_emp, p_model):
    """ Calculates the loglikelihood for current model parameters"""

    return np.dot(p_emp[p_model>0], np.log(p_model[p_model>0]))

def fitMaxEntropyModel(model_terms, p_emp, term_tags=[], verbose=1):
    """ Fit a maximum entropy model """

    # Exptectation values for each model term (constraints)
    E_emp = np.dot(p_emp, model_terms)

    # Initialize lambdas to zeros (initial guess)
    n_lambdas = model_terms.shape[1]
    lambdas = np.zeros(n_lambdas)

    # Get initial predicted probabilitites
    p_model = getModelDistribution(lambdas, model_terms)

    # Get the initial loglikelihood
    log_lik = []
    log_lik.append(logLikelihood(p_emp, p_model))

    # Gradient ascent with successive upated along the newton direction
    n_steps = 0
    converged = False
    while not converged:

        # Get the model's expectation values
        E_model = np.dot(p_model, model_terms)

        # Calculate the gradient and the Hessian
        gradient = E_emp - E_model
        h_term1 = np.dot(model_terms.T, (p_model[:, np.newaxis]*model_terms))
        h_term2 = np.outer(E_model, E_model)
        hessian = h_term1 - h_term2

        # Line search along the Newton direction
        newton_direction = np.dot(np.linalg.inv(hessian), gradient)
        step_lengths = np.arange(0.1, 1.01, 0.1)
        log_lik_tmp = np.zeros(step_lengths.size)
        for i in range(step_lengths.size):
            lambdas_tmp = lambdas + step_lengths[i]*newton_direction
            p_model_tmp = getModelDistribution(lambdas_tmp, model_terms)
            log_lik_tmp[i] = logLikelihood(p_emp, p_model_tmp)

        # Update lambdas using the best step length
        lambdas += step_lengths[log_lik_tmp.argmax()]*newton_direction

        # Model probabilitites with updated lambdas
        p_model = getModelDistribution(lambdas, model_terms)

        # Update the loglikelihood and check for convergence
        log_lik.append(logLikelihood(p_emp, p_model))
        if np.abs(log_lik[-1] - log_lik[-2]) < 1e-10:
            converged = True
        n_steps += 1

    if verbose:
        print('{:10}'.format('') + ''.join(['{:>8}'.format(tag) for tag in term_tags]))
        print('{:10}'.format('Empirical') + ''.join(['{:8.3f}'.format(val) for val in E_emp]))
        print('{:10}'.format('Model') + ''.join(['{:8.3f}'.format(val) for val in E_model]))

    return p_model, lambdas, log_lik

def naiveEntropy(p):
    """ Naive entropy calculation """

    return -np.sum(p[p>0]*np.log2(p[p>0]))

def plotModelFit(p_emp, p_model, x, log_lik):
    """ Plot Newton iterations and the fitted model"""

    # Open figure
    plt.figure(figsize=(15, 3))
    # Subplot 1
    ax = plt.subplot(1, 2, 1)
    ax.axhline(logLikelihood(p_emp, p_emp), c='gray', ls=':')  # Max value if p_emp == p_model
    ax.plot(np.arange(len(log_lik)), log_lik, 'ko')  # Progress per iteration
    ax.set_xlabel('Newton iteration step')
    ax.set_ylabel('Loglieklihood')
    # Subplot 2
    ax = plt.subplot(1, 2, 2)
    ax.bar(x, p_emp, alpha=0.4, label='$p_\mathrm{emp.}(x)$')
    ax.bar(x, p_model, color='r', alpha=0.4, label='$p_\mathrm{model}(x)$')
    ax.plot(x, p, 'ko', label='$p_\mathrm{true}(x)$')
    ax.legend(loc=0, frameon=False)
    ax.set_xlabel('$x$')
    ax.set_ylabel('$p(x)$');
```

#### Exponential distribution

```python
n_outcomes = 21
n_observations = 200
mu = 3.

# The true distribution is an exponential distribution
x = np.arange(n_outcomes)
p = 1/mu * np.exp(-x/mu)
p /= p.sum()

# Get a sample of n_observations from the true distribution
sample_idx = sampleDistribution(p, n_observations)

# Get the response patterns and the model terms
patterns = generateResponsePatterns(x, sample_idx.shape[1])
model_terms, term_tags = generateModelTerms(patterns, order=2)

# Empirical pattern probabilitites
p_emp = getEmpiricalDistribution(x[sample_idx], patterns)

# Fit the maximum entropy model
p_model, lambdas, log_lik = fitMaxEntropyModel(model_terms, p_emp, term_tags)

# Plot results
plotModelFit(p_emp, p_model, x, log_lik)
```

#### Normal distribution

```python
n_outcomes = 21
n_observations = 200
mu = -7.
sigma = 2.

# The true distribution is an normal distribution
#x = np.arange(n_outcomes)
x = np.arange(-10, 11, 1)
p = np.exp(-(x-mu)**2/(2*sigma**2))
p /= p.sum()

# Get a sample of n_observations from the true distribution
sample_idx = sampleDistribution(p, n_observations)

# Get the response patterns and the model terms
patterns = generateResponsePatterns(x, sample_idx.shape[1])
model_terms, term_tags = generateModelTerms(patterns, order=2)

# Empirical pattern probabilitites
p_emp = getEmpiricalDistribution(x[sample_idx], patterns)

# Fit the maximum entropy model
p_model, lambdas, log_lik = fitMaxEntropyModel(model_terms, p_emp, term_tags)

# Plot results
plotModelFit(p_emp, p_model, x, log_lik)
```

#### Two-dimensional normal distribution

```python
n_outcomes = 21
n_observations = 10000
mu = (n_outcomes-1) / 2.*np.ones([2, 1])
var = 20.
cov = 15.

# Get the true probability distribution
cov_mat = np.array([[var, cov], [cov, var]])
[x1, x2,] = np.meshgrid(range(n_outcomes), range(n_outcomes))
p = np.zeros([n_outcomes, n_outcomes])
for i in range(n_outcomes**2):
    x_tmp = np.array([x1.ravel()[i]-mu[0], x2.ravel()[i]-mu[1]])
    p.ravel()[i] = np.exp(-1/2 * np.dot(x_tmp.T, np.dot(np.linalg.inv(cov_mat), x_tmp)))
p /= p.sum()

# Get a sample of n_observations from the true distribution
sample_idx = sampleDistribution(p, n_observations)

# Get the response patterns and the model terms
patterns = generateResponsePatterns(np.arange(n_outcomes), sample_idx.shape[1])
model_terms, term_tags = generateModelTerms(patterns, order=2)

# Empirical pattern probabilitites
p_emp = getEmpiricalDistribution(sample_idx, patterns)

# Fit the maximum entropy model
p_model, lambdas, log_lik = fitMaxEntropyModel(model_terms, p_emp, term_tags)

# Plot results
plt.figure(figsize=(15, 3))
# Subplot 1
ax = plt.subplot(1, 3, 1)
ax.imshow(p)
ax.invert_yaxis()
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_title('$p_\mathrm{true}(x)$')
# Subplot 2
ax = plt.subplot(1, 3, 2)
ax.imshow(p_emp.reshape(n_outcomes, n_outcomes))
ax.invert_yaxis()
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_title('$p_\mathrm{emp}(x)$')
# Subplot 3
ax = plt.subplot(1, 3, 3)
ax.imshow(p_model.reshape(n_outcomes, n_outcomes))
ax.invert_yaxis()
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_title('$p_\mathrm{model}(x)$');
```

#### Entropy for two-dimensional normal distribution

```python
relevance_lim = 1e-2
n_dist_params = np.sum(p>relevance_lim)
sample_sizes = np.arange(1, 11)*n_dist_params
n_repetitions = 100

# Entropies
H_true = naiveEntropy(p)
H_emp = np.zeros([n_repetitions, sample_sizes.size])
H_model = np.zeros([n_repetitions, sample_sizes.size])

# Get the response patterns and the model terms
patterns = generateResponsePatterns(np.arange(n_outcomes), sample.shape[1])
model_terms, term_tags = generateModelTerms(patterns, order=2)

for size_idx in range(sample_sizes.size):

    n_observations = sample_sizes[size_idx]
    for rep_idx in range(n_repetitions):

        # Get a sample of n_observations from the true distribution
        sample_idx = sampleDistribution(p, n_observations)

        # Empirical pattern probabilitites
        p_emp = getEmpiricalDistribution(sample_idx, patterns)
        H_emp[rep_idx, size_idx] = naiveEntropy(p_emp)

        # Fit the maximum entropy model
        p_model, lambdas, log_lik = fitMaxEntropyModel(model_terms, p_emp, term_tags, verbose=0)
        H_model[rep_idx, size_idx] = naiveEntropy(p_model)

# Plotting
plt.figure(figsize=(7.5, 3))
plt.errorbar(sample_sizes, H_emp.mean(axis=0), yerr=H_emp.std(axis=0), fmt='ro', label='$H_\mathrm{emp}\pm\sigma$')
plt.errorbar(sample_sizes, H_model.mean(axis=0), yerr=H_model.std(axis=0), fmt='bo', label='$H_\mathrm{model}\pm\sigma$')
plt.axhline(H_true, ls=':', c='gray', label='$H_\mathrm{true}$')
plt.legend(loc=4, frameon=False)
plt.xlabel('Number of observations')
plt.ylabel('Entropy ($H$)');

```

InformationTheory/Part 3, maximum entropy models.ipynb


# 1. High order pathology for the Stiffness matrix

For this chapter, we will need the [GeLaTo](https://github.com/pyccel/GeLaTo) library.

In this example, we show the dependency of the **stiffness** symbol with respect to the spline degree in 1d, we will also give some asymptotic expansion of the basic operators: **stiffness**, **mass** and **advection**.

```python
# ... needed imports
from numpy import linspace, zeros, pi
import matplotlib.pyplot as plt
%matplotlib inline

from IPython.display import Math

from sympy import Symbol
from sympy import Lambda

from gelato.glt import glt_symbol
from gelato.glt import glt_lambdify
from gelato.calculus   import Dot, Grad, dx

from spl.fem.splines import SplineSpace
```

We define the weak form, as:

```python
x = Symbol('x')

u = Symbol('u')
v = Symbol('v')

a_s = Lambda((x,v,u), Dot(Grad(u), Grad(v)))
```

```python
V = SplineSpace(2, grid=linspace(0., 1., 32+1))
```

```python
from IPython.display import Math
from gelato.printing.latex import glt_latex

symbol = glt_symbol(a_s, space=V, evaluate=True)
Math(glt_latex(symbol))
```

$$\left( \left ( x, \quad \theta_1\right ) \mapsto - \frac{64}{3} \cos{\left (\theta_1 \right )} - \frac{32}{3} \cos{\left (2 \theta_1 \right )} + 32 \right)$$

```python
from gelato.fem.utils    import compile_symbol
symbol_s = compile_symbol('symbol_1', a_s, V, backend='fortran')
```

```python
# ... make the symbol a numeric function, that can be evaluated
f = glt_lambdify(symbol)

nx = 100
x = linspace(  0, 1, nx)
t = linspace(-pi,pi, nx)
w = f(x,t)
# ...

# ... example of symbol evaluation
e = zeros(nx)
symbol_s(x, t, e)
# ...

```

```python
from numpy import allclose

allclose(e,w)
```

    True

```python

```

The following function allows to plot the eigenvalues for a given *grid* and different spline **degrees**:

```python
def plot_symbols(expr, degrees=[2, 3], ne=64):

    colors = ["r", "b", "g", "y", "m", "k", "c"]

    for i,p in enumerate(degrees):
        # ...  create a finite element space
        V = SplineSpace(p, grid=linspace(0., 1., ne+1))
        # ...

        # ... compute the GLT symbol
        symbol = glt_symbol(expr, space=V, evaluate=True)
        # ...

        # ... make the symbol a numeric function, that can be evaluated
        f = glt_lambdify(symbol)

        nx = 100
        x1 = linspace(  0, 1, nx)
        t1 = linspace(-pi,pi, nx)
        w = f(x1,t1)
        # ...

        plt.plot(t1, w.real, "-"+colors[i], label="$p=" + str(p) + "$")
        plt.legend(loc=9);
```

```python
plot_symbols(a_s, degrees=[2, 3, 4, 5])
```

As expected, we notice the convergence to 0 at $| \theta | = \pi$.

We consider now the weak formulation related to the **mass** matrix:

```python
x = Symbol('x')

u = Symbol('u')
v = Symbol('v')

# weak form for the mass matrix
a_m = Lambda((x,v,u), u * v)
```

We will use the following function to plot the division of the symbol of the **stiffness** by the symbol of the **mass**  of degree $p-1$

```python
def plot_symbols(expr_s, expr_m, degrees=[2, 3], ne=64):

    colors = ["r", "b", "g", "y", "m", "k", "c"]

    for i,p in enumerate(degrees):
        # ...  create a finite element space
        V = SplineSpace(p, grid=linspace(0., 1., ne+1))
        W = SplineSpace(p-1, grid=linspace(0., 1., ne+1))
        # ...

        # ... compute the GLT symbol
        symbol_s = glt_symbol(expr_s, space=V, evaluate=True)
        symbol_m = glt_symbol(expr_m, space=W, evaluate=True)
        # ...

        # ... make the symbol a numeric function, that can be evaluated
        f_s = glt_lambdify(symbol_s)
        f_m = glt_lambdify(symbol_m)

        nx = 100
        x = linspace(  0, 1, nx)
        t = linspace(-pi,pi, nx)
        w = f_s(x,t) / f_m(x,t)
        # ...

        plt.plot(t, w.real, "-"+colors[i], label="$p=" + str(p) + "$")
        plt.legend(loc=9);
```

```python
plot_symbols(a_s, a_m, degrees=[2, 3, 4, 5])
```

We see now that the *new symbol* is not sensitive to the *B-Spline* degree.

In fact, this symbol corresponds to the $S M^{-1}$ (and $M^{-1} S$), if we denote by $S$ and $M$ the *stiffness* and *mass* matrices respectively.

## Symbolic expression and asymptotic expansion

In this section, we give the asymptotic expansion of the symbol associated to the **mass**, **stiffness** and **advection** matrices.

We start by defining our weak formulations:

```python
x = Symbol('x')

u = Symbol('u')
v = Symbol('v')

a_m = Lambda((x,v,u), u * v)
a_s = Lambda((x,v,u), Dot(Grad(u), Grad(v)))
a_a = Lambda((x,v,u), dx(v) * u)
```

```python
# ...
p  = 3
ne = 16
# ...

# ...  create a finite element space
V = SplineSpace(p, grid=linspace(0., 1., ne+1))
# ...

# ... compute the GLT symbol
symbol_m = glt_symbol(a_m, space=V, evaluate=True)
symbol_s = glt_symbol(a_s, space=V, evaluate=True)
symbol_a = glt_symbol(a_a, space=V, evaluate=True)
# ...
```

We can now evaluate, symbolicaly, the symbols at the sampling points, using the following function, and print them using *Latex*:

```python
from gelato.printing.latex import glt_latex
from IPython.display import Math

t1 = Symbol("t1")

# ...
def to_latex(expr):
    m = Symbol("m")
    N = Symbol("N")
    pi = Symbol("pi")
    return glt_latex(expr.subs({t1: 2 * pi * m / N}))
# ...
```

---
**Note**:
since **expr** is a sympy **Lambda** expression, we need to use its *expr* member

---

```python
# symbol of the mass matrix
Math(to_latex(symbol_m.expr))
```

$$\frac{397}{13440} \cos{\left (\frac{2 m}{N} \pi \right )} + \frac{1}{336} \cos{\left (\frac{4 m}{N} \pi \right )} + \frac{1}{40320} \cos{\left (\frac{6 m}{N} \pi \right )} + \frac{2396825387}{79999999672}$$

```python
# symbol of the stiffness matrix
Math(to_latex(symbol_s.expr))
```

$$- 4 \cos{\left (\frac{2 m}{N} \pi \right )} - \frac{32}{5} \cos{\left (\frac{4 m}{N} \pi \right )} - \frac{4}{15} \cos{\left (\frac{6 m}{N} \pi \right )} + \frac{32}{3}$$

```python
# symbol of the advection matrix
Math(to_latex(symbol_a.expr))
```

$$- i \left(- \frac{49}{72} \sin{\left (\frac{2 m}{N} \pi \right )} - \frac{7}{45} \sin{\left (\frac{4 m}{N} \pi \right )} - \frac{1}{360} \sin{\left (\frac{6 m}{N} \pi \right )}\right)$$

We now give the asymptotic expansion of the GLT symbol arount $0$, up to the order $8$:

```python
# asymptotic expansion of the mass symbol
Math(to_latex(symbol_m.expr.series(t1,0,8)))
```

$$- \frac{4 m^{6} \pi^{6}}{189 N^{6}} + \frac{19 m^{4} \pi^{4}}{360 N^{4}} - \frac{m^{2} \pi^{2}}{12 N^{2}} + \frac{1574999993543}{25199999896680} + \mathcal{O}\left(\frac{m^{8} \pi^{8}}{N^{8}}; \left ( N, \quad m, \quad \pi\right )\rightarrow \left ( 0, \quad 0, \quad 0\right )\right)$$

```python
# asymptotic expansion of the stiffness symbol
Math(to_latex(symbol_s.expr.series(t1,0,8)))
```

$$\frac{2432 m^{6} \pi^{6}}{45 N^{6}} - \frac{256 m^{4} \pi^{4}}{3 N^{4}} + \frac{64 m^{2}}{N^{2}} \pi^{2} + \mathcal{O}\left(\frac{m^{8} \pi^{8}}{N^{8}}; \left ( N, \quad m, \quad \pi\right )\rightarrow \left ( 0, \quad 0, \quad 0\right )\right)$$

```python
# asymptotic expansion of the advection symbol
Math(to_latex(symbol_a.expr.series(t1,0,8)))
```

$$- \frac{128 i m^{7} \pi^{7}}{189 N^{7}} + \frac{76 i m^{5} \pi^{5}}{45 N^{5}} - \frac{8 i m^{3} \pi^{3}}{3 N^{3}} + \frac{2 i}{N} m \pi + \mathcal{O}\left(\frac{m^{8} \pi^{8}}{N^{8}}; \left ( N, \quad m, \quad \pi\right )\rightarrow \left ( 0, \quad 0, \quad 0\right )\right)$$

```python

```

```python

```

```python
fs = glt_lambdify(symbol_s)
fm = glt_lambdify(symbol_m)
fa = glt_lambdify(symbol_a)

nx = 100
x1 = linspace(  0, 1, nx)
t1 = linspace(-pi,pi, nx)

s1 = fs(x1,t1)
m1 = fm(x1,t1)
a1 = fa(x1,t1)
```

```python
w = a1**2 + m1 * s1
plt.plot(t1, w.real);
```

```python
s2 = s1
m2 = m1
a2 = a1

w = (m1*s2 + m2*s1)**2 + 4*(a1**2*m2**2 + a2**2*m1**2)

plt.plot(t1, w.real);

```

```python
# ...
p1  = 3 ; p2 = 3
ne1 = 64 ; ne2 = 64
# ...

# ...  create a finite element space
V1 = SplineSpace(p1, grid=linspace(0., 1., ne1+1))
V2 = SplineSpace(p2, grid=linspace(0., 1., ne2+1))
# ...

# ... compute the GLT symbol
symbol_m1 = glt_symbol(a_m, space=V1, evaluate=True)
symbol_s1 = glt_symbol(a_s, space=V1, evaluate=True)
symbol_a1 = glt_symbol(a_a, space=V1, evaluate=True)

symbol_m2 = glt_symbol(a_m, space=V2, evaluate=True)
symbol_s2 = glt_symbol(a_s, space=V2, evaluate=True)
symbol_a2 = glt_symbol(a_a, space=V2, evaluate=True)
# ...

# ...
fs1 = glt_lambdify(symbol_s1)
fm1 = glt_lambdify(symbol_m1)
fa1 = glt_lambdify(symbol_a1)

fs2 = glt_lambdify(symbol_s2)
fm2 = glt_lambdify(symbol_m2)
fa2 = glt_lambdify(symbol_a2)
# ...

x1 = x2 = linspace(  0, 1, 100)
t1 = t2 = linspace(-pi,pi, 100)

# ...
s1 = fs1(x1,t1)
m1 = fm1(x1,t1)
a1 = fa1(x1,t1)

s2 = fs2(x2,t2)
m2 = fm2(x2,t2)
a2 = fa2(x2,t2)
# ...
```

```python
w = (m1*s2 + m2*s1)**2 + 4*(a1**2*m2**2 + a2**2*m1**2)
print(min(w.real))
plt.plot(t1, w.real);
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```

```python

```


# MAT 221 Calculus I

## April 9, 2020

Today's Agenda:
1. Continuous Function
2. Intermediate Value Theorem
3. Exercises

# Product, Quotient and Power Rules

Several rules exist for finding the derivatives of functions with several components such as $x \space sin \space x$. With these rules and the chain rule, which will be explored later, any derivative of a function can be found (assuming they exist). There are five rules that help simplify the computation of derivatives, of which each will be explored in turn.

## Sum Rule

The most straightforward of the derivation rules is the sum rule, which is stated as:

$$ \text{The derivative of} \space u(x) + v(x) \space \text{is} \space \frac{d}{dx} (u + v) = \frac{du}{dx} + \frac{dv}{dx} $$

## Rule of Linearity

The sum rule can be considered a case of a larger concept called linearity. For example, linearity can be seen in the derivative of the function: $\frac{d}{dx}5x - cos x = 5 + sin \space x$ and all other similar functions. This rule is known as the Rule of Linearity and follows from [linear combinations](https://en.wikipedia.org/wiki/Linear_combination). Where $a$ and $b$ are real numbers, the rule is stated as the following:

$$ \text{The derivative of} \space au(x) + bv(x) \space \text{is} \space \frac{d}{dx}(au + bv) = a \frac{du}{dx} + b \frac{dv}{dx} $$

## Product Rule

The product rule helps deal with the frequently occurring situation of the product of two functions, such as $sin \space x + cos \space x$ and is stated as:

$$ \text{The deriative of the product of two functions,} \space u(x)v(x) \space \text{is} \space \frac{d}{dx}(uv) = u \frac{dv}{dx} + v \frac{du}{dx} $$.

The product rule can be extended to more than the product of two functions. For example, the derivative of the product of three functions would be:

$$ \frac{d}{dx} \space (uvw) = uv \frac{dw}{dx} + uw \frac{dv}{dx} + vw \frac{du}{dx} $$

## Reciprocal Rule

The reciprocal rule is more or less a case of the powerful quotient rule. The case of the reciprocal rule helps us find the derivative of $\frac{1}{v(x)}$ much easier. We can state the rule as the following:

$$\text{The derivative of the function} \space \frac{1}{v(x)} \space \text{is} \space \frac{d}{dx} \space \frac{1}{v(x)} = \frac{\frac{-dv}{dx}}{v^2}$$

## Quotient Rule

The quotient rule is an important derivation technique that allows us to more efficiently find the deriative of $\frac{u(x)}{v(x)}$. We can state the quotient rule as follows:

$$ \text{The derivative of} \space \frac{u(x)}{v(x)} \space \text{is} \space \frac{v \frac{du}{dx} - u \frac{dv}{dx}}{v^2} $$

Last but certainly not least is the power rule.

## Power Rule

The power rule gives us the ability to find the deriative of a function of the form $(u(x))^n$ and can be stated as:

$$ \text{The deriative of} \space \Big[u(x) \Big]^n \space \text{is} \space n \Big[u(x) \Big ]^{n-1} \frac{du}{dx} $$

Okay! With those rules stated and out of the way, we are now able to solve the vast majority of deriatives we will ever encounter! We test our hand at using the rules to calculate the deriative of several functions with the following examples.

## Examples

The examples presented here are borrowed from Gilbert Strang's amazing book [*Calculus*](https://ocw.mit.edu/ans7870/resources/Strang/Edited/Calculus/Calculus.pdf), which can be found for free in PDF form on the MIT OCW site. We also take advantage of the wonderful [SymPy](http://www.sympy.org/en/index.html) library for symbolic computation to check our answers to the example problems.

We import the functions from the SymPy library that we will require.

```python
from sympy import symbols, diff, sin, cos, simplify, init_printing
```

The `init_printing()` [function](http://docs.sympy.org/latest/tutorial/printing.html) is used to output the results of SymPy in LaTeX.

```python
init_printing()
```

First initialize a [symbol](http://docs.sympy.org/latest/modules/core.html#symbols) variable for $x$ to be used with SymPy's computation engine.

```python
x = symbols('x')
```

Example 1: Find the derivative of $(x + 1)(x - 1)$

We could factor and take the derivative of the resulting polynomial without having to use the product rule:

$$ (x + 1)(x - 1) = x^2 - x + x + 1 = x^2 + 1 $$

$$ \frac{d}{dx} x^2 + 1 = 2x $$

We can also find the deriative of the function with the product rule:

$$ \frac{d}{dx} (uv) = u \frac{dv}{dx} + v \frac{du}{dx} $$

$$ u = x + 1, \qquad v = x - 1 $$

$$ \frac{d}{dx} = (x + 1)(1) + (x - 1)(1) = 2x $$

Example 2: Find the derivative of $\frac{1}{1 + x} + \frac{1}{1 + sin \space x}$

Here we take advantage of the reciprocal rule, $\frac{d}{dx} \space \frac{1}{v(x)} = \frac{\frac{-dv}{dx}}{v^2}$. Starting with $\frac{1}{1 + x}$:

$$ \frac{d}{dx} \space \frac{1}{1 + x} = \frac{-1}{(1 + x)^2} $$

Then the second piece of the function, $\frac{1}{1 + sin \space x}$

$$ \frac{d}{dx} \space \frac{1}{1 + sin \space x} = \frac{-cos \space x}{(1 + sin \space x)^2} $$

Then putting the two pieces together gives us the derivative of the function:

$$ \frac{d}{dx} \frac{1}{1 + x} + \frac{1}{1 + sin \space x} = \frac{-1}{(1 + x)^2} - \frac{cos \space x}{(1 + sin \space x)^2} $$

We can verify this result using SymPy:

```python
diff(1 / (1 + x) + 1 / (1 + sin(x)))
```

Example 3: Find the derivative of $(x - 1)(x - 2)(x - 3)$

We can find the derivative using the product rule, $\frac{d}{dx} \space (uvw) = uv \frac{dw}{dx} + uw \frac{dv}{dx} + vw \frac{du}{dx}$

$$ u = (x - 1), \qquad v = (x - 2), \qquad w = (x - 3) $$

$$ \frac{d}{dx} (uvw) = (x - 1)(x - 2)(1) + (x - 1)(x - 2)(1) + (x - 2)(x - 3)(1) $$

Though not absolutely necessary, finish by factoring and simplifying the derivative.

$$ x^2 - 2x - x + 2 + x^2 - 3x - x + 3 + x^2 - 3x - 2x + 6 $$

$$ 3x^2 - 12x + 11 $$

Verifying with SymPy:

```python
diff((x - 1) * (x - 2) * (x - 3))
```

We can also check to see if our simplifying of the function was accurate using SymPy's `simplify` function.

```python
simplify(diff((x - 1) * (x - 2) * (x - 3)))
```

Example 4: Find the derivative of $\frac{x^3 + 1}{x + 1} + \frac{cos \space x}{sin \space x}$

Using the sum rule, we can find the derivatives of each piece and then add the derivatives together. Starting with $\frac{x^3 + 1}{x + 1}$, we apply the quotient rule:

$$ \frac{d}{dx} \space \frac{u(x)}{v(x)} = \frac{v \frac{du}{dx} - u \frac{dv}{dx}}{v^2} $$

$$ u = x^3 + 1, \qquad v = x + 1 $$

$$ \frac{d}{dx} \space \frac{x^3 + 1}{x + 1} = \frac{(x + 1)(3x^2) - (x^3 + 1)(1)}{(x + 1)^2} = 2x - 1 $$

We'll take the easy way out on factoring the above derivative and use SymPy:

```python
simplify(((x + 1) * (3 * x ** 2) - (x ** 3 + 1)) / (x + 1) ** 2)
```

Then we find the derivative of the second piece of the equation, $\frac{\cos{x}}{\sin{x}}$. Again, we take advantage of the quotient rule:

$$ u = \cos{x}, \qquad v = \sin{x} $$

$$ \frac{d}{dx} \frac{\cos{x}}{\sin{x}} = \frac{(\sin{x})(-\sin{x}) - (\cos{x})(\sin{x})}{(\sin{x})(\sin{x})} $$

$$ \frac{d}{dx} \frac{\cos{x}}{\sin{x}} = \frac{-\sin^2{x} - \cos^2{x}}{\sin^2{x}} $$

With the Pythagorean identity, $\sin^2{x} + \cos^2{x} = 1$, we can simplify the derivative to:

$$ -\frac{1}{\sin^2{x}} = -\csc^2{x} $$

Adding the two pieces of the equation together using the sum rule gives us the deriative:

$$ 2x - 1 - \frac{1}{\sin{x}} = 2x - 1 - \csc{x} $$

## References

Strang, G. (2010). Calculus. Wellesley, MA: Wellesley-Cambridge.

Pythagorean trigonometric identity. (2017, October 4). In Wikipedia, The Free Encyclopedia.
From https://en.wikipedia.org/w/index.php?title=Pythagorean_trigonometric_identity&oldid=803778637


Quantum Error Correction

## Imports

```python
%matplotlib inline
```

```python
from IPython.display import display
```

```python
from sympy import init_printing
init_printing(use_latex=True)
```

```python
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

5 qubit code

```python
M0 = Z(1)*X(2)*X(3)*Z(4)
M0
```

```python
M1 = Z(2)*X(3)*X(4)*Z(0)
M1
```

```python
M2 = Z(3)*X(4)*X(0)*Z(1)
M2
```

```python
M3 = Z(4)*X(0)*X(1)*Z(2)
M3
```

These operators should mutually commute.

```python
gate_simp(Commutator(M0,M1).doit())
```

And square to the identity.

```python
for o in [M0,M1,M2,M3]:
    display(gate_simp(o*o))
```

Codewords

```python
zero = Rational(1,4)*(1+M0)*(1+M1)*(1+M2)*(1+M3)*IntQubit(0, 5)
zero
```

```python
qapply(4*zero)
```

```python
one = Rational(1,4)*(1+M0)*(1+M1)*(1+M2)*(1+M3)*IntQubit(2**5-1, 5)
one
```

```python
qapply(4*one)
```

The encoding circuit

```python
encoding_circuit = H(3)*H(4)*CNOT(2,0)*CNOT(3,0)*CNOT(4,0)*H(1)*H(4)*\
                   CNOT(2,1)*CNOT(4,1)*H(2)*CNOT(3,2)*CNOT(4,2)*H(3)*\
                   H(4)*CNOT(4, 3)*Z(4)*H(4)*Z(4)
```

```python
circuit_plot(encoding_circuit, nqubits=5, scale=0.5);
```

```python
represent(4*encoding_circuit, nqubits=5)
```


```python
import numpy as np
import matplotlib.pyplot as plot
from sympy import *
%matplotlib inline
```

```python
x = symbols('x')
y = x**2 + (6 * x) - 3
solve(y, x)
```

    [-3 + 2*sqrt(3), -2*sqrt(3) - 3]

```python
print(-3 + 2*np.sqrt(3))
print(-2*np.sqrt(3) - 3)
plot(y, xlim=(-1, 1), ylim=(-1, 1))
```

```python

```


```python
from __future__ import print_function
import numpy as np
import math
from IPython.display import display
from sympy import *
init_printing()
import matplotlib.pyplot as plt
%matplotlib inline

def display_eq(a,e):
    if isinstance(a, str):
            a = Symbol(a)
    display(Eq(a,e))
```

## Gaussian Type Orbitals

Gaussian-type orbitals are frequently used for single particle orbitals in QMC.

```python
x,y,z = symbols('x y z')
alpha = Symbol('alpha', positive=True, real=True)
r = Symbol('r',real=True,nonnegative=True)
i,j,k = symbols('i j k',integer=True)
N = Symbol('N')
```

The form for the primitive orbital is

```python
gto_sym = N * x**i * y**j * z**k * exp(-alpha *r**2)
display(gto_sym)
```

The normalization ($N$) is given by

```python
n1 = factorial(i)*factorial(j)*factorial(k)
n2 = factorial(2*i)*factorial(2*j)*factorial(2*k)
norm_sym = (2*alpha/pi)**(3/S(4)) * sqrt((8*alpha)**(i+j+k)*n1/n2)
norm_sym
```

Sometimes the normalization is written in terms of a double factorial.  The expression can be converted between notations using identities like $\frac{(2i)!}{!i!2^i} = (2i-1)!!$.

For more information, see the [Double Factorial entry at MathWorld](http://mathworld.wolfram.com/DoubleFactorial.html)

QMCPACK splits the normalization between the radial and angular parts.  The normalization can be derived.

### Check normalization

Derive the normalization for the simplest case.

```python
# Derive the normalization for the simplest case (i=j=k=0)
gto000 = gto_sym.subs({N:1,i:0,j:0,k:0})
display_eq('phi_000',gto000)
```

```python
# Integrate the radial part
val_int = integrate(r*r*gto000*gto000, (r,0,oo))
display(val_int)
```

```python
# Normalization for just the radial part
radial_norm = 1/sqrt(val_int)
display_eq('N_radial',radial_norm)
```

```python
# The full norm includes the factor of 4*pi from the angular integration
full_norm = 1/sqrt(4*pi*val_int)
display_eq('N_000',full_norm)
```

```python
# Compare with the previous expression for N
norm_sym.subs({i:0,j:0,k:0})
```

### Higher angular momentum
Now check the normalization for higher values of $L=i+j+k$.

```python
# L = i+j+k acts like angular momentum state
L = Symbol('L', integer=True, positive=True)
# Use just powers of x, general enough for now
gtoL00 = gto_sym.subs({N:1,i:L,j:0,k:0})
display_eq('phi_L00',gtoL00)
```

```python
# Will need some values for the spherical integral
theta = Symbol('theta')
phi = Symbol('phi')
x_in_spherical_coords = r*sin(theta)*cos(phi)
surface_element = r*r*sin(theta)
display_eq('x',x_in_spherical_coords)
surface_element
```

```python
# substitute x to get the integrand in spherical coordinates
e_sph = gtoL00.subs(Symbol('x'), x_in_spherical_coords)
#display(e_sph)
e_int = e_sph*e_sph*surface_element
e_int
```

```python
# Perform radial integral
val_L00 = integrate(e_int,(r,0,oo))
val_L00
```

```python
# Replace the Gamma function, from https://en.wikipedia.org/wiki/Gamma_function
gamma_half = 2**(-2*L) *sqrt(pi) * factorial(2*L)/factorial(L)
display_eq(gamma(L+S.Half),gamma_half)
```

```python
# use gamma(x+1) = x * gamma(x)  and the previous formula
tmp00 = val_L00.subs(gamma(L+3*S.Half), (L+S.Half)*gamma_half)
display(tmp00)
norm_radial_L00 = 1/sqrt(tmp00.rewrite(factorial))
display(norm_radial_L00)

# Remove sin and cos terms
tmp = simplify(norm_radial_L00.subs({sin(theta):1,cos(phi):1}))
tmp
```

```python
# Integral for general L doesn't seem to work in Sympy, but any particular value of L works
integrate(val_L00.subs(L,1), (theta, 0, pi),(phi,0,2*pi))
```

### Sum of primitive basis functions

```python
c = IndexedBase('c')
alpha2 = IndexedBase('alpha')
norm2 = IndexedBase('N')
i = Symbol('i',integer=True)
# Fix to size 3, could be general
cg_sym = Sum(norm2[i]*c[i]*gto000.subs(alpha,alpha2[i]),(i,1,3))
cg_sym
```

```python
# STO-3G for Hydrogen
h_alpha = [3.42525091, 0.62391373, 0.16885540]
h_coeff = [0.15432897, 0.53532814, 0.44463454]
nbasis = len(h_coeff)

cg_unroll = cg_sym.doit()
for idx in range(nbasis):
    cg_unroll = cg_unroll.subs(c[idx+1], h_coeff[idx])
    cg_unroll = cg_unroll.subs(alpha2[idx+1],h_alpha[idx])
    #cg_unroll = cg_unroll.subs(norm2[idx+1],norm_sym.subs({i:0,j:0,k:0,alpha:h_alpha[idx]}))
    print(h_coeff[idx],h_alpha[idx],radial_norm.subs(alpha,h_alpha[idx]).evalf())
    cg_unroll = cg_unroll.subs(norm2[idx+1],radial_norm.subs({alpha:h_alpha[idx]}))

cg_unroll.evalf()
```

```python
# Evaluate at a concrete value for r
cg_unroll.subs(r,1.3).evalf()
```

```python
# Can check derivatives as well
d_cg = diff(cg_unroll, r)
d_cg.subs(r, 1.3).evalf()
```

```python
# Plot the primitive gaussians, the overall GTO, and the STO it mimics

nplot = 161
delta_plot = .05
rvals = np.zeros(nplot)
yvals = np.zeros(nplot)
indvals = np.zeros((3,nplot))
stovals = np.zeros(nplot)
for i in range(nplot):
    rval = i*delta_plot - 4.0
    y = cg_unroll.subs(r,rval).evalf()
    rvals[i] = rval
    yvals[i] = y
    stovals[i] = 2.0*math.exp(-abs(rval))
    for j in range(3):
        val = h_coeff[j]*gto000.subs({alpha:h_alpha[j], r:rval})*radial_norm.subs({alpha:h_alpha[j]}).evalf()
        #print(val)
        indvals[j,i] = val

plt.plot(rvals, yvals, label='H GTO')
plt.plot(rvals, stovals, label='H STO')
plt.plot(rvals, indvals[0,:], label= r'$\alpha=%.2f$'%h_alpha[0])
plt.plot(rvals, indvals[1,:], label= r'$\alpha=%.2f$'%h_alpha[1])
plt.plot(rvals, indvals[2,:], label= r'$\alpha=%.2f$'%h_alpha[2])
plt.legend()
```

```python
# Compute normalization for STO for previous plot
sto00 = exp(-r)
val = integrate(sto00*sto00*r*r,(r,0,oo))
1/math.sqrt(val)
```

```python

```