# 1.1 The First Programming Encounter: a Formula

## 1.1.1 Using a Program as a Calculator

```python
print(5*0.6 - 0.5*9.81*0.6**2)
```

    1.2342

## 1.1.5 Warning About Typing Program Text

```python
%%capture

# Code (illegal):
"""
write 5*0.6 - 0.5*9.81*0.6^2
"""

# Result:
"""
File "<ipython-input-3-a4896c6d3f86>", line 1
    write 5*0.6 - 0.5*9.81*0.6^2
          ^
SyntaxError: invalid syntax
"""
```

## 1.1.7 Using Variables

```python
v0 = 5
g = 9.81
t = 0.6
y = v0*t - 0.5*g*t**2
print(y)
```

    1.2342

## 1.1.8 Names of Variables

```python
initial_velocity = 5
acceleration_of_gravity = 9.81
TIME = 0.6
VerticalPositionOfBall = initial_velocity*TIME - \
                         0.5*acceleration_of_gravity*TIME**2
print(VerticalPositionOfBall)
```

    1.2342

## 1.1.10 Comments

```python
# Program for computing the height of a ball in vertical motion.
v0 = 5   # initial velocity
g = 9.81 # acceleration of gravity
t = 0.6  # time
y = v0*t - 0.5*g*t**2  # vertical position
print(y)
```

    1.2342

```python
# -*- coding: ASCII -*-
# halløchen
```

scheint trotzdem zu klappen... sonst:

```python
# -*- coding: utf-8 -*-
```

## 1.1.11 Formatting Text and Numbers

```python
print('At t=%g s, the height of the ball is %.2f m.' % (t, y))
```

    At t=0.6 s, the height of the ball is 1.23 m.

*Format string syntax*:

```python
print("At t={t:g} s, the height of the ball is {y:.2f} m.".format(t=t, y=y))
```

    At t=0.6 s, the height of the ball is 1.23 m.

```python
print("""
At t={t:f} s, a ball with
initial velocity v0={v0:.3E} m/s
is located at the height {y:.2f} m.
""".format(t=t, v0=v0, y=y))
```

    At t=0.600000 s, a ball with
    initial velocity v0=5.000E+00 m/s
    is located at the height 1.23 m.

```python
# Meiner Meinung nach eine schönere Alternative
x = 1234567890
print(f"x = {x:g}")
```

    x = 1.23457e+09

# 1.2 Computer Science Glossary

```python
v0 = 3; g = 9.81; t = 0.6
y = v0*t - 0.5*g*t**2
print(y)
```

    0.034199999999999786

```python
# mathematically false, but in programming just an assignment:
y = y + 3
```

```python
y = 3
print(y)
y = y + 4
print(y)
y = y*y
print(y)
```

    3
    7
    49

```python
v0=3;g=9.81;t=0.6;y=v0*t-0.5*g*t**2;print(y)
```

    0.034199999999999786

# 1.3 Another Formula: Celsius-Fahrenheit Conversion

## 1.3.1 Potential Error: Integer Division

```python
C = 21
F = (9//5)*C + 32
print(F)
# or:
print(9//5*C + 32)

# adjusted for python 3: now // remains floor (integer) division, and / is always float division
# floor division (a//b) calculates the largest integer c so that c*b <= a ("how many times does b fit into a")
```

    53
    53

## 1.3.3 Avoiding Integer Division

```python
from __future__ import division
print(9/5*C + 32)
# wow it works!
```

    69.80000000000001

```python
print(float(2))
```

    2.0

# 1.4 Evaluating Standard Mathematical Functions

## 1.4.1 Example: Using the Square Root Function

```python
v0 = 5
g = 9.81
yc = 0.2
import math
t1 = (v0 - math.sqrt(v0**2 - 2*g*yc))/g
t2 = (v0 + math.sqrt(v0**2 - 2*g*yc))/g
print("At t=%g s and %g s, the height is %g m." % (t1, t2, yc))
```

    At t=0.0417064 s and 0.977662 s, the height is 0.2 m.

```python
import math as m
# m is now the name of the math module
v = m.sin(m.pi)
from math import log as ln
v = ln(5)
from math import sin as s, cos as c, log as ln
x = 1234
v = s(x)*c(x) + ln(x)
print(v)
```

    6.637346500400129

```python
import math
m = math
ln = m.log
s = m.sin
c = m.cos
v = s(x)*c(x) + ln(x)
print(v)
```

    6.637346500400129

## 1.4.2 Example: Computing with $\text{sinh}(x)$

```python
from math import sinh, exp, e, pi
x = 2*pi
r1 = sinh(x)
r2 = 0.5*(exp(x) - exp(-x))
r3 = 0.5*(e**x - e**(-x))
print(r1, r2, r3)
```

    267.74489404101644 267.74489404101644 267.7448940410163

## 1.4.3 A First Glimpse of Rounding Errors

```python
print("%.16f %.16f %.16f" % (r1,r2,r3))
```

    267.7448940410164369 267.7448940410164369 267.7448940410163232

# 1.5 Interactive Computing

## 1.5.2 Type Conversion

```python
X = 123.9999
print(X, type(X))
X = int(X)
print(X, type(X))
# Decimals are truncated
```

    123.9999 <class 'float'>
    123 <class 'int'>

## 1.5.3 IPython

```python
a = 21
a
```

    21

```python
_ / 7
```

    3.0

```python
__ + 4
```

    25

```python
# !command für system-befehle. (in der ipython-konsole) like:
# !fortune
# !date
# !ls
```

# 1.6 Complex Numbers

## 1.6.1 Complex Arithmetics in Python

```python
u = 2.5 + 3j
u
```

    (2.5+3j)

```python
%%capture

# Code (illegal):
"""
u = 2.5 + j3
"""

# Result:
"""
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-31-a2fc771f4bd2> in <module>()
----> 1 u = 2.5 + j3
      2 u

NameError: name 'j3' is not defined
"""
```

```python
v = 2
u * v
```

    (5+6j)

```python
u**-6/7
```

    (2.0846166182505192e-05+3.446647537619421e-05j)

```python
a = 2; b = 3
c = complex(a, b)
c
```

    (2+3j)

```python
print(f"Re(c) = {c.real}, Im(c) = {c.imag}, c* = {c.conjugate()}")
```

    Re(c) = 2.0, Im(c) = 3.0, c* = (2-3j)

## 1.6.2 Complex Functions in Python

```python
%%capture

# Code (illegal):
"""
from math import sin
r = sin(c)
"""

# Result:
"""
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-51-771d0302db05> in <module>()
      1 from math import sin
----> 2 r = sin(c)

TypeError: can't convert complex to float
"""
```

```python
# Korrekte implementierung mit cmath:
from cmath import sin, exp, pi
print(f"{sin(c):g}")
print(f"{exp(pi*1j):.4f}")
```

    9.1545-4.16891j
    -1.0000+0.0000j

## 1.6.3 Unified Treatment of Complex and Real Functions

Für Berechnungen, wo komlexe und reelle Zahlen gemischt vorkommen, kann man eine andere Library benutzen:

```python
from numpy.lib.scimath import sqrt # bzw. from numpy.lib.scimath import * für alle funktionen.

# Die gleichen Funktionen die hier enthalten sind, sind auch in scipy und scitools.std enthalten:
# from scipy import *
# from scitools.std import *
```

### `math` (float)

```python
from math import sqrt

sqrt(4)    # float
```

    2.0

```python
%%capture

# Code (illegal):
"""
sqrt(-1)
"""

# Result:
"""
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-79-75be91ddb76a> in <module>()
----> 1 sqrt(-1)

ValueError: math domain error
"""
```

### `cmath` (complex)

```python
from cmath import sqrt

print(sqrt(4))     # complex
print(sqrt(-1))    # complex
```

    (2+0j)
    1j

### `numpy.lib.scimath` (mixed)

```python
from numpy.lib.scimath import sqrt

print(sqrt(4))     # float
print(sqrt(-1))    # complex
```

    2.0
    1j

Roots of a quadratic funtion:

```python
a = 1; b = 2; c = 100    # polynomial coefficients
from numpy.lib.scimath import sqrt
r1 = (-b + sqrt(b**2 - 4*a*c))/(2*a)
r2 = (-b - sqrt(b**2 - 4*a*c))/(2*a)
print(r1, r2)
```

    (-1+9.94987437107j) (-1-9.94987437107j)

```python
c = -20    # updated coefficient
r1 = (-b + sqrt(b**2 - 4*a*c))/(2*a)
r2 = (-b - sqrt(b**2 - 4*a*c))/(2*a)
print(r1, r2)
```

    3.58257569496 -5.58257569496

# 1.7 Symbolic Computing

## 1.7.1 Basic Differentiation and Integration

```python
from sympy import (
    symbols,      # define symbols for symbolic math
    diff,         # differentiate expressions
    integrate,    # integrate expressions
    Rational,     # define rational numbers
    lambdify,     # turn symbolic expr. into Python functions
    )

t, v0, g = symbols('t v0 g')
y = v0*t - Rational(1,2)*g*t**2
dydt = diff(y, t)
dydt
```

    -g*t + v0

```python
print("acceleration:", diff(y, t, t))
```

    acceleration: -g

```python
y2 = integrate(dydt, t)
y2
```

    -g*t**2/2 + t*v0

```python
# lambdify:
v = lambdify((t, v0, g),       # arguments in v
             dydt,             # symbolic expression
             dummify=False)    # damit argumente genannt werden dürfen, ohne dies geht nur v(a,b,c) nicht v(t=a,v0=b,g=c)

v(t = 0, v0 = 5, g = 9.81)
```

    5.0

## 1.7.2 Equation Solving

```python
# solve(e, x): solve expression e to x (for e = 0)
from sympy import solve
roots = solve(y, t)
roots
```

    [0, 2*v0/g]

```python
# inserting values with subs:
y1 = y.subs(t, roots[0])
y2 = y.subs(t, roots[1])
print(y1, y2)
```

    0 0

## 1.7.3 Taylor Series and More

```python
from sympy import exp, sin, cos

# taylor series of e^t of order 3 around t=0
f = exp(t)
f.series(t, 0, 3)
```

    1 + t + t**2/2 + O(t**3)

```python
# latex encoding
from sympy import latex
print(latex(f.series(t, 0, 7)))
```

    1 + t + \frac{t^{2}}{2} + \frac{t^{3}}{6} + \frac{t^{4}}{24} + \frac{t^{5}}{120} + \frac{t^{6}}{720} + \mathcal{O}\left(t^{7}\right)

Result:
$$1 + t + \frac{t^{2}}{2} + \frac{t^{3}}{6} + \frac{t^{4}}{24} + \frac{t^{5}}{120} + \frac{t^{6}}{720} + \mathcal{O}\left(t^{7}\right)$$

```python
# expanding and simplifying expressions

from sympy import symbols, sin, cos, simplify, expand
x, y = symbols('x y')
f = -sin(x)*sin(y) + cos(x)*cos(y)
print(simplify(f))
print(expand(sin(x+y), trig=True)) # requires a trigonometric hint
```

    cos(x + y)
    sin(x)*cos(y) + sin(y)*cos(x)

 # 1.8 Summary

## 1.8.2 : Example: Trajectory of a Ball

### Solution: Onno Eberhard

```python
from sympy import Symbol, lambdify, tan, cos, pi

# Symbols and constants
x = Symbol('x')            # horizontal coordinate [m]
v0 = Symbol('v0')          # initial velocity [km / h]
theta = Symbol('theta')    # angle to x-axis of initial velocity [degrees]
y0 = Symbol('y0')          # initial height [m]
g = 9.81                   # acceleration of gravity [m / s^2]

# Unit conversions
theta = theta * pi / 180    # theta: [degrees] -> [rad]
v0 = v0 * 1000 / 60 / 60    # v0: [km / h] -> [m / s]

# The formula
y = x*tan(theta) - (g * x**2)/(2 * v0**2 * cos(theta)**2) + y0
f = lambdify('x, v0, theta, y0', y, dummify = False)

# Result
y1 = f(x = 0.5, v0 = 15, theta = 60, y0 = 1)
print(f"y1 = {y1:g} m")
```

    y1 = 1.5835 m

### Solution: Book

```python
g = 9.81    # m/s**2
v0 = 15     # km/h
theta = 60  # degrees
x = 0.5     # m
y0 = 1      # m

print("""\
v0    = %.1f km/h
theta = %d degrees
y0    = %.1f m
x     = %.1f m\
""" % (v0, theta, y0, x))

from math import pi, tan, cos
# Convert v0 to m/s and theta to radians
v0 = v0/3.6
theta = theta*pi/180

y = x*tan(theta) - 1/(2*v0**2)*g*x**2/((cos(theta))**2) + y0

print('y     = %.1f m' % y)
```

    v0    = 15.0 km/h
    theta = 60 degrees
    y0    = 1.0 m
    x     = 0.5 m
    y     = 1.6 m

Chapter 1 - Computing with Formulas.ipynb


#Funzione convergente al limite L
Data la seguente definizione di limte

$$
\forall \epsilon > 0; \exists \delta_{\epsilon} > 0: \\
x \in (A\setminus\{x0\}\cap I(x_{0};\delta_{\epsilon}))\Rightarrow \\
f(x)-L| < \epsilon \\
f(x) \in I(L; \epsilon)
$$

per comodità diamo una nomeclatura per l'insime a cui appartiene x

\begin{align}
A^*(x_0; \delta)& = (A\setminus\{x_0\}) \cap I(x_0;\delta)\\
A^*(x_0; \delta)& = A \cap I(x_0;\delta)\\
\end{align}

Di conseguenza semplifico la notazione di convergenza a un limite L

$$
\forall \epsilon > 0; \exists \delta_\epsilon > 0: \\
f(A^*(x_0; \delta_\epsilon)) \subseteq I(L; \epsilon)
$$

Data questa notazione di convergenza a un limite L possiamo semplificare la definizione di funzione continua

$$
\forall \epsilon \gt 0: \exists \delta_\epsilon \gt 0: \\
f(A(x_0; \delta_\epsilon)) \subseteq I(f(x); \epsilon)
$$

# Funzioni continue

###Polinomio primo grado

Prendiamo un esempio di funzione affine continua

$$f(x) = mx+q$$

Questa è l'equazione di una retta su un piano avente m come coefficente angolare.
Prendendo $m=0$ questa è ovviamente continua

Andiamo quindi a chiederci se è continua con $m\ne0$
$$
\forall \epsilon \gt 0 \\
|f(x)-f(x_0)| \lt \epsilon \\
|f(x)-f(x_0)| = |m||x-x_0| \lt \epsilon \\
|x-x_0| \lt \epsilon/|m| \\
\delta_\epsilon = \epsilon/|m|
$$

Avendo definito che $m\ne0$ ne risulta che $\delta_\epsilon$ è sempre definito e quindi la funzione è continua in tutti i suoi punti

### Funzione radice quadrata

Analizzo ora un caso più complesso
$$y = \sqrt(x)\\ x \gt 0%$$

```
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 1000, 10000)
y = np.sqrt(x)

plt.plot(x, y)
```

Anche in questo caso voglio chiedermi se la funzione è continua.
Il dubbio è nell'intorno dell'origine dove intuisco che il coefficente angolare in quel punto sembra tendere all'infinito

Per arrivare a questo risultato affronto prima la dimostrazione di un'altra disequazione che poi mi aiuterà nel compito

\begin{align}
0 \le x_1 \lt x_2 \Rightarrow \\
\sqrt{x_2} - \sqrt{x_1} & \le \sqrt{x_2 - x_1} \\
x_2 - x_1 - 2\sqrt{x_2x_1} & \le x_2 - x_1 \\
2x_1 & \le 2\sqrt{x_2x_1} \\
x_1 & \le \sqrt{x_2x_1} \\
x_1^2 & \le x_1x_2 \\
x_1 & \le x_2 \\
CVD
\end{align}

Ora posso procedere alla dimostrazione di continuità

$$
y = \sqrt{x}, A = [0, +\infty) \\
\forall \epsilon \gt 0; \exists x_0 \lt x \Rightarrow \\
|f(x) - f(x_0)| \lt \epsilon \\
\sqrt{x} - \sqrt{x_0} \lt \epsilon \\
$$
Dalla disequazione precedente otteniamo
$$
\sqrt{x} - \sqrt{x_0} \le \sqrt{x-x_0} \lt \epsilon \\
x - x_0 \lt \epsilon^2 \\
\delta_\epsilon = \epsilon^2
$$

#Funzioni divergenti
Come per le funzioni convergenti che tendono a un valore $L$ andiamo a definire le funzioni divergenti.
Queste possono divergere sia positivamente che negativamente.
Qui riporto la definizione di funzione che diverge positivamente:

$$
\forall M \gt 0 \exists \delta_M \gt 0\\
x \in A^*(x_0, \delta_M) \Rightarrow f(x) > M\\
$$

#Definizione di limite che tende a infinito
Anche per questa definizione vale sia il tendere a $+\infty$ che $+\infty$
Inizio col caso di $+\infty$, quindi analizzo una funzione definita su un dominio $A$ illimitato superiormente, condizione che possiamo indicare come $$ supA = +\infty $$
Posso quindi definire il limite che tende a infinito come segue
$$
\lim_{x\to +\infty}f(x) = L;\\
\forall \epsilon \gt 0: \exists \delta_\epsilon \gt 0 \Rightarrow \\
\forall x \in A \cap (\delta_\epsilon, +\infty) \Rightarrow \\
f(x) \in I(L,\epsilon); |f(x)-L| \lt \epsilon
$$
Il limite $x\to -\infty$ è definito in maniera equivalente

##Funzione $1/x$
Vado quindi ad analizzare il limite di $y = 1/x$
$$
1/x - 1/x0 \lt \epsilon \\
x - x0 \lt 1/\epsilon \\
\lim_{x\to +\infty} = 1/\epsilon = 0 \\
$$
Posso visualizzare questa relazione in questo modo.
Quando $x \gt 1/\epsilon$ allora $f(x) \in (-L/\epsilon, L/\epsilon)$ con $L = 0$.
Vado quindi a rappresentare su un grafico la mia funzione e 2 rette
$$
r_1 = -\epsilon \\
r_2 = \epsilon \\
$$
E potrò vedere come qualunque $\epsilon$ scelgo, se visualizzo almeno l'intorno $(-1/\epsilon, 1/\epsilon)$ la mia funzione entrerà tra le due rette di riferimento.

```
import matplotlib.pyplot as plt
from matplotlib.widgets import Slider
import numpy as np
from IPython.display import Math

e = 0.0001
x = np.linspace(-2/e, 2/e, 1000)
y = 1/x
f = plt.plot(x, y)
s1 = plt.plot(x, (x/x)+(e-1))
s2 = plt.plot(x, (x/x)-(e+1))
axis = plt.axis((-2/e, 2/e, -e*10, e*10))
```

#Limiti destro e sinistro
Dato l'esempio precedente ora posso chiedermi qual'è $\lim_{x\to 0}f(x)$?
Se analizzo solo i valori $x > 0$ allora è facile vedere che il limite è $+\infty$.
Per i valori $x < 0$ noto però che il limite è $-\infty$
Non posso quindi definire un unico limite per $x\to 0$
Sfruttando il fatto che l'insieme $R$ è ordinato vado come prima cosa ad aggiungere una nuova notazione.
$$
A^+(x_0) := \{x \in A; x \gt x_0\} \\
A^-{x_0} := \{x \in A; x \lt x_0\}
$$
Quindi posso definire il limite "destro" come il limite della funzione ristretta all'insieme $A^+{x_0}$
$$
g(x) = f(x); x \in A^+{x_0} \\
\lim_{x\to x^+_0}f(x) =
\lim_{x\downarrow 0}f(x) =
\lim_{x\to0}g(x)
$$

Allo stesso modo posso definire il limite "sinistro" restrigento il dominio della funzione solo all'insieme $A^-{x_0}$

##Unicità del limite
Posso quindi notare che il limite in un punto esiste se e solo se in quel punto esistono e sono coincidenti sia i limiti destro e sinistro.
E in questo caso il limite sarà uguale al limite destro e sinistro.

Di conseguenza se un limite esiste è unico.

Algebra 12 Estensione della nozione di limite.ipynb


# Bias-Variance Decomposition

Bias variance decomposition of machine learning algorithms for various loss functions.

> `from mlxtend.evaluate import bias_variance_decomp`

## Overview

Often, researchers use the terms *bias* and *variance* or "bias-variance tradeoff" to describe the performance of a model -- i.e., you may stumble upon talks, books, or articles where people say that a model has a high variance or high bias. So, what does that mean? In general, we might say that "high variance" is proportional to overfitting, and "high bias" is proportional to underfitting.

Anyways, why are we attempting to do this bias-variance decomposition in the first place? The decomposition of the loss into bias and variance helps us understand learning algorithms, as these concepts are correlated to underfitting and overfitting.

To use the more formal terms for bias and variance, assume we have a point estimator $\hat{\theta}$ of some parameter or function $\theta$. Then, the bias is commonly defined as the difference between the expected value of the estimator and the parameter that we want to estimate:

$$
\text{Bias} = E[\hat{\theta}] - \theta.
$$

If the bias is larger than zero, we also say that the estimator is positively biased, if the bias is smaller than zero, the estimator is negatively biased, and if the bias is exactly zero, the estimator is unbiased. Similarly, we define the variance as the difference between the expected value of the squared estimator minus the squared expectation of the estimator:

$$
\text{Var}(\hat{\theta}) = E\big[\hat{\theta}^2\big] - \bigg(E\big[\hat{\theta}\big]\bigg)^2.
$$

Note that in the context of this lecture, it will be more convenient to write the variance in its alternative form:

$$
\text{Var}(\hat{\theta}) = E[(E[{\hat{\theta}}] - \hat{\theta})^2].
$$

To illustrate the concept further in context of machine learning ...

Suppose there is an unknown target function or "true function" to which we do want to approximate. Now, suppose we have different training sets drawn from an unknown distribution defined as "true function + noise." The following plot shows different linear regression models, each fit to a different training set. None of these hypotheses approximate the true function well, except at two points (around x=-10 and x=6). Here, we can say that the bias is large because the difference between the true value and the predicted value, on average (here, average means "expectation of the training sets" not "expectation over examples in the training set"), is large:

The next plot shows different unpruned decision tree models, each fit to a different training set. Note that these hypotheses fit the training data very closely. However, if we would consider the expectation over training sets, the average hypothesis would fit the true function perfectly (given that the noise is unbiased and has an expected value of 0). As we can see, the variance is very large, since on average, a prediction differs a lot from the expectation value of the prediction:

### Bias-Variance Decomposition of the Squared Loss

We can decompose a loss function such as the squared loss into three terms, a variance, bias, and a noise term (and the same is true for the decomposition of the 0-1 loss later). However, for simplicity, we will ignore the noise term.

Before we introduce the bias-variance decomposition of the 0-1 loss for classification, let us start with the decomposition of the squared loss as an easy warm-up exercise to get familiar with the overall concept.

The previous section already listed the common formal definitions of bias and variance, however, let us define them again for convenience:

$$
\text{Bias}(\hat{\theta}) = E[\hat{\theta}] - \theta, \quad \text{Var}(\hat{\theta}) = E[(E[{\hat{\theta}}] - \hat{\theta})^2].
$$

Recall that in the context of these machine learning lecture (notes), we defined

- the true or target function as $y = f(x)$,
- the predicted target value as $\hat{y} = \hat{f}(x) = h(x)$,
- and the squared loss as $S = (y - \hat{y})^2$. (I use $S$ here because it will be easier to tell it apart from the $E$, which we use for the *expectation* in this lecture.)

**Note that unless noted otherwise, the expectation is over training sets!**

To get started with the squared error loss decomposition into bias and variance, let use do some algebraic manipulation, i.e., adding and subtracting the expected value of $\hat{y}$ and then expanding the expression using the quadratic formula $(a+b)^2 = a^2 + b^2 + 2ab)$:

$$
\begin{equation}
\begin{split}
S = (y - \hat{y})^2 \\
(y - \hat{y})^2 &= (y - E[{\hat{y}}] + E[{\hat{y}}] - \hat{y})^2 \\
&= (y-E[{\hat{y}}])^2 + (E[{\hat{y}}] - y)^2 + 2(y - E[\hat{y}])(E[\hat{y}] - \hat{y}).
\end{split}
\end{equation}
$$

Next, we just use the expectation on both sides, and we are already done:

$$
\begin{align}
E[S] &= E[(y - \hat{y})^2] \\
E[(y - \hat{y})^2]
&= (y-E[{\hat{y}}])^2 + E[(E[{\hat{y}}] - \hat{y})^2]\\
&= \text{[Bias]}^2 + \text{Variance}.
\end{align}
$$

You may wonder what happened to the "$2ab$" term ($2(y - E[\hat{y}])(E[\hat{y}] - \hat{y})$) when we used the expectation. It turns that it evaluates to zero and hence vanishes from the equation, which can be shown as follows:

$$
\begin{align}
E[2(y - E[{\hat{y}}])(E[{\hat{y}}] - \hat{y})]  &= 2 E[(y - E[{\hat{y}}])(E[{\hat{y}}] - \hat{y})] \\
&=  2(y - E[{\hat{y}}])E[(E[{\hat{y}}] - \hat{y})] \\
&= 2(y - E[{\hat{y}}])(E[E[{\hat{y}}]] - E[\hat{y}])\\
&= 2(y - E[{\hat{y}}])(E[{\hat{y}}] - E[{\hat{y}}]) \\
&= 0.
\end{align}
$$

So, this is the canonical decomposition of the squared error loss into bias and variance. The next section will discuss some approaches that have been made to decompose the 0-1 loss that we commonly use for classification accuracy or error.

The following figure is a sketch of variance and bias in relation to the training error and generalization error -- how high variance related to overfitting, and how large bias relates to underfitting:

### Bias-Variance Decomposition of the 0-1 Loss

Note that decomposing the 0-1 loss into bias and variance components is not as straight-forward as for the squared error loss. To quote Pedro Domingos, a well-known machine learning researcher and professor at University of Washington:

> "several authors have proposed bias-variance decompositions related to zero-one loss (Kong & Dietterich, 1995; Breiman, 1996b; Kohavi & Wolpert, 1996; Tibshirani, 1996; Friedman, 1997). However, each of these decompositions has significant shortcomings.". [1]

In fact, the paper this quote was taken from may offer the most intuitive and general formulation at this point. However, we will first, for simplicity, go over Kong & Dietterich formulation [2] of the 0-1 loss decomposition, which is the same as Domingos's but excluding the noise term (for simplicity).

The table below summarizes the relevant terms we used for the squared loss in relation to the 0-1 loss. Recall that the 0-1 loss, $L$, is 0 if a class label is predicted correctly, and one otherwise. The main prediction for the squared error loss is simply the average over the predictions $E[\hat{y}]$ (the expectation is over training sets), for the 0-1 loss Kong & Dietterich and Domingos defined it as the mode. I.e., if a model predicts the label one more than 50% of the time (considering all possible training sets), then the main prediction is 1, and 0 otherwise.

|               -              | Squared Loss                    | 0-1 Loss                    |
|------------------------------|---------------------------------|-----------------------------|
| Single loss                  | $(y - \hat{y})^2$               | $L(y, \hat{y})$             |
| Expected loss                | $E[(y - \hat{y})^2]$            | $E[L(y, \hat{y})]$          |
| Main prediction $E[\hat{y}]$ | mean (average)                  | mode                        |
| Bias$^2$                     | $(y-E[{\hat{y}}])^2$            | $L(y, E[\hat{y}])$          |
| Variance                     | $E[(E[{\hat{y}}] - \hat{y})^2]$ | $E[L(\hat{y}, E[\hat{y}])]$ |

Hence, as result from using the mode to define the main prediction of the 0-1 loss, the bias is 1 if the main prediction does not agree with the true label $y$, and 0 otherwise:

$$
Bias = \begin{cases}
1 \text{ if }  y \neq E[{\hat{y}}], \\
0 \text{ otherwise}.
\end{cases}
$$

The variance of the 0-1 loss is defined as the probability that the predicted label does not match the main prediction:

$$
Variance = P(\hat{y} \neq E[\hat{{y}}]).
$$

Next, let us take a look at what happens to the loss if the bias is 0. Given the general definition of the loss, loss = bias + variance, if the bias is 0, then we define the loss as the variance:

$$
Loss = 0 + Variance = Loss = P(\hat{y} \neq y) = Variance = P(\hat{y} \neq E[\hat{{y}}]).
$$

In other words, if a model has zero bias, it's loss is entirely defined by the variance, which is intuitive if we think of variance in the context of being proportional overfitting.

The more surprising scenario is if the bias is equal to 1. If the bias is equal to 1, as explained by Pedro Domingos, the increasing the variance can decrease the loss, which is an interesting observation. This can be seen by first rewriting the 0-1 loss function as

$$
Loss = P(\hat{y} \neq y) = 1 - P(\hat{y} = y).
$$

(Note that we have not done anything new, yet.) Now, if we look at the previous equation of the bias, if the bias is 1, we have  $ y \neq E[{\hat{y}}]$. If $y$ is not equal to the main prediction, but $y$ is also is equal to $\hat{y}$, then $\hat{y}$ must be equal to the main prediction. Using the "inverse" ("1 minus"), we can then write the loss as

$$
Loss = P(\hat{y} \neq y) = 1 - P(\hat{y} = y) = 1 - P(\hat{y} \neq E[{\hat{y}}]).
$$

Since the bias is 1, the loss is hence defined as "loss = bias - variance" if the bias is 1 (or "loss = 1 - variance"). This might be quite unintuitive at first, but the explanations Kong, Dietterich, and Domingos offer was that if a model has a very high bias such that it main prediction is always wrong, increasing the variance can be beneficial, since increasing the variance would push the decision boundary, which might lead to some correct predictions just by chance then. In other words, for scenarios with high bias, increasing the variance can improve (decrease) the loss!

### References

- [1] Domingos, Pedro. "A unified bias-variance decomposition." Proceedings of 17th International Conference on Machine Learning. 2000.
- [2] Dietterich, Thomas G., and Eun Bae Kong. Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Technical report, Department of Computer Science, Oregon State University, 1995.

## Example 1 -- Bias Variance Decomposition of a Decision Tree Classifier

```python
from mlxtend.evaluate import bias_variance_decomp
from sklearn.tree import DecisionTreeClassifier
from mlxtend.data import iris_data
from sklearn.model_selection import train_test_split

X, y = iris_data()
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.3,
                                                    random_state=123,
                                                    shuffle=True,
                                                    stratify=y)

tree = DecisionTreeClassifier(random_state=123)

avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
        tree, X_train, y_train, X_test, y_test,
        loss='0-1_loss',
        random_seed=123)

print('Average expected loss: %.3f' % avg_expected_loss)
print('Average bias: %.3f' % avg_bias)
print('Average variance: %.3f' % avg_var)
```

    Average expected loss: 0.062
    Average bias: 0.022
    Average variance: 0.040

For comparison, the bias-variance decomposition of a bagging classifier, which should intuitively have a lower variance compared than a single decision tree:

```python
from sklearn.ensemble import BaggingClassifier

tree = DecisionTreeClassifier(random_state=123)
bag = BaggingClassifier(base_estimator=tree,
                        n_estimators=100,
                        random_state=123)

avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
        bag, X_train, y_train, X_test, y_test,
        loss='0-1_loss',
        random_seed=123)

print('Average expected loss: %.3f' % avg_expected_loss)
print('Average bias: %.3f' % avg_bias)
print('Average variance: %.3f' % avg_var)
```

    Average expected loss: 0.048
    Average bias: 0.022
    Average variance: 0.026

## Example 2 -- Bias Variance Decomposition of a Decision Tree Regressor

```python
from mlxtend.evaluate import bias_variance_decomp
from sklearn.tree import DecisionTreeRegressor
from mlxtend.data import boston_housing_data
from sklearn.model_selection import train_test_split

X, y = boston_housing_data()
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.3,
                                                    random_state=123,
                                                    shuffle=True)

tree = DecisionTreeRegressor(random_state=123)

avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
        tree, X_train, y_train, X_test, y_test,
        loss='mse',
        random_seed=123)

print('Average expected loss: %.3f' % avg_expected_loss)
print('Average bias: %.3f' % avg_bias)
print('Average variance: %.3f' % avg_var)
```

    Average expected loss: 31.917
    Average bias: 13.814
    Average variance: 18.102

For comparison, the bias-variance decomposition of a bagging regressor is shown below, which should intuitively have a lower variance than a single decision tree:

```python
from sklearn.ensemble import BaggingRegressor

tree = DecisionTreeRegressor(random_state=123)
bag = BaggingRegressor(base_estimator=tree,
                       n_estimators=100,
                       random_state=123)

avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(
        bag, X_train, y_train, X_test, y_test,
        loss='mse',
        random_seed=123)

print('Average expected loss: %.3f' % avg_expected_loss)
print('Average bias: %.3f' % avg_bias)
print('Average variance: %.3f' % avg_var)
```

    Average expected loss: 18.593
    Average bias: 15.354
    Average variance: 3.239

## API

```python
with open('../../api_modules/mlxtend.evaluate/bias_variance_decomp.md', 'r') as f:
    s = f.read()
print(s)
```

    ## bias_variance_decomp

    *bias_variance_decomp(estimator, X_train, y_train, X_test, y_test, loss='0-1_loss', num_rounds=200, random_seed=None)*

    estimator : object
    A classifier or regressor object or class implementing a `fit`
    `predict` method similar to the scikit-learn API.

    - `X_train` : array-like, shape=(num_examples, num_features)

        A training dataset for drawing the bootstrap samples to carry
        out the bias-variance decomposition.

    - `y_train` : array-like, shape=(num_examples)

        Targets (class labels, continuous values in case of regression)
        associated with the `X_train` examples.

    - `X_test` : array-like, shape=(num_examples, num_features)

        The test dataset for computing the average loss, bias,
        and variance.

    - `y_test` : array-like, shape=(num_examples)

        Targets (class labels, continuous values in case of regression)
        associated with the `X_test` examples.

    - `loss` : str (default='0-1_loss')

        Loss function for performing the bias-variance decomposition.
        Currently allowed values are '0-1_loss' and 'mse'.

    - `num_rounds` : int (default=200)

        Number of bootstrap rounds for performing the bias-variance
        decomposition.

    - `random_seed` : int (default=None)

        Random seed for the bootstrap sampling used for the
        bias-variance decomposition.

    **Returns**

    - `avg_expected_loss, avg_bias, avg_var` : returns the average expected

        average bias, and average bias (all floats), where the average
        is computed over the data points in the test set.


# Advection

## Conservation of mass

Imagine a fluid flowing in a narrow tube.  We'll use $q$ to indicate the density of the fluid and $u$ to indicate its velocity.  Both of these are functions of space and time: $q = q(x,t)$; $u=u(x,t)$.  The total mass in the section of tube $[x_1,x_2]$ is

\begin{equation}
\int_{x_1}^{x_2} q(x,t) dx.
\end{equation}

This total mass can change in time due to fluid flowing in or out of this section of the tube.  We call the rate of flow the *flux*, and represent it with the function $f(q)$.  Thus the net rate of flow of mass into (or out of) the interval $[x_1,x_2]$ at time $t$ is

$$f(q(x_1,t)) - f(q(x_2,t)).$$

We just said that this rate of flow must equal the time rate of change of total mass; i.e.

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = f(q(x_1,t)) - f(q(x_2,t)).$$

Now since $\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx = f(q(x_2,t)) - f(q(x_1,t))$, we can rewrite this as

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = -\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx.$$

Under certain smoothness assumptions on $q$, we can move the time derivative inside the integral.  We'll also put everything on the left side, to obtain

$$\int_{x_1}^{x_2} \left(\frac{\partial}{\partial t}q(x,t) + \frac{\partial}{\partial x} f(q)\right) dx = 0.$$

Since this integral is zero for *any* choice of $x_1,x_2$, it must be that the integrand (the expression in parentheses) is actually zero *everywhere*!  Therefore we can write the **differential conservation law**

$$q_t + f_x = 0.$$

Here and throughout the book, we use subscripts to denote partial derivatives.
This equation expresses the fact that the total mass is conserved -- since locally the mass can change only due to a net inflow or outflow.

## Advection
In order to solve the conservation law above, we need an expression for the flux, $f$.  The rate of flow is just mass times velocity: $f=u q$.  Thus we obtain the **continuity equation**

$$q_t + (uq)_x = 0.$$

In general, we need another equation to determine the velocity $u(x,t)$.  In later chapters we'll exam models in which the velocity depends on the density (or other properties), but for now let's consider the simplest case, in which all of the fluid flows at a single, constant velocity $u(x,t)=a$.  Then the continuity equation becomes the **advection equation**

\begin{align} \label{adv:advection}
    q_t + a q_x = 0.
\end{align}

This equation has a very simple solution.  If we are given the density $q(x,0)=q_0(x)$ at time zero, then the solution is just

\begin{align} \label{adv:solution}
    q(x,t) = q_0(x-at).
\end{align}

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
mpl.rcParams['font.size'] = 12

from exact_solvers import advection_figures
```

## Characteristics
Notice that the solution is constant along the line $x-at=C$, for each value of $C$.  These are parallel, straight lines in the $x-t$ plane, with slope $1/a$, as shown in the figure below.

```python
advection_figures.figure1()
```

We can think of the initial values $q_0(x)$ being transmitted along these lines; we sometimes say that *information* is transmitted along characteristics.

The next figure shows how smooth initial data (a Gaussian pulse of positive density) would propagate to the right along characteristics.

```python
advection_figures.figure2()
```

For more complicated hyperbolic problems, we may have multiple sets of characteristics, they may not be parallel, and the solution may not be constant along them.  But it will still be the case that information is propagated along characteristics.  The idea that information propagates at finite speed is an essential property of hyperbolic PDEs.

## The Riemann problem
The Riemann problem consists of a hyperbolic PDE (like (\ref{adv:advection}) together with piecewise constant initial data consisting of two states.  For convenience, we place the interface (or jump) at $x=0$ and refer to the left state (for $x<0$) as $q_l$ and the right state (for $x>0$) as $q_r$.  We thus have

\begin{align}
    q_0(x) & = \begin{cases} q_l & x<0 \\ q_r & x>0. \end{cases}
\end{align}

For the advection equation, the solution to the Riemann problem is immediately obvious; it is simply a special case of (\ref{adv:solution}).  The discontinuity initially at $x=0$ moves at speed $a$.  We have

\begin{align}
    q(x,t) & = \begin{cases} q_l & x-at<0 \\ q_r & x-at>0. \end{cases}
\end{align}

```python
advection_figures.figure3()
```

Notice how the initial discontinuity remains at the point defined by the characteristic coming from $x=0$ at $t=0$.


# Simple model using double exponentials

```python
from sympy import *
```

```python
from IPython.display import display, Markdown
```

```python
init_printing()
```

```python
t, P, e_r, e_d, delta_e, rho_e, g_e, i_r, i_d, delta_i, rho_i, g_i, b = symbols('t P \\tau_{er} \\tau_{ed} \\delta_e \\rho_e \\bar{g}_e \\tau_{ir} \\tau_{id} \\delta_i \\rho_i \\bar{g}_i \\beta', positive=True, real=True)
```

```python
SymbolDict = {t: "Time (ms)", P: "Proportion of $g_i/g_e$", e_r: "Excitatory Rise (ms)", e_d: "Excitatory Fall (ms)", delta_e: "Excitatory onset time (ms)", rho_e: "Excitatory $tau$ ratio (fall/rise)", g_e: "Excitatory max conductance", i_r : "Inhibitory Rise (ms)", i_d : "Inhibitory Fall(ms)", delta_i:"Inhibitory onset time(ms)", rho_i: "Inhibitory $tau$ ratio (fall/rise)", g_i: "Inhibitory max conductance", b: "Inhibitory/Excitatory $tau$ rise ratio"}
```

```python
estimateDict = { P: (1.9,2.1), e_r: (1.5,5), e_d: (8,20), delta_e: (0,0), rho_e: (2,7), g_e: (0.02,0.25), i_r : (1.5,5), i_d : (14,60), delta_i:(3,8), rho_i: (5,20), g_i:(0.04,0.5) , b:(0.5,5) }
```

```python
averageEstimateDict = {key: pow(value[0]*value[1],0.5) for key,value in estimateDict.items()}
```

```python
#averageEstimateDict = {key: ((value[0]+value[1])/2.) for key,value in estimateDict.items()}
```

```python
print "| Variable  |  Meaning |  Range |"
print "|---|---|---|"
print "|$t$|Time (ms)|0-100|"
for i in [P, e_r, e_d, delta_e, rho_e, g_e, i_r, i_d, delta_i, rho_i, g_i, b]:
    print "|${}$|{}|{}-{}|".format(i, SymbolDict[i], estimateDict[i][0], estimateDict[i][1])
```

    | Variable  |  Meaning |  Range |
    |---|---|---|
    |$t$|Time (ms)|0-100|
    |$P$|Proportion of $g_i/g_e$|1.9-2.1|
    |$\tau_{er}$|Excitatory Rise (ms)|1.5-5|
    |$\tau_{ed}$|Excitatory Fall (ms)|8-20|
    |$\delta_e$|Excitatory onset time (ms)|0-0|
    |$\rho_e$|Excitatory $tau$ ratio (fall/rise)|2-7|
    |$\bar{g}_e$|Excitatory max conductance|0.02-0.25|
    |$\tau_{ir}$|Inhibitory Rise (ms)|1.5-5|
    |$\tau_{id}$|Inhibitory Fall(ms)|14-60|
    |$\delta_i$|Inhibitory onset time(ms)|3-8|
    |$\rho_i$|Inhibitory $tau$ ratio (fall/rise)|5-20|
    |$\bar{g}_i$|Inhibitory max conductance|0.04-0.5|
    |$\beta$|Inhibitory/Excitatory $tau$ rise ratio|0.5-5|

| Variable  |  Meaning |  Range |
|---|---|---|
|$t$|Time (ms)|0-100|
|$P$|Proportion of $g_i/g_e$|1.9-2.1|
|$\tau_{er}$|Excitatory Rise (ms)|1.5-5|
|$\tau_{ed}$|Excitatory Fall (ms)|8-20|
|$\delta_e$|Excitatory onset time (ms)|0-0|
|$\rho_e$|Excitatory $tau$ ratio (fall/rise)|2-7|
|$\bar{g}_e$|Excitatory max conductance|0.02-0.25|
|$\tau_{ir}$|Inhibitory Rise (ms)|1.5-5|
|$\tau_{id}$|Inhibitory Fall(ms)|14-60|
|$\delta_i$|Inhibitory onset time(ms)|3-15|
|$\rho_i$|Inhibitory $tau$ ratio (fall/rise)|5-20|
|$\bar{g}_i$|Inhibitory max conductance|0.04-0.5|
|$\beta$|Inhibitory/Excitatory $tau$ rise ratio|0.5-5|

### Double exponential to explain the net synaptic conductance.

```python
alpha = exp(-(t-delta_e)/e_d) - exp(-(t-delta_e)/e_r)
```

```python
alpha
```

```python
#alpha = alpha.subs(e_d, (rho_e*e_r)).doit()
```

```python
alpha_prime = alpha.diff(t)
```

```python
alpha_prime
```

```python
theta_e = solve(alpha_prime,t) # Time to peak
```

```python
theta_e = logcombine(theta_e[0])
```

```python
theta_e
```

```python
N(theta_e.subs(averageEstimateDict))
```

```python
alpha_star = simplify(alpha.subs(t, theta_e).doit())
```

```python
alpha_star
```

```python
#alpha_star = simplify(alpha) # Replacing e_d/e_r with tau_e
```

### Finding maximum of the curve and substituting ratio of taus

```python
alpha_star
```

```python
E = Piecewise((0, t < delta_e), (g_e * (alpha/alpha_star), True))
```

### Final equation for Excitation normalized to be maximum at $g_e$

```python
E
```

### Verifying that E Behaves

```python
E_check = N(E.subs(averageEstimateDict))
```

```python
E_check.free_symbols
```

```python
plot(E_check,(t,0,100))
```

### Doing the same with inhibition

```python
I = E.xreplace({g_e: g_i, rho_e: rho_i, e_r:i_r, e_d: i_d, delta_e: delta_i})
```

```python
I
```

### Verifying that I Behaves

```python
I_check = N(I.subs(averageEstimateDict))
```

```python
plot(-I_check,(t,0,100))
```

### Now finding the control peak using difference of these double-exponentials

```python
C =  E*() - I
```

```python
C
```

```python
# 1. Get rid of the normalization factor.
# 2. The functions must approximate each other when close to 0.
# 3. The precision of the parameters in the equations must increase with g_e.
```

### Substituting excitatory and inhibitory ratios and putting $\delta_e$ to zero.

```python
#C = C.subs({g_i: g_e*P, i_r : e_r*b}) # Replacing g_i with P*ge
C = C.subs({delta_e:0})
```

```python
C_check = N(C.subs(averageEstimateDict))
```

```python
C_check
```

### Verifying that C behaves

```python
plot(C_check,(t,0,100))
```

```python
#C_check = N(C.subs({rho_e:7, rho_i: 15}))
```

```python
C_check
```

```python
C_check = C_check.subs(averageEstimateDict)
```

```python
plot(C_check,(t,0,100))
```

```python
C_prime = diff(C,t)
```

```python
C_prime_check = N(C_prime.subs(averageEstimateDict))
```

```python
plot(C_prime_check,(t,0,100))
```

```python
C_prime_prime = diff(C_prime,t)
```

```python
C_prime_prime_check = N(C_prime_prime.subs(averageEstimateDict))
```

```python
plot(C_prime_prime_check,(t,0,100))
```

```python
#simplify(C.subs(t,log(T)))
```

```python
#C.subs(delta_i, 1/g_e)
```

```python
#x, denominator = cse(simplify(C_prime.as_numer_denom()))
```

```python
#T = symbols('T')
```

```python
simplify(C_prime)
```

#### Explicit solving this equation doesn't work

### Trying to use lambert function

```python
a,b,c,d = -t/e_r, -t/e_d, -(t - delta_i)/i_r, -(t - delta_i)/i_d
```

```python
alpha_star
```

```python
W = lambda z: z*exp(z)
```

```python
LambertW(W(a))
```

```python
C = g_e*(exp(LambertW(W(a))) - exp(LambertW(W(b))))/alpha_star - g_i*(exp(LambertW(W(c))) - exp(LambertW(W(d))))/alpha_star.subs({ e_r:i_r, e_d: i_d})
```

```python
C
```

```python
C.diff(t)
```

```python
t_star = solve(expand(C.diff(t)),t)
```

```python
a
```

```python
exp(a).diff(t)
```

```python
mpmath.lambertw(1)
```

    mpf('0.56714329040978384')

```python
C.diff(t)
```

```python
powsimp(C.subs({-t/e_r:a, -t/e_d:b, -(t - delta_i)/i_r:c, -(t - delta_i)/i_d:d}))
```

```python

```

```python
piecewise_C_star = simplify(ratsimp(factor(C_prime))).args
```

```python
C_star_1 = simplify(piecewise_C_star[1][0])
```

```python
C_star_1.args
```

```python
simplify(solveset(C_star_1.args[5],t).doit())
```

```python
factor(C_star_1).collect(exp(t)).args
```

```python
expand_log(factor(C_prime))
```

```python
denominator
```

```python
-x[5]/((i_d*exp(x[13]*x[9])) - (i_r*(exp(x[13]/i_d))))
```

```python
j,k = symbols({'J','K'})
```

```python
new_eq = simplify(C_prime.subs({e_d:e_r*((j+1)/(j-1)), i_d:i_r*((k+1)/(k-1))}))
```

```python
refine(powsimp(new_eq.as_numer_denom()))
```

```python
eq_1 = latex("\bar{g}_e*\tau_{er}*(-P*\rho_e*\rho_i**(\rho_i/(\rho_i - 1) - 1 + 1/(\rho_i - 1))*(\rho_e**(\rho_e/(\rho_e - 1)) - \rho_e**(1/(\rho_e - 1)))*(\rho_i*exp((\delta_i - t)/(\beta*\tau_{er})) - exp((\delta_i - t)/(\beta*\rho_i*\tau_{er})))*exp(t*(\rho_e + 1)/(\rho_e*\tau_{er}))/(\beta*(\rho_i**(\rho_i/(\rho_i - 1)) - \rho_i**(1/(\rho_i - 1)))) + \rho_e**((\rho_e + 1)/(\rho_e - 1))*(\rho_e*exp(t/(\rho_e*\tau_{er})) - exp(t/\tau_{er})))")
```

```python
eq_1
```

```python
C.diff(t).diff(t)
```

```python
t_star = (delta_i - (b*e_r*log((P*(rho_i+1)*rho_e)/((rho_e+1)*rho_i))))/(b+1)
```

```python
t_star.subs(averageEstimateDict)
```

```python
N(delta_i.subs(averageEstimateDict))
```

### Unfortunately this is not possible: Since the $\tau_{decay}$ will not contribute to the first peak, we can eliminate them.

```python
erise = Piecewise((0, t < delta_e), (g_e * (exp(-(t-delta_e)/e_r)/alpha_star), True))
```

```python
efall = Piecewise((0, t < delta_e), (g_e * (exp(-(t-delta_e)/e_d)/alpha_star), True))
```

```python
irise = erise.subs({g_e: g_i, rho_e: rho_i, e_r:i_r, e_d: i_d, delta_e: delta_i})
```

```python
ifall = efall.subs({g_e: g_i, rho_e: rho_i, e_r:i_r, e_d: i_d, delta_e: delta_i})
```

```python
C = C.subs({g_i: g_e*P, i_r : e_r*b}) # Replacing g_i with P*ge
C = C.subs({delta_e:0})
```

```python
C
```

```python
C_check = C.subs({P:0, delta_i:2})
```

```python
C_check
```

```python
C
```

```python
C.diff(t)
```

```python
averageEstimateDict
```

```python
C_check = N(C.subs(averageEstimateDict))
```

```python
C_check
```

### Verifying that C behaves

```python
plot(C_check,(t,0,100))
```

```python
plot(erise.subs(averageEstimateDict),(t,0,100))
```

```python
plot(((efall-erise)-(-irise)).subs(averageEstimateDict),(t,0,100))
```

```python
plot(((efall-erise)-(ifall-irise)).subs(averageEstimateDict),(t,0,100))
```

```python
plot((E-I).subs(averageEstimateDict),(t,0,100))
```

```python
C_check = N(C.subs({rho_e:7, rho_i: 15}))
```

```python
C_check
```

```python
C_check = C_check.subs(averageEstimateDict)
```

```python
plot(C_check,(t,0,100))
```

```python
C_prime = diff(C,t)
```

```python
C_prime_check = N(C_prime.subs(averageEstimateDict))
```

```python
plot(C_prime_check,(t,0,100))
```

```python
C_prime_prime = diff(C_prime,t)
```

```python
C_prime_prime_check = N(C_prime_prime.subs(averageEstimateDict))
```

```python
plot(C_prime_prime_check,(t,0,100))
```

```python
simplify(C)
```

```python

```

```python

```

```python
C_prime = diff(C,t)
```

```python
C_prime
```

```python
theta_C = solve(C_prime, t)
```

```python
theta_C
```

```python
C_star = C.subs(t, theta_C[0])
```

```python
C_star = C_star.subs(delta_e,0.) # Putting excitatory delay to zero
```

### Assuming that certain ratios are more than one and substituting

```python
C_star = C_star.subs({i_d: (i_r*tau_i)+i_r, e_d: (e_r*tau_e)+e_r, g_i: g_e*P, i_r:e_r*b}) # Replacing e_d/e_r with tau_e
```

```python
C_star = cancel(powsimp(factor(C_star), deep=True))
```

```python
C_star = C_star.collect([g_e, delta_i, P])
```

```python
#C_star1 = limit(limit(C_star, (1/tau_e), 0), (1/tau_i),0)
```

```python
#simplify(C_star.subs({e_r: 4., i_r: 2.73, g_i:P*g_e, e_d: g_e*b, i_d : g_e*g, delta_i: theta/g_e }))
```

```python
#cancel(C_star1.together(deep=True))
```

```python
C_star.free_symbols
```

```python
#tau_e1, tau_i1 = symbols('\\tau_{e1} \\tau_{i1}', real=True, positive=True)
```

```python
#simplify(C_star.subs({tau_e:tau_e1+1, tau_i:tau_i1+1}))
```

```python
C_star = simplify(C_star)
```

```python
C_star
```

```python
cse(C_star)
```

```python
cse(simplify(diff(C_star,g_e)))
```

```python
x = Symbol('x')
```

```python
y = x**(1/x)
```

```python
y.subs(x,40).evalf()
```

```python
theta_C_nice = simplify(theta_C[0].subs({i_d: (i_r*tau_i)+i_r, e_d: (e_r*tau_e)+e_r, g_i: g_e*P}))
```

```python
cse(cancel(expand(theta_C_nice)))
```

```python
theta_C_nice
```

```python
limit(x/(x-1),x,5)
```

```python
log(-2)
```

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
from scipy import integrate
from scipy.integrate import solve_ivp
import sympy
import mpmath
from mpl_toolkits.mplot3d.axes3d import Axes3D
import myastro.orbit as ob
import myastro.coord as co
import myastro.timeutil as tc
from toolz import pipe
```

    Bad key "text.kerning_factor" on line 4 in
    /home/anybody/apps/anaconda3/envs/py37astro/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
    or from the matplotlib source distribution

```python
x = sympy.symbols("x")
y = sympy.Function("y")
f = y(x)**2 + x
```

```python
f_np = sympy.lambdify((y(x),x),f)
```

```python
y0 = 0
xp = np.linspace(0, 1.9, 100)
yp = integrate.odeint(f_np, y0, xp)
xm = np.linspace(0, -5, 100)
ym = integrate.odeint(f_np, y0, xm)
```

```python
 def plot_direction_field(x, y_x, f_xy, x_lim=(-5, 5), y_lim=(-5, 5), ax=None):
     f_np = sympy.lambdify((x, y_x), f_xy, 'numpy')
     x_vec = np.linspace(x_lim[0], x_lim[1], 20)
     y_vec = np.linspace(y_lim[0], y_lim[1], 20)
     if ax is None:
        _, ax = plt.subplots(figsize=(4, 4))
     dx = x_vec[1] - x_vec[0]
     dy = y_vec[1] - y_vec[0]

     for m, xx in enumerate(x_vec):
        for n, yy in enumerate(y_vec):
             Dy = f_np(xx, yy) * dx
             Dx = 0.8 * dx**2 / np.sqrt(dx**2 + Dy**2)
             Dy = 0.8 * Dy*dy / np.sqrt(dx**2 + Dy**2)
             ax.plot([xx - Dx/2, xx + Dx/2],[yy - Dy/2, yy + Dy/2], 'b', lw=0.5)
     ax.axis('tight')
     ax.set_title(r"$%s$" % (sympy.latex(sympy.Eq(y(x).diff(x), f_xy))),fontsize=18)
     return ax
```

```python
fig, ax = plt.subplots(1, 1, figsize=(4, 4))
plot_direction_field(x, y(x), f, ax=ax)
ax.plot(xm, ym, 'b', lw=2)
ax.plot(xp, yp, 'r', lw=2)
```

```python
a, b, c, d = 0.4, 0.002, 0.001, 0.7
def f(xy, t):
    x, y = xy
    return [a * x - b * x * y, c * x * y - d * y]
```

```python
xy0 = [600, 400]
t = np.linspace(0, 50, 250)
xy_t = integrate.odeint(f, xy0, t)
```

```python
xy_t.shape
```

    (250, 2)

```python
def f(t, y, args):
    m1, k1, g1, m2, k2, g2 = args
    return [y[1], - k1/m1 * y[0] + k2/m1 * (y[2] - y[0]) -g1/m1 * y[1], y[3], - k2/m2 * (y[2] - y[0]) - g2/m2 * y[3]]
```

```python
m1, k1, g1 = 1.0, 10.0, 0.5
m2, k2, g2 = 2.0, 40.0, 0.25
args = (m1, k1, g1, m2, k2, g2)
y0 = [1.0, 0, 0.5, 0]
t = np.linspace(0, 20, 1000)
```

```python
r = integrate.ode(f)
```

```python
r.set_integrator('lsoda')
r.set_initial_value(y0, t)
r.set_f_params(args)
```

```python
import numpy as np
from scipy.integrate import solve_ivp
```

```python
sol = solve_ivp(lambda t, y: t-y, [0, 15], [2])
```

```python
plt.plot(sol.t, sol.y[0], 'k--s')

```

```python
sol.success
```

    True

```python
t_mjd = pipe(co.epochformat2jd("2020.05.13.5"),tc.jd2mjd)
print (t_mjd)
r_venus, v_venus = ob.planet_helioeclipt_rv("Venus",t_mjd)
```

    58982.5

```python
a_venus = ob.calc_accelaration(t_mjd,r_venus)
```

```python
a_venus
```

    array([ 3.35866638e-04,  4.63490039e-04, -1.30506670e-05])

```python
def my_f(t,y):
    acc = ob.calc_accelaration(t,y[0:3])
    return np.concatenate((y[3:6],acc))
    #return np.array([v1,v2,v3,acc[0],acc[1],acc[2]])
```

```python
y0 = np.array([r_venus[0],r_venus[1],r_venus[2],v_venus[0],v_venus[1],v_venus[2]])
y0 = np.concatenate((r_venus,v_venus))

```

```python
sol = solve_ivp(my_f,(t_mjd,t_mjd+500),y0)
```

```python
a = np.array([0,1,2,3,4,5])
```

```python
a[3:6]
```

    array([3, 4, 5])

```python
np.concatenate((a[3:6],a[0:3]))
```

    array([3, 4, 5, 0, 1, 2])

```python
plt.plot(sol.t, sol.y[0], 'k--s')
```

```python
sol
```

      message: 'The solver successfully reached the end of the integration interval.'
         nfev: 116
         njev: 0
          nlu: 0
          sol: None
       status: 0
      success: True
            t: array([58982.5       , 58982.69463174, 58984.64094914, 59004.10412312,
           59035.52136368, 59071.41046562, 59104.51932098, 59138.6965941 ,
           59170.74162723, 59201.95634269, 59232.90889038, 59261.71915742,
           59293.83006402, 59324.59607838, 59353.36065734, 59383.76932541,
           59411.2487118 , 59438.59445994, 59467.38578135, 59482.5       ])
     t_events: None
            y: array([[-4.21742867e-01, -4.18543847e-01, -3.85875427e-01,
            -1.30119291e-02,  5.48556909e-01,  6.89722778e-01,
             2.34765298e-01, -4.29955274e-01, -6.73029598e-01,
            -3.22282972e-01,  3.03669914e-01,  6.58892008e-01,
             4.70456332e-01, -1.29000712e-01, -5.85903225e-01,
            -5.20890904e-01, -1.15563895e-02,  5.02732555e-01,
             5.91066938e-01,  4.03946229e-01],
           [-5.81981354e-01, -5.84306603e-01, -6.06578458e-01,
            -7.20708708e-01, -4.71213698e-01,  2.06875521e-01,
             6.66456759e-01,  5.37583519e-01, -6.26839316e-02,
            -5.91276480e-01, -6.02109273e-01, -1.39424988e-01,
             4.76525115e-01,  6.44508714e-01,  2.64949144e-01,
            -3.63578564e-01, -6.31608749e-01, -3.79515592e-01,
             2.13182434e-01,  4.77635213e-01],
           [ 1.63870986e-02,  1.61707108e-02,  1.39810903e-02,
            -9.09654851e-03, -3.80960972e-02, -3.69776326e-02,
            -4.44211861e-03,  3.21583164e-02,  3.79841815e-02,
             1.05197742e-02, -2.57522804e-02, -3.99300482e-02,
            -2.06387888e-02,  1.62514556e-02,  3.74328978e-02,
             2.50924930e-02, -7.96381569e-03, -3.41987409e-02,
            -3.11974336e-02, -1.67849733e-02],
           [ 1.64036693e-02,  1.64687902e-02,  1.70921874e-02,
             2.02540010e-02,  1.31249372e-02, -6.07882983e-03,
            -1.94507363e-02, -1.59162985e-02,  2.21500880e-03,
             1.85058228e-02,  1.86819064e-02,  4.25326354e-03,
            -1.51053480e-02, -2.06810366e-02, -8.49106699e-03,
             1.25844724e-02,  2.17127916e-02,  1.30055246e-02,
            -7.50141982e-03, -1.67165454e-02],
           [-1.19920821e-02, -1.19016940e-02, -1.09787833e-02,
            -4.66761793e-04,  1.52901117e-02,  1.91874866e-02,
             6.27402713e-03, -1.34458198e-02, -2.09771492e-02,
            -1.00609144e-02,  9.45903226e-03,  2.04783081e-02,
             1.44711673e-02, -4.69745744e-03, -1.98456860e-02,
            -1.76533905e-02, -2.74791256e-04,  1.73651893e-02,
             2.02917753e-02,  1.38830287e-02],
           [-1.11051614e-03, -1.11303939e-03, -1.13640690e-03,
            -1.17525210e-03, -5.48529991e-04,  6.12987286e-04,
             1.20824049e-03,  7.34813060e-04, -4.14460648e-04,
            -1.20545197e-03, -9.48890923e-04,  3.43648260e-05,
             1.06947478e-03,  1.12931946e-03,  2.18838865e-04,
            -9.67476705e-04, -1.25680471e-03, -5.13259675e-04,
             7.10191412e-04,  1.15442292e-03]])
     y_events: None

```python
sol.t
```

    array([58982.5       , 58982.69463174, 58984.64094914, 59004.10412312,
           59035.52136368, 59071.41046562, 59104.51932098, 59138.6965941 ,
           59170.74162723, 59201.95634269, 59232.90889038, 59261.71915742,
           59293.83006402, 59324.59607838, 59353.36065734, 59383.76932541,
           59411.2487118 , 59438.59445994, 59467.38578135, 59482.5       ])

```python
sol.y.shape
```

    (6, 20)

```python
sol.y
```

    array([[-4.21742867e-01, -4.18543847e-01, -3.85875427e-01,
            -1.30119291e-02,  5.48556909e-01,  6.89722778e-01,
             2.34765298e-01, -4.29955274e-01, -6.73029598e-01,
            -3.22282972e-01,  3.03669914e-01,  6.58892008e-01,
             4.70456332e-01, -1.29000712e-01, -5.85903225e-01,
            -5.20890904e-01, -1.15563895e-02,  5.02732555e-01,
             5.91066938e-01,  4.03946229e-01],
           [-5.81981354e-01, -5.84306603e-01, -6.06578458e-01,
            -7.20708708e-01, -4.71213698e-01,  2.06875521e-01,
             6.66456759e-01,  5.37583519e-01, -6.26839316e-02,
            -5.91276480e-01, -6.02109273e-01, -1.39424988e-01,
             4.76525115e-01,  6.44508714e-01,  2.64949144e-01,
            -3.63578564e-01, -6.31608749e-01, -3.79515592e-01,
             2.13182434e-01,  4.77635213e-01],
           [ 1.63870986e-02,  1.61707108e-02,  1.39810903e-02,
            -9.09654851e-03, -3.80960972e-02, -3.69776326e-02,
            -4.44211861e-03,  3.21583164e-02,  3.79841815e-02,
             1.05197742e-02, -2.57522804e-02, -3.99300482e-02,
            -2.06387888e-02,  1.62514556e-02,  3.74328978e-02,
             2.50924930e-02, -7.96381569e-03, -3.41987409e-02,
            -3.11974336e-02, -1.67849733e-02],
           [ 1.64036693e-02,  1.64687902e-02,  1.70921874e-02,
             2.02540010e-02,  1.31249372e-02, -6.07882983e-03,
            -1.94507363e-02, -1.59162985e-02,  2.21500880e-03,
             1.85058228e-02,  1.86819064e-02,  4.25326354e-03,
            -1.51053480e-02, -2.06810366e-02, -8.49106699e-03,
             1.25844724e-02,  2.17127916e-02,  1.30055246e-02,
            -7.50141982e-03, -1.67165454e-02],
           [-1.19920821e-02, -1.19016940e-02, -1.09787833e-02,
            -4.66761793e-04,  1.52901117e-02,  1.91874866e-02,
             6.27402713e-03, -1.34458198e-02, -2.09771492e-02,
            -1.00609144e-02,  9.45903226e-03,  2.04783081e-02,
             1.44711673e-02, -4.69745744e-03, -1.98456860e-02,
            -1.76533905e-02, -2.74791256e-04,  1.73651893e-02,
             2.02917753e-02,  1.38830287e-02],
           [-1.11051614e-03, -1.11303939e-03, -1.13640690e-03,
            -1.17525210e-03, -5.48529991e-04,  6.12987286e-04,
             1.20824049e-03,  7.34813060e-04, -4.14460648e-04,
            -1.20545197e-03, -9.48890923e-04,  3.43648260e-05,
             1.06947478e-03,  1.12931946e-03,  2.18838865e-04,
            -9.67476705e-04, -1.25680471e-03, -5.13259675e-04,
             7.10191412e-04,  1.15442292e-03]])

```python
sol
```

      message: 'The solver successfully reached the end of the integration interval.'
         nfev: 116
         njev: 0
          nlu: 0
          sol: None
       status: 0
      success: True
            t: array([58982.5       , 58982.69463174, 58984.64094914, 59004.10412312,
           59035.52136368, 59071.41046562, 59104.51932098, 59138.6965941 ,
           59170.74162723, 59201.95634269, 59232.90889038, 59261.71915742,
           59293.83006402, 59324.59607838, 59353.36065734, 59383.76932541,
           59411.2487118 , 59438.59445994, 59467.38578135, 59482.5       ])
     t_events: None
            y: array([[-4.21742867e-01, -4.18543847e-01, -3.85875427e-01,
            -1.30119291e-02,  5.48556909e-01,  6.89722778e-01,
             2.34765298e-01, -4.29955274e-01, -6.73029598e-01,
            -3.22282972e-01,  3.03669914e-01,  6.58892008e-01,
             4.70456332e-01, -1.29000712e-01, -5.85903225e-01,
            -5.20890904e-01, -1.15563895e-02,  5.02732555e-01,
             5.91066938e-01,  4.03946229e-01],
           [-5.81981354e-01, -5.84306603e-01, -6.06578458e-01,
            -7.20708708e-01, -4.71213698e-01,  2.06875521e-01,
             6.66456759e-01,  5.37583519e-01, -6.26839316e-02,
            -5.91276480e-01, -6.02109273e-01, -1.39424988e-01,
             4.76525115e-01,  6.44508714e-01,  2.64949144e-01,
            -3.63578564e-01, -6.31608749e-01, -3.79515592e-01,
             2.13182434e-01,  4.77635213e-01],
           [ 1.63870986e-02,  1.61707108e-02,  1.39810903e-02,
            -9.09654851e-03, -3.80960972e-02, -3.69776326e-02,
            -4.44211861e-03,  3.21583164e-02,  3.79841815e-02,
             1.05197742e-02, -2.57522804e-02, -3.99300482e-02,
            -2.06387888e-02,  1.62514556e-02,  3.74328978e-02,
             2.50924930e-02, -7.96381569e-03, -3.41987409e-02,
            -3.11974336e-02, -1.67849733e-02],
           [ 1.64036693e-02,  1.64687902e-02,  1.70921874e-02,
             2.02540010e-02,  1.31249372e-02, -6.07882983e-03,
            -1.94507363e-02, -1.59162985e-02,  2.21500880e-03,
             1.85058228e-02,  1.86819064e-02,  4.25326354e-03,
            -1.51053480e-02, -2.06810366e-02, -8.49106699e-03,
             1.25844724e-02,  2.17127916e-02,  1.30055246e-02,
            -7.50141982e-03, -1.67165454e-02],
           [-1.19920821e-02, -1.19016940e-02, -1.09787833e-02,
            -4.66761793e-04,  1.52901117e-02,  1.91874866e-02,
             6.27402713e-03, -1.34458198e-02, -2.09771492e-02,
            -1.00609144e-02,  9.45903226e-03,  2.04783081e-02,
             1.44711673e-02, -4.69745744e-03, -1.98456860e-02,
            -1.76533905e-02, -2.74791256e-04,  1.73651893e-02,
             2.02917753e-02,  1.38830287e-02],
           [-1.11051614e-03, -1.11303939e-03, -1.13640690e-03,
            -1.17525210e-03, -5.48529991e-04,  6.12987286e-04,
             1.20824049e-03,  7.34813060e-04, -4.14460648e-04,
            -1.20545197e-03, -9.48890923e-04,  3.43648260e-05,
             1.06947478e-03,  1.12931946e-03,  2.18838865e-04,
            -9.67476705e-04, -1.25680471e-03, -5.13259675e-04,
             7.10191412e-04,  1.15442292e-03]])
     y_events: None

```python
sol.t[19]
```

    59482.5

```python
sol.y[:,19]
```

    array([ 0.40394623,  0.47763521, -0.01678497, -0.01671655,  0.01388303,
            0.00115442])

```python
sol
```

      message: 'The solver successfully reached the end of the integration interval.'
         nfev: 116
         njev: 0
          nlu: 0
          sol: None
       status: 0
      success: True
            t: array([58982.5       , 58982.69463174, 58984.64094914, 59004.10412312,
           59035.52136368, 59071.41046562, 59104.51932098, 59138.6965941 ,
           59170.74162723, 59201.95634269, 59232.90889038, 59261.71915742,
           59293.83006402, 59324.59607838, 59353.36065734, 59383.76932541,
           59411.2487118 , 59438.59445994, 59467.38578135, 59482.5       ])
     t_events: None
            y: array([[-4.21742867e-01, -4.18543847e-01, -3.85875427e-01,
            -1.30119291e-02,  5.48556909e-01,  6.89722778e-01,
             2.34765298e-01, -4.29955274e-01, -6.73029598e-01,
            -3.22282972e-01,  3.03669914e-01,  6.58892008e-01,
             4.70456332e-01, -1.29000712e-01, -5.85903225e-01,
            -5.20890904e-01, -1.15563895e-02,  5.02732555e-01,
             5.91066938e-01,  4.03946229e-01],
           [-5.81981354e-01, -5.84306603e-01, -6.06578458e-01,
            -7.20708708e-01, -4.71213698e-01,  2.06875521e-01,
             6.66456759e-01,  5.37583519e-01, -6.26839316e-02,
            -5.91276480e-01, -6.02109273e-01, -1.39424988e-01,
             4.76525115e-01,  6.44508714e-01,  2.64949144e-01,
            -3.63578564e-01, -6.31608749e-01, -3.79515592e-01,
             2.13182434e-01,  4.77635213e-01],
           [ 1.63870986e-02,  1.61707108e-02,  1.39810903e-02,
            -9.09654851e-03, -3.80960972e-02, -3.69776326e-02,
            -4.44211861e-03,  3.21583164e-02,  3.79841815e-02,
             1.05197742e-02, -2.57522804e-02, -3.99300482e-02,
            -2.06387888e-02,  1.62514556e-02,  3.74328978e-02,
             2.50924930e-02, -7.96381569e-03, -3.41987409e-02,
            -3.11974336e-02, -1.67849733e-02],
           [ 1.64036693e-02,  1.64687902e-02,  1.70921874e-02,
             2.02540010e-02,  1.31249372e-02, -6.07882983e-03,
            -1.94507363e-02, -1.59162985e-02,  2.21500880e-03,
             1.85058228e-02,  1.86819064e-02,  4.25326354e-03,
            -1.51053480e-02, -2.06810366e-02, -8.49106699e-03,
             1.25844724e-02,  2.17127916e-02,  1.30055246e-02,
            -7.50141982e-03, -1.67165454e-02],
           [-1.19920821e-02, -1.19016940e-02, -1.09787833e-02,
            -4.66761793e-04,  1.52901117e-02,  1.91874866e-02,
             6.27402713e-03, -1.34458198e-02, -2.09771492e-02,
            -1.00609144e-02,  9.45903226e-03,  2.04783081e-02,
             1.44711673e-02, -4.69745744e-03, -1.98456860e-02,
            -1.76533905e-02, -2.74791256e-04,  1.73651893e-02,
             2.02917753e-02,  1.38830287e-02],
           [-1.11051614e-03, -1.11303939e-03, -1.13640690e-03,
            -1.17525210e-03, -5.48529991e-04,  6.12987286e-04,
             1.20824049e-03,  7.34813060e-04, -4.14460648e-04,
            -1.20545197e-03, -9.48890923e-04,  3.43648260e-05,
             1.06947478e-03,  1.12931946e-03,  2.18838865e-04,
            -9.67476705e-04, -1.25680471e-03, -5.13259675e-04,
             7.10191412e-04,  1.15442292e-03]])
     y_events: None

```python
idx = len(sol.t) -1
sol.t[idx]
sol.y[:,idx]
```

    array([ 0.40394623,  0.47763521, -0.01678497, -0.01671655,  0.01388303,
            0.00115442])

```python
r_sol = sol.y[:,idx][:3]
```

```python
v_sol = sol.y[:,idx][3:6]
```

```python
r_sol
```

    array([ 0.40394623,  0.47763521, -0.01678497])

```python
v_sol
```

    array([-0.01671655,  0.01388303,  0.00115442])

```python
from myastro.cluegen import Datum

class Coordinates(Datum):
    x: int
    y: int

a = Coordinates(2, 3)

```

```python
step = 2
start = 1
end = 9
samples = 4 + 1
interval = np.linspace(start,end,samples,endpoint=True)
print (interval)
```

    [1. 3. 5. 7. 9.]

```python
def my_range (start, stop, step, include_start = True, include_end = True):
    result = []
    i = 0
    while start + i * step <= stop:
        result.append(start + i * step)
        i += 1
    if include_end :
        if result[-1] != stop :
            result.append(stop)
    else :
        if result[-1] == stop :
            result = result[:-1]
    if not include_start :
        result = result[1:]
    return result

```

```python
# init    from to
# intervalo 1  my_range(init,from,step,include_end=False)
# intervalo 2  my_range(from,to,step,include_end=True)

int1 = my_range(3,10,2,include_end=False)
int2 = my_range(10,20,2,include_end=True)
print (int1)
print (int2)
```

    [3, 5, 7, 9]
    [10, 12, 14, 16, 18, 20]
    None
    None

```python
my_range(1,20,2,include_start=False,include_end=True)
```

    [3, 5, 7, 9, 11, 13, 15, 17, 19, 20]

```python
# from    to   init
# intervalo 1  reversed my_range(to,init,step,include_end=True, include_start=False)
# intervalo 2 reversed my_range(from,to, include_end=True

int1 = reversed(my_range(10,20,3,include_start=False, include_end=True))
int2 = reversed(my_range(2,10,2,include_end=True))
print (list(int1))
print (list(int2))

```

    [20, 19, 16, 13]
    [10, 8, 6, 4, 2]

```python

```


# Burgers equation

* Physical space
\begin{align}
u_t + uu_x = \mu u_{xx}
\end{align}
* Spectral space: $\hat{u} = \mathscr{F}\{u\}$
\begin{align}
\hat{u_t} =  -\mu k_x^{2}\hat{u}
                                       - \mathscr{F}\{ {\mathscr{F}^{-1}\{ \hat{u} \} \mathscr{F}^{-1} \{ i k_x \hat{u} \} } \}
\end{align}

# Imports

```python
import numpy as np
import matplotlib.pyplot as plt
from rkstiff.grids import construct_x_kx_rfft
from rkstiff.derivatives import dx_rfft
from rkstiff.if34 import IF34
from rkstiff.etd35 import ETD35
%matplotlib inline
```

# Helper graph functions

```python
def plotResults(u0,uf,u0FFT,ufFFT):
    """
    Helper function that plots initial 1D-array along with final propagated result.
    Also plots the initial spectral (fft) 1D-array along with final propagated spectral result.

    INPUTS
        u0 - initial 1D-array
        uf - final propagated 1D-array
        u0FFT - initial spectral 1D-array
        uFFT - final propagated spectral 1D-array

    """

    fig = plt.figure(figsize=(12,8))
    ax1 = fig.add_subplot(2,2,1)
    ax2 = fig.add_subplot(2,2,2,sharey = ax1)
    ax3 = fig.add_subplot(2,2,3)
    ax4 = fig.add_subplot(2,2,4,sharey = ax3)
    ax1.plot(x,u0)
    ax2.plot(x,uf)
    ax3.plot(kx,np.abs(u0FFT)**2)
    ax4.plot(kx,np.abs(ufFFT)**2)
    ax3.set_yscale('log')
    ax4.set_yscale('log')
    fig.tight_layout()

def waterfall(x,t,u,**kwargs):
    if 'figsize' in kwargs:
        fig = plt.figure(figsize=kwargs['figsize'])
    else:
        fig = plt.figure(figsize=(6,6))
    ax = fig.add_subplot(1,1,1,projection='3d')
    ax.w_xaxis.set_pane_color((0,0,0,0))
    ax.w_yaxis.set_pane_color((0,0,0,0))
    ax.w_zaxis.set_pane_color((0,0,0,0))
    for i,snapshot in enumerate(u):
        ax.plot(x,t[i]*np.ones_like(x),snapshot,color='black')
    plt.xlim([x[0],x[-1]])
    plt.ylim([t[0],t[-1]])
    plt.tight_layout()
    return ax
```

# Construct grids

```python
# uniform grid spacing, field assumed to be real-valued -> construct_x_kx_rfft
N = 8192
a,b = -np.pi, np.pi
x,kx = construct_x_kx_rfft(N,a,b)
```

# Linear operator and nonlinear function

```python
mu = 0.0005
L = -mu*kx**2
def NL(uf):
    u = np.fft.irfft(uf)
    ux = np.fft.irfft(1j*kx*uf)
    return -np.fft.rfft(u*ux)
```

# Initial field to be propagated

```python
u0 = np.exp(-10*np.sin(x/2)**2)
u0FFT = np.fft.rfft(u0)
plt.plot(x,u0)
```

# Initialize IF34 solver

```python
solver = IF34(linop=L,NLfunc=NL,epsilon=1e-8)
```

# Propagate step by step

```python
h = 0.01
uFFT = u0FFT.copy()
t = 0
while t < 1:
    uFFT,h,h_suggest = solver.step(uFFT,h)
    t += h
    # use suggested step
    h = h_suggest

print('tf = ',t)
uf = np.fft.irfft(uFFT)
plotResults(u0,uf,u0FFT,uFFT)
```

# Propagate from time t0 to tf

```python
# store_data -> propagated field stored in solver.u at times solver.t
# store_freq -> propagated field values stored on every store_freq step (default is every step)
uFFT = solver.evolve(u0FFT,t0=0,tf=1,store_data=True,store_freq=50)
```

# Graph result

```python
U = []
for u in solver.u:
    U.append(np.fft.irfft(u))
U = np.array(U)
t = np.array(solver.t)
ax = waterfall(x,t,U,figsize=(8,8))
ax.grid(False)
ax.axis(False)
ax.view_init(62,-69)
```


# 05 Simple Bayesian Linear Regression

In this notebook, we will develop a bayesian linear regression model that has a zero mean isotropic gaussian prior. On some simulated data (linear and nonlinear ones), MLP, MAP and Bayesian Inference will be compared.

From our lecture slides,

In the linear regression model we have data $D$, of $n$ observations
$D=\left\{ ({\bf x}_1, y_1), ({\bf x}_2,y_2), \ldots, ({\bf x}_n, y_n) \right\} $ where ${\bf x}$
denotes an input vector of dimension $d$ (features) and $y$ denotes a scalar output (target values).
All data points are combined into a $(d) \times n$ matrix $X$.
The model that determines the relationship between inputs and
output is given by

$$ y   = \bf x^{T} {\bf \theta} + \epsilon $$

where ${\bf \theta}$ is a vector of parameters of the linear model. And the additive noise $\epsilon$ is iid Gaussian with
zero mean and variance $\sigma_y^2$

\begin{equation}
\epsilon \sim N(0, \sigma^2_y)
\end{equation}

```python
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
import random
random.seed(10)
```

# 1. Linear Problem

```python
# Generate linear toy dataset
def linear_data(x):
    y = -0.4 + 0.8*x
    return y
# Create training and test points
sigma_y   = 0.8  # fixed and known noise level
num_train = 30
dataset_linear = {}
dataset_linear['X_train'] = np.random.uniform(-2, 2, num_train)
dataset_linear['y_train'] = linear_data(dataset_linear['X_train']) + np.random.normal(0, sigma_y, num_train)
dataset_linear['X_test'] = np.linspace(-4, 4, 10*num_train)
dataset_linear['y_test'] = linear_data(dataset_linear['X_test'])
```

```python
# Plot dataset
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2.5, xmin =-2.5)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

### Linear basis function

x will be converted to [1, x]. So that the bias can be included in the above equation. The data matrix $\mathbf{X}$ shape will be $(d+1) \times n$

```python
def phi_linear(x):
    z = np.ones((x.shape[0],))
    res = np.stack((z,x), axis=1)
    return res
```

```python
phi_train = phi_linear(dataset_linear['X_train'])
```

```python
phi_train.shape
```

    (30, 2)

## MLE Closed-Form Solution

The equation is given as:
$\hat{\theta}_{ml}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$

```python
def mle_compute(phi_x, y):
    #phi_x is the feature matrix
    #y is the label
    Sigma_N = np.dot(phi_x.T, phi_x)
    theta  = np.dot(np.dot(np.linalg.inv(Sigma_N), phi_x.T), y)
    return theta
```

```python
theta = mle_compute(phi_train, dataset_linear['y_train'] )
```

```python
phi_test =  phi_linear(dataset_linear['X_test'])
y_predict = phi_test.dot(theta)
```

```python
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2.5, xmin =-2.5)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_test'], y_predict, color='green', linewidth=1, label="MLE Predict Function")
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

## MAP Closed-Form Solution

In the Bayesian framework inference we need to specify a prior
over the parameters that expresses our belief about the parameters
before we take any measurements. A wise choice is a zero-mean isotropic gaussian

$\mathbf{\theta}\sim\mathcal{N}(\mathbf{0}, \tau_0^2\mathbf{I}_{d \times d})$

We can now continue with the standard Bayesian formalism

\begin{eqnarray}
 p(\bf \theta| \bf y,X) &\propto& p(\bf y | X, \bf \theta) \, p(\bf \theta) \nonumber \\
                       &\propto& \exp{ \left(- \frac{1}{2 \sigma_y^2}(\bf y-X^T \bf \theta)^T(\bf y - X^T \bf \theta) \right)}
                        \exp{\left( -\frac{1}{2 \tau_0^2} \bf \theta^T \bf \theta \right)}  \nonumber \\
 \end{eqnarray}

In the next step we `complete the square' and obtain

\begin{equation}
p(\bf \theta| \bf y,X)  \propto  \exp \left( -\frac{1}{2} (\bf \theta - \bf \theta_n)^T \mathbf{V}_n^{-1}(\bf \theta - \bf \theta_n )  \right)
\end{equation}

MAP Solution will be the mean of the posterior distribution

\begin{equation}
\hat{\theta}_{map}=(\frac{\sigma_y^2}{\tau_0^2}\mathbf{I}_{d\times d}+\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}
\end{equation}

```python
def map_compute(phi_x, y, prior_tau, sigma_y=0.5):
    """
    phi_x is the training feature matrix
    y is the training label
    prior_tau denotes the uncertainty behind our prior belief over model parameters
    sigma_y is the noisy level
    """
    Sigma_N = np.dot(phi_x.T, phi_x) + pow(sigma_y,2)/pow(prior_tau,2) * np.identity(phi_x.shape[1])
    theta  = np.dot(np.dot(np.linalg.inv(Sigma_N), phi_x.T), y)
    return theta
```

```python
prior_tau = 0.4
theta_2  = map_compute(phi_train, dataset_linear['y_train'], prior_tau, sigma_y)
```

```python
y_predict_map = phi_test.dot(theta_2)
```

```python
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2.5, xmin =-2.5)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_test'], y_predict, color='green', linewidth=1, label="MLE Predict Function")
plt.plot(dataset_linear['X_test'], y_predict_map, color='blue', linewidth=1.5, label="MAP Predict Function")
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

## Bayesian Analytics Solution

The posterior distribution follows Gaussian

$$ p(\bf \theta| X, {\bf y}) \sim {\cal N}(\theta_n , V_n ) $$

where ${\theta}_{n}=(\frac{\sigma_y^2}{\tau_0^2}\mathbf{I}_{d\times d}+\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$ and $V_n = \sigma_y^2(\frac{\sigma_y^2}{\tau_0^2}\mathbf{I}_{d\times d}+\mathbf{X}^T\mathbf{X})^{-1}$

```python
def bayes_compute(phi_x, y, prior_tau, sigma_y):
    Sigma_N = np.dot(phi_x.T, phi_x) + pow(sigma_y,2)/pow(prior_tau,2) * np.identity(phi_x.shape[1])
    inver_sigma = np.linalg.inv(Sigma_N)
    theta  = np.dot(np.dot(inver_sigma, phi_x.T), y)
    var_theta = pow(sigma_y,2)*inver_sigma
    return theta, var_theta
```

```python
#positer prediction
theta_mean, var_theta  = bayes_compute(phi_train, dataset_linear['y_train'], prior_tau, sigma_y)
```

```python
var_theta
```

    array([[0.01887496, 0.00088748],
           [0.00088748, 0.01531415]])

#### Sample from posterior distribution

```python
#sample from positier distribution
n_sample = 30
z = np.random.multivariate_normal(mean=theta_mean, cov=var_theta, size=n_sample)
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2.5, xmin =-2.5)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
for idx, one_model in enumerate(z):
    y_predict_sample = phi_test.dot(one_model)
    if idx == 0:
        plt.plot(dataset_linear['X_test'], y_predict_sample, color='blue', linewidth=0.5,label='Sampled Models')
    else:
        plt.plot(dataset_linear['X_test'], y_predict_sample, color='blue', linewidth=0.5)
plt.legend()
plt.show()
```

#### Bayesian prediction

To make predictions for a test case we average over all possible parameter predictive distribution
values, weighted by their posterior probability. This is in contrast to non Bayesian schemes, where a single parameter is typically chosen by some criterion.

Thus the predictive distribution at some $x_{new}$ is given by averaging the output of all possible linear models w.r.t. the  posterior

\begin{eqnarray}
p(y_{new} | x_{new}, {\bf D}) &=& \int p({\bf y}_{new}| {\bf x}_{new}, {\bf \theta} ) p(\bf \theta| D)d\theta \nonumber \\
                                    &=& \mathcal{N}(y_{new}|x_{new}^T\theta_n, \sigma_y^2+x_{new}^T\mathbf{V}_nx_{new}),
\end{eqnarray}

which is again Gaussian, with a mean given by the posterior mean multiplied by the test input
and the variance is a quadratic
form of the test input with the posterior covariance matrix, showing that the
predictive uncertainties grow with the magnitude of the test input, as one would
expect for a linear model.

```python
y_preds = []
std_preds = []
for testdata in phi_test:
    y_pred = testdata.T.dot(theta_mean)
    std_pred = np.sqrt(sigma_y*sigma_y+testdata.T.dot(var_theta).dot(testdata))
    y_preds.append(y_pred)
    std_preds.append(std_pred)
```

```python
y_preds = np.array(y_preds)
std_preds = np.array(std_preds)
```

```python
plt.figure(figsize=(7,5))
plt.xlim(xmax = 4, xmin =-4)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.plot(dataset_linear['X_test'], y_preds, color='blue', label="BLR average")
plt.fill_between(dataset_linear['X_test'], y_preds-1*std_preds, y_preds+1*std_preds, color='lightblue', label='1 std. int.')
plt.legend()
plt.show()
```

# 2. Non-Linear Problem

```python
# Generate linear toy dataset
def sinus_data(x):
    y = 0.6*np.sin(2*np.pi*x) + 0.4*x
    return y
# Create training and test points
sigma_y   = 0.3  # fixed and known noise level
num_train = 50
dataset_linear = {}
dataset_linear['X_train'] = np.random.uniform(-2, 2, num_train)
# add noise on training data
dataset_linear['y_train'] = sinus_data(dataset_linear['X_train']) + np.random.normal(0, sigma_y, num_train)
dataset_linear['X_test'] = np.linspace(-4, 4, 10*num_train)
dataset_linear['y_test'] = sinus_data(dataset_linear['X_test'])
```

```python
# Plot dataset
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2, xmin =-2)
plt.ylim(ymax = 2, ymin = -2)
plt.plot(dataset_linear['X_test'], dataset_linear['y_test'], color='red', linewidth=2, label="True Function")
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

#### Polynomial basis function

```python
# Define basis function
from sklearn.preprocessing import PolynomialFeatures
def phi_polynomial(x, D=5):
    """ Polynomial Basis Functions

    Args:
      x: (float) 1D input

    Returns:
      (array) polynomial features of x
    """
    poly = PolynomialFeatures(degree=D)
    return poly.fit_transform(x.reshape(-1, 1))
```

```python
phi_train = phi_polynomial(dataset_linear['X_train'])
phi_test = phi_polynomial(dataset_linear['X_test'])
```

##### Polynomial Fits for Different Degrees

Understand the relationship between overfitting and model complexity

```python
degree = 15 #vary the degree here
phi_train = phi_polynomial(dataset_linear['X_train'], degree)
phi_test = phi_polynomial(dataset_linear['X_test'],   degree)
theta = mle_compute(phi_train, dataset_linear['y_train'])
y_predict_mle = phi_test.dot(theta)
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2, xmin =-2)
plt.ylim(ymax = 2, ymin = -2)

plt.plot(dataset_linear['X_test'], y_predict_mle, color='green', linewidth=1.5, label="MLE Predictions with {} degress".format(degree))
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

## Maximum-A-Posterior Estimation

Adding a regularizer is identical to using a prior distribution $p(\theta)$   on the model parameters

```python
prior_tau1 = 0.1
theta_map1 = map_compute(phi_train, dataset_linear['y_train'], prior_tau1, sigma_y)
```

```python
prior_tau2 = 1.5
theta_map2 = map_compute(phi_train, dataset_linear['y_train'], prior_tau2, sigma_y)
```

```python
y_predict_map1 = phi_test.dot(theta_map1)
y_predict_map2 = phi_test.dot(theta_map2)
```

```python
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2, xmin =-2)
plt.ylim(ymax = 2, ymin = -2)
plt.plot(dataset_linear['X_test'], y_predict_map1, color='blue', linewidth=1.5, label="MAP Prediction with prior tau: {} and polynomial degree {}".format(prior_tau1, degree))
plt.plot(dataset_linear['X_test'], y_predict_map2, color='red', linewidth=1.5, label="MAP Prediction with prior tau: {} and polynomial degree {}".format(prior_tau2, degree))
plt.plot(dataset_linear['X_test'], y_predict_mle, color='green', linewidth=1.5, label="MLE Predictions with {} degress".format(degree))
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.legend()
plt.show()
```

### Bayesian Regression

```python
#positer prediction
theta_mean, var_theta  = bayes_compute(phi_train, dataset_linear['y_train'], prior_tau1, sigma_y)
```

```python
#sample from positier distribution
n_sample = 30
z = np.random.multivariate_normal(mean=theta_mean, cov=var_theta, size=n_sample)
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2, xmin =-2)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
for idx, one_model in enumerate(z):
    y_predict_sample = phi_test.dot(one_model)
    if idx == 0:
        plt.plot(dataset_linear['X_test'], y_predict_sample, color='blue', linewidth=0.5,label='Sampled Models')
    else:
        plt.plot(dataset_linear['X_test'], y_predict_sample, color='blue', linewidth=0.5)
plt.legend()
plt.show()
```

```python
y_preds = []
std_preds = []
for testdata in phi_test:
    y_pred = testdata.T.dot(theta_mean)
    std_pred = np.sqrt(sigma_y*sigma_y+testdata.T.dot(var_theta).dot(testdata))
    y_preds.append(y_pred)
    std_preds.append(std_pred)
```

```python
y_preds = np.array(y_preds)
std_preds = np.array(std_preds)
```

```python
plt.figure(figsize=(7,5))
plt.xlim(xmax = 2.1, xmin =-2.2)
plt.ylim(ymax = 4, ymin = -4)
plt.plot(dataset_linear['X_train'], dataset_linear['y_train'], 'o', color='black', label='Training points')
plt.plot(dataset_linear['X_test'], y_preds, color='blue', label="BLR average")
plt.fill_between(dataset_linear['X_test'], y_preds-1*std_preds, y_preds+1*std_preds, color='lightblue', label='1 std. int.')
plt.legend()
plt.show()
```

05  Simple Bayesian Linear Regression.ipynb


<figure>
  <IMG SRC="gfx/Logo_norsk_pos.png" WIDTH=100 ALIGN="right">
</figure>

# Particle in a two-dimensional box
*Roberto Di Remigio*, *Luca Frediani*

After discussing and experimenting with the one-dimensional particle in a box model, we now move on to the two-dimensional case. The particle is now confined into a two-dimensional box with sides $L_x$ and $L_y$ long by the appropriate potential energy operator:

\begin{equation}
V(x, y) =
\begin{cases}
0 \quad\quad \text{if} \,\, 0\leq x \leq L_x \,\, \text{and} \,\, 0 \leq y \leq L_y \\
+\infty \quad\quad \text{otherwise}
\end{cases}
\end{equation}

Notice that $L_x$ can differ from $L_y$. In the general case, the particle can be confined inside
a rectangular well. The geometry of the potential will determine the properties of the solutions.

How does the quantum particle behave? We need to find the **eigenfunctions** and **eigenvalues** of the **Hamiltonian operator**, that is we have to solve the following ordinary differential equation:
\begin{equation}
-\frac{\hbar^2}{2m}\left(\frac{\mathrm{\partial}^2}{\mathrm{\partial}x^2}
+ \frac{\mathrm{\partial}^2}{\mathrm{\partial}y^2}\right)
\psi_{nm}(x,y) = E_{nm}\psi_{nm}(x,y)
\end{equation}
with **boundary conditions**:
\begin{equation}
\begin{aligned}
  \psi_{nm}(0, y) &= 0 \\
  \psi_{nm}(L_x, y) &= 0
\end{aligned}
\end{equation}
and:
\begin{equation}
\begin{aligned}
  \psi_{nm}(x, 0) &= 0 \\
  \psi_{nm}(x, L_y) &= 0
\end{aligned}
\end{equation}
You will notice that, not only the eigenfunctions now depend on two **degrees of freedom** (the $x$ and $y$ coordinates) but they also carry **two** quantum numbers $n$ and $m$.

Given that the kinetic energy operator is **separable**, an acceptable form for the solutions is
the product of one-dimensional states:
\begin{equation}
\psi_{nm}(x, y) = \psi_{n}(x)\psi_{m}(y)
\end{equation}
that is, states that are eigenfunctions of the one-dimensional particle in a box problem.
A more explicit form is:

\begin{equation}
\psi_{nm}(x, y) = \sqrt{\frac{2}{L_x}}\sin\left(\frac{n\pi x}{L_x}\right)
\sqrt{\frac{2}{L_y}}\sin\left(\frac{m\pi y}{L_y}\right) \quad \forall n, m \neq 0
\end{equation}

We can then derive the form of the eigenvalues by inserting this form of the wavefunction into the Schrödinger equation:
\begin{equation}
E_{nm} = \frac{h^2}{8M}\left(\frac{n^2}{L_x^2} + \frac{m^2}{L_y^2} \right)
\end{equation}
Of course, if the box is square the expression for the eigenvalues would simplify to:
\begin{equation}
E_{nm} = \frac{h^2}{8ML^2}\left(n^2 + m^2\right) \quad \forall n, m\neq 0
\end{equation}

## Exercise 1: Normalization

The one-dimensional eigenfunctions $\psi_n(x)$ and $\psi_m(y)$ are orthonormal. What about the two-dimensional eigenfunctions $\psi_{nm}$? Are they orthogonal? Are they normalized?
Given a linear combination of two-dimensional normalized, eigenfunctions, is it still normalized? That is,
is
\begin{equation}
\Psi(x, y) = \psi_{11}(x, y) + \psi_{21}(x, y)
\end{equation}
normalized? If not, find the normalization constant.

Define also a function to calculate the value of the two-dimensional eigenfunctions on a grid of points. We will use this function to plot the eigenfunctions.
The function should take the following arguments: the quantum numbers $n$ and $m$, the box lengths $L_x$ and $L_y$, the NumPy arrays with $x$ and $y$ values:

```Python
def eigenfunction2D(n, m, Lx, Ly, x, y):
    """ Normalized eigenfunction for the 2D particle in a box.

    n -- the quantum number, relative to the x axis
    m -- the quantum number, relative to the y axis
    Lx -- the size of the box on the x axis
    Ly -- the size of the box on the y axis
    x -- the NumPy array with the x values
    y -- the NumPy array with the y values
    """
```

Once this function is defined, we can obtain the respective probability distribution by taking its square.
**Hint** Notice that you can re-use the function for the one-dimensional particle in a box to write this one!

## Interval: 3D plots with `matplotlib`

Since these will be 3D plots, the `matplotlib` commands are slightly more complicated.
The following commands will set up two 3D plots side by side. Put the plot of the eigenfunction on the left panel and the probability density on the right panel.

```Python
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm
import numpy as np
from scipy.constants import *
# make sure we see it on this notebook
%matplotlib inline

# Generate points on x and y axes
x = np.linspace(0, pi, 100)
y = np.linspace(0, pi, 100)
# Generate grid in the xy plane
X, Y = np.meshgrid(x, y)
# Tell matplotlib to create a figure with two panels
fig = plt.figure(figsize=plt.figaspect(0.5))
# Tell matplotlib to add axes for a plot on the left panel
ax = fig.add_subplot(1, 2, 1, projection='3d')
# Generate function values for the first plot
Z = (np.sin(X)*np.cos(Y)).T
max_val = np.max(Z)
ax.plot_surface(X, Y, Z, rstride=8, cstride=8, alpha=0.3)
# Contour plots on the xy plane
cset = ax.contour(X, Y, Z, zdir='z', offset=-max_val)

ax.set_xlabel('X')
ax.set_xlim([0, L])
ax.set_ylabel('Y')
ax.set_ylim([0, L])
ax.set_zlabel('Z')
ax.set_zlim(-max_val, max_val)

# Tell maplotlib to add axes for a plot on the right panle
ax = fig.add_subplot(1, 2, 2, projection='3d')
Z1 = ((np.sin(X)*np.cos(Y)).T )**2
max_val = np.max(Z1)
ax.plot_surface(X, Y, Z1, rstride=8, cstride=8, alpha=0.3)
# Contour plots on the xy plane
cset = ax.contour(X, Y, Z1, zdir='z', offset=-max_val)

plt.show()

```

```python
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
from matplotlib import cm
import numpy as np
from scipy.constants import *
# make sure we see it on this notebook
%matplotlib inline

# Generate points on x and y axes
x = np.linspace(0, pi, 100)
y = np.linspace(0, pi, 100)
# Generate grid in the xy plane
X, Y = np.meshgrid(x, y)
# Tell matplotlib to create a figure with two panels
fig = plt.figure(figsize=plt.figaspect(0.5))
# Tell matplotlib to add axes for a plot on the left panel
ax = fig.add_subplot(1, 2, 1, projection='3d')
# Generate function values for the first plot
Z = (np.sin(X)*np.cos(Y)).T
max_val = np.max(Z)
ax.plot_surface(X, Y, Z, rstride=8, cstride=8, alpha=0.3)
# Contour plots on the xy plane
cset = ax.contour(X, Y, Z, zdir='z', offset=-max_val)

ax.set_xlabel('X')
ax.set_xlim([0, pi])
ax.set_ylabel('Y')
ax.set_ylim([0, pi])
ax.set_zlabel('Z')
ax.set_zlim(-max_val, max_val)

# Tell maplotlib to add axes for a plot on the right panle
ax = fig.add_subplot(1, 2, 2, projection='3d')
Z1 = ((np.sin(X)*np.cos(Y)).T)**2
max_val = np.max(Z1)
ax.plot_surface(X, Y, Z1, rstride=8, cstride=8, alpha=0.3)
# Contour plots on the xy plane
cset = ax.contour(X, Y, Z1, zdir='z', offset=-max_val)
```

## Exercise 2: Eigenfunction or not?

Given the following linear combinations of two-dimensional eigenfunctions:
\begin{equation}
\begin{aligned}
\psi_a(x, y) &= \psi_{11}(x, y) + \psi_{22}(x, y) \\
\psi_b(x, y) &= \psi_{11}(x, y) - \psi_{22}(x, y) \\
\psi_c(x, y) &= \psi_{23}(x, y) + \psi_{32}(x, y) \\
\psi_d(x, y) &= \psi_{23}(x, y) - \psi_{32}(x, y) \\
\end{aligned}
\end{equation}
Are these eigenfunctions of the two-dimensional particle in a box Hamiltonian when the box is rectangular ($L_x\neq L_y$)? What happens when the box is square ($L_x = Ly$)?
Normalize all the linear combinations and plot them, together with their probability distributions.

## Exercise 3: Hybridization

The eigenfunctions for the particle in a square box can be used to visualize orbitals similar to the $sp$ and $sp^2$ hybridized orbitals, as explained [here].
Hybridized orbitals are linear combinations of eigenfunctions for a given problem that exhibit a peculiar structure of the probability density.
Plot the wavefunctions and probability densities for the following linear combinations:
\begin{equation}
\Psi(x, y) = \psi_{11}(x, y) + \psi_{21}(x, y) \quad\quad \Psi(x, y) = \psi_{11}(x, y) - \psi_{21}(x, y)
\end{equation}

[here]: http://pubs.acs.org/doi/abs/10.1021/ed067p866