# Basic Bayesian Inference

Statistical inference is a process of learning from incomplete or imperfect (error-contaminated) data. Can account for this "imperfection" using either a sampling model or a measurement error model.

### Statistical hypothesis testing

The *de facto* standard for statistical inference is statistical hypothesis testing. The goal of hypothesis testing is to evaluate a **null hypothesis**. There are two possible outcomes:

- reject the null hypothesis
- fail to reject the null hypothesis

Rejection occurs when a chosen test statistic is higher than some pre-specified threshold valuel; non-rejection occurs otherwise.

Notice that neither outcome says anything about the quantity of interest, the **research hypothesis**.

Setting up a statistical test involves several subjective choices by the user that are rarely justified based on the problem or decision at hand:

- statistical test to use
- null hypothesis to test
- significance level

Choices are often based on arbitrary criteria, including "statistical tradition" (Johnson 1999). The resulting evidence is indirect, incomplete, and typically overstates the evidence against the null hypothesis (Goodman 1999).

Most importantly to applied users, the results of statistical hypothesis tests are very easy to misinterpret.

### Estimation

Instead of testing, a more informative and effective approach for inference is based on **estimation** (be it frequentist or Bayesian). That is, rather than testing whether two groups are different, we instead pursue an estimate of *how different* they are, which is fundamentally more informative.

Additionally, we include an estimate of **uncertainty** associated with that difference which includes uncertainty due to our lack of knowledge of the model parameters (*epistemic uncertainty*) and uncertainty due to the inherent stochasticity of the system (*aleatory uncertainty*).

# An Introduction to Bayesian Statistical Analysis

Though many of you will have taken a statistics course or two during your undergraduate (or graduate education, most of those who have will likely not have had a course in *Bayesian* statistics. Most introductory courses, particularly for non-statisticians, still do not cover Bayesian methods at all. Even today, Bayesian courses (similarly to statistical computing courses!) are typically tacked onto the curriculum, rather than being integrated into the program.

In fact, Bayesian statistics is not just a particular method, or even a class of methods; it is an entirely **different paradigm** for doing statistical analysis.

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of **probability statements**

This confers several benefits to the analyst, including:

- ease of interpretation, summarization of uncertainty
- can incorporate uncertainty in parent parameters
- easy to calculate summary statistics

### Bayesian vs Frequentist Statistics: *What's the difference?*

Any statistical inferece paradigm, Bayesian or otherwise, involves at least the following:

1. Some **unknown quantities** about which we are interested in learning or testing. We call these *parameters*.
2. Some **data** which have been observed, and hopefully contain information about.
3. One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### The Frequentist World View

- The **data** that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model **parameters** are considered **fixed**. A parameter's true value is uknown and fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

\\[f(y | \theta)\\]

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

Frequentist inference typically involves deriving **estimators** for the unknown parameters. Estimators are formulae that return estimates for particular estimands, as a function of data. They are selected based on some chosen optimality criterion, such as *unbiasedness*, *variance minimization*, or *efficiency*.

> For example, lets say that we have collected some data on the prevalence of autism spectrum disorder (ASD) in some defined population. Our sample includes \\(n\\) sampled children, \\(y\\) of them having been diagnosed with autism. A frequentist estimator of the prevalence \\(p\\) is:

>

> $$\hat{p} = \frac{y}{n}$$
>

> Why this particular function? Because it can be shown to be unbiased and minimum-variance.

It is important to note that, in a frequentist world, new estimators need to be derived for every estimand that is introduced.

### The Bayesian World View

- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**. In some cases, it is useful to consider parameters as having been sampled from probability distributions.

This implies the following form:

## $$p(\theta | y)$$

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

## Bayes' Formula

Now that we have some probability under our belt, we turn to Bayes' formula. While frequentist statistics uses different estimators for different problems, Bayes formula is the **only estimator** that Bayesians need to obtain estimates of unknown quantities that we care about.

The equation expresses how our belief about the value of \\(\theta\\), as expressed by the **prior distribution** \\(P(\theta)\\) is reallocated following the observation of the data \\(y\\).

The innocuous denominator \\(P(y)\\) usuallt cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\):

\\[Pr(\theta|y) = \frac{Pr(y|\theta)Pr(\theta)}{\int Pr(y|\theta)Pr(\theta) d\theta}\\]

The intractability of this integral is one of the factors that has contributed to the under-utilization of Bayesian methods by statisticians.

### Priors

Once considered a controversial aspect of Bayesian analysis, the prior distribution characterizes what is known about an unknown quantity before observing the data from the present study. Thus, it represents the information state of that parameter. It can be used to reflect the information obtained in previous studies, to constrain the parameter to plausible values, or to represent the population of possible parameter values, of which the current study's parameter value can be considered a sample.

### Likelihood functions

The likelihood represents the information in the observed data, and is used to update prior distributions to posterior distributions. This updating of belief is justified becuase of the **likelihood principle**, which states:

> Following observation of \\(y\\), the likelihood \\(L(\theta|y)\\) contains all experimental information from \\(y\\) about the unknown \\(\theta\\).

Bayesian analysis satisfies the likelihood principle because the posterior distribution's dependence on the data is **only through the likelihood**. In comparison, most frequentist inference procedures violate the likelihood principle, because inference will depend on the design of the trial or experiment.

Remember from the density estimation section that the likelihood is closely related to the probability density (or mass) function. The difference is that the likelihood varies the parameter while holding the observations constant, rather than *vice versa*.

## Bayesian Inference, in 3 Easy Steps

Gelman et al. (2013) describe the process of conducting Bayesian statistical analysis in 3 steps.

### Step 1: Specify a probability model

As was noted above, Bayesian statistics involves using probability models to solve problems. So, the first task is to *completely specify* the model in terms of probability distributions. This includes everything: unknown parameters, data, covariates, missing data, predictions. All must be assigned some probability density.

This step involves making choices.

- what is the form of the sampling distribution of the data?
- what form best describes our uncertainty in the unknown parameters?

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns
sns.set_context('notebook')

RANDOM_SEED = 20090425
```

```python
from pymc3 import Poisson

x = Poisson.dist(mu=1)
samples = x.random(size=10000)
```

```python
samples.mean()
```

    0.9859

```python
plt.hist(samples, bins=len(set(samples)));
```

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}
E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2
\end{align}$$

```python
from pymc3 import Normal

y = Normal.dist(mu=-2, sd=4)
samples = y.random(size=10000)
```

```python
samples.mean()
```

    -2.0303670533348503

```python
samples.std()
```

    3.972885354039681

```python
plt.hist(samples);
```

### Step 2: Calculate a posterior distribution

The mathematical form \\(p(\theta | y)\\) that we associated with the Bayesian approach is referred to as a **posterior distribution**.

> posterior /pos·ter·i·or/ (pos-tēr´e-er) later in time; subsequent.

Why posterior? Because it tells us what we know about the unknown \\(\theta\\) *after* having observed \\(y\\).

This posterior distribution is formulated as a function of the probability model that was specified in Step 1. Usually, we can write it down but we cannot calculate it analytically. In fact, the difficulty inherent in calculating the posterior distribution for most models of interest is perhaps the major contributing factor for the lack of widespread adoption of Bayesian methods for data analysis. Various strategies for doing so comprise this tutorial.

**But**, once the posterior distribution is calculated, you get a lot for free:

- point estimates
- credible intervals
- quantiles
- predictions

### Step 3: Check your model

Though frequently ignored in practice, it is critical that the model and its outputs be assessed before using the outputs for inference. Models are specified based on assumptions that are largely unverifiable, so the least we can do is examine the output in detail, relative to the specified model and the data that were used to fit the model.

Specifically, we must ask:

- does the model fit data?
- are the conclusions reasonable?
- are the outputs sensitive to changes in model structure?

---
## References and Resources

- Goodman, S. N. (1999). Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130(12), 995–1004. http://doi.org/10.7326/0003-4819-130-12-199906150-00008
- Johnson, D. (1999). The insignificance of statistical significance testing. Journal of Wildlife Management, 63(3), 763–772.
- Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis, Third Edition. CRC Press.


# Problem set 2

Getting the mnist dataset from keras and importing other libraries that would help in this problem statement.

Sources: https://www.youtube.com/watch?v=81ZGOib7DTk
https://www.datacamp.com/community/tutorials/svm-classification-scikit-learn-python
https://towardsdatascience.com/write-markdown-latex-in-the-jupyter-notebook-10985edb91fd, https://nlp.stanford.edu/IR-book/html/htmledition/support-vector-machines-the-linearly-separable-case-1.html#:~:text=The%20SVM%20in%20particular%20defines,the%20margin%20of%20the%20classifier.&text=A%20classifier%20with%20a%20large%20margin%20makes%20no%20low%20certainty%20classification%20decisions.

TASK 1: Write code to train a multi-class support vector classifier with dot-product kernel and 1-norm
soft margin using the MNIST training data set.Then report the performance using MNIST test data set. There is a hyper-parameter that sets the trade-off between the margin and the training error | tune this hyper-parameter through cross-validation.

Linear Kernel: A linear kernel can be used as normal dot product any two given observations. The product between two vectors is the sum of the multiplication of each pair of input values.

```python
import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
import math
from sklearn import svm
```

Loading the testing(10000) and training data(60000) samples, creating SVM classifier, training the model using training sets, and also predicting the data.

```python
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000,784) #had to reshape it to make equal dimensions
x_test = x_test.reshape(10000,784) #had to reshape it to make equal dimensions
print(x_train.shape, y_train.shape, x_test.shape, y_test.shape)
clf = svm.LinearSVC()
clf.fit(x_train, y_train) #training the data
y_pred = clf.predict(x_test) #predict the response of test dataset

```

    (60000, 784) (60000,) (10000, 784) (10000,)

    C:\Users\lovey\anaconda3\lib\site-packages\sklearn\svm\_base.py:976: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      warnings.warn("Liblinear failed to converge, increase "

In this the accuracy of dataset is calculated for that I am importing metrics from sklearn

```python
from sklearn import metrics
accuracy = metrics.accuracy_score(y_test, y_pred)
print("Accuracy:", accuracy*100, "%")
```

    Accuracy: 87.26 %

TASK2:

Identify the Lagrange dual problem of the following primal problem:
Given Features $(x_1,y_1),....(x_N,y_N)$ $where y_1,...y_N\in{-1,1}$
\begin{equation}
Minimize \; w^T.w+C\sum_{i=1}^N\xi_i \;
\
Subject\;to\;y_i(w^T.x_i)\geq1-\xi_i and \;\;\xi_i\geq0
\end{equation}
inequality constraints can be expressed as and also lagrange form is:
\begin{equation}
2-y_i(w^T.x_i)-\xi_i\leq0 where\ 1-\xi\leq0
\\
L(w,\alpha) = w^T.w+C\sum_{i=1}^N\xi_i + \alpha_1(2-y_i(w^T.x_i)-\xi_i)+\alpha_2(1-\xi)
\end{equation}
where $\alpha_1$ & $\alpha_2$ are the Lagrange Multipliers where $\alpha_i\geq0$.
Set the partial derivative function to Zero.
\begin{equation}
\partial_wL = 2w - \alpha_1y_ix_i= 0
\\
w = \frac{\alpha_1y_ix_i}{2}
\end{equation}
The margin is:
\begin{equation}
\gamma = y\frac{w^Tw+b}{||w||}
\end{equation}

The SVM in particular defines the criterion to be looking for a decision surface that is maximally far away from any data point. This distance from the decision surface to the closest data point determines the margin of the classifier.

Benefits of maximizing the margin is this allows to handle minor  errors in the data sets.

Solving Dual problem advantages: In the Dual problem we take $\alpha_i$ which help us to decide the importance of the subjected features also computation is fast as well as optimzation is easy.

Solving Primal Problem Disadvantages: we only obtain the optimal $w$ also Computing the scalar product of $w^Tx$ which makes the computation slow.


# Spectral decomposition

This work by Jephian Lin is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
from scipy import linalg as LA
import sympy
```

```python
def spec_decom(A, tol=0.0001):
    n = A.shape[0]
    vals,vecs = LA.eigh(A)

    inds = np.where((vals[1:] - vals[:-1]) > tol)[0]
    starts = np.hstack([np.array([0]), inds+1])
    ends = np.hstack([inds+1, np.array([n])])

    dist_vals = vals[starts]
    projs = np.zeros((len(dist_vals), n, n))
    i = 0
    for s,e in zip(starts,ends):
        space = vecs[:,s:e]
        projs[i] = space.dot(space.T)
        i += 1

    return dist_vals, projs
```

## Main idea

A matrix $A$ is **symmetric** if $A^\top = A$.

Every symmetric matrix $A$ can be written as
$$ A = \sum_{i=1}^q \lambda P_i$$
such that $\sum_{i=1}^q P_i = I$, $P_iP_j = O$ if $i\neq j$, and $P_i^2 = P_i$.
This is called the **spectral decomposition** of $A$.

Each $P_i$ is in fact a projection matrix onto a space $E_i$.
For any vector ${\bf v}$ in $E_i$, $A{\bf v} = \lambda_i{\bf v}$.
We call $\lambda_i$ an **eigenvalue** ,
call ${\bf v}$ an **eigenvector** ,
and $E_i$ an **eigenspace** of $A$.

## Side stories

- eigenvalue, eigenvector, eigenspace
- spectral decomposition = projection then scale

## Experiments

###### Exercise 1
Let
```python
A = np.ones((5,5), dtype=float) - np.eye(5, dtype=float)
v1 = np.ones((5,), dtype=float)
v2 = np.array([1,-1,0,0,0], dtype=float)
```

###### 1(a)
Check if ${\bf v}_1$ is an eigenvector of $A$.
If yes, what is the eigenvalue?

```python
### your answer here
```

###### 1(b)
Check if the $i$-th entry of ${\bf v}_2$ is zero then the $i$-th entry of $A{\bf v}_2$ is also zero.

```python
### your answer here
```

###### 1(c)
Check if ${\bf v}_2$ is an eigenvector of $A$.
If yes, what is the eigenvalue?

```python
### your answer here
```

###### Exercise 2
Let
```python
A = np.array([[2,-1,-1],
              [-1,2,-1],
              [-1,-1,2]]) / 3
v1 = np.array([1,1,1])
v2 = np.array([1,-1,0])
```
Since checking the zero is tiring, we will use a different approach to test whether a vector is an eigenvector.
The following are equivalent:
- $A{\bf v} = \lambda{\bf v}$ for some $\lambda$
- $A{\bf v}$ is a multiple of ${\bf v}$
- $A{\bf v}$ and ${\bf v}$ are parallel (zero vectors are considered to be parallel to any vector)
- $|\langle A{\bf v}, {\bf v}\rangle| = \|A{\bf v}\|\|{\bf v}\|$

###### 2(a)
Compute $|\langle A{\bf v}_1, {\bf v}_1\rangle|$ and $\|A{\bf v}_1\|\|{\bf v}_1\|$.
Then see if they are close.

```python
### your answer here
```

###### 2(b)
Do the same for ${\bf v}_2$.

```python
### your answer here
```

###### 2(c)
Run and understand the code below.

```python
%matplotlib notebook

tol = 0.1
vs = 5*np.random.randn(3,10000)
Avs = A.dot(vs)
dots = np.abs(np.sum(vs * Avs, axis = 0)) ### |<Avi, vi>|'s
Avs_length = np.linalg.norm(Avs, axis=0) ### ||Avi||'s
vs_length = np.linalg.norm(vs, axis=0) ### ||vi||'s
diff = np.abs( dots - Avs_length*vs_length)
mask = (diff < tol) ### mask for potential eigenvectors
eigvecs = vs[:,mask]

ax = plt.axes(projection='3d')
ax.set_xlim(-5,5)
ax.set_ylim(-5,5)
ax.set_zlim(-5,5)
ax.scatter(eigvecs[0], eigvecs[1], eigvecs[2])
```
What is the plane in the output?
Change `tol` to 1 or 2.  Then you will see a vague straight line appears.  Why?

```python
### your answer here
```

## Exercises

###### Exercise 3
Let
```python
A = np.array([[0,1,0,0,0,1],
              [1,0,1,0,0,0],
              [0,1,0,1,0,0],
              [0,0,1,0,1,0],
              [0,0,0,1,0,1],
              [1,0,0,0,1,0]])
vals,projs = spec_decom(A)
```
Here `vals` are all the $\lambda_i$'s and `projs` are all the $P_i$'s.

###### 3(a)
Check if $A = \sum_{i=0}^3\lambda_iP_i$.
Can you do it without any for loop?
Note:  You might need `np.set_printoptions(precision=1, suppress=True)`.

```python
### your answer here
```

###### 3(b)
Let
```python
v = np.ones((6,))
```
Check if $A{\bf v} = \sum_{i=0}^3 \lambda_i P_i{\bf v}$.

```python
### your answer here
```

###### 3(c)
Pick any $0\leq i,j\leq 3$.
Check if $P_iP_j = O$.

```python
### your answer here
```

###### 3(d)
Pick any $0\leq i\leq 3$.
Check if $P_i^2 = P_i$.

```python
### your answer here
```

##### Exercise 4
Let
```python
%matplotlib inline
plt.axis('equal')
A = np.array([[2,1],
              [1,2]])
t = np.linspace(0, 2*np.pi, 36)
xs = np.cos(t)
ys = np.sin(t)
vs = np.vstack([xs, ys])
Avs = A.dot(vs)
```

###### 4(a)
Plot a red point at the origin.
Plot the points (columns) in `vs` using `c=t` .
Then plot the points (columns) in `Avs` using `c=t` .
Interpret the output using the spectral decomposition.

```python
### your answer here
```

###### 4(b)
Do the same with
```python
A = np.array([[1,1],
              [1,1]])
```

```python
### your answer here
```

##### Exercise 5
Let
```python
A = np.array([[0,1,0,0,0,1],
              [1,0,1,0,0,0],
              [0,1,0,1,0,0],
              [0,0,1,0,1,0],
              [0,0,0,1,0,1],
              [1,0,0,0,1,0]])
vals,projs = spec_decom(A)

k = 3
```

###### 5(a)
Let
```python
B = k*A
vals_B,projs_B = spec_decom(B)
```
Find a relation between `vals` and `vals_B` using `k` .
Find a relation between `projs` and `projs_B` .

```python
### your answer here
```

###### 5(b)
Let
```python
B = A + k*np.eye(6)
vals_B,projs_B = spec_decom(B)
```
Find a relation between `vals` and `vals_B` using `k` .
Find a relation between `projs` and `projs_B` .

```python
### your answer here
```

###### 5(c)
It looks like $\lambda=1$ is an eigenvalue of $A$.
Use `sympy.Matrix` to find the null space of $A - \lambda I$.
Find a projection matrix of it, and compare it with the corresponding projection matrix in `projs` .

```python
### your answer here
```

Mathematics/Linear Algebra/0.0b Supplement of MIT Course - Concept Explanations with Problems_and_Questions to Solve/09-Spectral-decomposition.ipynb


# Quantization of Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Spectral Shaping of the Quantization Noise

The quantized signal $x_Q[k]$ can be expressed by the continuous amplitude signal $x[k]$ and the quantization error $e[k]$ as

\begin{equation}
x_Q[k] = \mathcal{Q} \{ x[k] \} = x[k] + e[k]
\end{equation}

According to the [introduced model](linear_uniform_quantization_error.ipynb#Model-for-the-Quantization-Error), the quantization noise can be modeled as uniformly distributed white noise. Hence, the noise is distributed over the entire frequency range. The basic concept of [noise shaping](https://en.wikipedia.org/wiki/Noise_shaping) is a feedback of the quantization error to the input of the quantizer. This way the spectral characteristics of the quantization noise can be modified, i.e. spectrally shaped. Introducing a generic filter $h[k]$ into the feedback loop yields the following structure

The quantized signal can be deduced from the block diagram above as

\begin{equation}
x_Q[k] = \mathcal{Q} \{ x[k] - e[k] * h[k] \} = x[k] + e[k] - e[k] * h[k]
\end{equation}

where the additive noise model from above has been introduced and it has been assumed that the impulse response $h[k]$ is normalized such that the magnitude of $e[k] * h[k]$ is below the quantization step $Q$. The overall quantization error is then

\begin{equation}
e_H[k] = x_Q[k] - x[k] = e[k] * (\delta[k] - h[k])
\end{equation}

The power spectral density (PSD) of the quantization error with noise shaping is calculated to

\begin{equation}
\Phi_{e_H e_H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \left| 1 - H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \right|^2
\end{equation}

Hence the PSD $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the quantizer without noise shaping is weighted by $| 1 - H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2$. Noise shaping allows a spectral modification of the quantization error. The desired shaping depends on the application scenario. For some applications, high-frequency noise is less disturbing as low-frequency noise.

### Example - First-Order Noise Shaping

If the feedback of the error signal is delayed by one sample we get with $h[k] = \delta[k-1]$

\begin{equation}
\Phi_{e_H e_H}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \left| 1 - \mathrm{e}^{\,-\mathrm{j}\,\Omega} \right|^2
\end{equation}

For linear uniform quantization $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sigma_e^2$ is constant. Hence, the spectral shaping constitutes a high-pass characteristic of first order. The following simulation evaluates the noise shaping quantizer of first order.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

w = 8  # wordlength of the quantized signal
xmin = -1  # minimum of input signal
N = 32768  # number of samples

def uniform_midtread_quantizer_w_ns(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -1)
    x[idx] = -1
    idx = np.where(x > 1 - Q)
    x[idx] = 1 - Q
    # linear uniform quantization with noise shaping
    xQ = Q * np.floor(x/Q + 1/2)
    e = xQ - x
    xQ = xQ - np.concatenate(([0], e[0:-1]))

    return xQ[1:]

# quantization step
Q = 1/(2**(w-1))
# compute input signal
np.random.seed(5)
x = np.random.uniform(size=N, low=xmin, high=(-xmin-Q))
# quantize signal
xQ = uniform_midtread_quantizer_w_ns(x, Q)
e = xQ - x[1:]
# estimate PSD of error signal
nf, Pee = sig.welch(e, nperseg=64)
# estimate SNR
SNR = 10*np.log10((np.var(x)/np.var(e)))
print('SNR = {:2.1f} dB'.format(SNR))

plt.figure(figsize=(10,5))
Om = nf*2*np.pi
plt.plot(Om, Pee*6/Q**2, label='estimated PSD')
plt.plot(Om, np.abs(1 - np.exp(-1j*Om))**2, label='theoretic PSD')
plt.plot(Om, np.ones(Om.shape), label='PSD w/o noise shaping')
plt.title('PSD of quantization error')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\hat{\Phi}_{e_H e_H}(e^{j \Omega}) / \sigma_e^2$')
plt.axis([0, np.pi, 0, 4.5]);
plt.legend(loc='upper left')
plt.grid()
```

**Exercise**

* The overall average SNR is lower than for the quantizer without noise shaping. Why?

Solution: The average power per frequency is lower that without noise shaping for frequencies below $\Omega \approx \pi$. However, this comes at the cost of a larger average power per frequency for frequencies above $\Omega \approx \pi$. The average power of the quantization noise is given as the integral over the PSD of the quantization noise. It is larger for noise shaping and the resulting SNR is consequently lower. Noise shaping is nevertheless beneficial in applications where a lower quantization error in a limited frequency region is desired.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


# 1. Electrical capacitance and energy of the mitochondrial inner membrane

A capacitor is an electrical circuit element that stores electrical energy in an electric field (Wikipedia REF). The mitochondrial inner membrane acts as an electrical capacitor to store energy in an electrostatic potential different between the milieux on the two sides the membrane.

Electrical capacitance is measured in units of charge per unit voltage, or in standard units, Coulombs per Volt (C/V), equivalently, a Farad. (1 F = 1 C/V)

## 1.1. What is the capacitance of mitochondrial inner membrane? Make a reasonable assumption about the size of a mitochondrion and assume that the inner membrane has 5 or 10 times more area than the outer membrane.

A typical range for the capacity density of a biological membrane is 0.5-1.0 $\mu$F cm$^{-2}$ [REF]. Let’s take a mitochondrion as a sphere with radius 1  $\mu$m. In this case the outer membrane would have a surface area of $4\pi r^2$, or approximately 12.5  $\mu$m$^2$. If the inner membrane has 10 times more area than the outer, then the area of the inner membrane is 125 $\mu$m$^2$. So, we can approximate the capacitance of this mitochondrion’s inner membrane as

$$
  c_{mito} = \left( 1 \frac{\mu{\rm F}} {{\rm cm}^2} \right)
             \left(  125 \, \mu{\rm m}^2  \right)
             \left( \frac{ 1 \, {\rm cm}^2} {10^8 \mu{\rm m}^2} \right) =
             1.25 \times 10^{-6} \mu{\rm F} .
$$

## 1.2. Express inner mitochondrial membrane capacitance in units of: (a.) farads per liter of mitochondria; (b.) moles per millivolt per unit liter of mitochondria.

Our one-micron radius spherical mitochondrion has a volume of approximately 4 $\mu$m$^3$, or $4 \times 10^{-15}$ L. Thus, the capacitance per untit volume is

$$
  \frac{1.25 \times 10^{-6} \mu{\rm F}}{4 \times 10^{-15} \, {\rm L}} =
  3.125 \times 10^{8} \frac{\mu{\rm F}}{{\rm L}} =
  3.125 \times 10^{2} \frac{{\rm F}}{{\rm L}} =
  3.125 \times 10^{2} \frac{{\rm C}}{{\rm V \, L}} =
  3.125 \times 10^{-1} \frac{{\rm C}}{{\rm mV \, L}}
$$

Using the fact that there are $1.036 \times 10^{−5}$ moles per coulomb, the capacitance can be expressed

$$
 3.24 \times 10^{-6} {\rm mol}/({\rm mV \, L}).
$$

## 1.3. If the electrostatic potential across the inner mitochondrial membrane is 180 mV, how much electrical energy per unit volume is stored in mitochondria? How much electrical energy is stored in mitochondria per unit volume of myocardium? Express your answer in joules per liter.

The total energy stored in a capacitor is $1/2\, c \Delta\Psi^2$, where $c$ is the capacitance and $\Delta\Psi$ is the voltage drop across the capacitance. (Can you derive this formula? Verify that the units are correct.) The density energy in the electrostatic potential across the mitochondrial inner membrane is computed

$$
  e_{mito} = \frac{1}{2} c_{mito} \Delta\Psi^2
  = \frac{3.125}{2} \times 10^{-1} \frac{{\rm C}}{{\rm mV \, L}} ( 180 \, {\rm mV)}) (0.180\, {\rm V})^2 \, ,
$$

which is roughly equal to 5 joule / L. The myocardium is approximately 1/3 by volume mitochondria. So the mitochondrial electrical energy per unit volume of myocardium is about 5/3 joule / L.

## 1.4. Approximately much electrical energy is stored in mitochondria in the whole human heart? How does that compare to the amount of energy supplied by a AA battery? How does it compare to the amount of mechanical work the heart does per unit time?

The heart is around 250 grams, or in terms of volume, approximately $1/4$ L. So the inner mitochondrial membrane of mitochondrial stores around $5/12$ of a joule of energy in the electrical potential.

For comparison a AA battery provides approximately 10 kJ of energy.

Another comparison: the LV does approximately 1 watt of work under resting baseline conditions. That is: 1 W = 1 joule / second. The $1/4$ joule of electrical energy stored in the inner mitochodondrial membrane is equivalent to how much mechanical work the heart does at rest in $1/4$second.

# 2. Converting electrical potential to the ATP hydrolysis chemical potential

The electrostatic energy potential across the mitochondrial inner membrane is used to drive the synthesis of ATP in the final step of oxidative ATP synthesis. The mammalian mitochondrial F1F0-ATPase synthesizes ATP in the mitochondrial matrix from ADP and inorganic phosphate, coupled to the translocation of protons (H+ ions) from outside to inside of the matrix. The chemical reaction stoichiometry can be expressed:

\begin{equation}\label{eq:ATPase1}
	{\rm MgADP}^{1-} + {\rm HPO}_4^{2-} + ({\rm H}^+)_{\rm inside} \leftrightharpoons
    {\rm MgATP}^{2-} + {\rm H}_2{\rm O},
\end{equation}

where the term $({\rm H}^+)_{\rm inside}$ indicates that a hydrogen ion from inside the matrix is covalently incorporated into the synthesized ATP. The species MgADP$^{1-}$ and MgATP$^{2-}$ are the magnesium-bound species of ADP and ATP. This chemical reaction is coupled to the transport of $n_A = 8/3$ protons across the inner membrane:

\begin{equation}\label{eq:ATPase2}
    n_A \, ({\rm H}^+)_{\rm outside} \leftrightharpoons n_A \, ({\rm H}^+)_{\rm inside}.
\end{equation}

## 2.1. Given a free magnesium concentration [Mg$^{2+}$] and hydrogen ion activity [H$^+$] = 10$^{\rm -pH}$, how can you compute the concentrations of MgADP$^{1-}$, MgATP$^{2-}$, and HPO$_4^{2-}$ in terms of the total concentrations of the reactants [$\Sigma$ADP], [$\Sigma$ATP], and [$\Sigma$Pi]? (You will need to account for binding of biochemical species to [Mg$^{2+}$] and [H$^+$].)

Let’s start with looking at the species of ATP:
\begin{eqnarray}
  [\Sigma{\rm ATP}] & = & [{\rm ATP}^{4-}] + [{\rm Mg}\cdot{\rm ATP}^{2-}] +
                          [{\rm H}\cdot{\rm ATP}^{3-}] + [{\rm K}\cdot{\rm ATP}^{3-}] \\
                    & = & [{\rm ATP}^{4-}]  + \frac{[{\rm Mg}^{2+}]}{K_{MgATP}}[{\rm ATP}^{4-}]  +
                          \frac{[{\rm H}^{+}]}{K_{HATP}}[{\rm ATP}^{4-}] +
                          \frac{[{\rm K}^{+}]}{K_{KATP}}[{\rm ATP}^{4-}]  \\
                    & = & [{\rm ATP}^{4-}]\left(1 + \frac{[{\rm Mg}^{2+}]}{K_{MgATP}} +
                          \frac{[{\rm H}^{+}]}{K_{HATP}} + \frac{[{\rm K}^{+}]}{K_{KATP}} \right)
\end{eqnarray}

(I am ignoring the two-cation-bound species.)

We can define something called the “binding polynomial” for each reactant:
\begin{eqnarray}
  P_{ATP} & = & 1 + \frac{[{\rm Mg}^{2+}]}{K_{MgATP}} +
                     \frac{[{\rm H}^{+}]}{K_{HATP}} + \frac{[{\rm K}^{+}]}{K_{KATP}} \\
  P_{ADP} & = & 1 + \frac{[{\rm Mg}^{2+}]}{K_{MgADP}} +
                     \frac{[{\rm H}^{+}]}{K_{HADP}} + \frac{[{\rm K}^{+}]}{K_{KADP}} \\
  P_{Pi}  & = & 1 + \frac{[{\rm Mg}^{2+}]}{K_{MgPi}} +
                     \frac{[{\rm H}^{+}]}{K_{HPi}} + \frac{[{\rm K}^{+}]}{K_{KPi}} \, ,
\end{eqnarray}
and
\begin{eqnarray}
  [\Sigma{\rm ATP}] & = & [{\rm ATP}^{4-}] P_{ATP} \\
  [\Sigma{\rm ADP}] & = & [{\rm ADP}^{3-}] P_{ADP} \\
  [\Sigma{\rm Pi}] & = & [{\rm HPO}_4^{2-}] P_{Pi} \, .
\end{eqnarray}

Furthermore
\begin{eqnarray}
  [{\rm Mg}\cdot{\rm ATP}^{2-}] & = & \frac{[{\rm Mg}^{2+}]/K_{MgATP}}{P_{ATP}}[\Sigma{\rm ATP}] \\
  [{\rm Mg}\cdot{\rm ADP}^{1-}] & = & \frac{[{\rm Mg}^{2+}]/K_{MgADP}}{P_{ADP}}[\Sigma{\rm ADP}] \\
  [{\rm HPO}_4^{2-}] & = & \frac{1}{P_{Pi}}[\Sigma{\rm Pi}] \\
\end{eqnarray}

## 2.2. Derive an expression for the Gibbs free energy change associated with reaction (2.1) in terms of the reference $\Delta G_r^0$, the concentrations of biochemical reactants, and the cation concentrations [Mg$^{2+}$] and [H$^+$]. What is the free energy of ATP hydrolysis in the mitochondrial matrix? Assume that pH = 7.2 and [$\Sigma$ADP] = 9.5 mM, [$\Sigma$ATP] = 0.5 mM, and [$\Sigma$Pi] = 1 mM.

Consider the ATP synthesis reaction
$$
  \rm ADP^{3-} + HPO_4^{2-} + H^+ \rightleftharpoons ATP^{4-} + H_2O \, ,
$$
with equilibrium mass action ratio
$$
  K_{eq} = e^{-\Delta G_r^0 / RT} = \left( \frac{[{\rm ATP}^{4-}]}
        {[{\rm H}^+][{\rm HPO}_4^{2-}][{\rm ADP}^{3-}]} \right)_{eq} \, .
$$
Substituting the expressions for the binding polynomials, we have
$$
  K_{eq} = e^{-\Delta G_r^0 / RT} = \left( \frac{[\Sigma{\rm ATP}]}
        {[\Sigma{\rm Pi}][\Sigma{\rm ADP}]} \right)_{eq}
        \frac{P_{ADP}P_{Pi}}{[{\rm H}^+]P_{ATP}}\, ,
$$

which yields and expression for the apparent equilibrium constant in terms of biochemical reactants
$$
  K'_{eq} = e^{-\Delta G_r^{\prime 0} / RT} = \left( \frac{[\Sigma{\rm ATP}]}
        {[\Sigma{\rm Pi}][\Sigma{\rm ADP}]} \right)_{eq} =
        K_{eq}\frac{[{\rm H}^+]P_{ATP}}{P_{ADP}P_{Pi}} \, .
$$

Using these equations, there are a couple of equivalent ways to express the Gibbs free enegy change for this reaction. One is based on the referenc chemical reaction

\begin{equation}
  \Delta G_r = \Delta G^0_r - RT \ln\left( \frac{[{\rm H}^+][{\rm HPO}_4^{2-}][{\rm ADP}^{3-}]}
                                                  {[{\rm ATP}^{4-}]}\right) \, .
\end{equation}

The other is
\begin{equation}
  \Delta G_r = \Delta G^{\prime 0}_r - RT \ln\left( \frac{[\Sigma{\rm Pi}][\Sigma{\rm ADP}]}
                                                  {[\Sigma{\rm ATP}]}\right) \, .
\end{equation}

To verify these two equations give the same value, take the following values for dissociation constants

| Reactant (L) | $K_{H-L}$                | $K_{K-L}$               | $K_{Mg-L}$                |
| ------------ | ------------------------ | ----------------------- | ------------------------- |
| ATP          | 2.757$\times 10^{-7}$ M  | 9.809$\times 10^{-2}$ M   | 8.430$\times 10^{-5}$ M   |
| ADP          | 4.106$\times 10^{-7}$ M  | 1.319$\times 10^{-1}$ M   | 7.149$\times 10^{-4}$ M   |
| PI           | 2.308$\times 10^{-7}$ M  | 3.803$\times 10^{-1}$ M  | 2.815$\times 10^{-2}$ M   |

and $\Delta_r^0 G = -4.67$ kJ/mole, which yields $K_{eq} = 6.1163$ at 37$^\circ$ C.

The Python code below verifies that both approaches yield the same value of 45.730 kJ/mole.

```python
# %%

import numpy

# Dissociation Constants
K_HATP = 2.757e-7
K_KATP = 9.809e-2
K_MATP = 8.430e-5
K_HADP = 4.106e-7
K_KADP = 1.319e-1
K_MADP = 7.149e-4
K_HPI  = 2.308e-7
K_KPI  = 3.803e-1
K_MPI  = 2.815e-2

# Other constants
Keq = 6.1163
DGr0 = -4670
RT = 310.15*8.3145
DGr0 = -RT*numpy.log(Keq)

# Concentrations
ATP = 0.5e-3
ADP = 9.5e-3
Pi  = 1e-3
Mg  = 1e-3
H   = 10**(-7.2)
K   = 100e-3

# binding polynomials:
P_ATP = 1 + H/K_HATP + K/K_KATP + Mg/K_MATP
P_ADP = 1 + H/K_HADP + K/K_KADP + Mg/K_MADP
P_PI = 1 + H/K_HPI + K/K_KPI + Mg/K_MPI

# reference species concentration
ATPs = ATP/P_ATP
ADPs = ADP/P_ADP
Pi_s = Pi/P_PI

# compute DeltaG_r in terms of reference species
DG1 = +DGr0 - RT*numpy.log(H*Pi_s*ADPs/ATPs)
print("First equation:",DG1," J/mole")

K_apparent = Keq*H*P_ATP/(P_ADP*P_PI)
DG2 = -RT*numpy.log(K_apparent) - RT*numpy.log(ADP*Pi/ATP)
print("Second equation:",DG2," J/mole")

ATPs, ATP
```

    First equation: 45729.93159766788  J/mole
    Second equation: 45729.93159766787  J/mole

    (3.543404140361549e-05, 0.0005)

This positive value for $\Delta_r G$ means that the reaction is not favorable to proceed in the forward driection. (In other words, ATP synthesis cannot proceed without being coupled to some other driving force.)

## 2.3. What is the free energy change of Equation (2.2) at $\Delta\Psi$ = 180 mV? How does the free energy change of Equation (2.1) compare to that of Equation (2.2)? How efficient is the transduction of electrical to chemical free energy in this step in ATP synthesis? (What is the ratio of energy stored in ATP to the total energy consumed?)

The electrical potential of 0.18 J/C can be converted to J/mole using the unit conversion factor $F = 96485.33$ C/mole, which is known as Faraday's constant:

\begin{equation}
  \Delta G_{H^+} = \left(0.18\,{\rm J/C}\right)\left( 96485.33\,{\rm C/mole} \right)
                 = 17367.35 \, {\rm J/mole} \, .
\end{equation}

(This equation ignores differences in pH on either side of the inner membrane.) This is the energy required (per mole) to move positive charges against the electrostatic gradient, from the inside to the outside of the matrix. In other words, there is an electrical force driving postive charge into the matrix.

Since ATP synthesis reaction in mitochondrial is coupled to $n_A = 8/3$ H$^+$ translocate to the matrix, the overall free energy change for the coupled transport and reaction process is

\begin{equation}
  \Delta G = \Delta G^{\prime 0}_r - RT \ln\left( \frac{[\Sigma{\rm Pi}][\Sigma{\rm ADP}]}
                                                  {[\Sigma{\rm ATP}]}\right)
                                   - n_A \Delta G_{H^+} \, .
\end{equation}

Plugging the numbers from Section (2.2) we get

\begin{eqnarray}
  \Delta G &=& +45.73 - (8/3)*17.37 \\
           &=& +45.73 -46.32  \\
           &=& -0.59 \, {\rm kJ/mole} \, ,
\end{eqnarray}

which means that, given the assumed concentrations, the system is operating close to equilibrium. The ratio of chemical energy stored in the ATP hydrolysis potential to electrical energy consumed is $45.73/46.32 \approx 98.7 \%$.

## 2.4. Given the concentrations assumed in 2.2, what is the minimum value of $\Delta\Psi$ at which ATP can be synthesized in the mitochondrial matrix?

At equilibrium the electrical energy consumed is equal to the ATP hydrolysis potential:
$$
 (8/3) \Delta G_{H^+} = 45.73 \, {\rm kJ/mole} ,
$$
yielding $\Delta\Psi = 177.7$ mV.

## 2.5. Assume that reaction (2.1) proceeds by simple mass-action kinetics, with a constant reverse rate kr. How does the forward rate constant necessarily depend on $\Delta\Psi$ for the reaction kinetics to be properly thermodynamically balanced?

To recap, the free energy for the ATP synthesis reaction
\begin{equation}
	{\rm ADP}^{3-} + {\rm HPO}_4^{2-} + ({\rm H}^+)_{\rm inside} \leftrightharpoons
    {\rm ATP}^{4-} + {\rm H}_2{\rm O},
\end{equation}
is
\begin{equation}
  \Delta G_r = \Delta G_r^0 - RT \log\left( \frac{[{\rm H}^+]_{\rm inside}[{\rm HPO}_4^{2-}][{\rm ADP}^{3-}]}
      {[{\rm ATP}^{4-}]} \right)
\end{equation}
where $\Delta G_r^0 \approx -4.6$ kJ/mole is the standard Gibbs free energy change for the reference chemical reaction. In mitochondria the ATP synthesis is coupled to proton transport from outside to inside of the matrix:
\begin{equation}
   n_A \left({\rm H}^+ \right)_{\rm outside} \leftrightharpoons n_A \left({\rm H}^+ \right)_{\rm inside}
\end{equation}
which is associated with a free energy change
\begin{equation}
  \Delta G_H = -n_A F\Delta\Psi - RT \log\left( \frac{[{\rm H}^+]_{\rm outside}}
          { [{\rm H}^+]_{\rm inside}} \right)^{n_A} \, .
\end{equation}
This free energy is sometimes called the proton motive force. The last term on the right-hand side represents the contribution to the proton-transport free energy from a gradient in pH across the inner mitochondrial membrane.

The free energy of coupled ATP synthsis and proton transport is
\begin{equation}
  \Delta G_r = \Delta G_r^0 - RT \log\left( \frac{[{\rm HPO}_4^{2-}][{\rm ADP}^{3-}]}
      {[{\rm ATP}^{4-}]} \right) -n_A F\Delta\Psi - RT \log \frac{\left( [{\rm H}^+]_{\rm outside}\right)^{n_A}}
          {\left( [{\rm H}^+]_{\rm inside}\right)^{n_A-1}}  \, .
\end{equation}

Chemical equilibrium occurs when $\Delta G_r = 0$:
\begin{equation}
  \left( \frac{[{\rm ATP}^{4-}]}{[{\rm HPO}_4^{2-}][{\rm ADP}^{3-}]}
       \right)_{eq} = \exp\left[-\left( \Delta G_r^0 - N_A F\Delta\Psi \right)/RT \right]\cdot
      \frac{\left( [{\rm H}^+]_{\rm outside}\right)^{n_A}}
          {\left( [{\rm H}^+]_{\rm inside}\right)^{n_A-1}} \, ,
\end{equation}
or in terms of sums of species
\begin{equation}
  K'_{F1F0ATPase} =
  \left( \frac{[\Sigma{\rm ATP}]}{[\Sigma{\rm Pi}][\Sigma{\rm ADP}]}
       \right)_{eq} = \exp\left[-\left( \Delta G_r^0 - N_A F\Delta\Psi \right)/RT \right]\cdot
       \frac{P_{ATP}}{P_{ADP} P_{Pi}}\cdot
      \frac{\left( [{\rm H}^+]_{\rm outside}\right)^{n_A}}
          {\left( [{\rm H}^+]_{\rm inside}\right)^{n_A-1}} \, .
\end{equation}

A simple mass-action expression for the reaction flux that satisfies this thermodynamic relation is
\begin{equation}
  J_{F1F0ATPase} = k_r\left( K'_{F1F0ATPase}[\Sigma{\rm Pi}][\Sigma{\rm ADP}] - [\Sigma{\rm ATP}] \right) .
\end{equation}

## 2.6. Write a simple program that simulates the kinetics of [$\Sigma$ADP], [$\Sigma$ATP] , and [$\Sigma$Pi] in the matrix given a fixed membrane potential, pH, and magnesium concentration, and given arbitrary initial conditions. How do the predicted steady-state concentrations depend on membrane potential, pH, and magnesium concentration?

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# init_conditions = ATP, ADP, P
# ATP, ADP, P = init_conditions
# Psi input in mV
def J_ATPase(psi, pH_inner, pH_outer, Mg, concentrations):
    # Define initial conditions
    ATP, ADP, Pi = concentrations

    # Assume k_r = 1
    k_r = 10
    N_A = 8/3
    # Dissociation Constants
    K_HATP = 2.757e-7
    K_KATP = 9.809e-2
    K_MATP = 8.430e-5
    K_HADP = 4.106e-7
    K_KADP = 1.319e-1
    K_MADP = 7.149e-4
    K_HPI  = 2.308e-7
    K_KPI  = 3.803e-1
    K_MPI  = 2.815e-2

    # H concentrations
    H_in = 10**(-pH_inner)
    H_out = 10**(-pH_outter)
    # print("H_out",H_out)
    # print("H_in", H_in)
    H = H_in

    # binding polynomials:
    P_ATP = 1 + H/K_HATP + Mg/K_MATP
    P_ADP = 1 + H/K_HADP + Mg/K_MADP
    P_PI = 1 + H/K_HPI + Mg/K_MPI

    # Other constants
    # Keq = 6.1163
    DGr0 = -4670
    RT = 310.15*8.3145
    # DGr0 = -RT*np.log(Keq)
    F = 96485.33  # C/mole
    # R = 8.3145

    # Define K'_F1F0ATPase
    k_prime = np.exp(-(DGr0 - F * psi/1000* N_A)/(RT))*P_ATP/(P_ADP*P_PI)*H_out**N_A / H_in**(N_A-1) #Psi / 1000 b/c input in mv
    return k_r*(k_prime*Pi * ADP - ATP)

```

```python
pH_outter = 7.2
pH_inner = 7.4
ADP = 9.5 # mM
ATP = 0.5 # mM

Mg  = 1e-3
psi = 180
ATP = 0.5e-3
ADP = 9.5e-3
Pi  = 1e-3
Mg  = 1e-3
H   = 10**(-7.2)
K   = 100e-3

pH_outter = 7.2
pH_inner = 7.4

init_conditions = np.array([ATP, ADP, Pi])
# J_ATPase(psi, pH_inner, pH_outter, Mg, init_conditions)
def dSTATES_dt(STATES, t, psi, pH_inner, pH_outter, Mg):
    rate = J_ATPase(psi, pH_inner, pH_outter, Mg, STATES)
    dATP = rate
    dADP = -rate
    dPi = 0
    return (dATP, dADP, dPi)
```

```python
pH_outter = 7.2
pH_inner = 7.4
Mg  = 1e-3
psi = 180

t = np.linspace(0,2)

dSTATES_dt(init_conditions, t, psi, pH_inner, pH_outter, Mg)

ode_results = odeint(dSTATES_dt,init_conditions, t, (psi, pH_inner, pH_outter, Mg)).T
plt.plot(t,ode_results[0], label = 'ATP')
plt.plot(t,ode_results[1], label = 'ADP')
plt.plot(t,ode_results[2], label = 'Pi')
plt.legend()
ode_results[:,-1][0]/ode_results[:,-1][1]
```

```python
RT*np.log(0.2)
```

    -4150.325422837771

### Mg Analysis

```python
Mg_array = np.linspace(1e-4, 1e-2)
SS_end = []
for Mg in Mg_array:
    ode_results = odeint(dSTATES_dt,init_conditions, t, (psi, pH_inner, pH_outter, Mg)).T
    SS_end.append(ode_results[:,-1])
SS_end= np.array(SS_end)

plt.figure()
plt.plot(Mg_array,SS_end[:,0])
Mg  = 1e-3
```

### Psi Analysis

```python
psi_array = np.linspace(120, 220)
SS_end = []
for psi in psi_array:
    ode_results = odeint(dSTATES_dt,init_conditions, t, (psi, pH_inner, pH_outter, Mg)).T
    SS_end.append(ode_results[:,-1])
SS_end= np.array(SS_end)

plt.figure()
plt.plot(psi_array,SS_end[:,0])
plt.plot(psi_array,SS_end[:,1])
plt.plot(psi_array,SS_end[:,2])
psi  = 180
```

### Inner pH

```python
pH_array = np.linspace(7, 8)
SS_end = []
for pH_inner in pH_array:
    ode_results = odeint(dSTATES_dt,init_conditions, t, (psi, pH_inner, pH_outter, Mg)).T
    SS_end.append(ode_results[:,-1])
SS_end= np.array(SS_end)

plt.figure()
plt.plot(pH_array,SS_end[:,0])
plt.plot(pH_array,SS_end[:,1])
plt.plot(pH_array,SS_end[:,2])
psi  = 180
```

## 2.7 Buiding on the model developed in the previous section, add an extramitochondrial cytoplasmic compartment coupled to the mitochondria with the kinetic model for the adenine nucleotide translocase (ANT) of Bazil et al. https://doi.org/10.1016/j.bpj.2015.09.036. (The model formulation and kinetic parameters are in the supplementary material for the paper.) With constant $\Delta\Psi$ and inorganic phosphate concentrations, simulate how matrix and cytoplasmic ATP and ADP concentration depend on cytoplasmic ATP hydrolysis rate.

```python

```

Python Code/Quantitative Aspects of Mitochondrial Function (3).ipynb


Si $\frac{dy}{dt}$ es una recta, entonces tiene la forma:

$$
y = mx + b
$$

y como tiene ordenada al origen $0$, $b = 0$:

$$
y = mx
$$

aun mas, la pendiente de esa recta es $\frac{1}{2}$, por lo que la ecuación queda:

$$
y = \frac{1}{2}x
$$

o escrito de otra manera:

$$
y = \frac{1}{2} x
$$

Si $\frac{dx}{dt}$ es una parabola, eso quiere decir que tiene la forma:

$$
\frac{dx}{dt} = a x^2 + b x + c
$$

o escrito de otra manera:

$$
x' = ax^2 + b^x + c
$$

tambien sabemos que la ecuación de una parabola vertical se puede escribir como:

$$
(x-h)^2 = 4 p (y - k)
$$

en donde el vertice de la parabola está en $(h, k)$ y su distancia focal es $p$.

Examinando el problema podemos ver que la distancia focal es $1$ y que el vertice es $(2k, -k^2)$, por lo que podemos escribir la ecuación de la parabola como:

$$
(x-2k)^2 = 1 (y + k^2)
$$

para nuestro sistema es:

$$
(x-2k)^2 = (y + k^2)
$$

$$
x^2 - 4kx + 4k^2 - k^2 = y
$$

$$
y = x^2 - 4kx + 3 k^2
$$

Igualando estos dos sistemas obtenemos:

$$
\frac{1}{2} x = x^2 - 4k x + 3 k^2
$$

por lo que:

$$
x^2 - \left(4k + \frac{1}{2}\right) x + 3 k^2 = 0
$$

$$
x_{1,2} = \frac{\left(4k + \frac{1}{2}\right) \pm \sqrt{\left(4k + \frac{1}{2}\right)^2 - 4 (3 k^2)}}{2}
$$

$$
x_{1,2} = 2k + \frac{1}{4} \pm \frac{1}{2}\sqrt{4k^2 + 4k +\frac{1}{4}}
$$

```
from sympy import var, init_printing, solve, Matrix, eye, symbols
init_printing()

from IPython.html.widgets import interact, fixed
from matplotlib.pyplot import plot
%matplotlib inline
```

```
var("x, k")
```

```
ec = (x**2 - 4*k*x + 3*k**2) - (0.5*x)
ec
```

```
sol = solve(ec, x)
```

```
x1 = sol[0]
x1
```

```
x2 = sol[1]
x2
```

```
y = 0.5*x
y
```

```
y1 = y.subs(x, x1)
```

```
y2 = y.subs(x, x2)
```

```
x1.subs(k, 1)
```

```
y1.subs(k, 1)
```

```
x2.subs(k, 1)
```

```
y2.subs(k, 1)
```

Si $\frac{dy}{dt} = 0$, entonces tenemos $y = \frac{1}{2} x$.

Esto quiere decir que:

$$
y - \frac{1}{2}x = 0 = \frac{dy}{dt} = f_2
$$

De la misma manera para $\frac{dx}{dt}$:

$$
y - x^2 + 4kx - 3 k^2 = 0 = \frac{dx}{dt} = f_1
$$

```
var("y")
```

```
f1 = y - x**2 + 4*k*x - 3*k**2
f1
```

```
f2 = y - 0.5*x
f2
```

```
theta = Matrix([[x], [y]])
f = Matrix([[f1], [f2]])
f
```

```
Jf = f.jacobian(theta)
Jf
```

```
var("s")
```

```
detsIJf = ((s*eye(2) - Jf).det()).collect(s)
detsIJf
```

```
solucion1 = (solve(detsIJf, s)[0].subs(x, x1)).simplify()
solucion1
```

```
solucion2 = (solve(detsIJf, s)[1].subs(x, x1)).simplify()
solucion2
```

```
sol1 = solucion1.subs(k, 1)
```

```
def funcion(ks = 0.25):
    sol1 = solucion1.subs(k, ks)
    sol2 = solucion2.subs(k, ks)
    a1, b1 = sol1.as_real_imag()
    a2, b2 = sol2.as_real_imag()
    plot([a1, a2], [b1, b2], "o")
```

```
interact(funcion, ks=(0.0, 10.0))
```

```
a1, b1 = sol1.as_real_imag()
```

```
plot(a1, b1)
```

```

```

IPythonNotebooks/Modelado y Simulacion/Examen I - Analisis de sistemas dinamicos.ipynb


# Univariate methods

Univariate methods take only one feature into consideration when splitting the node into leafs. In this section we cover two univariate methods:
- CART,
- C4.5.

There are more univariate methods, but show only two examples that use different split methods. In this notebook we show two methods with a different approach to splitting. The first one build a binary tree and the second generates a non-binary tree.

```python
import math
import numpy as np
import pydot
import copy
from math import log
```

We should restore the ``data_set`` and ``labels`` from the previous notebook.

```python
%store -r data_set
%store -r labels
```

## CART

CART stands for Classification and Regression Trees. It generates a binary tree and consist of three steps:
1. Calculate the gini index for each feature
2. Take the lowest value ofωand split the node into two child nodes
3. Repeat the steps until we have all child nodes

Before we come to the method itself, we should define the leaf.

```python
class BinaryLeaf:

    def __init__(self, elements, labels, ids):
        self.L = None
        self.R = None
        self.elements = elements
        self.split_feature = None
        self.labels = labels
        self.completed = False
        self.ids = ids

    def set_R(self, Rleaf):
        self.R = Rleaf

    def set_L(self, Lleaf):
        self.L = Lleaf

    def set_elements(self, elements):
        self.elements = elements

    def get_elements(self):
        return self.elements

    def set_p(self, threshold):
        self.p = threshold

    def get_L(self):
        return self.L

    def get_R(self):
        return self.R

    def set_completed(self):
        self.completed = True

    def is_completed(self):
        return self.completed

    def get_labels(self):
        return self.labels

    def set_split(self, feature):
        self.split_feature = feature

    def get_split(self):
        return self.split_feature

    def set_ids(self, ids):
        self.ids = ids

    def get_ids(self):
        return self.ids
```

The variables that we gonna use is the ``labels_count``, in other words the number of classes. We need ``ids`` to track the split.

```python
labels_count = len(np.unique(labels))

ids = list(range(len(data_set)))
root = BinaryLeaf(data_set, labels, ids)
current_node = root
```

There are several helper functions that are next used by our CART method. We use the methods to:
- ``get_unique_labels`` - return the unique labels in a leaf,
- ``get_unique_values`` - return unique values in a leaf,
- ``is_leaf_completed`` - check if a leaf needs to be split or not,
- ``find_leaf_not_completed`` - returns the leaf that needs to be split.

```python
def get_unique_labels(labels):
    return np.unique(np.array(labels)).tolist()

def get_unique_values(elements):
    features_number = len(elements[0])
    unique = []
    for i in range(features_number):
        features_list = []
        for j in range(len(elements)):
            features_list.append(elements[j][i])
        unique.append(np.unique(np.array(features_list)))
    return unique

def is_leaf_completed(node):
    if node.is_completed():
        if node.get_L() != None and not node.get_L().is_completed():
            return node.get_L()
        elif node.get_R() != None and not node.get_R().is_completed():
            return node.get_R()
        elif node.get_L() == None and node.get_R() == None:
            return None
        elif node.get_L().is_completed() or node.get_R().is_completed():
            new_node = is_leaf_completed(node.get_L())
            if new_node == None:
                return is_leaf_completed(node.get_R())
            else:
                return new_node
        else:
            return None
    return node

def find_leaf_not_completed(root):
    return is_leaf_completed(root)
```

The split method below return possible split leafs.

```python
def get_split_candidates(unique_values):
    split_list = []
    for i in range(len(unique_values)):
        current_list = []
        temp_list = copy.deepcopy(unique_values)
        current_list.append(temp_list[i])
        del temp_list[i]
        current_list.append(temp_list)
        split_list.append(current_list)
    return split_list
```

The methods below return the ``get_number_of_labels_for_value`` return exactly what the name says.

```python
def get_number_of_labels_for_value(elements, column_id, label):
    count = 0
    if not isinstance(elements, list):
        elements_list = [elements]
    else:
        elements_list = elements

    column_elements = get_node_elements_column(column_id)

    for i in range(len(elements_list)):
        for j in range(len(column_elements)):
            if column_elements[j] == elements_list[i]:
                if current_node.labels[j] == label:
                    count = count + 1
    return count
```

Get values for feature ``column_id``.

```python
def get_node_elements_column(column_id):
    return np.array(current_node.elements)[..., column_id].tolist()

def count_number_of_elements(elements, column_id):
    count = 0
    if isinstance(elements, list):
        column_elements = get_node_elements_column(column_id)
        for i in range(len(elements)):
            count = count + column_elements.count(elements[i])
    else:
        count = count + get_node_elements_column(column_id).count(elements)
    return count
```

Calculate the $\phi $ for given feature.

```python
def calculate_omega(elements, column_id):
    t_l = count_number_of_elements(elements[0], column_id)
    t_r = count_number_of_elements(elements[1], column_id)
    p_l = t_l * 1.0 / len(current_node.elements) * 1.0
    p_r = t_r * 1.0 / len(current_node.elements) * 1.0

    sum_p = 0
    labels = get_unique_labels(current_node.labels)
    for i in range(labels_count):
        p_class_t_l = (get_number_of_labels_for_value(elements[0], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[0], column_id) * 1.0)
        p_class_t_r = (get_number_of_labels_for_value(elements[1], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[1], column_id) * 1.0)
        sum_p = sum_p + math.fabs(p_class_t_l - p_class_t_r)
    return 2.0 * p_l * p_r * sum_p
```

Method that checks if the new leaf does not need to be split again.

```python
def check_completed(labels, elements):
    ratio = len(get_unique_labels(labels))
    if ratio == 1:
        return True
    elements = sorted(elements)
    duplicated = [elements[i] for i in range(len(elements)) if i == 0 or elements[i] != elements[i - 1]]
    if len(duplicated) == 1:
        return True
    return False
```

Split method:

```python
def split_node(current_node, value, split_id, split_history):
    left_leaf = []
    left_leaf_labels = []
    left_leaf_ids = []
    right_leaf = []
    right_leaf_labels = []
    right_leaf_ids = []
    for i in range(len(current_node.elements)):
        if current_node.elements[i][split_id] == value:
            left_leaf.append(current_node.elements[i])
            left_leaf_labels.append(current_node.labels[i])
            left_leaf_ids.append(current_node.ids[i])
        else:
            right_leaf.append(current_node.elements[i])
            right_leaf_labels.append(current_node.labels[i])
            right_leaf_ids.append(current_node.ids[i])
    if len(right_leaf_labels) == 0 or len(left_leaf_labels) == 0:
        current_node.set_completed()
        return current_node, split_history
    split_history.append([str(current_node.ids), str(left_leaf_ids)])
    split_history.append([str(current_node.ids), str(right_leaf_ids)])
    current_node.set_L(BinaryLeaf(left_leaf, left_leaf_labels, left_leaf_ids))
    current_node.set_R(BinaryLeaf(right_leaf, right_leaf_labels, right_leaf_ids))
    current_node.set_split(split_id)
    current_node.set_completed()
    if check_completed(left_leaf_labels, left_leaf):
        current_node.L.set_completed()
    if check_completed(right_leaf_labels, right_leaf):
        current_node.R.set_completed()
    return current_node, split_history
```

Get the current node that needs to be split:

```python
def get_current_node():
    return find_leaf_not_completed()
```

Tree building method:

```python
def build(root_node):
    current_node = root_node
    stop_criterion = False
    split_history = []
    while stop_criterion == False:
        unique_values = get_unique_values(current_node.get_elements())
        max_unique_id = 0
        max_split_id = 0
        max_value = 0
        for i in range(len(unique_values)):
            if len(unique_values[i]) == 1:
                continue
            split_candidates = get_split_candidates(unique_values[i].tolist())
            for j in range(len(split_candidates)):
                current_value = calculate_omega(split_candidates[j], i)
                if max_value < current_value:
                    max_unique_id = i
                    max_split_id = j
                    max_value = current_value
        current_node, split_history = split_node(current_node, unique_values[max_unique_id][max_split_id], max_unique_id, split_history)
        new_node = find_leaf_not_completed(root_node)
        if new_node != None:
            current_node = new_node
        else:
            stop_criterion = True
    return root_node, split_history
```

Execution of the ``build`` method returns the tree and the split history. The second variable can be used to plot the tree.

```python
cart_tree, split_history_cart = build(current_node)
```

We can store the history to use it in other notebooks:

```python
%store split_history_cart
```

    Stored 'split_history_cart' (list)

The plot function is very simple:

```python
def plot_tree(split_history):
    tree = pydot.Dot(graph_type='graph')
    for split in split_history:
        new_edge = pydot.Edge(split[0], split[1])
        tree.add_edge(new_edge)
    tree.write('cart_tree.png', format='png')

plot_tree(split_history_cart)
```

We can display it:

```python
from IPython.display import Image
Image(filename='cart_tree.png')
```

## C4.5

In C4.5 method we generate a non-binary tree. As in the previous example, we should define the Leaf.

```python
class Leaf:

    def __init__(self, elements, labels, ids):
        self.child_leafs = []
        self.elements = elements
        self.labels = labels
        self.completed = False
        self.ids = ids

    def get_elements(self):
        return self.elements

    def set_child_leafs(self, new_leafs):
        self.child_leafs = new_leafs

    def set_completed(self):
        self.completed = True

    def is_completed(self):
        return self.completed

    def get_labels(self):
        return self.labels

    def get_child_leafs(self):
        return self.child_leafs

    def get_ids(self):
        return self.ids
```

We need ``ids`` to track the split and set the root of the tree.

```python
ids = list(range(len(data_set)))
root = Leaf(data_set, labels, ids)
current_node = root
```

There are several helper functions that are next used by our CART method. We use the methods to:
- ``get_unique_labels`` - return the unique labels in a leaf,
- ``get_unique_values`` - return unique values in a leaf,
- ``is_leaf_completed`` - check if a leaf needs to be split or not,
- ``find_leaf_not_completed`` - returns the leaf that needs to be split,
- ``get_current_node`` - return the leaf that we should currently work on (split).

```python
def get_unique_labels(labels):
    return np.unique(np.array(labels)).tolist()

def get_unique_values(elements):
    features_number = len(elements[0])
    unique = []
    for i in range(features_number):
        features_list = []
        for j in range(len(elements)):
            features_list.append(elements[j][i])
        unique.append(np.unique(np.array(features_list)))
    return unique

def is_leaf_completed(node):
    if node.is_completed():
        child_nodes = node.get_child_leafs()
        if len(child_nodes) == 0:
            return None
        is_child_to_return = False
        for i in range(len(child_nodes)):
            if not child_nodes[i].is_completed():
                return child_nodes[i]
            else:
                new_node = is_leaf_completed(child_nodes[i])
                if new_node != None:
                    is_child_to_return=True
        if is_child_to_return:
            return new_node
    return node

def find_leaf_not_completed(root_node):
    return is_leaf_completed(root_node)

def get_current_node(root):
    return find_leaf_not_completed(root)
```

We can calcualte the entropy:
\begin{equation}
E(X)=-\sum_{i=1}^{m}p_{i}\log_{2}p_{i}.
\end{equation}

```python
def calculate_entropy(labels):
    unique_labels, labels_count = np.unique(labels, return_counts=True)
    entropy = 0
    size = len(labels)
    for i in range(len(unique_labels)):
        if labels_count[i] > 0:
            log2 = log((labels_count[i] * 1.0) / (size * 1.0), 2)
        else:
            log2 = 0.0
        entropy = entropy - 1.0 * ((labels_count[i] * 1.0) / size) * log2
    return entropy
```

The method below finds the candiate to split using the entropy.

```python
def calculate_split_candidate_entropy(full_entropy, labels, elements, unique_labels, unique_elements, iter):
    split_entropy = 0
    for i in range(len(unique_elements)):
        indices = np.where(np.array(elements)[..., iter].tolist() == unique_elements[i])
        unique_size = len(indices[0].tolist())
        filtered_labels = np.array(labels)[indices]
        for j in range(len(unique_labels)):
            labels_count = filtered_labels.tolist().count(unique_labels[j])
            if labels_count > 0:
                log2 = log((labels_count * 1.0) / (unique_size * 1.0), 2)
            else:
                log2 = 0.0
            split_entropy = split_entropy - 1.0 * (
                    (labels_count * 1.0) / unique_size * 1.0) * log2 * unique_size * 1.0 / len(elements) * 1.0
    return (full_entropy - split_entropy)

```

Split the leaf:

```python
def split(current_node, split_values, column_id, split_history):
    new_leafs = []
    for i in range(len(split_values)):
        indices = np.where(np.array(current_node.get_elements())[..., column_id].tolist() == split_values[i])
        new_leaf_elements = np.array(current_node.get_elements())[indices].tolist()
        new_leaf_labels   = np.array(current_node.get_labels())[indices].tolist()
        new_leaf_ids = np.array(current_node.get_ids())[indices].tolist()
        new_leaf = Leaf(new_leaf_elements,new_leaf_labels, new_leaf_ids)
        split_history.append([str(current_node.ids), str(new_leaf_ids)])
        if len(np.unique(new_leaf_labels)) == 1:
            new_leaf.set_completed()
        new_leafs.append(new_leaf)
    current_node.set_child_leafs(new_leafs)
    current_node.set_completed()
    return current_node, split_history
```

Finally, we can build the tree as below:

```python
def build(root):
    stop_criterion = False
    split_history = []
    current_node = root
    unique_labels = get_unique_labels(root.get_labels())
    while stop_criterion == False:
        unique_values = get_unique_values(current_node.get_elements())
        full_entropy = calculate_entropy(current_node.get_labels())
        max_entropy_id = 0
        max_entropy_value = 0
        for i in range(len(unique_values)):
            split_entropy = calculate_split_candidate_entropy(full_entropy,
                                                                      current_node.get_labels(),
                                                                      current_node.get_elements(),
                                                                      unique_labels,
                                                                      unique_values[i], i)
            if split_entropy > max_entropy_value:
                max_entropy_id = i
                max_entropy_value = split_entropy
        current_node, split_history = split(current_node, unique_values[max_entropy_id], max_entropy_id, split_history)
        new_node = get_current_node(root)
        if new_node != None:
            current_node = new_node
        else:
            stop_criterion = True
    return root, split_history
```

The building methods returns the same variables as in the previous example:

```python
c45_tree, split_history_c45 = build(root)
```

It is worth to save the history.

```python
%store split_history_c45
```

    Stored 'split_history_c45' (list)

```python

```


## Problem Sheet Question 2a

The general form of the population growth differential equation
\begin{equation} y^{'}=ty^3-y, \ \ (0 \leq t \leq 2) \end{equation}
with the initial condition
\begin{equation}y(0)=1.\end{equation}
For N=4
\begin{equation} y(x_1)= 0.5.\end{equation}

### 2-step Adams Bashforth

The 2-step Adams Bashforth difference equation is
\begin{equation}w^{0}_{i+1} = w_{i} + \frac{h}{2}(3f(t_i,w_i)-f(t_{i-1},w_{i-1})) \end{equation}
\begin{equation}w^{0}_{i+1} = w_{i} + \frac{h}{2}(3(t_iw_i^3-w_i)-(t_{i-1}w_{i-1}^3-w_{i-1})) \end{equation}
### 3-step Adams Moulton
\begin{equation}w^{1}_{i+1} = w_{i} + \frac{h}{12}(5f(t_{i+1},w^{0}_{i+1})+8f(t_{i},w_{i})-f(t_{i-1},w_{i-1})) \end{equation}

\begin{equation} w^{1}_{i+1} = w_{i} + \frac{h}{12}(5(t_{i+1}(w^0_{i+1})^3-w^0_{i+1})+8(t_{i}w_{i}^3-w_{i})-(t_{i-1}w_{i-1}^3-w_{i-1})). \end{equation}

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

```python
def myfun_ty(t,y):
    return y*y*y*t-y

#PLOTS
def Adams_Bashforth_Predictor_Corrector(N,IC):

    x_end=2
    x_start=0
    INTITIAL_CONDITION=IC
    h=x_end/(N)
    N=N+2;
    t=np.zeros(N)
    w_predictor=np.zeros(N)
    w_corrector=np.zeros(N)

    Analytic_Solution=np.zeros(N)
    k=0
    w_predictor[0]=INTITIAL_CONDITION
    w_corrector[0]=INTITIAL_CONDITION
    Analytic_Solution[0]=INTITIAL_CONDITION
    t[0]=x_start
    t[1]=x_start+1*h
    t[2]=x_start+2*h
    w_predictor[1]=0.5
    w_corrector[1]=0.5

    for k in range (2,N-1):
        w_predictor[k+1]=w_corrector[k]+h/2.0*(3*myfun_ty(t[k],w_corrector[k])-myfun_ty(t[k-1],w_corrector[k-1]))
        w_corrector[k+1]=w_corrector[k]+h/12.0*(5*myfun_ty(t[k+1],w_predictor[k+1])+8*myfun_ty(t[k],w_corrector[k])-myfun_ty(t[k-1],w_corrector[k-1]))
        t[k+1]=t[k]+h

    fig = plt.figure(figsize=(10,4))
    # --- left hand plot
    ax = fig.add_subplot(1,2,1)
    plt.plot(t,w_predictor,color='red')
    #ax.legend(loc='best')
    plt.title('Predictor h=%s'%(h))

    # --- right hand plot
    ax = fig.add_subplot(1,2,2)
    plt.plot(t,w_corrector,color='blue')
    plt.title('Corrector')

    # --- titled , explanatory text and save
    fig.suptitle(r"$y'=ty^3-y$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
    print('time')
    print(t)
    print('Predictor')
    print(w_predictor)
    print('Corrector')
    print(w_corrector)
```

```python
Adams_Bashforth_Predictor_Corrector(10,1)
```

```python

```

Chapter 04 - Multistep Methods/4_Problem_Sheet/406b_Problem_Sheet.ipynb


```python
import sympy as sym
import matplotlib.pyplot as plt
import stanpy as stp

x = sym.Symbol("x")
l = 4  # m
E = 2.1e8
b, ha, hb = 0.2, 0.3, 0.4  # m
hx = ha + (hb - ha) / l * x  # m
cs = stp.cs(b=b, h=hx)

s = {"E": E, "cs": cs, "q": 10, "l": l, "bc_i": {"w": 0, "M": 0, "H": 0}, "bc_k": {"w": 0, "phi": 0}}

print(cs)
print("I_y(0) = ", cs["I_y"](0))
print("I_y(l) = ", cs["I_y"](l))

offset = 0.2
fig, ax = plt.subplots()
stp.plot_cs(ax, b, hx.subs(x, 0))
stp.plot_cs(ax, b, hx.subs(x, l), dy=0.3)

ax.set_xlim(-offset, b + offset)
ax.set_ylim(-offset, float(hx.subs(x, l)) + offset)
ax.grid(linestyle=":")
ax.axis('equal')

plt.show()

fig, ax = plt.subplots()
stp.plot_system(ax, s, render=True, facecolor="gray", alpha=0.5, render_scale=0.4)
ax.grid(linestyle=":")
ax.axis('equal')

plt.show()
```

```python

```

```python

```

```python

```

```python

```

```python

```


```python
import fractions
import math
import random

import numpy as np
import sympy as sp
import cirq
```

```python
#elements of Z_n
def Z_n(n: int) -> List[int]:
    assert n > 1
    modd = [1]
    for x in range(2, n):
        if math.gcd(x, n) == 1:
            modd.append(x)
    return modd

n = 89
print("The multiplicative group modulo n = is:", Z_n(n))
```

    The multiplicative group modulo n = is: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88]

```python
#Computing the order of an element of Z_n, classsically
def classical(x: int, n: int) -> Optional[int]:
    if x < 2 or x >= n or math.gcd(x, n) > 1:
        raise ValueError(f"Invalid x={x} for modulus n={n}.")

    # Find order.
    p, q = 1, x
    while q != 1:
        q = (x * q) % n
        p += 1
    return p

n = 89
x = 12
p = classical(x, n)
print(p)
```

    8

```python
#computing the order of an element of Z_n, using quantum approach
def quantum(x: int, n: int) -> Optional[int]:
    if x < 2 or n <= x or math.gcd(x, n) > 1:
        raise ValueError(f'Invalid x={x} for modulus n={n}.')
```

```python
"""Functions for factoring from start to finish."""
def find_factor_of_prime_power(n: int) -> Optional[int]:
    for k in range(2, math.floor(math.log2(n)) + 1):
        c = math.pow(n, 1 / k)
        c1 = math.floor(c)
        if c1**k == n:
            return c1
        c2 = math.ceil(c)
        if c2**k == n:
            return c2
    return None

def find_factor(
    n: int,
    order_finder: Callable[[int, int], Optional[int]] = quantum,
    max_attempts: int = 30
) -> Optional[int]:

    if sp.isprime(n):
        print("n is prime!")
        return None

    if n % 2 == 0:
        return 2

    c = find_factor_of_prime_power(n)
    if c is not None:
        return c

    for _ in range(max_attempts):
        x = random.randint(2, n - 1)
        c = math.gcd(x, n)
        if 1 < c < n:
            return c

        # Compute the order r of x modulo n using the order finder.
        r = order_finder(x, n)
        if r is None:
            continue
        if r % 2 != 0:
            continue
        y = x**(r // 2) % n
        assert 1 < y < n
        c = math.gcd(y - 1, n)
        if 1 < c < n:
            return c
    return None
```

```python
# Number to factor: a very huge number
n = 99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999
z = find_factor(n, order_finder=quantum)
y = n // z

print("Factoring n = pq =", n)
print("p =", z)
print("q =", y)
```

    Factoring n = pq = 99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999
    p = 3
    q = 33333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333

```python
"""Check the answer is correct."""
z * y == n
```

    True

```python

```


# 08/05/15  Example of variational inference using Theano (for Markov Random Fields)!

## Written by Andreas Damianou, James Hensman, Srijith Prabhakaran, Varvara Logacheva, Alan Saul, Zhenwen Dai

This is a simple example on using Theano to implement a simple probabilistic model. The greatest advantage of using Theano, is that we don't have to compute the analytic gradients for optimising the model.

This notebook is indended to demonstrate the basic functionality, errors/bugs are likely to be present!! If you find bugs/errors, please contact one of the authors!

## Markov Random Fields

A Markov Random Field (MRF) is an undirected graphical model. We consider nodes in the graph to take values in the set $\{+1, -1\}$. A subset of these nodes are instantiations of the latent variable $\mathbf{x}$, and another subset are instantiations of the observed set $\mathbf{y}$. Both $x_i, y_i$ take values from $\{+1, -1\}$. We can equivalently write $\mathbf{x} \in \Re^{ND\times 1}$ as a matrix $\mathbf{X} \in \Re^{N \times D}$, in order to define neighbouring relationships between elements (e.g. $\mathbf{X}_{2,2}$ has as neighbours $\mathbf{X}_{1,2},\mathbf{X}_{2,1},\mathbf{X}_{2,3}$ and $\mathbf{X}_{3,2}$). Similarly for $\mathbf{y}$.

Sources:

1) Bishop's book (PRML), p. 383

2) MacKay's book (Inf. Theory, Inference and learning algorithms), Variational Inference section (Ising model).

The energy function for the model is written as:
\begin{equation}
E(\mathbf{x}, \mathbf{y}) = h \sum_i x_i - \beta \sum_{\{i,j\}} x_i x_j - \eta \sum_i x_i y_i  \; \; \; \text{(1)}
\end{equation}
where $h,\beta,\eta$ are scalar parameters and $\{i,j\}$ in the second summation refers to nodes that are neighbours.

The joint distribution over $\mathbf{x}, \mathbf{y}$ given the parameters (omitted for clarity), is given from:

\begin{equation}
p( \mathbf{x}, \mathbf{y} ) = \frac{1}{Z} \exp{ \left(-E(\mathbf{x}, \mathbf{y})\right)}, \; \; \; \text{(2)}
\end{equation}
where $Z$ is the normalisation constant (known as partition function) obtained by summing over all possible values of $\mathbf{x}$.

Next, we will implement the energy function in Theano.

## Implementation of the energy function in Theano

```python
# Import Theano
import theano
# Tensor is the most useful module to import for our purposes. Let's make a shorthand
from theano import tensor as T
# Let's import other useful libraries
import numpy as np
import scipy as sp
import Image
import pylab as pb
%pylab inline

pb.ion()
```

    Couldn't import dot_parser, loading of dot files will not be possible.
    Populating the interactive namespace from numpy and matplotlib

    WARNING: pylab import has clobbered these variables: ['linalg', 'draw_if_interactive', 'random', 'power', 'info', 'fft']
    `%matplotlib` prevents importing * from pylab and numpy

Tip: if running Theano from the terminal through ipython, it might be helpful for debugging to open ipython using:

THEANO_FLAGS='optimize=None' ipython

Next we define the actual theano function. The expression to be evaluated (the energy function) is not trivially implemented in one line of code, therefore we might need intermediate variables. However, in the end all we're interested in, is the final symbolic expression for the energy function, and not the intermediate symbolic variables. Therefore, we might create the symbolic function within a python function:

```python
def build_theano_energy_func():
    """
    All symbolic objects need to be declared with their type before being used.
    Here, imatrix is a matrix of integers, d of doubles etc
    """
    X = T.imatrix()
    Y = T.imatrix()
    b = T.dscalar()
    h = T.dscalar()
    e = T.dscalar()

    # Implement the second part of the energy function, requiring summation
    # over neighbours. A trick (thanks to J. Hensman!) is to do it by summing
    # the two "shifted" matrices by columns and rows to get partial sums, which
    # are then summed together
    sum_cols  = T.sum(X[:,:-1]*X[:,1:])
    sum_rows  = T.sum(X[:-1,:]*X[1:,:])
    sum_XX    = sum_cols + sum_rows

    # The last term of the energy function
    sumXY = T.sum(X*Y)

    # The first term plus everything put together
    energy_function = h*T.sum(X) - b*sum_XX - e*sumXY

    # The object to be returned is a theano function. Inputs to this function
    # are given as a list, and those are the latents, the observed and the
    # scalar paramters
    return theano.function([X,Y,b,h,e],energy_function,allow_input_downcast=True)

```

Let's test it:

```python
# Create toy data
N = 10
D = 5

# Random values from the choice of -1,1
Yval  = np.random.choice([-1,1], (N,D))
Xval  = np.random.choice([-1,1], (N,D))
RSval = np.random.randn(N,D)
b_init = 1
h_init = 1
e_init = 1
```

```python
f  = build_theano_energy_func()
f
```

    <theano.compile.function_module.Function at 0x7f01f5921e10>

Successful compilation is a good sign, but many errors actually occur during runtime. Let's call the function then:

```python
f(Xval, Yval, b_init, h_init, e_init)
```

    array(-7.0)

Seems to work!

## Mean-field variational inference

But to train the model, what we want is to not simply evaluate the energy function, but to find the posterior of the parameters given the data. The full posterior is $p(\mathbf{x}, \beta,h,\eta | \mathbf{y})$. However, to evaluate it we'd need to find the normalising constant $Z$. This is a very difficult problem, because evaluating $Z$ requires marginalisation over $\mathbf{x}$, i.e. averaging over all possible states for that variable. This cannot be achieved analytically, but instead we can resort to mean-field variational inference. For a certain type of approximating variational distributions, we can approximate the posterior for $\mathbf{x}$. However, to approximate the full posterior is not easy, because the partition function is a constant with respect to $\mathbf{x}$ (since it's marginalised out), but still includes $\beta, h, \eta$.

Therefore, we will assume that these parameters are given and we will only focus on finding the approximate posterior $p(\mathbf{x} | \beta, h, \eta, \mathbf{y})$. We will also omit the dependence on $\beta, h, \eta$ further on, for clarity.

For a variational distribution $q(\mathbf{x})$, we want to find:
\begin{align}
  & \text{KL}(q(\mathbf{x}) || p(\mathbf{x} | \mathbf{y}) \\
= & \text{KL}\left( q(\mathbf{x}) || \frac{p(\mathbf{x} , \mathbf{y})}{p(\mathbf{y})} \right) \\
= & \langle \log q(\mathbf{x}) - \log p(\mathbf{x},\mathbf{y}) \rangle_{q(\mathbf{x})} + \log p(\mathbf{y}) \\
& \langle \log q(\mathbf{x}) + E(\mathbf{x},\mathbf{y}) \rangle_{q(\mathbf{x})} + \log Z +\log p(\mathbf{y})
\end{align}

To optimise this expression w.r.t. $\mathbf{x}$, we drop the two last terms that do not depend on it, and obtain our final objective:
\begin{equation}
\mathcal{F} = \langle \log q(\mathbf{x}) \rangle_{q(\mathbf{x})} + \langle E(\mathbf{x},\mathbf{y}) \rangle_{q(\mathbf{x})} \; \; \; \text{(3)}
\end{equation}

For this expression to be tractable, we select a variational distribution as a product of Bernoulli's:
\begin{equation}
q(\mathbf{x}) = \prod_i q(x_i) \; \; \; \text{(4)}
\end{equation}
where:

$q(x_i) = \rho_i,  \ \; \; \; \; \; \; \; $  if $x_i=+1$

$q(x_i) = (1-\rho_i)$, if $x_i=-1$

Given this distribution, the first term of $\mathcal{F}$ is simply:
\begin{equation}
 \langle \log q(\mathbf{x}) \rangle_{q(\mathbf{x})} = \rho_i \log \rho_i + (1-\rho_i) \log(1-\rho_i) \; \; \; \text{(5)}
\end{equation}

The second term is found by replacing the energy function $E(\mathbf{x},\mathbf{y})$ from eq. (1), and itself gives us three terms:

\begin{align}
\langle E(\mathbf{x},\mathbf{y}) \rangle_{q(\mathbf{x})} &= \langle h \sum_i x_i + \beta \sum_{\{i,j\}} x_i x_j + \eta \sum_i x_i y_i \rangle_{q(\mathbf{x})}  \\
&= h \sum_i (2\rho_i-1 ) + \beta \sum_{\{i,j\}} \left( (2\rho_i-1)(2\rho_j-1)\right) + \eta \sum_i (2\rho_i)y_i \; \; \; \text{(6)}
\end{align}

Replacing eq. 5,6) into (3), we get the final bound, which we will write with two indexes to facilitate translation into python. Specifically, the $c$ and $r$ are used to denote any consistent way of summing over all columns and rows of a matrix $\mathbf{R}$ which contains the variational parameters. The matrix notation comes from the fact that $\mathbf{R}$ can be stored as an $N \times D$ matrix instead of a long $ND \times 1$ vector. Further, $\{ i,j \}$ denotes neighbouring elements, as before. Therefore, we have:

\begin{align}
\mathcal{F} &= \sum_{c,r} \left[ \rho_{c,r} \log \rho_{c,r} + (1-\rho_{c,r}) \log( 1-\rho_{c,r}) + (h+\eta y_{c,r})(2 \rho_{c,r}-1) \right] \\
& + \beta \sum_{\{i,j\}} (2\rho_i-1)(2\rho_j-1) \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text{(7)}
\end{align}

And here's how to implement this in Theano:

```python
def build_expected_energy_func():
    # Declare all the variables to be used.

    # N and D are given as parameters, so that we know how to reshape RV
    N  = T.iscalar()
    D  = T.iscalar()
    # RV is the vector of \rho's, serialised with shape NDx1
    RV = T.dvector()
    Y  = T.imatrix()
    b  = T.dscalar()
    h  = T.dscalar()
    e  = T.dscalar()

    # RS is RV reshaped into a matrix (which makes the neighbouring connections clear)
    RS = T.reshape(RV, (N,D))

    # R must have entries between 0 and 1. This can be forced by having RS to be free
    # parameters and push them through a sigmoid.
    R = T.nnet.sigmoid(RS)

    # This computation is for eq. (7), using the same trick as for the energy function.
    sum_ij = T.sum(R*T.log(R) + (1-R) * T.log(1-R) + (h + e*Y)*(2*R-1))
    sum_ij_neighbor_cols = T.sum((2*R[:,:-1]-1)*(2*R[:,1:]-1))
    sum_ij_neighbor_rows = T.sum((2*R[:-1,:]-1)*(2*R[1:,:]-1))
    sum_ij_neighbor = sum_ij_neighbor_rows + sum_ij_neighbor_cols

    expected_energy_function = sum_ij + b*sum_ij_neighbor

    # first part are inputs to the built function. Then a list of outputs. Our function
    # will return as outputs the objective and the gradients.
    return theano.function([RV,Y,b,h,e, N, D],[expected_energy_function, theano.grad(expected_energy_function,RV)],allow_input_downcast=True)

```

As can be seen, once we implement the objective, then the gradient can be simply returned by calling one more function! Another trick used, is to represent the parameters separately in the optimiser and separately internally. Specifically, the optimiser sees a vector of real values, and the programme then transforms them to a matrix of elements in [0,1], firstly through a reshaping and then through pushing through the sigmoid.\

Test the syntax:

```python
Ef = build_expected_energy_func()
```

Test running it:

```python
Ef(RSval.flatten(), Yval, b_init, h_init, e_init, N, D)

options = dict()
options['disp']=True

res=sp.optimize.minimize(Ef, RSval.flatten(), (Yval, b_init, h_init, e_init, N, D), jac=True, options=options)#, bounds=optBounds)

optimised_params = res['x']
```

    Optimization terminated successfully.
             Current function value: -86.039966
             Iterations: 412
             Function evaluations: 455
             Gradient evaluations: 455

It works! Now let's move to an application.

## Image denoising

We'll attempt to use the MRF as an image denoiser, as in Bishop's book. To begin with, we'll load the image of the the incredibly complex flag of Japan:

```python
import urllib, cStringIO
URL='http://upload.wikimedia.org/wikipedia/en/thumb/9/9e/Flag_of_Japan.svg/1280px-Flag_of_Japan.svg.png'
file = cStringIO.StringIO(urllib.urlopen(URL).read())
img = Image.open(file).convert('L')
```

We can optionally reshape the image. We will also "binarize" it.

```python
im_reshape=False

if im_reshape:
    perc=.25
    im = np.asarray(img)
    N,D = im.shape
    img = img.resize((int(np.ceil(D*perc)),int(np.ceil(N*perc))),Image.ANTIALIAS)
    im = np.asarray(img)
    N,D = im.shape
else:
    im = np.asarray(img)
    N,D = im.shape

# Normalise pixel values between 0 and 1
im = im / 255.

# Shift spectrum from 0,1 to -1, 1
im = np.where(im < 0.4, -1, 1)

pb.imshow(im,cmap=pb.cm.gray)
pb.title('Original image')
pb.show()

```

We'll add noise to the picture (20% noise, feel free to change the parameters).
Then, we'll initlise the R parameters. One way to do it, is based on the data. There are other suggested ways (commented out), feel free to explore.

The parameters are hardcoded and suboptimal, you can try different values too.

```python
# Noise is added randomly at 20% of pixels.
noise = np.random.rand(N,D)
noise = np.where(noise > 0.8, -1, 1)

Yval = im * noise

# Initialise the R parameters.
RSval = Yval #np.ones_like(Yval)*0.5#Yval # np.random.randn(N,D)  # Init

b_init = -50
h_init = 0
e_init = 2.1

Ef = build_expected_energy_func()

Ef(RSval.flatten(), Yval, b_init, h_init, e_init, N, D)

options = dict()
options['disp']=True
options['maxiter']=300

# This is a bit buggy, sometimes the output is displayed in the terminal instead of inline here...
res=sp.optimize.minimize(Ef, RSval.flatten(), (Yval, b_init, h_init, e_init, N, D), jac=True, method='L-BFGS-B', options=options )

```

Now display the results.

```python
fig = pb.figure(figsize=(15,12))
fig1=fig.add_subplot(121)
fig1.matshow(Yval,cmap=pb.cm.gray)
pb.title('Noisy image')

RSVopt = res['x']
A = T.dmatrix()
sig = theano.function([A], T.nnet.sigmoid(A))
Ropt = sig(RSVopt.reshape(N,D))

fig2=fig.add_subplot(122)
fig2.matshow(Ropt, cmap=pb.cm.gray)
pb.title('Cleaned image')

```

```python

```

```python

```


# Hyperbolic Functions and Their Derivatives

$\newcommand{\sech}{sech}$
$\newcommand{\csch}{csch}$

Hyperbolic functions are analogs of trigonometric functions in that hyperbolic functions are related to hyperbola while trigonometric functions are related to the circle. In fact, the hyperbolic functions are named very similarly to their corresponding trigonometric functions. For example, $\sinh{}$ is the hyperbolic sine and $\cosh{}$ is the hyperbolic cosine.

The following is a list of definitions of the hyperbolic functions:

$$ \sinh{x} = \frac{e^x - e^{-x}}{2} \qquad \cosh{x} = \frac{e^x + e^{-x}}{2} \qquad \tanh{x} = \frac{\sinh{x}}{\cosh{x}} $$

$$ \sech{x} = \frac{1}{\cosh{x}} \qquad \csch{x} = \frac{1}{\sinh{x}} \qquad \coth{x} = \frac{\cosh{x}}{\sinh{x}} $$

### Hyperbolic Identities

The following hyperbolic identities are similar to trigonometric identities, of which the most common are listed here:

$$ \sinh{-x} = -\sinh{x} \qquad \cosh{-x} = \cosh{x} $$

$$ \cosh^2{x} - \sinh^2{x} = 1 \qquad 1 - \tanh^2{x} = \sech^2{x} $$

$$ \sinh{x + y} = \sinh{x} \cosh{y} + \cosh{x} + \sinh{y} \qquad \cosh{x + y} = \cosh{x} \cosh{y} + \sinh{x} \sinh{y} $$

### Derivatives of Hyperbolic Functions

As we saw before with the hyperbolic identities, the derivatives of hyperbolic functions are also similar to the derivatives of trigonometric functions.

$$ \frac{d}{dx} \sinh{x} = \cosh{x} \qquad \frac{d}{dx} \cosh{x} = \sinh{x} $$

$$ \frac{d}{dx} \tanh{x} = \sech^2{x} \qquad \frac{d}{dx} \csch{x} = -\csch{x} \coth{x} $$

$$ \frac{d}{dx} \sech{x} = -\sech{x} \tanh{x} \qquad \frac{d}{dx} \coth{x} = -\csch^2{x} $$

### Inverse Hyperbolic Functions

```python

```

```python

```

```python
from sympy import symbols, diff, simplify, sin, cos, log, tan, sqrt, init_printing
from mpmath import *

init_printing()
x = symbols('x')
y = symbols('y')
```

```python

```

## References

Stewart, J. (2007). Essential calculus: Early transcendentals. Belmont, CA: Thomson Higher Education.

Weisstein, Eric W. "Hyperbolic Functions." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/HyperbolicFunctions.html


## Normalized Differential Cross Section

In 1928 Klein and Nishina  applied Dirac’s relativistic theory of
the electron to the Compton effect to obtain improved cross sections. The differential
cross section for photon scattering at angle $\phi$ per unit solid angle and per electron,
corresponding to Equation:
\begin{equation}
    \frac{d{_e\sigma}}{d\Omega_\phi}=\frac{r_0^2}{2}       (1+ \text{cos}^2\phi)
    \label{TH}
\end{equation}
from Thomson’s theory, may be written in the form
\begin{equation}
    \frac{d{_e\sigma}}{d\Omega_\phi}=\frac{r_0^2}{2}\bigg       (\frac{h\nu'}{h\nu}\bigg)^2\bigg       (\frac{h\nu}{h\nu'}+\frac{h\nu'}{h\nu}-\text{sin}^2\phi\bigg)
    \label{TH1}
\end{equation}
where, $ h\nu'=h\nu\frac{1}{1+\varepsilon (1+\text{cos}\phi)}$, $\varepsilon = \frac{h\nu}{m_ec^2}$ and  $m_ec^2$=the energy equivalence of the electron rest mass(511 keV);

r$_0$ = classical electron radius(= 2.8 $\times$ 10$^{-15}$m)

h$\nu$ = incident photon energy

h$\nu'$ = Compton scattered photon energy
and
$\phi$ = polar scattering angle in the photon coordinate
system

The differential solid angle is represented
by $d\Omega = 2\pi \text{sin}\phi d\phi$  assuming all azimuthal
angles are equally probable. This crosssection
can be expressed as cross-section
based on polar angle        ($\phi$) given by the relation:
$\frac{d{_e\sigma}}{d\phi}=\pi r_0^2\bigg(\frac{h\nu'}{h\nu}\bigg)^2\bigg(\frac{h\nu}{h\nu'}+\frac{h\nu'}{h\nu}-\text{sin}^2\phi\bigg)\text{sin}\phi$

If h$\nu$ $\approx$ h$\nu'$
$\frac{d{_e\sigma}}{d\Omega_\phi}=\frac{r_0^2}{2}(2-\text{sin}^2\phi)=\frac{r_0^2}{2}(1+\text{cos}^2\phi)$

The total K-N cross section per electron ($_e\sigma$):
$$
_e\sigma=2\pi \int_{\phi=0}^\pi  \frac{d{_e\sigma}}{d\Omega_\phi} \text{sin}\phi  \text{d}\phi
    \label{KN}
$$

$$
=\pi r_0^2 \int_{0}^\pi \bigg(\frac{h\nu'}{h\nu}\bigg)^2\bigg(\frac{h\nu}{h\nu'}+\frac{h\nu'}{h\nu}-\text{sin}^2\phi\bigg)\text{sin}\phi  \text{d}\phi$$
\begin{equation}
_e\sigma    =2\pi r_0^2 \bigg[\frac{1+\alpha}{\alpha^2} \bigg       (\frac{2       (1+\alpha)}{1+2\alpha}-\frac{ln       (1+2\alpha)}{\alpha}\bigg)+\frac{ln       (1+2\alpha)}{2\alpha}-\frac{1+3\alpha}{       (1+2\alpha)^2}\bigg]
    \label{KN1}
\end{equation}
Where $\alpha=\varepsilon=\frac{h\nu}{m_ec^2}$, in which h$\nu$ is to be expressed in MeV and m$_e$c$^2$ = 0.511 MeV.
The relation between  the electron scattering crosssection and the attenuation coefficient       ($\mu$) is $\mu$=n$_{v}$$\times$$\sigma_e$, where  n$_{v}$ is number of electron per unit volume.
## References:
### Attix, F. H. (2008). Introduction to radiological physics and radiation dosimetry. John Wiley & Sons.
### https://chem.libretexts.org/Courses/Sacramento_City_College/SCC%3A_CHEM_330_-_Adventures_in_Chemistry_(Alviar-Agnew)/11%3A_Nuclear_Chemistry/11.02%3A_Nuclear_Equations

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
import math
from statistics import stdev
phi=np.arange(180)
Einit=0.8 #Energy in MeV; h\nu

Escat=Einit/(1+Einit/0.511*(1-np.cos(phi*math.pi/180))) # Scattering Energy

dsigma_dphi=math.pi*(2.818e-15)**2*(Escat/Einit)**2*(Einit/Escat+Escat/Einit-np.sin(phi*math.pi/180)**2)*np.sin(phi*math.pi/180)

plt.plot(phi/180,dsigma_dphi/max(dsigma_dphi))
plt.fill_between(phi/180, dsigma_dphi/max(dsigma_dphi), color= 'red', alpha=0.2,hatch='/')
plt.xlabel(r'$\phi$, normalized')
plt.ylabel('$d\sigma$/d\u03A6, normalized')
plt.title('Compton Differential Cross-Section, 800KeV, photons')
plt.xlim(0,1)
plt.ylim(0,1)
plt.show()
print(max(dsigma_dphi))
```

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
import math
from statistics import stdev
phi=np.arange(180)
Einit=8 #Energy in MeV

Escat=Einit/(1+Einit/0.511*(1-np.cos(phi*math.pi/180))) # Scattering Energy

dsigma_dphi=math.pi*(2.818e-13)**2*(Escat/Einit)**2*(Einit/Escat+Escat/Einit-np.sin(phi*math.pi/180)**2)*np.sin(phi*math.pi/180)

plt.plot(phi/180,dsigma_dphi/max(dsigma_dphi))
plt.fill_between(phi/180, dsigma_dphi/max(dsigma_dphi), color= 'yellow', alpha=0.2,hatch='/')
plt.xlabel('\u03A6, normalized') # Unicode Character of Symbol
plt.ylabel('$d\sigma$/d\u03A6, normalized')
plt.title('Compton Differential Cross-Section, 8MeV, photons')
plt.xlim(0,1)
plt.ylim(0,1)
plt.show()
print(max(dsigma_dphi))
```

```python

```

Normalized Differential Cross Section.ipynb