Table of Contents

1  Objectives

2  What Has Calculus Done For You

3  Differentiation

3.1  Common Derivatives

3.2  sympy

4  Integration

4.1  Common Integrals

4.2  sympy

5  The Chain Rule

5.1  Exercise:

6  Partial Differentiation

```python
# conda install -c anaconda sympy
```

> NOTE: `sympy` is not a package that comes included in the `learn-env`. You can install it by running the command below though here in the notebook or in your terminal (with `learn-env` activated). You might have to restart the kernel after installation.

```python
import numpy as np
from matplotlib import pyplot as plt
from sympy import *
from sympy.abc import x, y
from scipy import stats
```

# Objectives

- Perform simple derivatives and indefinite integrals
- Use the Chain Rule to construct derivatives of complex functions
- Construct partial derivatives for functions of multiple variables

# What Has Calculus Done For You

We have already had occasion to use calculus in a few places. Calculus shows us:

- that the mean of a group of numbers is the number $n$ that minimizes the sum of squared differences $\Sigma(p-n)^2$ for each number $p$ in the group;
- that the median of a group of numbers is the number $n$ that minimizes the sum of absolute differences $\Sigma|p-n|$ for each number $p$ in the group;
- how to find the coefficients for a linear regression optimization problem.

The two main tools of calculus are **differentiation** and **integration**. For functions of one dimension:

- Differentiation gives us the *slope* of the function at any point.
- Integration gives us the *area under the curve* of the function between any two points.

Surprisingly, these two operations turn out to be inverses of one another in the sense that the derivative of the integral of a given function takes us back to the initial function:

$\frac{d}{dx}[\int^x_a f(t) dt] = f(x)$. This is known as the First Fundamental Theorem of Calculus.

# Differentiation

To find the slope of a function *at a point*, we imagine calculating the slope of the function between two points, and then gradually bringing those two points together. Consider the slope of the function $y=x^2$ at the point $x=100$.

We'll calculate the slope of the parabola between $x_1=100$ and $x_2=1000$, and then slowly move $x_2$ close to $x_1$:

```python
10**(7/3)
```

    215.44346900318845

```python
X = np.logspace(3, 2, 11)

fig, ax = plt.subplots()
ax.plot(X, X**2, lw=5, c='k', alpha=0.4)
for x_ in X[:-1]:
    ax.plot([100, x_], [10000, x_**2], 'r-.')
```

```python
# This will show how the slope approaches the value
# of the derivative

slopes = []
for x_ in X[:-1]:
    slopes.append((x_**2-10000) / (x_-100))

fig, ax = plt.subplots()
ax.plot(X[:-1], slopes, label='approximation')
ax.scatter(X[-1], 200, label='value of derivative at $x=100$', c='r')
ax.set_xlabel('X')
ax.set_ylabel('slope')
ax.set_title('slope between x=100 and X')
plt.legend();
```

## Common Derivatives

Here is a list of rules for some common derivative patterns.

```python
#first twwo and third from the bottom good to remember
#2nd from bottom also probably see
#bottom one is quotient rule, good also
```

$\large\frac{d}{dx}[cf(x)] = cf'(x)$

$\rightarrow$ Example: $\frac{d}{dx}[2x] = 2\frac{d}{dx}[x] = (2)(1) = 2$

$\large\frac{d}{dx}[x^n] = nx^{n - 1}$

$\rightarrow$ Example: $\frac{d}{dx}[x^4] = 4x^3$

$\large\frac{d}{dx}[sin(x)] = cos(x)$

$\large\frac{d}{dx}[cos(x)] = -sin(x)$

$\large\frac{d}{dx}[a^x] = a^xln(a)$

$\large\frac{d}{dx}[log_bx)] = \frac{1}{xln(b)}$

$\rightarrow$ Example: $\frac{d}{dx}[ln(x)] = \frac{1}{xln(e)} = \frac{1}{x}$

$\large\frac{d}{dx}[f(x) + g(x)] = f'(x) + g'(x)$

$\rightarrow$ Example: $\frac{d}{dx}[2x + 4x^2] = 2 + 8x$

$\large\frac{d}{dx}[f(x)g(x)] = f(x)g'(x) + g(x)f'(x)$

$\rightarrow$ Example: $\frac{d}{dx}[(x+1)(x-1)] = (x+1)(1) + (x-1)(1) = 2x$

$\large\frac{d}{dx}\left[\frac{f(x)}{g(x)}\right] = \frac{g(x)f'(x) - f(x)g'(x))}{(g(x))^2}$

$\rightarrow$ Example: $\frac{d}{dx}\left[\frac{x+1}{x-1}\right] = \frac{(x-1)(1) - (x+1)(1)}{(x-1)^2} = -\frac{2}{(x-1)^2}$

## `sympy`

The `sympy` package can be helpful:

```python
diff(sin(x), x)
```

$\displaystyle \cos{\left(x \right)}$

```python
diff(exp(2*x))
```

$\displaystyle 2 e^{2 x}$

# Integration

Integration is how we calculate the area under a curve. If the curve is a probability density function, then the area under this curve will be equal to 1:

```python
X = np.linspace(0, 5, 51)
for pt in X:
    print(stats.norm.cdf(pt))
```

    0.5
    0.539827837277029
    0.579259709439103
    0.6179114221889527
    0.6554217416103242
    0.6914624612740131
    0.7257468822499265
    0.758036347776927
    0.7881446014166034
    0.8159398746532405
    0.8413447460685429
    0.8643339390536173
    0.8849303297782918
    0.9031995154143897
    0.9192433407662289
    0.9331927987311419
    0.945200708300442
    0.955434537241457
    0.9640696808870742
    0.9712834401839983
    0.9772498680518208
    0.9821355794371834
    0.9860965524865014
    0.9892758899783242
    0.9918024640754038
    0.9937903346742238
    0.9953388119762813
    0.9965330261969594
    0.997444869669572
    0.998134186699616
    0.9986501019683699
    0.9990323967867817
    0.9993128620620841
    0.9995165758576162
    0.9996630707343231
    0.9997673709209645
    0.9998408914098424
    0.9998922002665226
    0.9999276519560749
    0.9999519036559824
    0.9999683287581669
    0.9999793424930875
    0.9999866542509841
    0.999991460094529
    0.9999945874560923
    0.9999966023268753
    0.9999978875452975
    0.9999986991925461
    0.999999206671848
    0.9999995208167234
    0.9999997133484281

How do you calculate the area of a shape with a curvy side? Imagine approximating the shape with rectangles, and then imagine making those rectangles narrower and narrower.

Again, let's work with the parabola $y=x^2$ between $x=100$ and $x=1000$:

```python
# This will show how we imagine ever narrower rectangles
# under the curve to approximate the area underneath it.

spacing = np.arange(3, 13)

X = [np.linspace(100, 1000, step) for step in spacing]
X_curve = np.linspace(100, 1000, 10000)

fig, ax = plt.subplots(10, figsize=(10, 30))

for num in spacing:
    ax[num-3].plot(X_curve, X_curve**2)
    for j in range(1, len(X[num-3])-1):
        ax[num-3].hlines(X[num-3][j]**2, X[num-3][j], X[num-3][j+1])
        ax[num-3].vlines(X[num-3][j], 0, X[num-3][j]**2)
    ax[num-3].set_xlabel(f'Area = {900/(num-1) * sum(X[num-3][1:-1]**2)}\n\
    For a=100, b=1000, $\int^b_ax^2=333000000$')
plt.tight_layout()
```

```python
# This will show the area of the rectangles as the number
# of rectangles increases.

spacing_longer = np.arange(3, 100)
X_longer = [np.linspace(100, 1000, step) for step in spacing_longer]

areas = [900 / (num-1) * sum(X_longer[num-3][1:-1]**2) for num in spacing_longer]

fig, ax = plt.subplots()

ax.hlines(333000000, 3, 99, label='333000000', color='r')
ax.plot(spacing_longer, areas, label='approximation')
ax.set_title('Area as a function of number of rectangles')
ax.set_xlabel('Number of rectangles')
ax.set_ylabel('Area')
plt.legend();
```

## Common Integrals

$\large\int cf(x)dx = c\int f(x)dx$

$\large\int x^ndx = \frac{x^{n+1}}{n+1}$

$\large\int sin(x)dx = -cos(x)$

$\large\int cos(x)dx = sin(x)$

$\large\int a^xdx = \frac{a^x}{ln(a)}$

$\large\int (f(x)dx+g(x))dx = \int f(x)dx + \int g(x)dx$

## `sympy`

```python
integrate(cos(x), x)
```

$\displaystyle \sin{\left(x \right)}$

```python
integrate(exp(2*x), x)
```

$\displaystyle \frac{e^{2 x}}{2}$

# The Chain Rule

$\large\frac{d}{dx}[f(g(x))] = f'(g(x))g'(x)$

That is: The derivative of a *composition* of functions is: the derivative of the first applied to the second, multiplied by the derivative of the second.

So if we know e.g. that $\frac{d}{dx}[e^x] = e^x$ and $\frac{d}{dx}[x^2] = 2x$, then we can use the Chain Rule to calculate $\frac{d}{dx}[e^{x^2}]$. We set $f(x) = e^x$ and $g(x) = x^2$, so the derivative must be:

$\large\frac{d}{dx}[e^{x^2}] = (e^{x^2})(2x) = 2xe^{x^2}$.

## Exercise:

Calculate the derivatives for the following compositions:

1. $\frac{d}{dx}[sin(4x)]$

     Answer

    $f(x) = sin(x)$

    $g(x) = 4x$

    So the derivative will be: $cos(4x)*4 = 4cos(4x)$

2. $\frac{d}{dx}[e^{sin(x)}]$

     Answer

    $f(x) = e^x$

    $g(x) = sin(x)$

    So the derivative will be: $e^{sin(x)}*cos(x) = cos(x)e^{sin(x)}$

# Partial Differentiation

Partial differentiation is required for functions of multiple variables. If e.g. I have some function $h = h(a, b)$, then I can consider how $h$ changes with respect to $a$ (while keeping $b$ constant)––that's $\frac{\partial h}{\partial a}$, and I can consider how $h$ changes with respect to $b$ (while keeping $a$ constant)––that's $\frac{\partial h}{\partial b}$. And so the rule is simple enough: If I'm differentiating my function with respect to some variable, I'll **treat all other variables as constants**.

Consider the following function:

$\large\xi(x, y, z) = x^2y^5z^3 - ze^{cos(xy)} + (yz)^3$;

for some parameters $x$, $y$, and $z$.

What are the partial derivatives of this function?

$\large\frac{\partial\xi}{\partial x} = ?$

        Check

    $2xy^5z^3 + yze^{cos(xy)}sin(xy)$

$\large\frac{\partial\xi}{\partial y} = ?$

        Check

    $5x^2y^4z^3 + xze^{cos(xy)}sin(xy) + 3y^2z^3$

$\large\frac{\partial\xi}{\partial z} = ?$

        Check

    $3x^2y^5z^2 - e^{cos(xy)} + 3y^3z^2$

```python
#cost is like RMSE
#gradient descent is like one big game of marco polo.
#say wanted price from sqft_living
#is each value of sqft_living randint(dollars)
    #then try each of these and call predict. What's our RMSE if we do this?
    #instead of randomly trying randomly, you get feedback from model. Partial differentiation.
    #you throw it in, 'marco!' model gives you an answer 'polo!'.
        #tells you which direction you should step in.
    #the idea is that you toss in a value, then get feedback and figure out which direction to move it.
    #the way you know this is by taking a partial derivative of the cost function

    #when you take a graddient there is a value that tells you whether to take a large or small step
    #lot of trial/feedback and repeat. update it, don't goi too fast/too far. little by little get closer
    #either way, you should get approximately the same answer.

#why one over the other?
    #they take different amounts of time to run

#linear regression is the model, there's different ways to solve it
```


Solution to: [Day 1: Quartiles](https://www.hackerrank.com/challenges/s10-quartiles)

Table of Contents

- Table of Contents
    - Wiki definition:
    - Dataset ranges
        - Quartile 1
        - Quartile 2
        - Quartile 3
- Solution
    - Process:
    - Implementation
    - Method NOTE:
    - Imports
    - Inputs
    - Median
    - Quantile Data
    - Format
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

## Wiki definition:
A quartile is a type of quantile which divides the number of data points into four more or less equal parts, or quarters.

- The first quartile (Q1) is defined as the middle number between the smallest number and the median of the data set. It is also known as the lower quartile or the 25th empirical quartile and it marks where 25% of the data is below or to the left of it (if data is ordered on a timeline from smallest to largest).
- The second quartile (Q2) is the median of a data set and 50% of the data lies below this point.
- The third quartile (Q3) is the middle value between the median and the highest value of the data set. It is also known as the upper quartile or the 75th empirical quartile and 75% of the data lies below this point.

Due to the fact that the data needs to be ordered from smallest to largest to compute quartiles, quartiles are a form of Order statistic.

## Dataset ranges
Because this problem asks to find the median for Q1, Q2, and Q3, we must establish the data range for each quartile.

### Quartile 1

\begin{equation}
\large
\mathrm{Q1} = \frac{\text{Q2} - \text{Min}}{2}
\end{equation}

### Quartile 2
\begin{equation}
\large
\mathrm{Q2} = \frac{\text{Max} - \text{Min}}{2}
\end{equation}

### Quartile 3
\begin{equation}
\large
\mathrm{Q3} = \frac{\text{Max} - \text{Q2}}{2}
\end{equation}

# Solution

## Process:
As an ordered statistic, we can derive Q1, Q2, and Q3 via finding the median of different sets of data.

## Implementation
To implement, I will create a median function and find Q2 (the median) of the data set.
Then, I will find the relevant data sets for Q1 and Q3, and calculate the median for those data sets as well.

## Method

*Notes:*
- There is no universal agreement on selecting quartile values for the discrete distributions.
- This solution is NOT inclusive, and thus does not include the median in the data used to calculate Q1 and Q3.

## Imports

```python
from typing import List, Tuple, Union
```

## Inputs

```python
def get_input() -> Tuple[int, List[int]]:
	"""Returns input for n: number of integers and nums: list of integers."""
	n = int(input())
	nums = sorted([int(val) for val in input().split()])
	assert n == len(nums)
	return (n, nums)
```

## Median

```python
def calc_median(nums: list) -> Union[int, float]:
	"""Returns the median value from a list of nums with length n."""
	mid, remainder = divmod(len(nums), 2)
	if remainder:
		return nums[mid]

	median = sum(nums[mid-1: mid+1]) / 2
	return format_int(median)
```

## Quantile Data

```python
def get_q_data(nums: list, quartile: str) -> List[int]:
	"""Returns list of data to calculate relevant quantile (Q1, Q3).

	quartile argument used to determine data boundary index for Q1 or Q3.
	"""
	assert quartile in (1, 3)
	n = len(nums)
	bound = 0 if quartile == 1 else n	# calculate data index boundary

	mid, remainder = divmod(n, 2)
	if remainder:
		mid = mid + 1 if (quartile == 3) else mid	# exclude middle data point in odd number solutions

	b1, b2 = sorted([bound, mid])
	return nums[b1:b2]
```

## Format

```python
def format_int(num: Union[float, int]) -> Union[float, int]:
	"""Formats decimal integer to integer else returns float"""
	if num.is_integer():
		return int(num)
	return num
```

## Main

```python
def main():
	## Input
	n, nums = get_input()

	## Q2
	q2 = calc_median(nums)

	## Get Q1&3 data
	q1_data = get_q_data(nums, quartile=1)
	q3_data = get_q_data(nums, quartile=3)

	## Calc Q1&3
	q1 = calc_median(q1_data)
	q3 = calc_median(q3_data)

	## Print
	quantiles = [q1, q2, q3]
	print(*quantiles, sep = '\n')
```

```python
if __name__ == "__main__":
	main()
```

    9
    3 7 8 5 12 14 21 13 18
    6
    12
    16


# Exercise session nº 4
---
# Orientation of Turing Patterns by Morphogen Gradients and Tissue Anisotropies

__*Sacha Ichbiah, 14/02/22, ENS Paris*__

This subject is extracted from :
> Tom W. Hiscock and Sean G. Megason., *Orientation of Turing-like Patterns by Morphogen Gradients and Tissue Anisotropies*, Cell Systems, 2015.
> https://doi.org/10.1016/j.cels.2015.12.001

> https://ipython-books.github.io/124-simulating-a-partial-differential-equation-reaction-diffusion-systems-and-turing-patterns/

__Periodic patterns__ are found in a wide variety of different organisms, forming across a large range of time and length scales. A number of different models have been proposed to explain the formation of these periodic patterns, most notably Turing’s __reaction-diffusion__ model in which periodicity is generated at the molecular scale. The canonical reaction-diffusion model involves two diffusing molecules: a __rapidly diffusing inhibitor__ molecule and a __slowly diffusing activator__ molecule. Provided that the activator stimulates production of both itself and its inhibitor sufficiently strongly, and that the inhibitor diffuses sufficiently more rapidly than the activator, periodic patterns can spontaneously emerge from an initially homogeneous pattern (termed a __Turing instability__).

The property that forms the focus of this work is __pattern orientation__. Stripes in animal pigment patterns often have a __stereotyped__ direction; e.g., wild-type zebrafish have __stripes parallel__ to their long axis during normal development. However, during aberrant patterning, for example, when the pattern regenerates after damage, or when the pattern proceeds in the absence of some organizing signal, the tight regulation of pattern orientaion is impaired, and we can observe __labyrinthine stripes__ on his body. This phenotype disruption shows that the orientation of the pattern requires to be __regulated__.

This study is based on a single variable description of a generic Turing instability: the __Swift-Hohenberg__ equation. With this equation, the authors find three different ways to orient reliably the stripes of the pattern. During this session, we will study three of them : the morphogen gradient, the tissue anisotropy, and growth.

---

# Part 1 - Unregulated Turing pattern

### Swift-Hohenberg equation

The Swift-Hohenberg equation can be written in the following way :

$\begin{equation}
\dfrac{\partial \phi}{\partial t} = a \phi - (\Delta + 1)^2\phi - \phi^3
\end{equation}$

We will simulate it on a $100 \times 100$ grid, corresponding to $\approx 10 $ wavelength, allowing for reasonable convergence times.

### __Question 0__
> Discretize the 2D laplacian $\Delta = \partial_{xx} + \partial_{yy}$ with a Taylor development. Discretize the differential equation with a forward-Euler scheme.

### __Question 1__
> Integrate this equation until convergence and plot the pattern obtained. Repeat the simulation with several different initial conditions to observe the pattern variation.

```python
#If necessary :
!pip install scikit-image
```

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm

%matplotlib inline

def show_patterns(U, ax=None):
    ax.imshow(U, cmap=plt.cm.copper,
              interpolation='bilinear',
              extent=[-1, 1, -1, 1])
    ax.set_axis_off()

#Simulation Parameters
size = 100  # size of the 2D grid
dx = 1 #.*(10**3) / size  # space step
dt = .01  # time step
T = dt*50000 #total duration of the simulation
n = int(T / dt)  # number of iterations
```

```python
fig, axes = plt.subplots(1, 5, figsize=(20, 4))
step_plot = n // 5

#Simulation Parameters
U = np.random.rand(size, size)*(10**-3)
a = 0.1

#Simulation Loop
for i in tqdm(range(n), total=n):

    # We update the variables.
    U = ...

    # We plot the state of the system at
    # different times.
    if i % step_plot == 0 and i < 5 * step_plot:
        ax = axes.flat[i // step_plot]
        show_patterns(U, ax=ax)
        ax.set_title(f'$t={i * dt:.2f}$')
```

# Part 2 - Stripes orientation with a morphogen gradient

We will change the equation by making $a$ vary with x :

$
\begin{equation}
\forall x \in [0,100] : a(x) = -0.2 + \dfrac{x}{500}
\end{equation}
$

### __Question 2__
> Integrate this varying parameter a to the previous model. What do you observe ?

```python
###Simulation parameters
baselinear = np.linspace(-0.1,0.1,size)
linear_potential = np.zeros((size,size))
for i in range(size):
    linear_potential[i] = baselinear

a = linear_potential.copy()

plt.figure(figsize=(4,3))
plt.imshow(a)
plt.title("Spatial gradient of a")
plt.colorbar()
```

```python
fig, axes = plt.subplots(1, 5, figsize=(20, 4))
U = np.random.rand(size, size)*(10**-3)

###Simulation loop :
for i in tqdm(range(n), total=n):

    # We update the variables.
    U = ...

    # We plot the state of the system at
    # different times.
    if i % step_plot == 0 and i < 5 * step_plot:
        ax = axes.flat[i // step_plot]
        show_patterns(U, ax=ax)
        ax.set_title(f'$t={i * dt:.2f}$')
```

# Part 3 - Stripes orientation with anisotropic diffusion

We will this time change the equation by replacing the term $(\Delta+1)^2 \text{ by }(\Delta+1)^2 - b \partial_x^2$  :

$\begin{equation}
\dfrac{\partial \phi}{\partial t} = a \phi - (\Delta + 1)^2\phi + b\partial_x^2\phi - \phi^3
\end{equation}$

With $a=0.1$ and $b =0.1$.

### __Question 3__
> Integrate this varying anisotropic diffusion to the model. What do you observe ?

```python
fig, axes = plt.subplots(1, 5, figsize=(20, 4))

###Simulation parameters
U = np.random.rand(size, size)*(10**-3)
a = 0.1
b = 0.1

###Simulation loop :
for i in tqdm(range(n), total=n):

    # We update the variables.
    U= ...

    # We plot the state of the system at
    # different times.
    if i % step_plot == 0 and i < 5 * step_plot:
        ax = axes.flat[i // step_plot]
        show_patterns(U, ax=ax)
        ax.set_title(f'$t={i * dt:.2f}$')
```

# Part 4 - Stripes orientation with anisotropic growth

We will this time make a pattern that grows. At each timestep, we will make the pattern grow (before integrating the Swift-Hohenberg equation as previously) by distorting the spatial lengthscale $x$ : $\phi(x,y,t+\delta t) = \phi(\dfrac{x}{1+b\delta t},y,t) $

$\begin{equation}
\dfrac{\partial \phi}{\partial t} = a \phi - (\Delta + 1)^2\phi - \phi^3
\end{equation}$

### __Question 4__
> Integrate this growth to the model, with $b = 0.001$ and $b = 0.003$. What do you observe ?

__Tip :__ Use the resize function of skimage to distort your Turing pattern.

```python
from skimage.transform import resize

fig, axes = plt.subplots(1, 5, figsize=(20, 4))
step_plot = n // 5

def show_patterns_long(U, ax=None):
    ax.imshow(U, cmap=plt.cm.copper,
              interpolation='bilinear')
    ax.set_axis_off()

#Simulation Parameters
U = np.random.rand(size, size)*(10**-3)
a = 0.1
b = 0.001

#Simulation Loop
current_size = size

for i in tqdm(range(n), total=n):

    # We distort U :
    dim_1 = ...
    dim_2 = ...
    U = resize(U, (dim_1,dim_2))

    # We update the variables
    U = ...

    # We plot the state of the system at
    # different times.
    if i % step_plot == 0 and i < 5 * step_plot:
        ax = axes.flat[i // step_plot]
        show_patterns_long(U, ax=ax)
        ax.set_title(f'$t={i * dt:.2f}$')
```

# Bonus : Stripes versus dots

If we add a term with a quadratic term, stripes will progressively be replaced by dots :

$\begin{equation}
\dfrac{\partial \phi}{\partial t} = a \phi + \delta \phi^2 - (\Delta + 1)^2\phi - \phi^3
\end{equation}$

### __Question 5__
> Integrate this quadratic term to the model, with $\delta \in [0.1,0.2,0.3]$. What do you observe ?

```python
fig, axes = plt.subplots(1, 5, figsize=(20, 4))
step_plot = n // 5

###Simulation parameters
U = np.random.rand(size, size)*(10**-3)
a = 0.1
delta = 0.3

###Simulation loop :
for i in tqdm(range(n), total=n):

    # We update the variables.
    U = ...

    # We plot the state of the system at
    # different times.
    if i % step_plot == 0 and i < 5 * step_plot:
        ax = axes.flat[i // step_plot]
        show_patterns(U, ax=ax)
        ax.set_title(f'$t={i * dt:.2f}$')
```

### Conclusion

Though the microscopic factor responsible for the turing instability can vary widly, these mechanisms studied previously are very general and can be applied on many systems. For a nice example on developmental biology showing how birds evolved a special organ to vocalize, see :

> E. Kingsley et al., *Identity and novelty in the avian syrinx*, PNAS, 2018.
> https://doi.org/10.1073/pnas.1804586115


```python
from ratter import *
```

```python
Ag = Material("Ag", refractive_index_value=0.050000+3.1309j)
air = Material("air", refractive_index_value=1)
```

```python
import sympy as sp
```

```python
d = sp.Symbol('d')
```

```python
l1 = Layer("air", air, sp.oo)
l2 = Layer("M1", Ag, 200)
l3 = Layer("space", air, d)
l4 = Layer("M2", Ag, 200)
l5 = Layer("air", air, sp.oo)

stack = Layerstack([l1,l2,l3,l4,l5])
```

```python
T = stack.transmittance_amplitude()*sp.conjugate(stack.transmittance_amplitude())
```

```python
fn = as_function_of(T, [LAMBDA_VAC, d])
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
wl = np.linspace(500, 510, num=1000)
```

```python
%matplotlib inline

plt.figure()

for d_ in [1e4, 1.5e4, 2e4]:
    plt.plot(wl, np.real(fn(wl, d_)), label=str(d_))

plt.legend()
```


# The cash matching problem

**Referencia:**
- Investment Science, David G. Luenberger, 2014.

## 0. Motivación

1. La clase pasada vimos una introducción a programación lineal.
 - Básicamente, ¿qué es?
 - ¿Cómo son las funciones que se maximizan?
 - De acuerdo a lo anterior, ¿qué tipo de problemas se resuelven con LP?
2. La optimización de portafolios es un problema de asignación de capital en diferentes opciones.
 - Si se tienen activos riesgosos, el problema de optimización que surge es uno de QP.
 - Con activos de renta fija (libres de riesgo, flujo de efectivo conocido en cantidad y en tiempo), el problema que surge es uno de LP, como vimos la clase pasada.
3. Aún así, dentro de la optimización de portafolios constituidos por activos libres de riesgo, hay varios subproblemas clásicos que se pueden plantear, uno de ellos es el **cash matching problem**

## 1. Conceptos básicos

Suponga que usted tiene unas obligaciones monetarias futuras (por ejemplo, dichas obligaciones podrían representar los pagos de anualidad de un plan de retiro, seguro de carro, consumo anual estimado del agua, entre otros).

Queremos invertir ahora, de forma que esas obligaciones se puedan pagar una vez se generen los cobros.

Para ello, planeamos comprar bonos con diferente madurez y utilizar los pagos cupón y pago final para atender las obligaciones.

En su forma más básica, se diseña un portafolio que, sin inversión adicional, provee el efectivo necesario cuando se requiera.

Matemáticamente, podemos formular este problema de la siguiente manera...

\begin{equation}
\begin{array}{ll}
\min_{x_1,\dots,x_n} & \sum_{j=1}^{m} p_jx_j \\
\text{s. a. }        & \sum_{j=1}^{m}c_{i,j}x_j\geq=y_i \text{ para } 1\leq i\leq n \\
                     & x_j \text{ para } 1\leq j\leq m,
\end{array}
\end{equation}
donde:
- $x_j$ es la cantidad del bono $j$ a ser comprada ($j=1,2,\dots,m$),
- $p_j$ es el precio del bono $j$,
- $y_i$ es la obligación en el periodo $i$ ($i=1,2,\dots,n$), y
- $c_{i,j}$ es el cupón del bono $j$ en el periodo $i$.

## 2. Ejemplo básico

- Queremos atender obligaciones de efectivo sobre un periodo de 6 años.
- Para este fin seleccionamos 10 bonos (y para simplicidad toda la contabilidad se lleva anual).
- La estructura de flujo de efectivo de cada bono se muestra en la columna correspondiente de la tabla.
- Los precios de cada bono están dados en el vector $p$.
- Las obligaciones están dadas en el vector $y$.
- Se supone que se pueden comprar fracciones de bono.

```python
import pandas as pd
import numpy as np
```

```python
table = pd.DataFrame(index=np.arange(6)+1, columns=np.arange(10)+1)
table.iloc[:,0] = [10,10,10,10,10,110]
table.iloc[:,1] = [7,7,7,7,7,107]
table.iloc[:,2] = [8,8,8,8,8,108]
table.iloc[:,3] = [6,6,6,6,106,0]
table.iloc[:,4] = [7,7,7,7,107,0]
table.iloc[:,5] = [5,5,5,105,0,0]
table.iloc[:,6] = [10,10,110,0,0,0]
table.iloc[:,7] = [8,8,108,0,0,0]
table.iloc[:,8] = [7,107,0,0,0,0]
table.iloc[:,9] = [100,0,0,0,0,0]
table
```

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10

      1
      10
      7
      8
      6
      7
      5
      10
      8
      7
      100

      2
      10
      7
      8
      6
      7
      5
      10
      8
      107
      0

      3
      10
      7
      8
      6
      7
      5
      110
      108
      0
      0

      4
      10
      7
      8
      6
      7
      105
      0
      0
      0
      0

      5
      10
      7
      8
      106
      107
      0
      0
      0
      0
      0

      6
      110
      107
      108
      0
      0
      0
      0
      0
      0
      0

```python
C = table.values
p = np.array([109, 94.8, 99.5, 93.1, 97.2, 92.9, 110, 104, 102, 95.2])
y = np.array([100, 200, 800, 100, 800, 1200])
```

Identificar las entradas de la función linprog y resolver...

```python

```

```python

```

```python

```

```python

```

¿Qué particularidades se encuentran?

## 3. Ejemplo

- La compañía XYZ desea invertir en un conjunto de 7 bonos para garantizar el cumplimient de obligaciones sobre los siguientes 14 años.
- La estructura de flujo de efectivo de cada bono se muestra en la columna correspondiente de la tabla.
- Los precios de cada bono están dados en el vector $p$.
- Las obligaciones están dadas en el vector $y$.
- Además de los bonos, se tiene una cuenta de ahorro cuya tasa de interés anual es del $3\%$.
- Cualquier excedente de efectivo después de cumplir los requerimientos de periodo se invierten en la cuenta de efectivo.
- La meta es determinar el portafolio de mínimo valor hoy para cumplir con todas las obligaciones
- Se supone que se pueden comprar fracciones de bono.

```python
table2 = pd.DataFrame(index=np.arange(14)+1, columns=np.arange(7)+1)
table2.iloc[:,0] = [60,60,60,60,1060,0,0,0,0,0,0,0,0,0]
table2.iloc[:,1] = [65,65,65,65,65,65,65,65,65,65,1065,0,0,0]
table2.iloc[:,2] = [75,75,75,75,75,75,75,75,75,75,75,75,75,1075]
table2.iloc[:,3] = [55,55,55,55,55,55,55,55,55,55,55,55,1055,0]
table2.iloc[:,4] = [80,80,80,80,80,80,80,80,80,1080,0,0,0,0]
table2.iloc[:,5] = [45,45,45,45,45,45,45,45,45,45,45,45,45,1045]
table2.iloc[:,6] = [100,100,100,100,100,100,1100,0,0,0,0,0,0,0]
table2
```

      1
      2
      3
      4
      5
      6
      7

      1
      60
      65
      75
      55
      80
      45
      100

      2
      60
      65
      75
      55
      80
      45
      100

      3
      60
      65
      75
      55
      80
      45
      100

      4
      60
      65
      75
      55
      80
      45
      100

      5
      1060
      65
      75
      55
      80
      45
      100

      6
      0
      65
      75
      55
      80
      45
      100

      7
      0
      65
      75
      55
      80
      45
      1100

      8
      0
      65
      75
      55
      80
      45
      0

      9
      0
      65
      75
      55
      80
      45
      0

      10
      0
      65
      75
      55
      1080
      45
      0

      11
      0
      1065
      75
      55
      0
      45
      0

      12
      0
      0
      75
      55
      0
      45
      0

      13
      0
      0
      75
      1055
      0
      45
      0

      14
      0
      0
      1075
      0
      0
      1045
      0

```python
C = table2.values
p = np.array([990, 1000, 1300, 1100, 1200, 990, 1250])
y = np.array([12000, 14000, 15000, 16000, 18000, 20000, 21000, 22000, 24000, 25000, 30000, 31000, 31000, 31000])
```

Formular el problema, identificar las entradas a la función linprog y resolver

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

Created with Jupyter by Esteban Jiménez Rodríguez.


# Computing the mutual information between the environment and the cellular response

```python
# Our numerical workhorses
import numpy as np
import scipy.optimize
import scipy.special
import scipy.integrate
from sympy import mpmath

# Import libraries to parallelize processes
from joblib import Parallel, delayed

# Import matplotlib stuff for plotting
import matplotlib.pyplot as plt
import matplotlib.cm as cm

# Import pickle to save data
import pickle

# Seaborn, useful for graphics
import seaborn as sns

# Import the utils for this project
import evolution_bits_utils as evo_utils

# favorite Seaborn settings for notebooks
rc={'lines.linewidth': 2,
    'axes.labelsize' : 16,
    'axes.titlesize' : 18,
    'axes.facecolor' : 'F4F3F6',
    'axes.edgecolor' : '000000',
    'axes.linewidth' : 1.2,
    'xtick.labelsize' : 13,
    'ytick.labelsize' : 13,
    'grid.linestyle' : ':',
    'grid.color' : 'a6a6a6'}
sns.set_context(rc=rc)

# Magic function to make matplotlib inline; other style specs must come AFTER
%matplotlib inline

# This enables SVG graphics inline (only use with static plots (non-Bokeh))
%config InlineBackend.figure_format = 'svg'

outputdir = '../../output/03_mutual_information/'
figdir = '../../fig/03_mutual_information/'
tmpdir = '../../tmp/'
```

## Computing the mutual information between environment and gene expression.

With the conditional probability defined theoretically using the chemical master equation approach we can finally tackle the main question of the work!

This probability $P(m \mid C)$ allows us to compute the mutual information between the substrate concentration $ C $ and the gene expression level enconded in the variable $ m $. The mutual information is given by
\begin{equation}
    I(m; C) = \int dC P(C) \int dm P(m \mid C) \log_2 \left[ \frac{P(m \mid C)}{P(m)} \right],
\end{equation}
where the probability $P(m)$ must obey
\begin{equation}
    P(m) = \int dC P(C) P(m \mid C).
\end{equation}

Lets define this function!

```python
def mutual_info(C, mRNA, PC_fun, logPmC_fun, params, cutoff=1E-10):
    '''
    Computes the mutual information between the environment and the gene
    expression level on a grid of values of C and mRNA.

    Parameters
    ----------
    C : array-like.
        discretized values of the concentration at which the numerical
        integral will be evaluated.
    mRNA : array-like.
        values of the mRNA copy number at which the numerical
        integral will be evaluated.
    PC_fun : function.
        function to determine the probability distribution of each of the
        elements of C.
        NOTE: When applied to C it must return an array of the same
        length.
    logPmC_fun : function.
        function to determine the conditional distribution logP(m|C). This in
        general will be one of the versions of the chemical master equation
        solutions, but it can be extended to any input-outpu function one
        wants to use.
        NOTE: When applied to C and mRNA it must return an array of the same
        length.
    params : dictionary.
        dictionary containing all the parameters to compute the mRNA
        distribution with the chemical-master equations approach.
        the parameters are:
        rep : float.
            repressor copy number per cell.
        ka, ki : float.
            dissociation constants for the active and inactive states
            respectively in the MWC model of the lac repressor.
        omega : float.
            energetic barrier between the inactive and the active state.
        kon : float.
            rate of activation of the promoter in the chemical master equation.
        k0 : float.
            diffusion limited rate of a repressor binding the promoter
        gamma : float.
            half-life time for the mRNA.
        r_gamma : float.
            average number of mRNA in the unregulated promoter.
    cutoff : float.
        necessary lower bound to determine when to ignore a term in the
        integral given the convention that 0 x log0 = 0.

    Returns
    -------
    The mutual information between the gene expession and the environment
    distribution in bits.
    '''
    # Since we'll need 2D integrals, make a mesh of variables
    CC, mm = np.meshgrid(C, mRNA)

    # Build P(m|C)
    PmC = np.exp(logPmC_fun(CC, mm, **params)).astype(float)

    # Build P(C)
    PC = PC_fun(C).astype(float)

    # Build P(m) by integrating P(C) * P(m|C) over C.
    Pm = scipy.integrate.simps(PC * PmC, x=C, axis=1).astype(float)
    Pm_tile = np.tile(Pm, (len(C), 1)).T

    # Make P(m|C) * log(P(m|C)) making sure no log(0).
    PmC_log_PmC = np.zeros_like(PmC)
    PmC_log_PmC[PmC > cutoff] = PmC[PmC > cutoff] * np.log2(PmC[PmC > cutoff])

    # Make P(m|C) * log(P(m)) making sure no log(0).
    PmC_log_Pm = np.zeros_like(PmC)
    PmC_log_Pm[Pm_tile > cutoff] = \
            PmC[Pm_tile > cutoff] * np.log2(Pm_tile[Pm_tile > cutoff])

    # Integrate over m
    #int_m = \
    #scipy.integrate.simps(PmC_log_PmC - PmC_log_Pm, x=mRNA, axis=0).astype(float)
    # Since the distribution over mRNA is discrete we have to do a simple sum
    # rather than a numerical integration over m
    int_m = np.sum(PmC_log_PmC - PmC_log_Pm, axis=0).astype(float)

    # Return integral over C
    return scipy.integrate.simps(PC * int_m, x=C).astype(float)
```

### Environment uniform distribution.

Let's first assume a uniform distribution over the different concentrations.

```python
def PC_unif(C):
    '''
    Returns a uniform PDF for an array C. Properly since it is a continuous
    variable the probability should be zero, but since we will be using Simpson's
    rule for numerical integration this funciton returns a numerical value > 0
    for P(C = c).

    Parameter
    ---------
    C : array-like.
        Concentrations at which evaluate the function

    Returns
    -------
    P(C) : array-like.
        evaluation of the PDF at each discrete point.
    '''
    return np.repeat(1 / (C.max() - C.min()), len(C))
```

Now let's set the parameters to evalute the mutual information. We will not include the repressor copy number since we will evaluate the mutual information for different values of this parameter.

```python
# Specify some parameters
mastereq_param = dict(ki=0.0012, ka=0.015, omega=1.1, kon=0.0069,
              k0=2.7e-3, gamma=0.00284, r_gamma=15.7)
```

Let's determine the integration grids and compute the mutual information!

```python
# Define the grid on which integrate
C_grid = np.linspace(0, 5, 50)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)

compute_MI = True
if compute_MI:
    # define the array of repressor copy numbers to evaluate the function in
    R_array = [0, 10, 100, 1000]
    MI_array = np.zeros_like(R_array)
    for i, r in enumerate(R_array):
        mastereq_param['rep'] = r
        MI_array[i] = \
        mutual_info(C=C_grid, mRNA=mRNA_grid,
                    PC_fun=PC_unif, logPmC_fun=evo_utils.log_p_m_mid_C,
                    params=mastereq_param).astype(float)
```

```python
MI_array
```

    array([0, 0, 0, 0])

For some reason I do not fully understand when assuming this distribution for the environments the integration fails to give a reasonable value. Must have to do with the range of values I am choosing.
I will come back to this issue later.

### $\log$ environment uniform distribution

A simple alternative is to assume that
\begin{equation}
    \log C \sim Unif \left[ \log C_{\min} , \log C_{\max} \right].
\end{equation}

Then we just calculate the mutual information as follows
\begin{equation}
    I(\log C; m) = \int d\log C \; P(\log C) \int dm \; P(m \mid \log C) \log \frac{P(m \mid \log C)}{P(m)},
\end{equation}
where
\begin{equation}
    P(m) = \int d \log C \; P(\log C) P(m \mid \log C).
\end{equation}

```python
def PlogC_unif(logC):
    '''
    Returns a uniform PDF for an array logC. Properly since it is a continuous
    variable the probability should be zero, but since we will be using Simpson's
    rule for numerical integration this funciton returns a numerical value > 0
    for P(logC = logc).

    Parameter
    ---------
    logC : array-like.
        Concentrations at which evaluate the function

    Returns
    -------
    P(C) : array-like.
        evaluation of the PDF at each discrete point.
    '''
    return np.repeat(1 / (logC.max() - logC.min()), len(logC))
```

```python
# Specify some parameters
mastereq_param = dict(ki=0.0012, ka=0.015, omega=1.1, kon=0.0069,
              k0=2.7e-3, gamma=0.00284, r_gamma=15.7)
```

### [$10^{-3}$ mM, $1$ mM]

```python
# Define the grid on which integrate
logC_grid = np.linspace(-3, 0, 100)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=logC_grid, mRNA=mRNA_grid,
        PC_fun=PlogC_unif, logPmC_fun=evo_utils.log_p_m_mid_logC,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-3mM_1mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(mastereq_param, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-3mM_1mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style('darkgrid', rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title('wt parameters, log unif distribution')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-3}$mM, $1$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
plt.savefig(figdir + 'MI_rgamma10_logunif_1E-3mM_1mM.pdf')
```

### [$10^{-6}$ mM, $1$ mM]

```python
# Define the grid on which integrate
logC_grid = np.linspace(-6, 0, 100)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=logC_grid, mRNA=mRNA_grid,
        PC_fun=PlogC_unif, logPmC_fun=evo_utils.log_p_m_mid_logC,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style(rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title('wt parameters, log unif distribution')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-6}$mM, $1$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
plt.tight_layout()
plt.savefig(figdir + 'MI_rgamma10_logunif_1E-6mM_1mM.pdf')
```

### Exponential distribution of environments.

In his paper Bialek mentions that they assume a probability distribution for the environments of the form
\begin{equation}
    P(C) \propto \exp (- 2 C).
\end{equation}

So let's try computing the mutual information using this distribution. In general wi will say
\begin{equation}
    P(C) = \tau \exp (- \tau C),
\end{equation}
where $\mathbf{E}(C) = \frac{1}{\tau}$

```python
def PC_expo(C, tau=2):
    '''
    Returns an exponential PDF for an array C. Properly since it is a
    continuous variable the probability should be zero, but since we
    will be using Simpson's rule for numerical integration this funciton
    returns a numerical value > 0 for P(C = c).

    Parameter
    ---------
    C : array-like.
        concentrations at which evaluate the function

    Returns
    -------
    P(C) : array-like.
        evaluation of the PDF at each discrete point.
    '''
    return tau * np.exp(- tau * C)
```

### $\tau = 2$,  [$10^{-8}$ mM, $5$ mM]

```python
# Define the grid on which integrate
C_grid = np.linspace(1E-8, 5, 100)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=C_grid, mRNA=mRNA_grid,
        PC_fun=PC_expo, logPmC_fun=evo_utils.log_p_m_mid_C,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_param_WT_expo_2_1E-8mM_5mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_param_WT_expo_2_1E-8mM_5mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style(rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title(r'wt parameters, expo dist $\tau = 2$')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-8}$mM, $5$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
plt.savefig(figdir + 'MI_param_WT_expo_2_1E-8mM_5mM.pdf')
```

### $\tau = 4$,  [$10^{-8}$ mM, $5$ mM]

```python
# Define the grid on which integrate
C_grid = np.linspace(1E-8, 5, 100)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

# define a lambda function to change the value of the parameter tau
# while still be able to feed it to the function
PC_tmp = lambda C: PC_expo(C, 4)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=C_grid, mRNA=mRNA_grid,
        PC_fun=PC_tmp, logPmC_fun=evo_utils.log_p_m_mid_C,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_param_WT_expo_4_1E-8mM_5mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_param_WT_expo_4_1E-8mM_5mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style(rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title(r'wt parameters, expo dist $\tau = 4$')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-8}$mM, $5$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
    plt.savefig(figdir + 'MI_param_WT_expo_4_1E-8mM_5mM.pdf')
```

### $\tau = 10$,  [$10^{-8}$ mM, $5$ mM]

```python
# Define the grid on which integrate
C_grid = np.linspace(1E-8, 5, 100)
# mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))
mRNA_grid = np.arange(0, 100)
# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

# define a lambda function to change the value of the parameter tau
# while still be able to feed it to the function
PC_tmp = lambda C: PC_expo(C, 10)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=C_grid, mRNA=mRNA_grid,
        PC_fun=PC_tmp, logPmC_fun=evo_utils.log_p_m_mid_C,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_param_WT_expo_10_1E-8mM_5mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_param_WT_expo_10_1E-8mM_5mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style(rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title(r'wt parameters, expo dist $\tau = 10$')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-8}$mM, $5$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
plt.savefig(figdir + 'MI_param_WT_expo_10_1E-8mM_5mM.pdf')
```

### Uniform distribution over the $\log$ concentration, but integrate over the linear scale.

Last but not least let's assume that again $\log C \sim Unif[C_\min, C_\max]$. But this time we will integrate over C rather than over $\log C$.

By a simple change of variables we can obtain this PDF
\begin{equation}
    f_C(c) = f_{\log C}(\log C) \lvert \frac{d \log C}{d C} \lvert.
\end{equation}
This gives us that the PDF for the concentration when it is uniformly distributed over log scale is given by
\begin{equation}
    f_C(C) = \frac{1}{C (\log C_{\max} - \log C_{\min})}, \forall C \in [\exp \log C_{\min}, \exp \log C_{\min}].
\end{equation}

```python
def PC_log_unif(C):
    '''
    Returns a PDF for an array C on which log C is uniformly distributed.
    Properly since it is a continuous variable the probability should be zero,
    but since we will be using Simpson's rule for numerical integration this
    funciton returns a numerical value > 0 for P(C = c).

    Parameter
    ---------
    logC : array-like.
        log_10 concentrations at which evaluate the function

    Returns
    -------
    P(C) : array-like.
        evaluation of the PDF at each discrete point.
    '''
    return 1 / C / (np.log(C).max() - np.log(C).min())
```

```python
# Define the grid on which integrate
logC_grid = np.linspace(-5, 1, 100)
mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=np.power(10, logC_grid), mRNA=mRNA_grid,
        PC_fun=PC_log_unif, logPmC_fun=evo_utils.log_p_m_mid_C,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)
    np.savetxt('../../tmp/MI_param_WT_C_log_unif.txt', MI_array)
```

```python
MI_array = np.loadtxt('../../tmp/MI_param_WT_C_log_unif.txt').flatten()
with sns.axes_style(rc=rc):
    plt.plot(R_array, MI_array)
    plt.xscale('log')
    plt.xlabel('repressor copy number')
    plt.ylabel('mutual information [bits]')
    plt.title('wild-type parameters, integrate over $C$ with $\log C$ uniform',
             fontsize=11)
```

Great! the fact that this is equal to the calculation we did with $I(\log C; m)$ confirms that it doesn't matter if we calculate the mutual information with respect to the $\log$ of the concentration or with the concentration itself since $\log$ is a one-to-one function and mutual information is invariant to these type of transformations.

# Arcsine distribution

An interesting bimodal distribution that we want to explore is the so-called arcsine distribution. This distribution has a PDF defined as
\begin{equation}
    P(C) = \frac{1}{\pi \sqrt{(C - C_\min)(C_\max - C)}},
\end{equation}
where $C_\min$ and $C_\max$ are the minimum and maximum concentrations allowed.

The idea for using this distribution came from a further down script where we calculate the channel capacity of the communication channel and the distribution that achieves capacity seems to be multimodal with high probability at low and at high concentrations.

```python
def PC_arcsine(C):
    '''
    Returns an arsine PDF for an array C. Properly since it is a
    continuous variable the probability should be zero, but since we
    will be using Simpson's rule for numerical integration this funciton
    returns a numerical value > 0 for P(C = c).

    Parameter
    ---------
    C : array-like.
        concentrations at which evaluate the function

    Returns
    -------
    P(C) : array-like.
        evaluation of the PDF at each discrete point.
    '''
    return 1 / (np.pi * np.sqrt((C - C.min()) * (C.max() - C)))
```

```python
C_grid = np.linspace(0, 5, 200)
_ = plt.plot(C_grid, PC_arcsine(C_grid))
```

```python
scipy.integrate.simps(PC_arcsine(C_grid), x=C_grid)
```

    inf

```python
# Specify some parameters
mastereq_param = dict(ki=0.0012, ka=0.015, omega=1.1, kon=0.0069,
              k0=2.7e-3, gamma=0.00284, r_gamma=15.7)

# Define the grid on which integrate
C_grid = np.linspace(0, 5, 200)
mRNA_grid = np.linspace(0, 100, 201)

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 3)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info(C=C_grid, mRNA=mRNA_grid,
        PC_fun=PC_arcsine, logPmC_fun=evo_utils.log_p_m_mid_C,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=1)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_param_WT_arcsine_0mM_5mM.pkl', 'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(C_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
    pklfile.close()
```

```python
pklfile = open(outputdir + 'MI_param_WT_arcsine_0mM_5mM.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)
pklfile.close()

fig = plt.figure()
ax = plt.subplot(111)
_ = ax.plot(R_array, MI_array)
ax.set_xlabel('repressor copy number')
ax.set_ylabel('mutual information [bits]')
ax.set_title(r'wild-type parameters, arcsine distribution')
# indicate the concentration range used for the calculation
ax.text(0.7, 0.05, r'[$0$mM, $5$mM]',
        horizontalalignment='left',
        verticalalignment='center',
        transform=ax.transAxes, fontsize=13)
```

    <matplotlib.text.Text at 0x1181c1ef0>

# Spectral integration.

One can greatly reduce the number of grid points you use without sacrificing accuracy by using **spectral integration**.

The key idea here is to sample the function at Chebyshev points and then use Clenshaw-Curtis weights to perform the quadrature.  Below are utility functions to do that.

(Credit to Justin Bois)

```python
def cheb_points_1d(n, xends=[-1.0, 1.0]):
    """
    Computes Chebyshev points for a 1-D Chebyshev grid, returned as ndarray.

    xends specifies the end points of the array.
    """

    x = np.cos(np.pi * np.array(np.arange(n)) / (n-1))
    x = (xends[1] - xends[0]) / 2.0 * x + (xends[1] + xends[0]) / 2.0

    return x

def clenshaw_curtis_weights(n):
    """
    Computes the weights to be applied in Curtis-Clenshaw integration,
    sampling at n assuming Chebyshev points.

    Adapted from Nick Trefethen's book: Spectral Methods in Matlab
    """

    n -= 1  # This is to stay consistent with our indexing

    theta = np.pi * np.arange(n+1) / n
    w = np.zeros_like(theta)
    v = np.ones(n-1)
    if n % 2 == 0:
        w[0] = 1.0 / (n**2 - 1)
        w[-1] = w[0]
        for k in range(1, n//2):
            v -= 2.0 * np.cos(2.0 * k * theta[1:-1]) / (4.0 * k**2 - 1)
        v -= np.cos(n * theta[1:-1]) / (n**2 - 1)
    else:
        w[0] = 1.0 / n**2
        w[-1] = w[0]
        for k in range(1, (n-1)//2 + 1):
            v -= 2.0 * np.cos(2.0 * k * theta[1:-1]) / (4.0 * k**2 - 1)

    w[1:-1] = 2.0 * v / n

    return w

def cheb_quad(y, w, domain_size):
    """
    Perform spectral integration given Clenshaw-Curtis weights. The
    weights are computed from clenshaw_curtis_weights().
    """
    return np.dot(w, y) * domain_size / 2
```

With these function, we can make new grid points and perform the integration.

```python
def mutual_info_spectral(C_range, mRNA, PC_fun, logPmC_fun, params,
                         n_points=64, cutoff=1E-10):
    '''
    Computes the mutual information between the environment and the gene
    expression level on a grid of values of C and mRNA.

    Parameters
    ----------
    C_range : array-like.
        range of concentrations that should be taken for the integration.
        For spectral integration we just give the range because the grid points
        at which the integral is evaluated are computed with Chebyshev's formula.
    mRNA : array-like.
        discretized values of the mRNA copy number at which the numerical
        integral will be evaluated.
    PC_fun : function.
        function to determine the probability distribution of each of the
        elements of C.
        NOTE: Should return n_points between C_range.min() and C_range.max()
    logPmC_fun : function.
        function to determine the conditional distribution logP(m|C). This in
        general will be one of the versions of the chemical master equation
        solutions, but it can be extended to any input-outpu function one
        wants to use.
        NOTE: When applied to C and mRNA it must return an array of the same
        length.
    params : dictionary.
        dictionary containing all the parameters to compute the mRNA
        distribution with the chemical-master equations approach.
        the parameters are:
        rep : float.
            repressor copy number per cell.
        ka, ki : float.
            dissociation constants for the active and inactive states
            respectively in the MWC model of the lac repressor.
        omega : float.
            energetic barrier between the inactive and the active state.
        kon : float.
            rate of activation of the promoter in the chemical master equation.
        k0 : float.
            diffusion limited rate of a repressor binding the promoter
        gamma : float.
            half-life time for the mRNA.
        r_gamma : float.
            average number of mRNA in the unregulated promoter.
    n_points : int.
        number of grid points used for the spectral integration. This is the
        number of Chebyshev points that are used for the integration.
    cutoff : float.
        necessary lower bound to determine when to ignore a term in the
        integral given the convention that 0 x log0 = 0.

    Returns
    -------
    The mutual information between the gene expession and the environment
    distribution in bits.
    '''
    # Convert C_range into a numpy array
    C_range = np.array(C_range)
    # Set the Chebyshev points and the Clenshaw curtis weights
    C_cheb= cheb_points_1d(n_points, [C_range.min(), C_range.max()])
    w = clenshaw_curtis_weights(n_points)

    # Since we'll need 2D integrals, make a mesh of variables
    CC, mm = np.meshgrid(C_cheb, mRNA)

    # Build P(m|C)
    PmC = np.exp(logPmC_fun(CC, mm, **params)).astype(float)

    # Build P(C)
    PC = PC_fun(C_cheb).astype(float)
    # Build P(m) by integrating P(C) * P(m|C) over C.
    Pm = cheb_quad((PC * PmC).T, w, C_cheb.max()-C_cheb.min()).astype(float)
    Pm_tile = np.tile(Pm, (n_points, 1)).T

    # Make P(m|C) * log(P(m|C)) making sure no log(0).
    PmC_log_PmC = np.zeros_like(PmC)
    PmC_log_PmC[PmC > cutoff] = PmC[PmC > cutoff] * np.log2(PmC[PmC > cutoff])

    # Make P(m|C) * log(P(m)) making sure no log(0).
    PmC_log_Pm = np.zeros_like(PmC)
    PmC_log_Pm[Pm_tile > cutoff] = \
            PmC[Pm_tile > cutoff] * np.log2(Pm_tile[Pm_tile > cutoff])

    # Integrate over m.
    # NOTE: since the distribution is only normalized for discrete values, we
    # actually sum over the values rather than integrating
    int_m = \
    np.sum(PmC_log_PmC - PmC_log_Pm, axis=0).astype(float)

    # Return integral over C
    return cheb_quad(PC * int_m, w, C_cheb.max()-C_cheb.min()).astype(float)
```

## Comparison between regular numerical integration and spectral integration.

Now that we implemented the spectral integration lets compare it with the regular one.

```python
# Specify some parameters
mastereq_param = dict(ki=0.0012, ka=0.015, omega=1.1, kon=0.0069,
              k0=2.7e-3, gamma=0.00284, r_gamma=15.7)

# Define the grid on which integrate
C_grid = np.linspace(0, 5, 200)
C_range = np.array([C_grid.max(), C_grid.min()])
mRNA_grid = np.arange(0, 50)
mastereq_param['rep'] = 10
```

```python
print('Regular integration:')
%timeit mutual_info(C=C_grid, mRNA=mRNA_grid, PC_fun=PC_expo, logPmC_fun=evo_utils.log_p_m_mid_C, params=mastereq_param)
print('Spectral integration:')
%timeit mutual_info_spectral(C_range=C_range, mRNA=mRNA_grid, PC_fun=PC_expo, logPmC_fun=evo_utils.log_p_m_mid_C, params=mastereq_param)
```

    Regular integration:
    1 loop, best of 3: 2.47 s per loop
    Spectral integration:
    1 loop, best of 3: 777 ms per loop

```python
mutual_info(C=C_grid, mRNA=mRNA_grid, PC_fun=PC_expo,
            logPmC_fun=evo_utils.log_p_m_mid_C, params=mastereq_param) - \
mutual_info_spectral(C_range=C_range, mRNA=mRNA_grid,
                     PC_fun=PC_expo, logPmC_fun=evo_utils.log_p_m_mid_C,
                     params=mastereq_param, n_points=64)
```

    0.062710998477379418

```python
# Define the grid on which integrate
logC_grid = np.linspace(-6, 0, 200)
logC_range = np.array([logC_grid.max(), logC_grid.min()])
mRNA_grid = np.arange(0, np.floor(mastereq_param['r_gamma'] * 1.7))

# define the array of repressor copy numbers to evaluate the function in
R_array = np.logspace(0, 3, 40)

compute_MI = True
if compute_MI:
# define a function to run in parallel the computation
    def MI_parallel(r):
        mastereq_param['rep'] = r
        return mutual_info_spectral(C_range=logC_range, mRNA=mRNA_grid,
        PC_fun=PlogC_unif, logPmC_fun=evo_utils.log_p_m_mid_logC,
        params=mastereq_param)
    MI_array = Parallel(n_jobs=6)(delayed(MI_parallel)(r) for r in R_array)

    # To be consistent and have reproducible results we will save the following:
    # 1. the mutual information array
    # 2. the repressor array
    # 3. the concentration array
    # 4. the mRNA array
    pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM_spectral.pkl',
                   'wb')
    pickle.dump(MI_array, pklfile)
    pickle.dump(R_array, pklfile)
    pickle.dump(logC_grid, pklfile)
    pickle.dump(mRNA_grid, pklfile)
```

```python
pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM_spectral.pkl', 'rb')
MI_array = pickle.load(pklfile)
R_array = pickle.load(pklfile)

with sns.axes_style(rc=rc):
    fig = plt.figure()
    ax = plt.subplot(111)
    _ = ax.plot(R_array, MI_array)
    ax.set_xscale('log')
    ax.set_xlabel('repressor copy number')
    ax.set_ylabel('mutual information [bits]')
    ax.set_title('wild-type parameters, log uniform distribution')
    # indicate the concentration range used for the calculation
    ax.text(0.7, 0.05, r'[$10^{-6}$mM, $1$mM]',
            horizontalalignment='left',
            verticalalignment='center',
            transform=ax.transAxes, fontsize=13)
#plt.savefig(figdir + 'MI_rgamma10_logunif_1E-6mM_1mM.pdf')
```

The plot looks exactly the same! I can't even believe it. Let's take a look on how different the values obtained with both integration procedures are.

```python
pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM_spectral.pkl', 'rb')
MI_spectral = pickle.load(pklfile)

pklfile = open(outputdir + 'MI_rgamma10_logunif_1E-6mM_1mM.pkl', 'rb')
MI_simps = pickle.load(pklfile)

np.array(MI_spectral) - np.array(MI_simps)
```

    array([  3.00713128e-10,   3.81172621e-10,   4.79775344e-10,
             5.99269856e-10,   7.42331627e-10,   9.11378945e-10,
             1.10837481e-09,   1.33464925e-09,   1.59079439e-09,
             1.87666993e-09,   2.19154739e-09,   2.53436316e-09,
             2.90403357e-09,   3.29976102e-09,   3.72131537e-09,
             4.16929602e-09,   4.64541494e-09,   5.15277476e-09,
             5.69606218e-09,   6.28152075e-09,   6.91660995e-09,
             7.60934338e-09,   8.36241598e-09,   9.18418053e-09,
             1.01732117e-08,   1.11127751e-08,   1.21723363e-08,
             1.33108278e-08,   1.45676428e-08,   1.55980021e-08,
             1.70741253e-08,   1.84796242e-08,   1.97058623e-08,
             2.11820990e-08,   2.23857638e-08,   2.36506886e-08,
             2.49200428e-08,   2.61260146e-08,   2.73129239e-08,
             2.82370647e-08])

This is amazing! Spectral integration is super fast. All of my computations of mutual information just got speed up by a factor of three.


# 1

### M1

$F = K_2^{-T}R_2[\vec{c_2}-\vec{c_1}]_x R_1^{-1} K_1^{-1} $

$\vec{e_1} = R_1 K_1 (\vec{c_2}-\vec{c_1})$

$\vec{c_2}-\vec{c_1} = R_1^{-1} K_1^{-1} \vec{e_1}$

тут ще десь константа є, але її можна відкинути з причини того що F можна множити на скаляр

$F = K_2^{-T}R_2[R_1^{-1} K_1^{-1} \vec{e_1}]_x R_1^{-1} K_1^{-1} $

$F = K_2^{-T}R_2 \frac{(R_1^{-1} K_1^{-1})^{-T}}{|(R_1^{-1} K_1^{-1})^{-T}|} [ \vec{e_1}]_x $

$M_1 = K_2^{-T}R_2 \frac{(K_1 R_1)^{T}}{|K_1^T|}$

$M_1 = K_2^{-T}R_2 \frac{(K_1 R_1)^{T}}{|K_1^T|}$

### M2

$F = K_2^{-T}R_2[\vec{c_2}-\vec{c_1}]_x R_1^{-1} K_1^{-1} $

$\vec{e_2} = R_2 K_2 (\vec{c_1}-\vec{c_2})$

$\vec{c_2}-\vec{c_1} = -R_2^{-1} K_2^{-1} \vec{e_1}$

$F = -K_2^{-T}R_2[R_2^{-1} K_2^{-1} \vec{e_2}]_x R_1^{-1} K_1^{-1} $

$F = -K_2^{-T}R_2[R_2^{-1} K_2^{-1} \vec{e_2}]_x R_2^{-1} K_2^{-1} K_2 R_2 R_1^{-1} K_1^{-1} $

$F = -K_2^{-T}R_2 \frac{(R_2^{-1} K_2^{-1})^{-T}}{(|R_2^{-1} K_2^{-1}|)^{-T}}[\vec{e_2}]_x R_2^{-1} K_2^{-1} K_2 R_2 R_1^{-1} K_1^{-1} $

$F = -\frac{1}{(|R_2^{-1} K_2^{-1}|)^{-T}}[\vec{e_2}]_x K_2 R_2 R_1^{-1} K_1^{-1} $

$F = -\frac{1}{(|K_2^{-1}|)^{-T}}[\vec{e_2}]_x K_2 R_2 R_1^{-1} K_1^{-1} $

$F = -[\vec{e_2}]_x \frac{K_2 R_2 R_1^{-1} K_1^{-1}}{(|K_2^{-1}|)^{-T}} $

$M_2 =  -\frac{K_2 R_2 R_1^{-1} K_1^{-1}}{|K_2|T}$

# 2

```python
from sympy import *
```

```python
a1,a2,a3,a4,a5,a6,a7,a8,a9 = symbols("a1,a2,a3,a4,a5,a6,a7,a8,a9".replace(","," "))
```

```python
R = Matrix([[a1,a2,a3],[a4,a5,a6],[a7,a8,a9]])
R
```

$\displaystyle \left[\begin{matrix}a_{1} & a_{2} & a_{3}\\a_{4} & a_{5} & a_{6}\\a_{7} & a_{8} & a_{9}\end{matrix}\right]$

```python
c1,c2,c3 = symbols("c1 c2 c3")
```

```python
z = Matrix([0,0,0])
c = Matrix([c1,c2,c3])

H = Matrix([[R, c],[z.T,Matrix([1])]])

H
```

$\displaystyle \left[\begin{matrix}a_{1} & a_{2} & a_{3} & c_{1}\\a_{4} & a_{5} & a_{6} & c_{2}\\a_{7} & a_{8} & a_{9} & c_{3}\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
Hi2 = simplify(H.inv()).subs(R.det(), 1)
Hi2
```

$\displaystyle \left[\begin{matrix}a_{5} a_{9} - a_{6} a_{8} & - a_{2} a_{9} + a_{3} a_{8} & a_{2} a_{6} - a_{3} a_{5} & - a_{2} a_{6} c_{3} + a_{2} a_{9} c_{2} + a_{3} a_{5} c_{3} - a_{3} a_{8} c_{2} - a_{5} a_{9} c_{1} + a_{6} a_{8} c_{1}\\- a_{4} a_{9} + a_{6} a_{7} & a_{1} a_{9} - a_{3} a_{7} & - a_{1} a_{6} + a_{3} a_{4} & a_{1} a_{6} c_{3} - a_{1} a_{9} c_{2} - a_{3} a_{4} c_{3} + a_{3} a_{7} c_{2} + a_{4} a_{9} c_{1} - a_{6} a_{7} c_{1}\\a_{4} a_{8} - a_{5} a_{7} & - a_{1} a_{8} + a_{2} a_{7} & a_{1} a_{5} - a_{2} a_{4} & - a_{1} a_{5} c_{3} + a_{1} a_{8} c_{2} + a_{2} a_{4} c_{3} - a_{2} a_{7} c_{2} - a_{4} a_{8} c_{1} + a_{5} a_{7} c_{1}\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
R.inv().subs(R.det(), 1)
```

$\displaystyle \left[\begin{matrix}a_{5} a_{9} - a_{6} a_{8} & - a_{2} a_{9} + a_{3} a_{8} & a_{2} a_{6} - a_{3} a_{5}\\- a_{4} a_{9} + a_{6} a_{7} & a_{1} a_{9} - a_{3} a_{7} & - a_{1} a_{6} + a_{3} a_{4}\\a_{4} a_{8} - a_{5} a_{7} & - a_{1} a_{8} + a_{2} a_{7} & a_{1} a_{5} - a_{2} a_{4}\end{matrix}\right]$

```python
Hi2[:3,:3] - R.inv().subs(R.det(), 1)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$

```python
-(R.inv().subs(R.det(), 1)) * c
```

$\displaystyle \left[\begin{matrix}c_{1} \left(- a_{5} a_{9} + a_{6} a_{8}\right) + c_{2} \left(a_{2} a_{9} - a_{3} a_{8}\right) + c_{3} \left(- a_{2} a_{6} + a_{3} a_{5}\right)\\c_{1} \left(a_{4} a_{9} - a_{6} a_{7}\right) + c_{2} \left(- a_{1} a_{9} + a_{3} a_{7}\right) + c_{3} \left(a_{1} a_{6} - a_{3} a_{4}\right)\\c_{1} \left(- a_{4} a_{8} + a_{5} a_{7}\right) + c_{2} \left(a_{1} a_{8} - a_{2} a_{7}\right) + c_{3} \left(- a_{1} a_{5} + a_{2} a_{4}\right)\end{matrix}\right]$

```python
simplify(Hi2[:-1,-1] + (R.inv().subs(R.det(), 1)) * c)
```

$\displaystyle \left[\begin{matrix}0\\0\\0\end{matrix}\right]$

### Офіційна відповідь

Для

$H =
\begin{bmatrix}
R &  \vec{c}\\
\vec{0}^T & 1
\end{bmatrix}
$

обернена матриця дорівнює

$H^{-1} =
\begin{bmatrix}
R^{-1} &  -R^{-1}\vec{c}\\
\vec{0}^T & 1
\end{bmatrix}
$

```python

```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Step Response

The response of an LTI system to a [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) as input signal is known as [*step response*](https://en.wikipedia.org/wiki/Step_response). It is defined as

\begin{equation}
h_\epsilon(t) = \mathcal{H} \{ \epsilon(t) \}
\end{equation}

The step response characterizes the properties of a system when the input signal is 'switched on' at $t=0$. It is related to the impulse response by

\begin{equation}
h_\epsilon(t) = \epsilon(t) * h(t) = \int_{-\infty}^{t} h(\tau) \; d\tau
\end{equation}

This implies that the impulse response is the derivative of the step response

\begin{equation}
h(t) = \frac{d h_\epsilon(t)}{dt}
\end{equation}

Using this result, the output signal $y(t) = \mathcal{H} \{ x(t) \}$ in terms of the step response reads

\begin{equation}
y(t) = x(t) * \frac{d h_\epsilon(t)}{dt} = \frac{d x(t)}{dt} * h_\epsilon(t)
\end{equation}

Since a Dirac impulse cannot be realized in practice, the step response is an alternative to measure the properties of an LTI system. It plays an import role in the theory of [control systems](https://en.wikipedia.org/wiki/Control_system).

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Concept discovery on MNIST

```python
import sys
sys.path.append('..')
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from sympy import simplify_logic
import time
from sklearn.metrics import accuracy_score
import pandas as pd
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.tree import _tree, export_text
from torchvision import datasets, transforms
from torch.utils.data import TensorDataset, DataLoader
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn.functional as F
from sklearn.preprocessing import OneHotEncoder
from sklearn.manifold import TSNE
from sklearn.cluster import KMeans
from sklearn.model_selection import train_test_split
import seaborn as sns

import lens
from lens.utils.base import validate_network, set_seed, tree_to_formula
from lens.utils.layer import prune_logic_layers
from lens import logic

results_dir = 'results_ll/mnist_h'
if not os.path.isdir(results_dir):
    os.makedirs(results_dir)

models_dir = 'models_ll/mnist_h'
if not os.path.isdir(models_dir):
    os.makedirs(models_dir)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
```

# Data

## Load MNIST data

```python
# MNIST problem
num_workers = 0
batch_size = 256
valid_size = 0.2
# Data augmentation for train data + conversion to tensor
train_transforms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))

])  # Data augmentation for test data + conversion to tensor
test_transforms = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])
train_data = datasets.MNIST(root='./data', train=True, download=True, transform=train_transforms)
test_data = datasets.MNIST(root='./data', train=False, download=True, transform=test_transforms)
```

## Generate data loaders

```python
# Finding indices for validation set
num_train = len(train_data)
indices = list(range(num_train))
# Randomize indices
np.random.shuffle(indices)
split = int(np.floor(num_train * valid_size))
train_index, test_index = indices[split:], indices[:split]  # Making samplers for training and validation batches
train_sampler = SubsetRandomSampler(train_index)
valid_sampler = SubsetRandomSampler(test_index)  # Creating data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, sampler=train_sampler,
                                           num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, sampler=valid_sampler,
                                           num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers)
```

# Model
## Define CNN architecture

```python
n_concepts = 16
```

```python
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # convolutional layers
        self.conv1 = torch.nn.Conv2d(1, 8, 3, padding=1)
        self.conv2 = torch.nn.Conv2d(8, n_concepts, 3, padding=1)
        self.log1 = lens.nn.XLogicConv2d(in_channels=n_concepts, channel_size=30, activation='leaky_relu')
        self.fc5 = torch.nn.Linear(n_concepts, 8)
        self.fc6 = torch.nn.Linear(8, 2)
        self.log2 = lens.nn.XLogic(2, top=True, activation='leaky_relu')
        # max pooling
        self.pool = torch.nn.MaxPool2d(2, 1, padding=1)
        self.filter_output = None

    def forward(self, x):
        # convolutional layers with ReLU and pooling
        x = self.pool(F.leaky_relu(self.conv1(x)))
        x = self.pool(F.leaky_relu(self.conv2(x)))
        self.filter_output = x
        x = self.log1(x)
        x = F.leaky_relu(self.fc5(x))
        x = F.leaky_relu(self.fc6(x))
        x = self.log2(x)
        return F.softmax(x, dim=1)
```

## Train model

### Whitening Loss function

```python
# the target covariance matrix is diagonal
cov_objective = torch.eye(n_concepts).to(device)

def whitening_loss(model):
    loss = 0
    for module in model.children():
        if isinstance(module, lens.nn.XLogicConv2d):
            # compute covariance matrix of activations
            cov = 1 / (n_concepts - 1) * torch.matmul(module.conceptizator.concepts.T, module.conceptizator.concepts)
            loss += torch.norm(cov - cov_objective, p=2)
    return loss, cov
```

### L1-regularization loss

```python
def l1_loss(model):
    loss = 0
    for module in model.children():
        if isinstance(module, lens.nn.XLogicConv2d):
            loss += torch.norm(module.weight, 1) + torch.norm(module.bias, 1)
    return loss
```

```python
model = Net()
model.cuda()

if os.path.isfile('./models_ll/mnist_h/trained_model_h3.pt'):
    # load trained model
    model.load_state_dict(torch.load('./models_ll/mnist_h/trained_model_h3.pt'))

else:
    epochs = 10
    set_seed(0)
    need_pruning = True

    # tracks validation loss change after each epoch
    minimum_validation_loss = np.inf

    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)
    criterion = torch.nn.CrossEntropyLoss()
    for epoch in range(1, epochs + 1):

        train_loss = 0
        valid_loss = 0

        # training steps
        model.train()
        for batch_index, (data, target) in enumerate(train_loader):
            # moves tensors to GPU
            data, target = data.cuda(), target.cuda() % 2
            # clears gradients
            optimizer.zero_grad()
            # forward pass
            output = model(data)  # .squeeze()
            # loss in batch
            loss = criterion(output, target) + 0.1 * whitening_loss(model)[0] + 0.001 * l1_loss(model)

            # backward pass for loss gradient
            loss.backward()
            # update paremeters
            optimizer.step()
            # update training loss
            train_loss += loss.item() * data.size(0)

        # validation steps
        model.eval()
        for batch_index, (data, target) in enumerate(valid_loader):
            # moves tensors to GPU
            data, target = data.cuda(), target.cuda() % 2
            # forward pass
            output = model(data)
            # loss in batch
            loss = criterion(output, target)
            # update validation loss
            valid_loss += loss.item() * data.size(0)

        # average loss calculations
        train_loss = train_loss / len(train_loader.sampler)
        valid_loss = valid_loss / len(valid_loader.sampler)

        # Display loss statistics
        print(f'Current Epoch: {epoch}\nTraining Loss: {round(train_loss, 6)}\nValidation Loss: {round(valid_loss, 6)}')

        # Saving model every time validation loss decreases
        if valid_loss <= minimum_validation_loss and epoch > 4:
            print(f'Validation loss decreased from {round(minimum_validation_loss, 6)} to {round(valid_loss, 6)}')
            torch.save(model.state_dict(), './models_ll/mnist_h/trained_model_h3.pt')
            minimum_validation_loss = valid_loss
            print('Saving New Model')

        # tracking test loss
        test_loss = 0.0
        class_correct = list(0. for i in range(2))
        class_total = list(0. for i in range(2))

        # test steps
        model.eval()
        for batch_index, (data, target) in enumerate(test_loader):
            # moves tensors to GPU
            data, target = data.cuda(), target.cuda() % 2
            # forward pass
            output = model(data)
            # loss in batch
            loss = criterion(output, target)

            # convert output probabilities to predicted class
            _, pred = torch.max(output, 1)
            # compare predictions to true label
            correct_tensor = pred.eq(target.data.view_as(pred))
            correct = np.squeeze(correct_tensor.numpy()) if not torch.cuda.is_available() else np.squeeze(
                correct_tensor.cpu().numpy())
            # calculate test accuracy for each object class
            for i in range(2):
                label = target.data[i]
                class_correct[label] += correct[i].item()
                class_total[label] += 1

        print(f'Test Accuracy of class "even": {round(100 * class_correct[0] / class_total[0], 2)}%')
        print(f'Test Accuracy of class "odd": {round(100 * class_correct[1] / class_total[1], 2)}%')

        cov_np = whitening_loss(model)[1].cpu().detach().numpy()
        plt.figure()
        sns.heatmap(cov_np)
        plt.show()

    model.load_state_dict(torch.load('./models_ll/mnist_h/trained_model_h3.pt'))
```

# Concepts' visualization

### Feed the network with validation data

```python
model.eval()
x_subset, y_subset, pred_subset = [], [], []
for i, (x_subset_i, y_subset_i) in enumerate(iter(valid_loader)):
    x_subset.append(x_subset_i)
    y_subset.append(y_subset_i)
    pred_subset.append(model.cpu()(x_subset_i))

x_subset = torch.cat(x_subset)
y_subset = torch.cat(y_subset)
pred_subset = torch.cat(pred_subset)

model.cpu()(x_subset)
symbols = model.log1.conceptizator.concepts
```

### Covariance matrix in the concept layer

```python
cov_objective = cov_objective.cpu()
cov_np = whitening_loss(model.cpu())[1].cpu().detach().numpy()

plt.figure()
sns.heatmap(cov_np)
plt.show()
```

## Visualize concepts

### Concept filters

```python
plt.figure(figsize=[10, 3])
max_c = 8
max_r = 2
for i in range(model.conv2.weight.size(0)):
    plt.subplot(max_r, max_c, i + 1)
    plt.title(f'concept {i}')
    plt.imshow(model.conv2.weight[i].sum(dim=0).detach().numpy())
    plt.axis('off')
plt.tight_layout()
plt.savefig('mnist_filters.png')
plt.show()
```

### Concept embeddings

```python
plt.figure(figsize=[10, 3])
max_c = 8
max_r = 2
for q, c in enumerate(torch.topk(symbols.abs(), k=1, dim=0)[1].T):
    plt.subplot(max_r, max_c, q + 1)
    plt.title(f'concept {q}')
    plt.imshow((model.filter_output[c, q] + 1).sum(dim=0).cpu().detach().numpy())
    plt.axis('off')
    if q >= (max_c * max_r - 1): break
plt.tight_layout()
plt.savefig('mnist_filters_output.png')
plt.show()
```

### Input images associated with concepts

```python
plt.figure(figsize=[10, 3])
max_c = 8
max_r = 2
for q, c in enumerate(torch.topk(symbols.abs(), k=100, dim=0)[1].T):
    plt.subplot(max_r, max_c, q + 1)
    plt.title(f'concept {q}')
    #     plt.imshow((x_subset[c, 0]+1).sum(dim=0).exp())
    plt.imshow((x_subset[c, 0] + 1).sum(dim=0))
    #     plt.figure(figsize=[10,2])
    #     for j, i in enumerate(c):
    #         plt.subplot(1, len(c), j+1)
    #         plt.imshow(x_subset[i, 0])
    plt.axis('off')
    if q >= (max_c * max_r - 1): break
plt.tight_layout()
plt.savefig('mnist_concepts.png')
plt.show()
```

# FOL formulas

```python
# threshold = 0.

# # positive class
# target_class = 1
# start = time.time()
# class_explanation, class_explanations = logic.explain_class(model.cpu(),
#                                                             x_subset,
#                                                             symbols>threshold,
#                                                             y_subset % 2,
#                                                             target_class=target_class,
#                                                             simplify=True,
#                                                             topk_explanations=5)
# elapsed_time = time.time() - start
# print(class_explanations['layer_5-neuron_1'])
# class_explanation = class_explanations['layer_5-neuron_1']

# # %%

# # # model.eval()
# # # x_subset, y_subset = next(iter(valid_loader))
# pred_subset = model.cpu()(x_subset)
# # # print(class_explanation)
# y_formula = lens.logic.metrics.predictions(class_explanation, target_class,
#                                          model.log1.conceptizator.concepts>threshold,
#                                          y_subset % 2, metric=accuracy_score)
# y_pred = pred_subset[:, 0].detach().cpu().numpy()
# y_true = (y_subset.to(device) % 2 < 0.5).detach().cpu().numpy()
# print(y_formula.shape)
# print(y_pred.shape)
# print(y_true.shape)

# # %%

# # concept_consistency, formula_consistency,
# accuracy_model = accuracy_score(y_pred>0.5, y_true)
# accuracy = accuracy_score(y_formula, y_true)
# fidelity = lens.logic.metrics.fidelity(y_formula>0.5, y_pred>0.5)
# complexity = lens.logic.metrics.complexity(class_explanation)
# print(f'Accuracy: {accuracy}')
# print(f'Fidelity: {fidelity}')
# print(f'Complexity: {complexity}')
```


# Expectations

## On scale

+ Given the parameters, constants, the period profit is around 1.5. Given that δ = 0.9, the value function should then be

$$ {1.5 \over (1- δ)} \approx 15$$

## On elasticities

+ As product becomes more inelastic, profit goes down. while prices go up
+ As product becomes more elastic, profit goes up, while prices go down

## On variance

+ With a risk neutral firm, it's not clear that a higher variance would reduce value
+ Higher variance should imply more variation in prices over time. Optimal price should vary with variance (fixing expected elasticity)

```python
%matplotlib inline

import matplotlib.pyplot as plt
import dill
import numpy as np
import pandas as pd
file_n = "2018-9-28vfi_dict.dill"  #"2018-9-22vfi_dict.dill"
with open('../data/' + file_n, 'rb') as file:
    data_d = dill.load(file)

import sys
sys.path.append('../')
import src

lambdas = src.generate_simplex_3dims(n_per_dim=data_d['n_of_lambdas_per_dim'])
price_grid = np.linspace(data_d['min_price'], data_d['max_price'])

policy = data_d['policy']
valueF = data_d['valueF']

lambdas_ext = src.generate_simplex_3dims(n_per_dim=30)
lambdas_ext.shape
```

```python
#Numbers taken from aguirragabiria_julia.ipynb

#Lambda1 become more relevant: which means demand becomes more elastic
#As product becomes more elastic, profit goes up
lambdas_selected = np.array([[0.0, 0.5, 0.5],
                    [0.1, 0.5, 0.4],
                    [0.2, 0.5, 0.3],
                    [0.3, 0.5, 0.2],
                    [0.4, 0.5, 0.1],
                    [0.5, 0.4, 0.1],
                    [0.6, 0.3, 0.1],
                    [0.7, 0.2, 0.1],
                    [0.8, 0.1, 0.1],
                    [0.9, 0.1, 0.0],
                    [1.0, 0.0, 0.0]])

valuef_values = valueF(lambdas_selected[:, 0:2])

fig, ax = plt.subplots(1, 1)
ax.plot(np.linspace(0, 1, num=11), valuef_values)
```

```python

```

## Analyze Value function as lambdas change value

Betas are given by : [-3. , -2.5, -2. ]

+ Thus, the first lambda is associated with -3 (more elastic)
+ The third one is associated with -2 (less elastic, more inelastic)

```python
lambdas_selected = np.array([[0.0, 0.5, 0.5],
                    [0.1, 0.5, 0.4],
                    [0.2, 0.5, 0.3],
                    [0.3, 0.5, 0.2],
                    [0.4, 0.5, 0.1],
                    [0.5, 0.4, 0.1],
                    [0.6, 0.3, 0.1],
                    [0.7, 0.2, 0.1],
                    [0.8, 0.1, 0.1],
                    [0.9, 0.1, 0.0],
                    [1.0, 0.0, 0.0]])

lambda1 = lambdas_selected[:, 0]
valuef_values = valueF(lambdas_selected[:, 0:2])

fig, ax = plt.subplots(1, 1)
ax.plot(lambda1, valuef_values)
ax.set_title("""Value function as the first lambda goes from 0 to 1
                \n (Demand becomes more elastic)
                """, size=16)
```

```python
lambdas_selected2 = np.array([[1., 0.],
                            [0.9, 0.1],
                            [0.8, 0.2],
                            [0.7, 0.3],
                            [0.6, 0.4],
                             [0.5, 0.5],
                             [0.4, 0.6],
                             [0.3, 0.7],
                             [0.2, 0.8],
                             [0.1, 0.9],
                             [0., 1.]
                            ])

lambda2 = lambdas_selected2[:, 1]
valuef_values = valueF(lambdas_selected2)

fig, ax = plt.subplots(1, 1)
ax.plot(lambda2, valuef_values)
ax.set_title("""Value function as the second lambda goes from 0 to 1
                \n (Demand becomes more inelastic)
                """, size=16)
```

## Analysis for myopic price and period profit

```python
from src import const

def period_profit(p, lambdas, betas_transition=const.betas_transition):
    """
    Not the right expected profit (expected value doesn't make epsilon go away)
    but it should be close enough
    """
    E_β = src.exp_b_from_lambdas(lambdas, betas_transition)
    logq = const.α + E_β*np.log(p)
    return (p-const.c)*np.e**logq

def myopic_price(lambdas: np.ndarray, betas_transition=const.betas_transition):
    """
    Given a lambda point, spits out optimal myopic price
    """
    #Elasticity implied by lambdas
    elasticity = np.dot(lambdas, betas_transition) #-2.2
    assert (elasticity < -1.0).all()
    return const.c / (1 + (1/elasticity))

def optimized_period_profit(lambdas):
    opt_p = myopic_price(lambdas)
    E_β = src.exp_b_from_lambdas(lambdas, const.betas_transition)
    logq = const.α + E_β*np.log(opt_p)

    return (opt_p-const.c)*np.e**logq
```

```python
lambdas_selected_3d = np.array([[0., 0.5, 0.5],
                            [0.1, 0.5, 0.4],
                            [0.2, 0.5, 0.2],
                            [0.3, 0.5, 0.2],
                            [0.4, 0.5, 0.1],
                             [0.5, 0.4, 0.1],
                             [0.6, 0.3, 0.1],
                             [0.7, 0.2, 0.1],
                             [0.8, 0.1, 0.1],
                             [0.9, 0.1, 0.],
                             [1., 0., 0.]
                            ])

lambda1 = lambdas_selected_3d[:, 0]
profit_values = optimized_period_profit(lambdas_selected_3d)

fig, ax = plt.subplots(1, 1)
ax.plot(lambda1, profit_values)
ax.set_title("Optimized period profit as lambda 1 goes from 0 to 1", size=16)
#from -2.2 to -3: As product becomes more elastic, prices go down. Profit goes up?
```

```python
E_β = src.exp_b_from_lambdas(lambdas_selected_3d, const.betas_transition)
```

```python
E_β
```

    array([-2.25, -2.35, -2.25, -2.55, -2.65, -2.7 , -2.75, -2.8 , -2.85,
           -2.95, -3.  ])

```python
const.betas_transition
```

    array([-3. , -2.5, -2. ])

```python
# Price goes down from  -2.2 to -3 (as product becomes more elastic)

lambda1 = lambdas_selected_3d[:, 0]
prices = myopic_price(lambdas_selected_3d, const.betas_transition)
fig, ax = plt.subplots(1, 1)
ax.plot(lambda1, prices)

ax.set_title("""
Prices over lambda1
\n
(as demand becomes more elastic)
""", size=15)
```

## Mathematical formulas for optimized price and profit

```python
from sympy import *
init_printing(use_unicode=True)
```

```python
p, c, α, β = symbols('p c α β', real=True)

logq = α + β*log(p)
q = exp(logq)
profit = (p-c)*q

optimal_price = solve(simplify(diff(profit, p)), p)[0]
optimal_price
```

```python
optimized_profit = (optimal_price - c)*exp(α + β*log(optimal_price))
simplify(diff(optimized_profit, β))
```

```python

def optimized_period_profit_over_elasticities(elasticities):
    opt_p = const.c*elasticities /(elasticities+1)
    logq = const.α + elasticities*np.log(opt_p)

    return (opt_p-const.c)*np.e**logq

def deriv_opt_profit_wrt_beta(β, α=1, c=0.5):
    num1 = c*np.exp(α + β*np.log( c*β /(β+1)  ))
    num2 = np.log(c*β /(β+1))
    num = num1*num2
    den = β + 1
    return -num/den

elasticities = np.linspace(-2.0, -5.0)
derivatives = deriv_opt_profit_wrt_beta(elasticities)
fig, [ax1, ax2] = plt.subplots(2, 1, figsize=(8, 10))
ax1.plot(elasticities, derivatives)
ax1.set_title("""
Derivative of Profit wrt elasticity. As product become more inelastic, price go up.\n
The derivative is always negative. So as product becomes more inelastic, value goes down""")

profits = optimized_period_profit_over_elasticities(elasticities)
ax2.plot(elasticities, profits)
ax2.set_title("""Profit wrt elasticity. As product becomes more inelastic, profit goes down.
              As product becomes more elastic, profit goes up.""")

fig.tight_layout()
```

```python
profit2 = (p-c)*exp(log(q))
mr = diff(profit, )
```

```python
def price_over_elast(elasticity):
    return const.c / (1 + (1/elasticity))
```

```python
elasticities = np.linspace(-2.0, -5.0)
prices = price_over_elast(elasticities)
fig, ax = plt.subplots(1, 1)
ax.plot(elasticities, prices)
ax.set_title("""
Prices. As product become more inelastic, price go up.\n
As product becomes more elastic, prices go down""")
```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## White Noise

### Definition

[White noise](https://en.wikipedia.org/wiki/White_noise) is a wide-sense stationary (WSS) random signal with constant power spectral density (PSD)

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = N_0
\end{equation}

where $N_0$ denotes the power per frequency. White noise draws its name from the analogy to white light. It refers typically to an idealized model of a random signal, e.g. emerging from measurement noise. The auto-correlation function (ACF) of white noise can be derived by inverse discrete-time Fourier transformation (DTFT) of the PSD

\begin{equation}
\varphi_{xx}[\kappa] = \mathcal{F}_*^{-1} \{ N_0 \} = N_0 \cdot \delta[\kappa]
\end{equation}

This result implies that white noise has to be a zero-mean random process. It can be concluded from the ACF that two neighboring samples $k$ and $k+1$ are uncorrelated. Hence they show no dependencies in the statistical sense. Although this is often assumed, the probability density function (PDF) of white noise is not necessarily given by the normal distribution. In general, it is required to additionally state the amplitude distribution when denoting a signal as white noise.

### Example - Amplifier Noise

Additive white Gaussian noise (AWGN) is often used as a model for amplifier noise. In order to evaluate if this holds for a typical audio amplifier, the noise $n[k]$ captured from a microphone preamplifier at full amplification with open connectors is analyzed statistically. For the remainder, a function is defined to estimate and plot the PDF and ACF of a given random signal.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats

def estimate_plot_pdf_acf(x, nbins=50, acf_range=30):
    '''Estimate and plot PDF/CDF of a given sample function.'''

    # compute and truncate ACF
    acf = 1/len(x) * np.correlate(x, x, mode='full')
    acf = acf[len(x)-acf_range-1:len(x)+acf_range-1]
    kappa = np.arange(-acf_range, acf_range)

    # plot PDF
    plt.figure(figsize=(10, 6))
    plt.subplot(121)
    plt.hist(x, nbins, density=True)
    plt.title('Estimated PDF')
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$\hat{p}_n(\theta)$')
    plt.grid()

    # plot ACF
    plt.subplot(122)
    plt.stem(kappa, acf, use_line_collection=True)
    plt.title('Estimated ACF')
    plt.ylabel(r'$\hat{\varphi}_{nn}[\kappa]$')
    plt.xlabel(r'$\kappa$')
    plt.axis([-acf_range, acf_range, 1.1*min(acf), 1.1*max(acf)])
    plt.grid()
```

Now the pre-captured noise is loaded and analyzed

```python
noise = np.load('../data/amplifier_noise.npz')['noise']
estimate_plot_pdf_acf(noise, nbins=100, acf_range=150)
```

Inspecting the PDF reveals that it fits quite well to a [normal distribution](important_distributions.ipynb#Normal-Distribution). The ACF consists of a pronounced peak. from which can be concluded that the samples are approximately uncorrelated. Hence, the amplifier noise can be modeled reasonably well as additive white Gaussian noise. The parameters of the normal distribution (mean $\mu_n$, variance $\sigma_n^2$) are estimated as

```python
mean = np.mean(noise)
variance = np.var(noise)

print('Mean:\t\t {0:1.3e} \nVariance:\t {1:1.3e}'.format(mean, variance))
```

    Mean:		 -1.537e-05
    Variance:	 1.140e-07

**Excercise**

* What relative level does the amplifier noise have when the maximum amplitude of the amplifier is assumed to be $\pm 1$?

Solution: The average power of a mean-free random signal is given by its variance, here $\sigma_\text{n}^2$. Due to the very low mean in comparison to the maximum amplitude, the noise can be assumed to be mean-free. Hence, the relative level of the noise is then given as $10*\log_{10}\left( \frac{\sigma_\text{n}^2}{1} \right)$. Numerical evaluation yields

```python
print('Level of amplifier noise: {:2.2f} dB'.format(10*np.log10(variance/1)))
```

    Level of amplifier noise: -69.43 dB

### Example - Generation of White Noise with Different Amplitude Distributions

Toolboxes for numerical mathematics like `Numpy` or `scipy.stats` provide functions to draw uncorrelated random samples with a given amplitude distribution.

**Uniformly distributed white noise**

For samples drawn from a zero-mean random process with uniform amplitude distribution, the PDF and ACF are estimated as

```python
np.random.seed(3)
estimate_plot_pdf_acf(np.random.uniform(size=10000)-1/2)
```

Lets listen to uniformly distributed white noise

```python
from scipy.io import wavfile
fs = 44100

x = np.random.uniform(size=5*fs)-1/2
wavfile.write('uniform_white_noise.wav', fs, np.int16(x*32768))
```

<audio src="./uniform_white_noise.wav" controls>Your browser does not support the audio element.</audio>[./uniform_white_noise.wav](./uniform_white_noise.wav)

**Laplace distributed white noise**

For samples drawn from a zero-mean random process with with Laplace amplitude distribution, the PDF and ACF are estimated as

```python
estimate_plot_pdf_acf(np.random.laplace(size=10000, loc=0, scale=1/np.sqrt(2)))
```

**Exercise**

* Do both random processes represent white noise?
* Estimate the power spectral density $N_0$ of both examples.
* How does the ACF change if you lower the length `size` of the random signal. Why?

Solution: Both processes represent white noise since the ACF can be approximated reasonably well as Dirac impulse $\delta[\kappa]$. The weight of the Dirac impulse is equal to $N_0$. In case of the uniformly distributed white noise $N_0 \approx \frac{1}{12}$, in case of the Laplace distributed white noise $N_0 \approx 1$. Decreasing the length `size` of the signal increases the statistical uncertainties in the estimate of the ACF.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.