# Loops

## for Loops

A [for loop](https://docs.python.org/3/reference/compound_stmts.html#for) allows us to execute a block of code multiple times with some parameters updated each time through the loop. A `for` loop begins with the `for` statement:

```python
iterable = [1,2,3]
for item in iterable:
    # code block indented 4 spaces
    print(item)
```

    1
    2
    3

The main points to observe are:

* `for` and `in` keywords
* `iterable` is a sequence object such as a list, tuple or range
* `item` is a variable which takes each value in `iterable`
* end `for` statement with a colon `:`
* code block indented 4 spaces which executes once for each value in `iterable`

For example, let's print $n^2$ for $n$ from 0 to 5:

```python
for n in [0,1,2,3,4,5]:
    square = n**2
    print(n,'squared is',square)
print('The for loop is complete!')
```

    0 squared is 0
    1 squared is 1
    2 squared is 4
    3 squared is 9
    4 squared is 16
    5 squared is 25
    The for loop is complete!

Copy and paste this code and any of the examples below into the [Python visualizer](http://www.pythontutor.com/visualize.html#mode=edit) to see each step in a `for` loop!

## while Loops

What if we want to execute a block of code multiple times but we don't know exactly how many times? We can't write a `for` loop because this requires us to set the length of the loop in advance. This is a situation when a [while loop](https://en.wikipedia.org/wiki/While_loop#Python) is useful.

The following example illustrates a [while loop](https://docs.python.org/3/tutorial/introduction.html#first-steps-towards-programming):

```python
n = 5
while n > 0:
    print(n)
    n = n - 1
```

    5
    4
    3
    2
    1

The main points to observe are:

* `while` keyword
* a logical expression followed by a colon `:`
* loop executes its code block if the logical expression evaluates to `True`
* update the variable in the logical expression each time through the loop
* **BEWARE!** If the logical expression *always* evaluates to `True`, then you get an [infinite loop](https://en.wikipedia.org/wiki/While_loop#Python)!

We prefer `for` loops over `while` loops because of the last point. A `for` loop will never result in an infinite loop. If a loop can be constructed with `for` or `while`, we'll always choose `for`.

## Constructing Sequences

There are several ways to construct a sequence of values and to save them as a Python list. We have already seen Python's list comprehension syntax. There is also the `append` list method described below.

### Sequences by a Formula

If a sequence is given by a formula then we can use a list comprehension to construct it. For example, the sequence of squares from 1 to 100 can be constructed using a list comprehension:

```python
squares = [d**2 for d in range(1,11)]
print(squares)
```

    [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

However, we can achieve the same result with a `for` loop and the `append` method for lists:

```python
# Intialize an empty list
squares = []
for d in range(1,11):
    # Append the next square to the list
    squares.append(d**2)
print(squares)
```

    [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

In fact, the two examples above are equivalent. The purpose of list comprehensions is to simplify and compress the syntax into a one-line construction.

### Recursive Sequences

We can only use a list comprehension to construct a sequence when the sequence values are defined by a formula. But what if we want to construct a sequence where the next value depends on previous values? This is called a [recursive sequence](https://en.wikipedia.org/wiki/Recursion).

For example, consider the [Fibonacci sequence](https://en.wikipedia.org/wiki/Fibonacci_number):

$$
x_1 = 1, x_2 = 1, x_3 = 2, x_4 = 3, x_5 = 5, ...
$$

where

$$
x_{n} = x_{n-1} + x_{n-2}
$$

We can't use a list comprehension to build the list of Fibonacci numbers, and so we must use a `for` loop with the `append` method instead. For example, the first 15 Fibonacci numbers are:

```python
fibonacci_numbers = [1,1]
for n in range(2,15):
    fibonacci_n = fibonacci_numbers[n-1] + fibonacci_numbers[n-2]
    fibonacci_numbers.append(fibonacci_n)
    print(fibonacci_numbers)
```

    [1, 1, 2]
    [1, 1, 2, 3]
    [1, 1, 2, 3, 5]
    [1, 1, 2, 3, 5, 8]
    [1, 1, 2, 3, 5, 8, 13]
    [1, 1, 2, 3, 5, 8, 13, 21]
    [1, 1, 2, 3, 5, 8, 13, 21, 34]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]
    [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610]

## Computing Sums

Suppose we want to compute the sum of a sequence of numbers $x_0$, $x_1$, $x_2$, $x_3$, $\dots$, $x_n$. There are at least two approaches:

1. Compute the entire sequence, store it as a list $[x_0,x_1,x_2,\dots,x_n]$ and then use the built-in function `sum`.
2. Initialize a variable with value 0 (and name it `result` for example), create and add each element in the sequence to `result` one at a time.

The advantage of the second approach is that we don't need to store all the values at once. For example, here are two ways to write a function which computes the sum of squares.

For the first approach, use a list comprehension:

```python
def sum_of_squares_1(N):
    "Compute the sum of squares 1**2 + 2**2 + ... + N**2."
    return sum([n**2 for n in range(1,N + 1)])
```

```python
sum_of_squares_1(4)
```

    30

For the second approach, use a `for` loop with the initialize-and-update construction:

```python
def sum_of_squares_2(N):
    "Compute the sum of squares 1**2 + 2**2 + ... + N**2."
    # Initialize the output value to 0
    result = 0
    for n in range(1,N + 1):
        # Update the result by adding the next term
        result = result + n**2
    return result
```

```python
sum_of_squares_2(4)
```

    30

Again, both methods yield the same result however the second uses less memory!

## Computing Products

There is no built-in function to compute products of sequences therefore we'll use an initialize-and-update construction similar to the example above for computing sums.

Write a function called `factorial` which takes a positive integer $N$ and return the factorial $N!$.

```python
def factorial(N):
    "Compute N! = N(N-1) ... (2)(1) for N >= 1."
    # Initialize the output variable to 1
    product = 1
    for n in range(2,N + 1):
        # Update the output variable
        product = product * n
    return product
```

Let's test our function for input values for which we know the result:

```python
factorial(2)
```

    2

```python
factorial(5)
```

    120

We can use our function to approximate $e$ using the Taylor series for $e^x$:

$$
e^x = \sum_{k=0}^{\infty} \frac{x^k}{k!}
$$

For example, let's compute the 100th partial sum of the series with $x=1$:

```python
sum([1/factorial(k) for k in range(0,101)])
```

    2.7182818284590455

## Searching for Solutions

We can use `for` loops to search for integer solutions of equations. For example, suppose we would like to find all representations of a positive integer $N$ as a [sum of two squares](https://en.wikipedia.org/wiki/Sum_of_two_squares_theorem). In other words, we want to find all integer solutions $(x,y)$ of the equation:

$$
x^2 + y^2 = N
$$

Write a function called `reps_sum_squares` which takes an integer $N$ and finds all representations of $N$ as a sum of squares $x^2 + y^2 = N$ for $0 \leq x \leq y$. The function returns the representations as a list of tuples. For example, if $N = 50$ then $1^2 + 7^2 = 50$ and $5^2 + 5^2 = 50$ and the function returns the list `[(1, 7),(5, 5)]`.

Let's outline our approach before we write any code:

1. Given $x \leq y$, the largest possible value for $x$ is $\sqrt{\frac{N}{2}}$
2. For $x \leq \sqrt{\frac{N}{2}}$, the pair $(x,y)$ is a solution if $N - x^2$ is a square
3. Define a helper function called `is_square` to test if an integer is square

```python
def is_square(n):
    "Determine if the integer n is a square."
    if round(n**0.5)**2 == n:
        return True
    else:
        return False

def reps_sum_squares(N):
    '''Find all representations of N as a sum of squares x**2 + y**2 = N.

    Parameters
    ----------
    N : integer

    Returns
    -------
    reps : list of tuples of integers
        List of tuples (x,y) of positive integers such that x**2 + y**2 = N.

    Examples
    --------
    >>> reps_sum_squares(1105)
    [(4, 33), (9, 32), (12, 31), (23, 24)]
    '''
    reps = []
    if is_square(N/2):
        # If N/2 is a square, search up to x = (N/2)**0.5
        max_x = round((N/2)**0.5)
    else:
        # If N/2 is not a square, search up to x = floor((N/2)**0.5)
        max_x = int((N/2)**0.5)
    for x in range(0,max_x + 1):
        y_squared = N - x**2
        if is_square(y_squared):
            y = round(y_squared**0.5)
            # Append solution (x,y) to list of solutions
            reps.append((x,y))
    return reps
```

```python
reps_sum_squares(1105)
```

    [(4, 33), (9, 32), (12, 31), (23, 24)]

What is the smallest integer which can be expressed as the sum of squares in 5 different ways?

```python
N = 1105
num_reps = 4
while num_reps < 5:
    N = N + 1
    reps = reps_sum_squares(N)
    num_reps = len(reps)
print(N,':',reps_sum_squares(N))
```

    4225 : [(0, 65), (16, 63), (25, 60), (33, 56), (39, 52)]

## Examples

### Prime Numbers

A positive integer is [prime](https://en.wikipedia.org/wiki/Prime_number) if it is divisible only by 1 and itself. Write a function called `is_prime` which takes an input parameter `n` and returns `True` or `False` depending on whether `n` is prime or not.

Let's outline our approach before we write any code:

1. An integer $d$ divides $n$ if there is no remainder of $n$ divided by $d$.
2. Use the modulus operator `%` to compute the remainder.
3. If $d$ divides $n$ then $n = d q$ for some integer $q$ and either $d \leq \sqrt{n}$ or $q \leq \sqrt{n}$ (and not both), therefore we need only test if $d$ divides $n$ for integers $d \leq \sqrt{n}$

```python
def is_prime(n):
    "Determine whether or not n is a prime number."
    if n <= 1:
        return False
    # Test if d divides n for d <= n**0.5
    for d in range(2,round(n**0.5) + 1):
        if n % d == 0:
            # n is divisible by d and so n is not prime
            return False
    # If we exit the for loop, then n is not divisible by any d
    # and therefore n is prime
    return True
```

Let's test our function on the first 30 numbers:

```python
for n in range(0,31):
    if is_prime(n):
        print(n,'is prime!')
```

    2 is prime!
    3 is prime!
    5 is prime!
    7 is prime!
    11 is prime!
    13 is prime!
    17 is prime!
    19 is prime!
    23 is prime!
    29 is prime!

Our function works! Let's find all the primes between 20,000 and 20,100.

```python
for n in range(20000,20100):
    if is_prime(n):
        print(n,'is prime!')
```

    20011 is prime!
    20021 is prime!
    20023 is prime!
    20029 is prime!
    20047 is prime!
    20051 is prime!
    20063 is prime!
    20071 is prime!
    20089 is prime!

### Divisors

Let's write a function called `divisors` which takes a positive integer $N$ and returns the list of positive integers which divide $N$.

```python
def divisors(N):
    "Return the list of divisors of N."
    # Initialize the list of divisors (which always includes 1)
    divisor_list = [1]
    # Check division by d for d <= N/2
    for d in range(2,N // 2 + 1):
        if N % d == 0:
            divisor_list.append(d)
    # N divides itself and so we append N to the list of divisors
    divisor_list.append(N)
    return divisor_list
```

Let's test our function:

```python
divisors(10)
```

    [1, 2, 5, 10]

```python
divisors(100)
```

    [1, 2, 4, 5, 10, 20, 25, 50, 100]

```python
divisors(59)
```

    [1, 59]

### Collatz Conjecture

Let $a$ be a positive integer and consider the recursive sequence where $x_0 = a$ and

$$
x_{n+1} = \left\\{ \begin{array}{cl} x_n/2 & \text{if } x_n \text{ is even} \\\\ 3x_n+1 & \text{if } x_n \text{ is odd}  \end{array} \\right.
$$

The [Collatz conjecture](https://en.wikipedia.org/wiki/Collatz_conjecture) states that this sequence will *always* reach 1. For example, if $a = 10$ then $x_0 = 10$, $x_1 = 5$, $x_2 = 16$, $x_3 = 8$, $x_4 = 4$, $x_5 = 2$ and $x_6 = 1$.

Write a function called `collatz` which takes one input parameter `a` and returns the sequence of integers defined above and ending with the first occurrence $x_n=1$.

```python
def collatz(a):
    "Compute the Collatz sequence starting at a and ending at 1."
    # Initialize list with first value a
    sequence = [a]
    # Compute values until we reach 1
    while sequence[-1] > 1:
        # Check if the last element in the list is even
        if sequence[-1] % 2 == 0:
            # Compute and append the new value
            sequence.append(sequence[-1] // 2)
        else:
            # Compute and append the new value
            sequence.append(3*sequence[-1] + 1)
    return sequence
```

Let's test our function:

```python
print(collatz(10))
```

    [10, 5, 16, 8, 4, 2, 1]

```python
collatz(22)
```

    [22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5, 16, 8, 4, 2, 1]

The Collatz conjecture is quite amazing. No matter where we start, the sequence always terminates at 1!

```python
a = 123456789
seq = collatz(a)
print("Collatz sequence for a =",a)
print("begins with",seq[:5])
print("ends with",seq[-5:])
print("and has",len(seq),"terms.")
```

    Collatz sequence for a = 123456789
    begins with [123456789, 370370368, 185185184, 92592592, 46296296]
    ends with [16, 8, 4, 2, 1]
    and has 178 terms.

Which $a < 1000$ produces the longest sequence?

```python
max_length = 1
a_max = 1
for a in range(1,1001):
    seq_length = len(collatz(a))
    if seq_length > max_length:
        max_length = seq_length
        a_max = a
print('Longest sequence begins with a =',a_max,'and has length',max_length)
```

    Longest sequence begins with a = 871 and has length 179

## Exercises

1. [Fermat's theorem on the sum of two squares](https://en.wikipedia.org/wiki/Fermat%27s_theorem_on_sums_of_two_squares) states that every prime number $p$ of the form $4k+1$ can be expressed as the sum of two squares. For example, $5 = 2^2 + 1^2$ and $13 = 3^2 + 2^2$. Find the smallest prime greater than $2019$ of the form $4k+1$ and write it as a sum of squares. (Hint: Use the functions `is_prime` and `reps_sum_squares` from this section.)

2. What is the smallest prime number which can be represented as a sum of squares in 2 different ways?

3. What is the smallest integer which can be represented as a sum of squares in 3 different ways?

4. Write a function called `primes_between` which takes two integer inputs $a$ and $b$ and returns the list of primes in the closed interval $[a,b]$.

5. Write a function called `primes_d_mod_N` which takes four integer inputs $a$, $b$, $d$ and $N$ and returns the list of primes in the closed interval $[a,b]$ which are congruent to $d$ mod $N$ (this means that the prime has remainder $d$ after division by $N$). This kind of list is called [primes in an arithmetic progression](https://en.wikipedia.org/wiki/Dirichlet%27s_theorem_on_arithmetic_progressions).

6. Write a function called `reciprocal_recursion` which takes three positive integers $x_0$, $x_1$ and $N$ and returns the sequence $[x_0,x_1,x_2,\dots,x_N]$ where

    $$
    x_n = \frac{1}{x_{n-1}} + \frac{1}{x_{n-2}}
    $$

7. Write a function called `root_sequence` which takes input parameters $a$ and $N$, both positive integers, and returns the $N$th term $x_N$ in the sequence:

    $$
    \begin{align}
    x_0 &= a \\\
    x_n &= 1 + \sqrt{x_{n-1}}
    \end{align}
    $$

    Does the sequence converge to different values for different starting values $a$?

8. Write a function called `fib_less_than` which takes one input $N$ and returns the list of Fibonacci numbers less than $N$.

9. Write a function called `fibonacci_primes` which takes an input parameter $N$ and returns the list of Fibonacci numbers less than $N$ which are also prime numbers.

10. Let $w(N)$ be the number of ways $N$ can be expressed as a sum of two squares $x^2 + y^2 = N$ with $1 \leq x \leq y$. Then

    $$
    \lim_{N \to \infty} \frac{1}{N} \sum_{n=1}^{N} w(n) = \frac{\pi}{8}
    $$

    Compute the left side of the formula for $N=100$ and compare the result to $\pi / 8$.

11. A list of positive integers $[a,b,c]$ (with $1 \leq a < b$) are a [Pythagorean triple](https://en.wikipedia.org/wiki/Pythagorean_triple) if $a^2 + b^2 = c^2$. Write a function called `py_triples` which takes an input parameter $N$ and returns the list of Pythagorean triples `[a,b,c]` with $c \leq N$.


```python
%matplotlib notebook

import pandas as pd
import numpy as np
import sklearn.datasets

import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = [8, 4]
```

```python
iris = sklearn.datasets.load_iris()

X = pd.DataFrame(iris.data, columns=iris.feature_names)

X_zscaled = (X - X.mean()) / X.std(ddof=1)

Y = pd.DataFrame(iris.target, columns=['target'])
Y['species'] = Y.apply(lambda r: iris.target_names[r])
```

## Multicollinearity Check

Using $corr(X)$, check to see that we some level of multicollinearity in the data, enough to warrant using PCA of visualization in reduced-rank principal component space. As a rule of thumb, the off-diagonal correlation values (either in the upper- or lower-triangle) should have absolute values of around 0.30 or so.

```python
corr = X_zscaled.corr()

tmp = pd.np.triu(corr) - np.eye(corr.shape[0])
tmp = tmp.flatten()
tmp = tmp[np.nonzero(tmp)]
tmp = pd.Series(np.abs(tmp))

print('Correlation matrix:\n\n{}\n\n'.format(corr.values))

print('Multicollinearity check using off-diagonal values:\n\n{}'.format(tmp.describe()))
```

    Correlation matrix:

    [[ 1.         -0.10936925  0.87175416  0.81795363]
     [-0.10936925  1.         -0.4205161  -0.35654409]
     [ 0.87175416 -0.4205161   1.          0.9627571 ]
     [ 0.81795363 -0.35654409  0.9627571   1.        ]]

    Multicollinearity check using off-diagonal values:

    count    6.000000
    mean     0.589816
    std      0.341915
    min      0.109369
    25%      0.372537
    50%      0.619235
    75%      0.858304
    max      0.962757
    dtype: float64

## PCA via Eigen-Decomposition

### Obtain eigenvalues, eigenvectors of $cov(X)$ via eigen-decomposition

From the factorization of symmetric matrix $S$ into orthogonal matrix $Q$ of the eigenvectors and diagonal matrix $\Lambda$ of the eigenvalues, we can likewise decompose $cov(X)$ (or in our case $corr(X)$ since we standardized our data).

\begin{align}
  S &= Q \Lambda Q^\intercal \\
  \Rightarrow X^\intercal X &= V \Lambda V^\intercal \\
\end{align}

where $V$ are the orthonormal eigenvectors of $X X^\intercal$.

We can normalize the eigenvalues to see how much variance is captured per each respective principal component. We will also calculate the cumulative variance explained. This will help inform our decision of how many principal components to keep when reducing dimensions in visualizing the data.

```python
eigenvalues, eigenvectors = np.linalg.eig(X_zscaled.cov())

eigenvalues_normalized = eigenvalues / eigenvalues.sum()

cumvar_explained = np.cumsum(eigenvalues_normalized)
```

### Reduce dimensions and visualize

To project the original data into principal component space, we obtain score matrix $T$ by taking the dot product of $X$ and the eigenvectors $V$.

\begin{align}
  T &= X V \\
\end{align}

```python
T = pd.DataFrame(X_zscaled.dot(eigenvectors))

# set column names
T.columns = ['pc1', 'pc2', 'pc3', 'pc4']

# also add the species label as
T = pd.concat([T, Y.species], axis=1)
```

We can visualize the original, 4D iris data of $X$ by using the first $k$ eigenvectors of $X$, projecting the original data into a reduced-rank $k$-dimensional principal component space.

\begin{align}
  T_{rank=k} &= X V_{rank=k}
\end{align}

```python
# let's try using the first 2 principal components
k = 2

# divide T by label
irises = [T[T.species=='setosa'],
          T[T.species=='versicolor'],
          T[T.species=='virginica']]

# define a color-blind friendly set of quantative colors
# for each species
colors = ['#1b9e77', '#d95f02', '#7570b3']

_, (ax1, ax2) = plt.subplots(1, 2, sharey=False)

# plot principal component vis-a-vis total variance in data
ax1.plot([1,2,3,4],
         eigenvalues_normalized,
         '-o',
         color='#8da0cb',
         label='eigenvalue (normalized)',
         alpha=0.8,
         zorder=1000)

ax1.plot([1,2,3,4],
         cumvar_explained,
         '-o',
         color='#fc8d62',
         label='cum. variance explained',
         alpha=0.8,
         zorder=1000)

ax1.set_xlim(0.8, 4.2)
ax1.set_xticks([1,2,3,4])
ax1.set_xlabel('Principal component')
ax1.set_ylabel('Variance')
ax1.legend(loc='center right', fontsize=7)
ax1.grid(color='#fdfefe')
ax1.set_facecolor('#f4f6f7')

# plot the reduced-rank score matrix representation
for group, color in zip(irises, colors):
    ax2.scatter(group.pc1,
                group.pc2,
                marker='^',
                color=color,
                label=group.species,
                alpha=0.5,
                zorder=1000)
ax2.set_xlabel(r'PC 1')
ax2.set_ylabel(r'PC 2')
ax2.grid(color='#fdfefe')
ax2.set_facecolor('#f4f6f7')
ax2.legend(labels=iris.target_names, fontsize=7)

plt.suptitle(r'Fig. 1: PCA via eigen-decomposition, 2D visualization')
plt.tight_layout(pad=3.0)
plt.show()
```

    <IPython.core.display.Javascript object>

### Relative weights of the original features in principal component space

* Each row of $V$ is a vector representing the relative weights for each of the original features for each principal component.
* Given reduced-rank $k$, the columns of $V_k$ are the weights for principal components $1, \dots, k$.
* Calculate the norm for each row of $V_k$, and normalize the results to obtain the relative weights for each original feature in principal component space.

```python
feature_norms = np.linalg.norm(eigenvectors[:, 0:k], axis=1)
feature_weights = feature_norms / feature_norms.sum()

msg = ('Using {} principal components, '
       'the original features are represented with the following weights:')
print(msg.format(k))
for feature, weight in zip(iris.feature_names, feature_weights):
    print('-  {}: {:0.3f}'.format(feature, weight))
```

    Using 2 principal components, the original features are represented with the following weights:
    -  sepal length (cm): 0.233
    -  sepal width (cm): 0.349
    -  petal length (cm): 0.211
    -  petal width (cm): 0.207

## PCA via Singular Value Decomposition

Singular value decomposition factors any matrix $A$ into right-singular vector matrix $U$; diagonal matrix of singular values $\Sigma$; and right-singular vector matrix $V$.

\begin{align}
  A &= U \Sigma V^\intercal \\
\end{align}

If we start with

\begin{align}
  X &= U \Sigma V^\intercal \\
  \\
  X^\intercal X &= (U \Sigma V^\intercal)^\intercal U \Sigma V^\intercal \\
  &= V \Sigma^\intercal U^\intercal U \Sigma V^\intercal \\
  &= V \Sigma^\intercal \Sigma V^\intercal \\
  \\
  \Rightarrow \Sigma^\intercal \Sigma &= \Lambda
\end{align}

```python
U, S, Vt = np.linalg.svd(X_zscaled)
```

```python
print(eigenvectors)
```

    [[ 0.52237162 -0.37231836 -0.72101681  0.26199559]
     [-0.26335492 -0.92555649  0.24203288 -0.12413481]
     [ 0.58125401 -0.02109478  0.14089226 -0.80115427]
     [ 0.56561105 -0.06541577  0.6338014   0.52354627]]

```python
print(Vt.T)
```

    [[ 0.52237162 -0.37231836  0.72101681  0.26199559]
     [-0.26335492 -0.92555649 -0.24203288 -0.12413481]
     [ 0.58125401 -0.02109478 -0.14089226 -0.80115427]
     [ 0.56561105 -0.06541577 -0.6338014   0.52354627]]

```python
Vt.T.shape
```

    (4, 4)

```python

```


```python
import sympy as sm
```

```python
from sympy.physics.vector import init_vprinting
init_vprinting(use_latex='mathjax', pretty_print=False)
```

```python
from IPython.display import Image
Image('fig/2rp_new.png', width=300)
```

```python
from sympy.physics.mechanics import dynamicsymbols
```

```python
theta1, theta2, l1, l2 = dynamicsymbols('theta1 theta2 l1 l2')
theta1, theta2, l1, l2
```

$\displaystyle \left( \theta_{1}, \  \theta_{2}, \  l_{1}, \  l_{2}\right)$

```python
px = l1*sm.cos(theta1) + l2*sm.cos(theta1 + theta2) # tip psition in x-direction
py = l1*sm.sin(theta1) + l2*sm.sin(theta1 + theta2) # tip position in y-direction
```

```python
# evaluating the jacobian matrix
a11 = sm.diff(px, theta1) # differentiate px with theta_1
a12 = sm.diff(px, theta2) # differentiate px with theta_2

a21 = sm.diff(py, theta1) # differentiate py with theta_1
a22 = sm.diff(py, theta2) # differentiate py with theta_2

J = sm.Matrix([[a11, a12], [a21, a22]]) # assemble into matix form
Jsim = sm.simplify(J) # simplified result
Jsim
```

$\displaystyle \left[\begin{matrix}- l_{1} \sin{\left(\theta_{1} \right)} - l_{2} \sin{\left(\theta_{1} + \theta_{2} \right)} & - l_{2} \sin{\left(\theta_{1} + \theta_{2} \right)}\\l_{1} \cos{\left(\theta_{1} \right)} + l_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} & l_{2} \cos{\left(\theta_{1} + \theta_{2} \right)}\end{matrix}\right]$

```python
# Manipulator singularities

Jdet = sm.det(Jsim) #determinant of the jacobian matrix
detJ = sm.simplify(Jdet)
detJ
```

$\displaystyle l_{1} l_{2} \sin{\left(\theta_{2} \right)}$

```python
sm.solve(detJ, (theta2)) # slove detJ for theta_2
```

$\displaystyle \left[ 0, \  \pi\right]$

```python
# This means the manipulator will be in singular configuration when the angle  θ2  is either zero or it is  ±π ,
Image('fig/2rp_sing_config1.png', width=300)  # θ2=0
```

```python
Image('fig/2rp_sing_config2.png', width=300) # θ2=±π
```

```python

```


```python
import sympy
from sympy import I, pi, oo
import numpy as np
sympy.init_printing(backcolor="Transparent")
```

```python
x = sympy.Symbol("x")
x.is_real is None
```

    True

```python
y = sympy.Symbol("y", real=True)
y.is_real
```

    True

```python
z = sympy.Symbol("z", imaginary=True)
z.is_real
```

    False

```python
x = sympy.Symbol("x")
y = sympy.Symbol("y", positive=True)
```

```python
sympy.sqrt(x ** 2)
```

```python
sympy.sqrt(y ** 2)
```

```python
n1 = sympy.Symbol("n")
n2 = sympy.Symbol("n", integer=True)
n3 = sympy.Symbol("n", odd=True)
```

```python
sympy.cos(n1 * pi)
```

```python
sympy.cos(n2 * pi)
```

```python
sympy.cos(n3 * pi)
```

```python
a, b, c = sympy.symbols("a, b, c", negative=True)
d, e, f = sympy.symbols("d, e, f", positive=True)
```

### Numbers

```python
i = sympy.Integer(19)
type(i)
```

    sympy.core.numbers.Integer

```python
i.is_Integer, i.is_real, i.is_odd
```

    (True, True, True)

```python
f = sympy.Float(2.3)
type(f)
```

    sympy.core.numbers.Float

```python
f.is_Integer, f.is_real, f.is_odd
```

    (False, True, False)

```python
i, f = sympy.sympify(19), sympy.sympify(2.3)
type(i), type(f)
```

    (sympy.core.numbers.Integer, sympy.core.numbers.Float)

```python
n = sympy.Symbol("n", integer=True)
```

```python
n.is_integer, n.is_Integer, n.is_positive, n.is_Symbol
```

    (True, False, None, True)

```python
i = sympy.Integer(19)
```

```python
i.is_integer, i.is_Integer, i.is_positive, i.is_Symbol
```

    (True, True, True, False)

```python
sympy.factorial(100)
```

```python
sympy.Float(0.3, 25)
```

```python
sympy.Rational(11, 13)
```

```python
r1 = sympy.Rational(2, 3)
r2 = sympy.Rational(4, 5)
```

```python
r1 * r2
```

```python
r1 / r2
```

### Functions

```python
x, y, z = sympy.symbols("x, y, z")
f = sympy.Function("f")
type(f)
```

    sympy.core.function.UndefinedFunction

```python
f(x)
```

```python
g = sympy.Function("g")(x, y, z)
g
```

```python
g.free_symbols
```

```python
sympy.sin
```

    sin

```python
sympy.sin(x)
```

```python
sympy.sin(pi * 1.5)
```

```python
n = sympy.Symbol("n", integer=True)
sympy.sin(pi * n)
```

```python
h = sympy.Lambda(x, x**2)
h
```

```python
h(5)
```

```python
h(1 + x)
```

```python
x = sympy.Symbol("x")
expr = 1 + 2 * x**2 + 3 * x**3
expr
```

### Simplify

```python
expr = 2 * (x**2 - x) - x * (x + 1)
expr
```

```python
sympy.simplify(expr)
```

```python
expr.simplify()
```

```python
expr
```

```python
expr = 2 * sympy.cos(x) * sympy.sin(x)
expr
```

```python
sympy.simplify(expr)
```

```python
expr = sympy.exp(x) * sympy.exp(y)
expr
```

```python
sympy.simplify(expr)
```

### Expand

```python
expr = (x + 1) * (x + 2)
sympy.expand(expr)
```

```python
sympy.sin(x + y).expand(trig=True)
```

```python
a, b = sympy.symbols("a, b", positive=True)
sympy.log(a * b).expand(log=True)
```

```python
sympy.exp(I*a + b).expand(complex=True)
```

```python
sympy.expand((a * b)**x, power_base=True)
```

```python
sympy.exp((a-b)*x).expand(power_exp=True)
```

### Factor, Collect and Combine

```python
sympy.factor(x**2 - 1)
```

```python
sympy.factor(x * sympy.cos(y) + sympy.sin(z) * x)
```

```python
sympy.logcombine(sympy.log(a) - sympy.log(b))
```

```python
expr = x + y + x * y * z
expr.collect(x)
```

```python
expr.collect(y)
```

```python
expr = sympy.cos(x + y) + sympy.sin(x - y)
expr.expand(trig=True).collect([
    sympy.cos(x),sympy.sin(x)
]).collect(sympy.cos(y) - sympy.sin(y))
```

```python
sympy.apart(1/(x**2 + 3*x + 2), x)
```

```python
sympy.together(1 / (y * x + y) + 1 / (1+x))
```

```python
sympy.cancel(y / (y * x + y))
```

```python
(x + y).subs(x, y)
```

```python
sympy.sin(x * sympy.exp(x)).subs(x, y)
```

```python
sympy.sin(x * z).subs({z: sympy.exp(y), x: y, sympy.sin: sympy.cos})
```

```python
sympy.sin(x * z).subs({z: sympy.exp(y), x: y, sympy.sin: sympy.cos})
```

```python
expr = x * y + z**2 *x
values = {x: 1.25, y: 0.4, z: 3.2}
expr.subs(values)
```

### Numerical evaluation

```python
sympy.N(1 + pi)
```

```python
sympy.N(pi, 50)
```

```python
(x + 1/pi).evalf(10)
```

```python
expr = sympy.sin(pi * x * sympy.exp(x))
[expr.subs(x, xx).evalf(3) for xx in range(0, 10)]
```

```python
expr_func = sympy.lambdify(x, expr)
expr_func(1.0)
```

```python
expr_func = sympy.lambdify(x, expr, 'numpy')
xvalues = np.arange(0, 10)
expr_func(xvalues)
```

    array([ 0.        ,  0.77394269,  0.64198244,  0.72163867,  0.94361635,
            0.20523391,  0.97398794,  0.97734066, -0.87034418, -0.69512687])

### Calculus

```python
f = sympy.Function('f')(x)
sympy.diff(f, x)
```

```python
sympy.diff(f, x, x)
```

```python
sympy.diff(f, x, 3)
```

```python
g = sympy.Function('g')(x, y)
g.diff(x, y)
```

```python
g.diff(x, 3, y, 2)
```

```python
expr = x**4 + x**3 + x**2 + x + 1
expr.diff(x)
```

```python
expr.diff(x, x)
```

```python
expr = (x + 1)**3 * y ** 2 * (z - 1)
expr.diff(x, y, z)
```

```python
expr = sympy.sin(x * y) * sympy.cos(x / 2)
expr.diff(x)
```

```python
expr = sympy.functions.special.polynomials.hermite(x, 0)
expr.diff(x).doit()
```

```python
d = sympy.Derivative(sympy.exp(sympy.cos(x)), x)
d
```

```python
d.doit()
```

```python
a, b, x, y = sympy.symbols("a, b, x, y")
f = sympy.Function("f")(x)
```

```python
sympy.integrate(f)
```

```python
sympy.integrate(f, (x, a, b))
```

```python
sympy.integrate(sympy.sin(x))
```

```python
sympy.integrate(sympy.sin(x), (x, a, b))
```

```python
sympy.integrate(sympy.exp(-x**2), (x, 0, oo))
```

```python
a, b, c = sympy.symbols("a, b, c", positive=True)
```

```python
sympy.integrate(a * sympy.exp(-((x-b)/c)**2), (x, -oo, oo))
```

```python
sympy.integrate(sympy.sin(x * sympy.cos(x)))
```

```python
expr = sympy.sin(x*sympy.exp(y))
sympy.integrate(expr, x)
```

```python
expr = (x + y)**2
sympy.integrate(expr, x)
```

```python
sympy.integrate(expr, (x, 0, 1), (y, 0, 1))
```

```python
x, y = sympy.symbols("x, y")
f = sympy.Function("f")(x)
```

```python
sympy.series(f, x)
```

$\displaystyle f{\left(0 \right)} + x \left. \frac{d}{d \xi} f{\left(\xi \right)} \right|_{\substack{ \xi=0 }} + \frac{x^{2} \left. \frac{d^{2}}{d \xi^{2}} f{\left(\xi \right)} \right|_{\substack{ \xi=0 }}}{2} + \frac{x^{3} \left. \frac{d^{3}}{d \xi^{3}} f{\left(\xi \right)} \right|_{\substack{ \xi=0 }}}{6} + \frac{x^{4} \left. \frac{d^{4}}{d \xi^{4}} f{\left(\xi \right)} \right|_{\substack{ \xi=0 }}}{24} + \frac{x^{5} \left. \frac{d^{5}}{d \xi^{5}} f{\left(\xi \right)} \right|_{\substack{ \xi=0 }}}{120} + O\left(x^{6}\right)$

```python
x0 = sympy.Symbol("{x_0}")
f.series(x, x0, n=2)
```

$\displaystyle f{\left({x_0} \right)} + \left(x - {x_0}\right) \left. \frac{d}{d \xi_{1}} f{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}={x_0} }} + O\left(\left(x - {x_0}\right)^{2}; x\rightarrow {x_0}\right)$

```python
f.series(x, x0, n=2).removeO()
```

$\displaystyle \left(x - {x_0}\right) \left. \frac{d}{d \xi_{1}} f{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}={x_0} }} + f{\left({x_0} \right)}$

```python
sympy.cos(x).series()
```

```python
sympy.sin(x).series()
```

```python
sympy.exp(x).series()
```

```python
(1/(1+x)).series()
```

```python
expr = sympy.cos(x) / (1 + sympy.sin(x * y))
expr.series(x, n=4)
```

```python
expr.series(y, n=4)
```

```python
sympy.limit(sympy.sin(x) / x, x, 0)
```

```python
f = sympy.Function('f')
x, h = sympy.symbols("x, h")
diff_limit = (f(x + h) - f(x))/h
sympy.limit(diff_limit.subs(f, sympy.cos), h, 0)
```

```python
sympy.limit(diff_limit.subs(f, sympy.sin), h, 0)
```

```python
expr = (x**2 - 3*x) / (2*x - 2)
p = sympy.limit(expr/x, x, sympy.oo)
q = sympy.limit(expr - p*x, x, sympy.oo)
p, q
```

```python
n = sympy.symbols("n", integer=True)
x = sympy.Sum(1/(n**2), (n, 1, oo))
x
```

```python
x.doit()
```

```python
x = sympy.Product(n, (n, 1, 7))
x
```

```python
x.doit()
```

```python
x = sympy.Symbol("x")
sympy.Sum((x)**n/(sympy.factorial(n)), (n, 1, oo)).doit().simplify()
```

```python
sympy.solve(x**2 + 2*x - 3)
```

```python
a, b, c = sympy.symbols("a, b, c")
sympy.solve(a * x**2 + b * x + c, x)
```

```python
sympy.solve(sympy.sin(x) - sympy.cos(x), x)
```

```python
sympy.solve(sympy.exp(x) + 2 * x, x)
```

```python
sympy.solve(x**5 - x**2 + 1, x)
```

```python
eq1 = x + 2 * y - 1
eq2 = x - y + 1
sympy.solve([eq1, eq2], [x, y], dict=True)

```

```python
eq1 = x**2 - y
eq2 = y**2 - x
sols = sympy.solve([eq1, eq2], [x, y], dict=True)
sols
```

```python
[eq1.subs(sol).simplify() == 0 and eq2.subs(sol).simplify() == 0 for sol in sols]
```

    [True, True, True, True]

### Matrix

```python
sympy.Matrix([1, 2])
```

$\displaystyle \left[\begin{matrix}1\\2\end{matrix}\right]$

```python
sympy.Matrix([[1, 2]])
```

$\displaystyle \left[\begin{matrix}1 & 2\end{matrix}\right]$

```python
sympy.Matrix([[1, 2], [3, 4]])
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
sympy.Matrix(3, 4, lambda m, n: 10 * m + n)
```

$\displaystyle \left[\begin{matrix}0 & 1 & 2 & 3\\10 & 11 & 12 & 13\\20 & 21 & 22 & 23\end{matrix}\right]$

```python
a, b, c, d = sympy.symbols("a, b, c, d")
M = sympy.Matrix([[a, b], [c, d]])
M
```

$\displaystyle \left[\begin{matrix}a & b\\c & d\end{matrix}\right]$

```python
M * M
```

$\displaystyle \left[\begin{matrix}a^{2} + b c & a b + b d\\a c + c d & b c + d^{2}\end{matrix}\right]$

```python
x = sympy.Matrix(sympy.symbols("x_1, x_2"))
M * x
```

$\displaystyle \left[\begin{matrix}a x_{1} + b x_{2}\\c x_{1} + d x_{2}\end{matrix}\right]$

```python
p, q = sympy.symbols("p, q")
M = sympy.Matrix([[1, p], [q, 1]])
M
```

$\displaystyle \left[\begin{matrix}1 & p\\q & 1\end{matrix}\right]$

```python
b = sympy.Matrix(sympy.symbols("b_1, b_2"))
b
```

$\displaystyle \left[\begin{matrix}b_{1}\\b_{2}\end{matrix}\right]$

```python
x = M.LUsolve(b)
x
```

$\displaystyle \left[\begin{matrix}b_{1} - \frac{p \left(- b_{1} q + b_{2}\right)}{- p q + 1}\\\frac{- b_{1} q + b_{2}}{- p q + 1}\end{matrix}\right]$

```python
x = M.inv() * b
x
```

$\displaystyle \left[\begin{matrix}\frac{b_{1}}{- p q + 1} - \frac{b_{2} p}{- p q + 1}\\- \frac{b_{1} q}{- p q + 1} + \frac{b_{2}}{- p q + 1}\end{matrix}\right]$

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# Linear Regression Derived

#### Cost function

$$ C = \sum_i (y_i - mx_i - c)^2 $$

#### Calculate the partial derivative with respect to $m$

$$
\begin{align}
\frac{\partial C}{\partial m} &= \sum_i 2(y_i - m x_i -c)(-x_i) \\
&= -2 \sum_i x_i (y_i - m x_i -c) \\
\end{align}
$$

#### Set derivative to zero

$$
\begin{align}
& \frac{\partial C}{\partial m} = 0 \\
\Rightarrow & -2 \sum_i x_i (y_i - m x_i -c) = 0 \\
\Rightarrow & \sum_i x_i y_i - m x_i x_i - x_i c = 0 \\
\Rightarrow & \sum_i x_i y_i - \sum_i m x_i x_i - \sum_i x_i c = 0 \\
\Rightarrow & \sum_i x_i y_i - \sum_i x_i c = \sum_i x_i x_i \\
\Rightarrow & \sum_i x_i y_i - c \sum_i x_i = m \sum_i x_i x_i \\
\Rightarrow & m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i}
\end{align}
$$

#### Calculate the partial derivative with respect to $c$

$$
\begin{align}
\frac{\partial C}{\partial c} &= \sum_i 2(y_i - m x_i -c)(-1) \\
&= -2 \sum_i (y_i - m x_i -c) \\
\end{align}
$$

#### Set the derivative to zero

$$
\begin{align}
& \frac{\partial C}{\partial c} = 0 \\
\Rightarrow & -2 \sum_i (y_i - m x_i - c) = 0 \\
\Rightarrow & \sum_i (y_i - m x_i - c) = 0 \\
\Rightarrow & \sum_i y_i - \sum_i m x_i - \sum_i c = 0 \\
\Rightarrow & \sum_i y_i - m \sum_i x_i - c \sum_i 1 = 0 \\
\Rightarrow & \sum_i y_i - m \sum_i x_i = c \sum_i 1 \\
\Rightarrow & c = \frac{\sum_i y_i - m \sum_i x_i}{\sum_i 1} \\
\Rightarrow & c = \frac{\sum_i y_i - m \sum_i x_i}{\sum_i 1} \\
\Rightarrow & c = \frac{\sum_i y_i}{\sum_i 1} - m \frac{\sum_i x_i}{\sum_i 1} \\
\Rightarrow & c = \bar{y} - m \bar{x} \\
\end{align}
$$

#### Combine the estimates

$$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\
& c = \bar{y} - m \bar{x} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - (\bar{y} - m \bar{x}) \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i  - m \bar{x} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i}  - m \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m + m \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i} = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(1 + \frac{\bar{x} \sum_i x_i}{\sum_i x_i x_i}) = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(\frac{\sum_i x_i x_i + \bar{x} \sum_i x_i}{\sum_i x_i x_i}) = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i} \\
& \Rightarrow m(\sum_i x_i x_i + \bar{x} \sum_i x_i) = \sum_i x_i y_i - \bar{y} \sum_i x_i \\
& \Rightarrow m = \frac{\sum_i x_i y_i - \bar{y} \sum_i x_i}{\sum_i x_i x_i + \bar{x} \sum_i x_i} \\
\end{align}
$$

#### End

```python
import numpy as np

w = np.arange(1.0, 16.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)

# Calculate the best values for m and c.
w_avg = np.mean(w)
d_avg = np.mean(d)

w_zero = w - w_avg
d_zero = d - d_avg

m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
c = d_avg - m * w_avg

print("m is %8.6f and c is %6.6f." % (m, c))

x, y, x_avg, y_avg = w, d, w_avg, d_avg
m2 = (np.sum(x * y) - y_avg * np.sum(x)) / (np.sum(x * x) - x_avg * np.sum(x))
c2 = y_avg - m2 * x_avg
m2, c2
```

    m is 4.728824 and c is 12.756653.

    (4.7288241778626832, 12.756652779225512)

```python
q1 = (np.sum(x * y) - y_avg * np.sum(x)) / (np.sum(x*x) - x_avg * np.sum(x))
q2 = (np.sum(x*y) - y_avg * np.sum(x) - x_avg * np.sum(y) + len(x) * x_avg * y_avg) / \
                                      (np.sum(x * x) - x_avg * np.sum(x) - x_avg * np.sum(x) + len(x) * x_avg * x_avg)
q3 = (np.sum(x * y) - (x.size * x_avg * y_avg)) / (np.sum(x * x) - x.size * x_avg * x_avg)

q1, q2, q3
```

    (4.7288241778626832, 4.7288241778626832, 4.7288241778626832)

```python

```


# Exploring the Discrete Fourier Transform

### George Tzanetakis, University of Victoria

In this notebook we will explore the Discrete Fourier Transform which is a fundamental algorithm in Digital Signal Processing. First let's look at the mathematical notation:

\begin{equation}
X_k = <x, e^{jtk2\pi /N}> = \sum_{t=0}^{N-1} x_t e^{-j t k 2 \pi/N}
\end{equation}

We start with a finite, length-N segment of a digital signal $x_0, x_1, \dots, x_{N-1}$. We define the inner product as expected:
\begin{equation}
<x,y> = \sum_{t=0}^{N-1} x_t y_t^*
\end{equation}
and the corresponding basis is N frequency phasors equally spaced in the desired rage from zero to the sampling frequency:
\begin{equation}
{e^{jtk2\pi/N}} = {cos(tk2\pi/N) + j sin(tk2\pi/N})\;\;\; \text{for} \;\; 0 \leq K \leq N-1
\end{equation}

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import IPython.display as ipd
```

Let's plot the real part of one of these basis functions - let's say for $k=4$.

```python
def plot_basis(t, x_re, x_im):
    # plot them
    plt.figure(figsize=(20,5))
    plt.subplot(121)
    plt.plot(t, x_re, lw=2, color='blue')
    plt.subplot(122)
    plt.plot(t, x_im, lw=2, color='red')

N = 512
t = np.arange(0,N)

# basis function real and imaginary parts for k=4
k = 1
x_re = np.cos(k*t*2*np.pi/N)
x_im = np.sin(k*t*2*np.pi/N)

plot_basis(t, x_re, x_im)

```

Let's plot the real and imaginary parts of a basis function with higher frequency with k=10

```python
# basis function real and imaginary parts for k=10
k = 10
x_re = np.cos(k*t*2*np.pi/N)
x_im = np.sin(k*t*2*np.pi/N)

plot_basis(t, x_re, x_im)
```

\begin{equation}
X_k = <x, e^{jtk2\pi /N}> = \sum_{t=0}^{N-1} x_t e^{-j t k 2 \pi/N}  = \sum_{t=0}^{N-1} x_t cos(tk2\pi/N) + \sum_{t=0}^{N-1} x_t  j sin(tk2\pi/N)
\end{equation}

Looking at the equation above you can see that for a specific k the complex spectrum value X_k can be computed by taking two inner products for the input signal $x_t$ with the real and imaginary parts of the corresponding basis function.

Let's consider as input a cosine signal of the same frequency as a particular pair of basis functions let's say with $k=4$ and look at the signal resulting from the point-wise multiplication of the input with the real part and the imagine part. As you can see the point-wise multiplication with the real part results in a signal that is centered above zero and therefore the inner product is positive (the value is shown with the straight black line in the plot). The point-wise multiplication with the imaginary part results in a signal that is centered at zero and therefore the inner product is zero (the value is shown with the straight black line in the plot). Recall from before when we examined using the inner product with a sinusoid of known frequency to estimate amplitude that the estimated amplitude is simply twice the inner product.

```python
def plot_product(t,x1,x2):
    prod = np.multiply(x1,x2)
    inner_prod = np.sum(prod) / len(prod)
    plt.figure(figsize=(20,5))
    plt.plot(t, x1,'.', lw=1, color='blue')
    plt.plot(t, x2,'.', lw=1, color='cyan')
    plt.plot(t, np.multiply(x1,x2), lw=4, color='red')
    ip_line = np.empty(len(prod))
    ip_line.fill(inner_prod)
    plt.plot(t, ip_line, lw=4, color='black')
    plt.plot(t, ip_line*2, lw=2, color='black')
    plt.legend(('x', 'x_re', 'point-wise product', 'inner product', 'amplitude estimate'), loc='upper right')

k=4
a = 0.6
x = a * np.cos(k*t*2*np.pi/N)
x_re = np.cos(k*t*2*np.pi/N)
x_im = np.sin(k*t*2*np.pi/N)
plot_product(t,x,x_re)
plot_product(t,x,x_im)

```

When the input is a sine wave the inner product with the real part of the basis is zero and the inner product with the imaginary part of the basis is positive.

```python
x = a * np.sin(k*t*2*np.pi/N)
plot_product(t,x,x_re)
plot_product(t,x,x_im)

```

Now let's consider the case of using as input a sinusoidal signal of different frequency let's say corresponding to k=10. As you can observe in this case the inner products with both the real and imaginary part of the basis functions are zero.

```python
k=10
x = a * np.sin(k*t*2*np.pi/N)
plot_product(t,x,x_re)
plot_product(t,x,x_im)
```

Now consider an input that is a mixure of two sinsoids. Notice that we are still able to correctly estimate the amplitude of the mixture component that corresponds to the basis function we are using.

```python
k=4
a1 = 0.8
x1 = a1 * np.sin(k*t*2*np.pi/N)
a2 = 0.4
k = 10
x2 = a2 * np.sin(k*t*2*np.pi/N)
x = x1+x2
plot_product(t,x,x_re)
plot_product(t,x,x_im)
```

Finally lets look at computing the Discrete Fourier Transform directly from the equation and viewing the magnitude spectrum for the mixture signal we examined above. The code is written so that the connection to measuring amplitude and phases using inner products with the real and imaginary parts of each basis element is emphasized and the code does not utilize the complex number type. Notice the normalization by 2/N so that the mangitude spectrum shows the estimated amplitudes of the input signal.

```python
def pedagogical_dft(x, N):
    X_re = np.zeros(N)       # array holding the real parts of the spectrum
    X_im = np.zeros(N)       # array holding the imaginary values of the spectrum
    for k in np.arange(0,N):
        for t in np.arange(0,N):
            X_re[k] += x[t] * np.cos(t * k * 2 * np.pi / N)   # inner product with real basis k
            X_im[k] += x[t] * np.sin(t * k * 2 * np.pi / N)   # inner product with imaginary basis k
    return (X_re, X_im)

def plot_mag_spectrum(Xmag):
    plt.figure(figsize=(20,5))
    n = np.arange(0,len(Xmag))
    plt.plot(n,Xmag)

N = 512
n = np.arange(0,N)

# Single sinusoid
k=50
x1 = 0.5 * np.sin(k*n*2*np.pi/N)

(X_re, X_im) = pedagogical_dft(x1, N)
Xmag = 2 * np.sqrt(X_re * X_re + X_im * X_im) /N
plot_mag_spectrum(Xmag)
```

```python
# Mixture sinusoid input
x2 = np.sin(50*n*2*np.pi/N) + 0.5 * np.sin(100 * n * 2 * np.pi/N) + 0.3 * np.sin(300 * n * 2 * np.pi/N)

(X_re, X_im) = pedagogical_dft(x2, N)
Xmag = 2 * np.sqrt(X_re * X_re + X_im * X_im) / N
plot_mag_spectrum(Xmag)

```

Finally let's check that we get similar results with the library implementation of the Fast Fourier Transform - notice the computation is much faster.

```python
X = np.fft.fft(x1)
Xmag = 2 * np.abs(X) / N
plot_mag_spectrum(Xmag)

X = np.fft.fft(x2)
Xmag = 2 * np.abs(X) / N
plot_mag_spectrum(Xmag)
```

```python
%%time
import numpy as np
import timeit

def pedagogical_dft(x, N):
    X_re = np.zeros(N)       # array holding the real parts of the spectrum
    X_im = np.zeros(N)       # array holding the imaginary values of the spectrum
    for k in np.arange(0,N):

        for t in np.arange(0,N):
            X_re[k] += x[t] * np.cos(t * k * 2 * np.pi / N)   # inner product with real basis k
            X_im[k] += x[t] * np.sin(t * k * 2 * np.pi / N)   # inner product with imaginary basis k
    return (X_re, X_im)

def test():
    x = np.zeros(512)
    pedagogical_dft(x,512)

test()
```

```python

```


# Unweighted and Weighted Means

```python
import numpy as np
import scipy.stats as stats

import matplotlib.pyplot as plt
import matplotlib.patches as mpathces
```

## Maximum Likelihood Estimator motivated "derivations"

### Unweighted Means

If we make $n$ identical statistically independent (isi) measurements of a random variable $x$, such that the measurements collected form data $\vec{x} = \left\{x_i, \cdots, x_n\right\}$, from a Gaussian (Normal) distribution,

$$
\begin{equation}
L\left(\vec{x}; \vec{\theta}\right) = \prod_{i=1}^{n} f(x_i; \mu, \sigma) = \frac{1}{(2\pi)^{n/2} \sigma^{n}} \exp\left(-\frac{1}{2\sigma^2} \sum_{i=1}^{n} \left(x_i - \mu\right)^2 \right)
\end{equation}
$$

then

$$
\begin{equation}
-\ln L = \frac{n}{2} \ln\left(2\pi\right) + n \ln \sigma + \frac{1}{2\sigma^2} \sum_{i=1}^{n}\left(x_i - \mu\right)^2
\end{equation}
$$

and so $L$ is maximized with respect to a variable $\alpha$ when $-\ln L$ is minimized,

$$
\begin{equation*}
\frac{\partial \left(-\ln L\right)}{\partial \alpha} = 0.
\end{equation*}
$$

Thus, $L$ is maximized when

$$
\begin{equation*}
\frac{\partial \left(-\ln L\right)}{\partial \mu} = -\frac{1}{\sigma^2} \sum_{i=1}^{n}\left(x_i - \mu\right) = 0,
\end{equation*}
$$

which occurs for

$$
\begin{equation*}
\sum_{i=1}^{n} x_i = n \mu,
\end{equation*}
$$

such that the best estimate for true parameter $\mu$ is

$$
\begin{equation}
\boxed{\hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x}\,}\,,
\end{equation}
$$

and $L$ is maximized when

$$
\begin{equation*}
\frac{\partial \left(-\ln L\right)}{\partial \sigma} = \frac{n}{\sigma} - \frac{1}{\sigma^3} \sum_{i=1}^{n} \left(x_i - \mu\right) = 0,
\end{equation*}
$$

which occurs for

$$
\begin{equation*}
n\sigma^2 = \sum_{i=1}^{n} \left(x_i - \mu\right)^2,
\end{equation*}
$$

which is

$$
\begin{equation*}
\sigma = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(x_i - \mu\right)^2}.
\end{equation*}
$$

However, $\mu$ is an unknown true parameter, and the best estimate of it is $\hat{\mu}$, which is in no
manner required to be equal to $\mu$. Thus, the best estimate of $\sigma$ is

$$
\begin{equation}
\boxed{\hat{\sigma}_{\hat{\mu}} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(x_i - \hat{\mu}\right)^2} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(x_i - \bar{x}\,\right)^2}\,}\,.
\end{equation}
$$

If the separation from the mean of each observation, $\left(x_i - \bar{x}\right) = \delta x = \text{constant}$, are the same then the uncertainty on the mean is found to be

$$
\begin{equation*}
\sigma_{\hat{\mu}} = \frac{\delta x}{\sqrt{n}},
\end{equation*}
$$

which is often referred to as the "standard error".

---
So, for a population of measurements sampled from a distribution, it can be said that the sample mean is

$$\mu = \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x},$$

and the standard deviation of the sample is

$$
\begin{equation*}
\sigma = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \left(x_i - \bar{x}\,\right)^2}.
\end{equation*}
$$

---

### Weighted Means

Assume that $n$ individual measurements $x_i$ are spread around (unknown) true value $\theta$ according to a Gaussian distribution, each with known width $\sigma_i$.

This then leads to the likelihood function

$$
\begin{equation*}
L(\theta) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi}\sigma_i} \exp\left(-\frac{\left(x_i - \theta\right)^2}{2\sigma_i^2} \right)
\end{equation*}
$$

and so negative log-likelihood

$$
\begin{equation}
-\ln L = \frac{1}{2} \ln\left(2\pi\right) + \ln \sigma_i + \frac{1}{2\sigma_i^2} \sum_{i=1}^{n}\left(x_i - \theta\right)^2.
\end{equation}
$$

As before, $L$ is maximized with respect to a variable $\alpha$ when $-\ln L$ is minimized,

$$
\begin{equation*}
\frac{\partial \left(-\ln L\right)}{\partial \alpha} = 0,
\end{equation*}
$$

and so $L$ is maximized with respect to $\theta$ when

$$
\begin{equation*}
\frac{\partial \left(-\ln L\right)}{\partial \theta} = -\sum_{i=1}^{n} \frac{x_i - \theta}{\sigma_i^2} = 0,
\end{equation*}
$$

which occurs for

$$
\begin{equation*}
\sum_{i=1}^{n} \frac{x_i}{\sigma_i^2} = \theta \sum_{i=1}^{n} \frac{1}{\sigma_i^2},
\end{equation*}
$$

which is

$$
\begin{equation}
\hat{\theta} = \frac{\displaystyle\sum_{i=1}^{n} \frac{x_i}{\sigma_i^2}}{\displaystyle\sum_{i=1}^{n}\frac{1}{\sigma_i^2}}.
\end{equation}
$$

Note that by defining "weights" to be

$$
\begin{equation*}
w_i = \frac{1}{\sigma_1^2},
\end{equation*}
$$

this can be expressed as

$$
\begin{equation}
\boxed{\hat{\theta} = \frac{\displaystyle\sum_{i=1}^{n} w_i\, x_i}{\displaystyle\sum_{i=1}^{n}w_i}}\,,
\end{equation}
$$

making the term "weighted mean" very transparent.

To find the standard deviation on the weighted mean, we first look to the variance, $\sigma^2$. [4]

$$
\begin{align*}
\sigma^2 &= \text{E}\left[\left(\hat{\theta} - \text{E}\left[\hat{\theta}\right]\right)^2\right] \\
    &= \text{E}\left[\left(\frac{\displaystyle\sum_{i=1}^{n} w_i\, x_i}{\displaystyle\sum_{i=1}^{n}w_i} - \text{E}\left[\frac{\displaystyle\sum_{i=1}^{n} w_i\, x_i}{\displaystyle\sum_{i=1}^{n}w_i}\right]\,\right)^2\right] \\
    &= \frac{1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \text{E} \left[ \displaystyle\left(\sum_{i=1}^{n} w_i\,x_i\right)^2 - 2 \displaystyle\left(\sum_{i=1}^{n} w_i\,x_i\right) \displaystyle\left(\sum_{i=j}^{n} w_j\, \text{E}\left[x_j\right]\right) + \displaystyle\left(\sum_{i=1}^{n} w_i\, \text{E}\left[x_i\right]\right)^2 \right] \\
    &= \frac{1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \text{E} \left[ \sum_{i,j}^{n} w_i\, x_i w_j\, x_j - 2 \sum_{i,j}^{n} w_i\, x_i w_j\, \text{E}\left[x_j\right] + \sum_{i,j}^{n} w_i\, \text{E}\left[x_i\right] w_j\, \text{E}\left[x_j\right] \right] \\
    &= \frac{1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \sum_{i,j}^{n} w_i w_j \left( \text{E}\left[ x_i x_j \right] - 2 \text{E}\left[ x_i \right]\text{E}\left[ x_j \right] + \text{E}\left[ x_i \right]\text{E}\left[ x_j \right] \right) \\
    &= \frac{1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \sum_{i,j}^{n} w_i w_j \left( \text{E}\left[ x_i x_j \right] - \text{E}\left[ x_i \right]\text{E}\left[ x_j \right] \right) \\
    &= \frac{1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \sum_{i,j}^{n} w_i w_j \,\text{Cov}\left( x_i, x_j \right) = \left\{
\begin{array}{ll}
\frac{\displaystyle1}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \displaystyle\sum_{i}^{n} \left( w_i \sigma_i \right)^2\,, & x_i \text{ and } x_j \text{ statistically independent}, \\
0\,, &\text{ otherwise},
\end{array}
\right. \\
    &= \frac{\displaystyle\sum_{i}^{n} \left( \sigma_i^{-2} \sigma_i \right)^2}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} = \frac{\displaystyle\sum_{i}^{n} w_i}{\displaystyle\left(\sum_{i=1}^{n} w_i\right)^2} \\
    &= \frac{\displaystyle 1}{\displaystyle\sum_{i=1}^{n} w_i}
\end{align*}
$$

Thus, it is seen that the standard deviation on the weighted mean is

$$
\begin{equation}
\boxed{\sigma_{\hat{\theta}} = \sqrt{\frac{\displaystyle 1}{\displaystyle\sum_{i=1}^{n} w_i}} = \left(\displaystyle\sum_{i=1}^{n} \frac{1}{\sigma_i^2}\right)^{-1/2}}\,.
\end{equation}
$$

Notice that in the event that the uncertainties are uniform for each observation, $\sigma_i = \delta x$, the above yields the same result as the unweighted mean. $\checkmark$

After this aside it is worth pointing out that [1] have a very elegant demonstration that

$$
\begin{equation*}
\sigma_{\hat{\theta}} = \left(\frac{\partial^2\left(- \ln L\right)}{\partial\, \theta^2}\right)^{-1/2} = \left(\displaystyle\sum_{i=1}^{n} \frac{1}{\sigma_i^2}\right)^{-1/2}.
\end{equation*}
$$

---
So, the average of $n$ measurements of quantity $\theta$, with individual measurements, $x_i$, Gaussianly distributed about (unknown) true value $\theta$ with known width $\sigma_i$, is the weighted mean

$$
\begin{equation*}
\hat{\theta} = \frac{\displaystyle\sum_{i=1}^{n} w_i\, x_i}{\displaystyle\sum_{i=1}^{n}w_i},
\end{equation*}
$$

with weights $w_i = \sigma_i^{-2}$, with standard deviation on the weighted mean

$$
\begin{equation*}
\sigma_{\hat{\theta}} = \sqrt{\frac{\displaystyle 1}{\displaystyle\sum_{i=1}^{n} w_i}} = \left(\displaystyle\sum_{i=1}^{n} \frac{1}{\sigma_i^2}\right)^{-1/2}.
\end{equation*}
$$

---

## Specific Examples

Given the measurements

$$
\vec{x} = \left\{10, 9, 11\right\}
$$

with uncertanties

$$\vec{\sigma_x} = \left\{1, 2, 3\right\}$$

```python
x_data = [10, 9, 11]
x_uncertainty = [1, 2, 3]
```

```python
numerator = sum(x / (sigma_x ** 2) for x, sigma_x in zip(x_data, x_uncertainty))
denominator = sum(1 / (sigma_x ** 2) for sigma_x in x_uncertainty)

print(f"hand calculated weighted mean: {numerator / denominator}")
```

Using [NumPy's `average` method](https://docs.scipy.org/doc/numpy/reference/generated/numpy.average.html)

```python
# unweighted mean
np.average(x_data)
```

```python
x_weights = [1 / (uncert ** 2) for uncert in x_uncertainty]
# weighted mean
weighted_mean = np.average(x_data, weights=x_weights)
print(weighted_mean)
```

```python
# no method to do this in NumPy!?
sigma = np.sqrt(1 / np.sum(x_weights))
print(f"hand calculated uncertaintiy on weighted mean: {sigma}")
```

```python
# A second way to find the uncertainty on the weighted mean
summand = sum((x * w) for x, w in zip(x_data, x_weights))
np.sqrt(np.average(x_data, weights=x_weights) / summand)
```

Let's plot the data now and take a look at the results

```python
def draw_weighted_mean(data, errors, w_mean, w_uncert):
    plt.figure(1)

    # the data to be plotted
    x = [i + 1 for i in range(len(data))]

    x_min = x[x.index(min(x))]
    x_max = x[x.index(max(x))]

    y = data
    y_min = y[y.index(min(y))]
    y_max = y[y.index(max(y))]

    err_max = errors[errors.index(max(errors))]

    # plot data
    plt.errorbar(x, y, xerr=0, yerr=errors, fmt="o", color="black")
    # plot weighted mean
    plt.plot((x_min, x_max), (w_mean, w_mean), color="blue")
    # plot uncertainty on weighted mean
    plt.plot(
        (x_min, x_max),
        (w_mean - w_uncert, w_mean - w_uncert),
        color="gray",
        linestyle="--",
    )
    plt.plot(
        (x_min, x_max),
        (w_mean + w_uncert, w_mean + w_uncert),
        color="gray",
        linestyle="--",
    )

    # Axes
    plt.xlabel("Individual measurements")
    plt.ylabel("Value of measruement")
    # view range
    epsilon = 0.1
    plt.xlim(x_min - epsilon, x_max + epsilon)
    plt.ylim([y_min - err_max, 1.5 * y_max + err_max])

    # ax = figure().gca()
    # ax.xaxis.set_major_locator(MaxNLocator(integer=True))

    # Legends
    wmean_patch = mpathces.Patch(
        color="blue", label=fr"Weighted mean: $\mu={w_mean:0.3f}$"
    )
    uncert_patch = mpathces.Patch(
        color="gray",
        label=fr"Uncertainty on the weighted mean: $\pm{w_uncert:0.3f}$",
    )
    plt.legend(handles=[wmean_patch, uncert_patch])

    plt.show()
```

```python
draw_weighted_mean(x_data, x_uncertainty, weighted_mean, sigma)
```

Now let's do this again but with data that are Normally distributed about a mean value

```python
true_mu = np.random.uniform(3, 9)
true_sigma = np.random.uniform(0.1, 2.0)
n_samples = 20

samples = np.random.normal(true_mu, true_sigma, n_samples).tolist()
gauss_errs = np.random.normal(2, 0.4, n_samples).tolist()

weights = [1 / (uncert ** 2) for uncert in gauss_errs]

draw_weighted_mean(
    samples,
    gauss_errs,
    np.average(samples, weights=weights),
    np.sqrt(1 / np.sum(weights)),
)
```

## References

1. [_Data Analysis in High Energy Physics_](http://eu.wiley.com/WileyCDA/WileyTitle/productCd-3527410589.html), Behnke et al., 2013, $\S$ 2.3.3.1
2. [_Statistical Data Analysis_](http://www.pp.rhul.ac.uk/~cowan/sda/), Glen Cowan, 1998
3. University of Marlyand, Physics 261, [Notes on Error Propagation](http://www.physics.umd.edu/courses/Phys261/F06/ErrorPropagation.pdf)
4. Physics Stack Exchange, [_How do you find the uncertainty of a weighted average?_](https://physics.stackexchange.com/questions/15197/how-do-you-find-the-uncertainty-of-a-weighted-average)


- The prime 41, can be written as the sum of six consecutive primes:
41 = 2 + 3 + 5 + 7 + 11 + 13

- This is the longest sum of consecutive primes that adds to a prime below one-hundred.

- The longest sum of consecutive primes below one-thousand that adds to a prime,
contains 21 terms, and is equal to 953.

- **Which prime, below one-million, can be written as the sum of the most consecutive primes?**

```python
from tqdm import tqdm
import numpy as np
from sympy import sieve

N = 1e6
sieve._reset()
sieve.extend(N)

primes = np.array(sieve._list)
prime_set = set(primes)
cs = np.cumsum(primes)

primes.shape
```

$f(i,n) = \sum_{i}^{i+n}P_i$

$S = \texttt{primes} < 1000000$

$\exists \ i^*, n^*: f(i^*, n^*) \in S \wedge \not \exists \ i', n' > n^*: \ f(i', n') \in S$

```python
d = -np.infty
p = None

for i in tqdm(range(len(primes))):
    for j in range(i, -1, -1):
        p_ = cs[i] - cs[j]
        if p_ > N: break

        if p_ in prime_set and (i - j) > d:
            d, p = i-j, p_
d, p
```


# 3. The Central Dogma of Molecular Biology

The __Central Dogma of Molecular Biology__  is the process by which the instructions in DNA are converted into a functional product. It its most simplest definition, it involves the following processes:

- _Replication_: DNA is copied during cell division so the two daughter cells inherit the same genetic information.

- _Transcription_: the information in the DNA of every cell is converted into small, portable RNA messages.

- _Translation_: During translation, these messages travel from where the DNA is in the cell nucleus to the ribosomes where they are ‘read’ to make specific proteins.

We will write a mathematical simplified version of the Central Dogma using the differential form of the Mass action Law. To do that, we will consider a gene that is regulated by a transcription factor $T$, and let M denote the RNA molecular species resulting from this gene transcription. Under the assumption that chemical kinetics can be used to model gene expression and gene regulation, we can write the following reaction:

\begin{align}
T+D &\overset{k_M}{\longrightarrow} M + T + D \tag{1} \\
M &\overset{\gamma_M}{\longrightarrow} \emptyset \tag{2}
\end{align}

- $[M]$ is the concentration of mRNA
- $[D]$ is the concentration of gene copies
- $[T]$ is the transcription factor
- $k_M$ is the maximum transcription rate of a gene copy
- $\gamma_M$ is the RNA degradation rate

Based on the Mass Action Law, and assuming that both `D` and `T` are constant, the equation governing the dynamics of the RNA concentration [M] is:

$$
\frac{\mathrm{d} [M]}{\mathrm{d} t}=  k_M [D]  [T]- \gamma_M [M] \tag{3}
$$

To solve this equation, we first separate the variables in both sides of the equation as:

$$\begin{align*}
   \frac{\mathrm{d} [M]}{\mathrm{d} t} + \gamma_M [M] &=  k_M [D] [T] \tag{4}\\
 \end{align*}$$

   We need to calculate the integrating factor, $e^{ \int p(x)dx }$, which in this case is $e^{\gamma_M \cdot t}$. We then multiply both terms in the previous equation by the integrating factor. In this case is simply
  $$\begin{align*}
   e^{\gamma_M \cdot t} \frac{\mathrm{d} [M]}{\mathrm{d} t} + \gamma_M [M] e^{\gamma_M \cdot t} &=  k_M [D] [T] e^{\gamma_M \cdot t}\tag{5}
 \end{align*}$$
 the first term of the equation is simply

  $$\begin{align*}
   \frac{\mathrm{d} ([M] e^{\gamma_M \cdot t})}{\mathrm{d} t} &=  k_M [D] [T] e^{\gamma_M \cdot t} \tag{6}
 \end{align*}$$

 we change the `dt` to the left side and then integrate both sides of the equation:
 $$\begin{align*}
  \int \mathrm{d} ([M] e^{\gamma_M \cdot t}) &= \int k_M [D] [T] e^{\gamma_M \cdot t} \mathrm{d} t\tag{7}\\
      [M] e^{\gamma_M \cdot t} &= \frac{k_M [D] [T]}{\gamma_M}  e^{ \gamma_M \cdot t} + C \tag{8}\\
 \end{align*}$$

 that rearranging terms becomes

 $$\begin{align*}
      [M]  &= \frac{k_M [D] [T]}{\gamma_M} + C \cdot e^{-\gamma_M \cdot t}\tag{9}\\
 \end{align*}$$

 to evaluate the integration constant, we use the initial value of `M`:

  $$\begin{align*}
      [M(0)]  &= \frac{k_M [D] [T]}{\gamma_M} + C \tag{10}\\
      C &= [M(0)] - \frac{k_M [D] [T]}{\gamma_M}  \tag{11}\\
 \end{align*}$$

 therefore

  $$\begin{align*}
      [M(t)]  &= \frac{k_M [D] [T]}{\gamma_M} + [M(0)] \cdot e^{-\gamma_M \cdot t} - \frac{k_M [D] [T]}{\gamma_M} \cdot e^{-\gamma_M \cdot t}\tag{12}\\
       [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{k_M [D] [T]}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{13}\\
 \end{align*}$$

Therefore, the dynamics of `mRNA` looks like this:

```julia
using Plots
gr()
```

    Plots.GRBackend()

```julia
k_M=1
γ_M=1
T=1
D=1
M₀=0.2
t=collect(0:0.1:10);
```

```julia
plot(t,t-> M₀*exp(-γ_M*t)+k_M*D*T/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,1))
title!("Central Dogma")
xaxis!("Time")
yaxis!("Concentration")
```

## Promoter leakining

This simplified approximation does not take into acount that promoters are not perfect, and that there is often a basal production of mRNA that is not negligible. The next set of inetactions includes the basal production of `M` in abseence of transcription factor `T` with a rate $\alpha_0$:

$$
\begin{align}
\emptyset &\overset{\alpha_0}{\longrightarrow} M \tag{14}\\
T+D &\overset{k_M}{\longrightarrow} M + T + D  \tag{15}\\
M &\overset{\gamma_M}{\longrightarrow} \emptyset \tag{16}\\
\end{align}
$$

Now, the equation governing the dynamics of the mRNA concentration [M] is:

$$
\frac{\mathrm{d} [M]}{\mathrm{d} t}=  \alpha_0 + k_M [D]  [T]- \gamma_M [M] \tag{17}
$$
Following the same steps as the previous equation, we obatin the following solution:

 $$\begin{align*}
      [M]  &= \frac{\alpha_0 + k_M [D] [T]}{\gamma_M} + C \cdot e^{-\gamma_M \cdot t}\tag{18}\\
 \end{align*}$$

  to evaluate the integration constant, we use the initial value of `M`:

  $$\begin{align*}
      [M(0)]  &= \frac{\alpha_0 + k_M [D] [T]}{\gamma_M} + C \tag{19}\\
      C &= [M(0)] - \frac{\alpha_0 + k_M [D] [T]}{\gamma_M}  \tag{20}\\
 \end{align*}$$

 therefore the final equation is

  $$\begin{align*}
      [M(t)]  &= \frac{\alpha_0 + k_M [D] [T]}{\gamma_M} + [M(0)] \cdot e^{-\gamma_M \cdot t} - \frac{\alpha_0 + k_M [D] [T]}{\gamma_M} \cdot e^{-\gamma_M \cdot t}\tag{21}\\
       [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + k_M [D] [T]}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{22}\\
 \end{align*}$$

Therefore, with promoter leaking the dynamics of `mRNA` looks like this:

```julia
α_0=0.5
plot(t,t-> M₀*exp(-γ_M*t)+(α_0 + k_M*D*T)/γ_M*(1-exp(-γ_M*t)),label="\\ M with \\alpha_0",seriestype=:line,ylims = (0,1.5))
plot!(t,t-> M₀*exp(-γ_M*t)+( k_M*D*T)/γ_M*(1-exp(-γ_M*t)),label="\\ M without \\alpha_0",seriestype=:line,ylims = (0,1.5))

title!("Central Dogma with Promoter Leaking")
xaxis!("Time")
yaxis!("Concentration")
```

## Hill Function and cooperativity

This first version of the model assumes that one molecule of transcription factor `T` results in one molecule of `mRNA`. This is not the case in transcription, since a molecule of transcription factor is able to recruit many molecules of `pRNA` and activate transcription of many `mRNA` molecules. Therefore, more molecules of transcition factor means more activation of the transcription of `M`. This way, we write the rate of production of `M` depending on the number of molecules of transciption factor `T` as a function:

$$
\frac{\mathrm{d} [M]}{\mathrm{d} t}=  \alpha_0 + k_M [D] \Phi ([T])- \gamma_M [M] \tag{23}
$$

$\Phi ([T])$ can be interpreted as the probability that a gene copy is transcribed at a given time, as a function of $[T]$. This dependence allows us to introduce the concept of a  transcription factor as activators (its presence results in activation of transcription) or as repressors (its presence results in repression of transcription).
Depending on this, the regulatory function $\Phi([T])$ will increase or decrease with `T`. As a first approximation, $\Phi([T])$ is often assumed as a step function., i.e, the gene is transcribed (or repressed) when the transcription factor is bounded.

```julia
T_vector= LinRange(0,2,100)
n=1000
K=1
ϕ=D.*T_vector.^n./(K.^n.+T_vector.^n)
P1=plot(T_vector,ϕ,lab="")
title!("Step  function activator")
xlabel!("Concentration of Transcription factor")
ylabel!("function ø")

ϕ=D.*K.^n./(K.^n.+T_vector.^n)
P2=plot(T_vector,ϕ,lab="")
title!("Step function repressor")
xlabel!("Concentration of Transcription factor")
ylabel!("function ø")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

 The analytical soultion will look quite similar to the previous equation, just replacing [$T$] by [$\Phi ([T])$]

$$\begin{align*}
       [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + k_M [D] \Phi ([T])}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{24}\\
 \end{align*}$$

 and the solution will depend on whether there is a value of `T` higher or lower than 1:

```julia
P1=plot(t,t-> M₀*exp(-γ_M*t)+(α_0 + k_M*D*0.)/γ_M*(1-exp(-γ_M*t)),label="\\ M with T=0",seriestype=:line,ylims = (0,1.5))
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + k_M*D*1.)/γ_M*(1-exp(-γ_M*t)),label="\\ M with T=1",seriestype=:line,ylims = (0,1.5))

title!("Central Dogma Activator Step Funct")
xaxis!("Time")
yaxis!("Concentration")

P2=plot(t,t-> M₀*exp(-γ_M*t)+(α_0 + k_M*D*1.)/γ_M*(1-exp(-γ_M*t)),label="\\ M with T=0",seriestype=:line,ylims = (0,1.5))
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + k_M*D*0.)/γ_M*(1-exp(-γ_M*t)),label="\\ M with T=1",seriestype=:line,ylims = (0,1.5))

title!("Central Dogma Repressor Step Func")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

We can see now that, when concentration of `T` is above a certain value, transcription is activated (for `T` as activator) or repressed (for `T`as repressors).  Binding and unbinding of`T` to the promoter is a highly dynamic procces, and more realistic scenario is that the probability of transcription of a gene `Ø([T],[D])` depends on the time the trascription factor `T` is bound to the DNA `D`. This bindng and unbinding can be written as a chemical interaction as

$$ [D_{inactive}] + [T] \overset{k_1}{\underset{k_2}{\longleftrightarrow}}  [D_{active}] \tag{25}$$

To obtain a more general approximation,  we will introduce the possibility that several molecuels of the transcription factor are required to activate transcription. This is known as `cooperativity` and is represented in the following scheme of interaction.

$$ [D_{inactive}] + n[T] \overset{k_1}{\underset{k_2}{\longleftrightarrow}}  [D_{Active}] \tag{26}$$

The dynamics of binding and unbinding of transcription factor to the DNA is much faster that the dynamics of transcription. Therefore, using separation of scales, we can consider that equilibrium is reached verty fast. Based on this assumption and on Mass Action Law, the concetrations at equilibrium satisfy the following relation:

$$[D_{inactive}] \cdot [T]^n= \frac{k_2}{k_1}  [D_{Active}]= K_D \cdot [D_{Active}] \tag{27}$$

where $K_D$ is the chemical equilibrium constant. We have also an extra constraint given by the fact that the total number of DNA copies `[D]` is fixed, therefore:

$$[D_{Active}] + [D_{Inactive}] = [D] \tag{28}$$

Combining the equations 27 and 28, we can write the ratio of active copies of the transcriotion factor as:

$$
\begin{align}
\frac{[D_{Active}]}{[D]}&=\frac{[T]^n [D_{Inactive}]}{K_D}\cdot \frac{1}{[D_{Active}]+[D_{Inactive}]} \tag{29}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{[T]^n }{K_D \frac{[D_{Inactive}]+[D_{Active}]}{[D_{Inactive}]}} \tag{30}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{[T]^n }{ K_D(\frac{[D_{Active}]}{[D_{Inactive}]}+1)} \tag{31}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{[T]^n}{\frac{K_D \cdot [D_{Active}]}{[D_{Inactive}]}+K_D} \tag{32}\\
\end{align}$$

and using again the relation in Eq. 27, we obtain

$$\frac{[D_{Active}]}{[D]}=\frac{[T]^n}{K_D+[T]^n}  \tag{33}$$

if we rewrite the equilibrium constant $K_D$ as a new constant to the power of `n`, we obatin:

$$h^{(1)}=\frac{[D_{Active}]}{[D]}=\frac{[T]^n}{K^n+[T]^n}  \tag{34}$$

This equation $h^{(1)}$ is the well known `Hill function`. It satisfies the following properties:

1. $h^{(1)}(0)=0 \tag{35}$

2. $h^{(1)}(T=K)=\frac{1}{2} \tag{36}$

3.  $\lim_{T\to \infty} h^{(1)}(T) = 1 \tag{37}$

4. The maximum slope  controlled by the parameter `n`, with is calles sigmoidicity. To study the value of the slope and its correlation with the value of `n`, we divide numerator and denominator by $K^n$ and define the variable x as $x= T/K$. This way the normalized version of the Hill function is:

$$
h^{(1)}=\frac{x^n}{1+x^n}\tag{38}
$$

we then calculate the slope by caluclating the value of the derivative of $h^{(1)}$ at the point $x=1$:

$$\begin{align}
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}&=\frac{n x^{n-1} (1+x^n)-x^n \cdot n x^{-1}}{(1+x^n)^2} \tag{39}\\
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}&=\frac{n x^{n-1} +x^n \cdot n x^{n-1}-x^n \cdot n x^{-1}}{(1+x^n)^2} \tag{40}\\
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}&=\frac{n x^{n-1}  +x^n \cdot n x^{n-1}-x^n \cdot n x^{-1}}{(1+x^n)^2} \tag{41}\\
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}&=\frac{n x^{n-1} }{(1+x^n)^2} \tag{42}\\
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}\big|_{x=1}&=\frac{n 1^{n-1} }{(1+1^n)^2} \tag{43}\\
\frac{\mathrm{d} h^{(1)}}{\mathrm{d} x}\big|_{x=1}&=\frac{n}{4} \tag{44}
\end{align}$$

```julia
n=5
ϕ=T_vector.^n./(K.^n.+T_vector.^n)
P1=plot(T_vector,ϕ,lab="Hill Function activator")

n=1000
ϕ=T_vector.^n./(K.^n.+T_vector.^n)
P1=plot!(T_vector,ϕ,lab="Step Function activator")

title!("Hill Function vs. Step function Activator")
xlabel!("Concentration of Transcription Factor")
ylabel!("fraction of active promoters")
```

For repressors,

$$ [D_{Active}] + n[T] \overset{k_1}{\underset{k_2}{\longleftrightarrow}}  [D_{Inactive}] \tag{45}$$

the equilibrium is now

$$[D_{Active}] \cdot [T]^n= \frac{k_2}{k_1}  [D_{Inactive}]= K_D \cdot [D_{Inactive}] \tag{46}$$
 and using the same conservation of of `D` we have

 $$
\begin{align}
\frac{[D_{Active}]}{[D]}&=\frac{K_D [D_{Inactive}]}{[T]^n}\cdot \frac{1}{[D_{Active}]+[D_{Inactive}]} \tag{47}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{K_D }{[T]^n \frac{[D_{Inactive}]+[D_{Active}]}{[D_{Inactive}]}} \tag{48}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{K_D }{[T]^n (\frac{[D_{Active}]}{[D_{Inactive}]}+1)} \tag{49}\\
\\
\frac{[D_{Active}]}{[D]}&=\frac{K_D }{\frac{[T]^n \cdot [D_{Active}]}{[D_{Inactive}]}+[T]^n} \tag{50}\\
\end{align}$$

and using again the relation in Eq 24, we obtain

$$\frac{[D_{Active}]}{[D]}=\frac{[K_D}{K_D+[T]^n}  \tag{51}$$

if we rewrite again the equilibrium constant $K_D$ as a new constant to the power of `n`, we obatin:

$$h^{(2)}=\frac{[D_{Active}]}{[D]}=\frac{K^n}{K^n+[T]^n} \tag{52}$$

This equation $h^{(1)}$ is now the `Hill function` for repressor molecules. It satisfies the following properties:

1. $h^{(2)}(0)=1 \tag{53}$

2. $h^{(2)}(K)=\frac{D}{2} \tag{54}$

3.  $\lim_{T\to \infty} h^{(2)}(T) = 0 \tag{55}$

4. The maximum slope  controlled by the parameter `n`, with is calles sigmoidicity. To study the value of the slope and its correlation with the value of `n`, we divide numerator and denominator by $K^n$ and define the variable x as $x= T/K$. This way the normalized version of the Hill function is:

$$
h^{(2)}=\frac{1}{1+x^n}\tag{56}
$$

we then calculate the slope by caluclating the value of the derivative of $h^{(2)}$ at the point $x=1$:

$$\begin{align}
\frac{\mathrm{d} h^{(2)}}{\mathrm{d} x}&=\frac{0 \cdot  (1+x^n) - 1 \cdot n x^{-1}}{(1+x^n)^2} \tag{57}\\
\frac{\mathrm{d} h^{(2)}}{\mathrm{d} x}&=- \frac{ n x^{-1}}{(1+x^n)^2} \tag{58}\\
\frac{\mathrm{d} h^{(2)}}{\mathrm{d} x}\big|_{x=1}&=- \frac{n 1^{n-1} }{(1+1^n)^2} \tag{59}\\
\frac{\mathrm{d} h^{(2)}}{\mathrm{d} x}\big|_{x=1}&=-\frac{n}{4} \tag{60}
\end{align}$$

```julia
n=5
ϕ=K.^n./(K.^n.+T_vector.^n)
P2=plot(T_vector,ϕ,lab="Hill Function repressor")

n=1000
ϕ=K.^n./(K.^n.+T_vector.^n)
P2=plot!(T_vector,ϕ,lab="step Function repressor")
title!("Hill Function vs. Step function repressor")
xlabel!("Concentration of Transcription factor")
ylabel!("fraction of active promoters")
```

The Hill functions as activator or repressor are a more realistic approach to introduce the effect of a transcription factor into the Central Dogma Model. This way, the dynamics of production of `mRNA` for activator is like this:

$$
\frac{\mathrm{d} [M]}{\mathrm{d} t}= \alpha_0 + k_M \frac{[T^n][D]}{K^n+[T^n]}-\gamma_M[M]= \alpha_0 +\alpha_M \frac{[T^n]}{K^n+[T^n]}-\gamma_M[M] \tag{61}
$$
and for repression, we have

$$
\frac{\mathrm{d} [M]}{\mathrm{d} t}= \alpha_0 + k_M \frac{K^n[D]}{K^n+[T^n]}-\gamma_M[M]= \alpha_0 + \alpha_M \frac{K^n}{K^n+[T^n]}-\gamma_M[M] \tag{62}
$$

where we defined  $\alpha_M= k_M [D]$ as the maximum transcription rate of mRNA `M`. The set of interactions is then generalized as:

$$
\begin{align}
\emptyset &\overset{\alpha_0}{\longrightarrow} M \tag{63}\\
\Psi (T,K) &\overset{α_M}{\longrightarrow} M \tag{64}\\
M &\overset{\gamma_M}{\longrightarrow} \emptyset \tag{65}
\end{align}
$$

illustrating that the production of `M` is defined by a function of the amount of transcription factor `T` and the constant of the Hill funcion `K` that corresponds to the half maximal concetration of transcripton factor. This function $\Psi$ can refer to the Hill function for activator or for repressor. The analytical solution can be dertived similarly as before, and the resulting equation is:

$$\begin{align*}
       [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + \alpha_M \frac{[T^n]}{K^n+[T^n]}}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{66}\\
           [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{67}
 \end{align*}$$

 for activators and repressors respectively:

```julia
α_M=k_M*D
K=1.
n=2.
T=0.5
ϕ=T.^n./(K.^n.+T.^n)
P1=plot(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=0.5",seriestype=:line,ylims = (0,1.5))
T=1
ϕ=T.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=1",seriestype=:line,ylims = (0,1.5))
T=2
ϕ=T.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=2",seriestype=:line,ylims = (0,1.5))
T=10
ϕ=T.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=10",seriestype=:line,ylims = (0,1.5))
T=50
ϕ=T.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=50",seriestype=:line,ylims = (0,1.5))

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

T=0.5
ϕ=K.^n./(K.^n.+T.^n)
P2=plot(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=0.5",seriestype=:line,ylims = (0,1.5))
T=1
ϕ=K.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=1",seriestype=:line,ylims = (0,1.5))
T=2
ϕ=K.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=2",seriestype=:line,ylims = (0,1.5))
T=10
ϕ=K.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=10",seriestype=:line,ylims = (0,1.5))
T=50
ϕ=K.^n./(K.^n.+T.^n)
plot!(t,t-> M₀*exp(-γ_M*t)+(α_0 + α_M * ϕ)/γ_M*(1-exp(-γ_M*t)),label="\\ T=50",seriestype=:line,ylims = (0,1.5))
title!("Central Dogma Repressor Hill")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

The Hill function combines the fact that one mRNA can produce may proteins, but also the dependence on the amount of trancription factor `T`. It also adds a saturation term for high concentrations of trancription factor, compared to the value of the Hill constant `K`.

## Protein dynamics

After analyzing the effect induced by the Hill function, the next step is to add the dynamics of the protein. The reactions that take place can be illustrated as:

$$
\begin{align}
\emptyset &\overset{\alpha_0}{\longrightarrow} M \tag{68}\\
\Psi (T,K) &\overset{α_M}{\longrightarrow} M \tag{69}\\
M &\overset{\gamma_M}{\longrightarrow} \emptyset \tag{70}\\
M &\overset{\alpha_P}{\longrightarrow} P \tag{71}\\
P &\overset{\gamma_P}{\longrightarrow} \emptyset \tag{72}
\end{align}
$$

where `P` is the concentration of protein, $\alpha_P$ and $\gamma_P$ correspond to syntehsis and degradation of `P`. Explicitely, for activators, the set of differential equations is:

$$\begin{align*}
       [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + \alpha_M \frac{[T^n]}{K^n+[T^n]}}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{73}\\
           \frac{\mathrm{d} [P]}{\mathrm{d} t} &=  \alpha_P [M]-\gamma_P[P] \tag{74}
 \end{align*}$$

and for repressors:

$$\begin{align*}
           [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{75}\\
           \frac{\mathrm{d} [P]}{\mathrm{d} t} &=  \alpha_P [M]-\gamma_P[P] \tag{76}
 \end{align*}$$

To solve analytically, we will proceed with separation of variables in eq. 76:

$$\begin{align*}
           \frac{\mathrm{d} [P]}{\mathrm{d} t}  + \gamma_P[P]&=  \alpha_P [M] \tag{77}
 \end{align*}$$

 We need to calculate the integrating factor, $e^{ \int p(x)dx }$, which in this case is $e^{ \int k_2dt }=e^{\gamma_P \cdot t}$. We then multiply both terms in the previous equation by the integrating factor. In this case is simply
 $$\begin{align*}
           \frac{\mathrm{d} [P]}{\mathrm{d} t} e^{\gamma_P \cdot t} + \gamma_P[P] e^{\gamma_P \cdot t}&=  \alpha_P [M] e^{\gamma_P \cdot t}\tag{78}
 \end{align*}$$

 the first term of the equation is simply

 $$\begin{align*}
           \frac{\mathrm{d} ([P]e^{\gamma_P \cdot t})}{\mathrm{d} t}  &=  \alpha_P [M] e^{\gamma_P \cdot t}\tag{78}
 \end{align*}$$

 now we substitute the solution for `M(t)` in Eq 66 and 67 into Eq 78:
 $$\begin{align*}
           \frac{\mathrm{d} ([P]e^{\gamma_P \cdot t})}{\mathrm{d} t}  &=  \alpha_P \big[ M(0) \cdot e^{-\gamma_M \cdot t} + \frac{W}{\gamma_M} (1  -  e^{-\gamma_M \cdot t}) e^{\gamma_P \cdot t}\big] \tag{78}
 \end{align*}$$

 where variable W is simply
 $$W=\alpha_0 + \alpha_M \cdot \Psi(T,K)$$

 This function $\Psi(T,K)$ can refer to the Hill function for activator or for repressor. Now we solve the differential equation:

  $$\begin{align*}
           \int \mathrm{d} ([P]e^{\gamma_P \cdot t})  &=  \alpha_P \int  e^{\gamma_P \cdot t} \big[M(0) \cdot e^{-\gamma_M \cdot t} dt  + \int  \frac{W}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})  dt \big]\tag{78}
 \end{align*}$$

We reorganize the second term:

   $$\begin{align*}
           \int \mathrm{d} ([P]e^{\gamma_P \cdot t})  &=  \alpha_P  \big[\int e^{\gamma_P \cdot t} M(0) \cdot e^{-\gamma_M \cdot t} dt  + \int  \frac{W}{\gamma_M} e^{\gamma_P \cdot t} dt  -  \int e^{\gamma_P \cdot t} e^{-\gamma_M \cdot t}  dt \big]\tag{78}\\
            \int \mathrm{d} ([P]e^{\gamma_P \cdot t})  &=  \alpha_P   \big[M(0) \int e^{(\gamma_P-\gamma_M) t} dt  +  \frac{W}{\gamma_M} \int e^{\gamma_P \cdot t} dt  -  \frac{W}{\gamma_M} \int e^{(\gamma_P-\gamma_M) t}  dt \big]\tag{79}
 \end{align*}$$

 We solve the integral

  $$\begin{align*}
           P(t)e^{\gamma_P \cdot t}  &=  \alpha_P  \big[ \frac{e^{(\gamma_P-\gamma_M)\cdot t}M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{e^{\gamma_P \cdot t}}{\gamma_P}- \frac{e^{(\gamma_P-\gamma_M)\cdot t}}{\gamma_P-\gamma_M}) \big] +C \tag{80}\\
             P(t) &=  \alpha_P  \big[ \frac{e^{-\gamma_M\cdot t}M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{1}{\gamma_P}- \frac{e^{-\gamma_M\cdot t}}{\gamma_P-\gamma_M}) \big] +C \cdot e^{-\gamma_P \cdot t}  \tag{81}
 \end{align*}$$

 Now we calculate the integration constant using the initial condition for the protein $P(t=0)=P[0]$:
  $$\begin{align*}
             P(0) &=  \alpha_P  \big[ \frac{M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{1}{\gamma_P}- \frac{1}{\gamma_P-\gamma_M}) \big] +C  \tag{81}
 \end{align*}$$
 which rearranging terms becomes:
  $$\begin{align*}
             P(0) &=  \alpha_P  \big[ \frac{M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{\gamma_P-\gamma_M-\gamma_P}{\gamma_P(\gamma_P-\gamma_M)}) \big] +C  \tag{82}\\
              P(0) &=  \alpha_P  \big[ \frac{M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{-\gamma_M}{\gamma_P(\gamma_P-\gamma_M)}) \big] +C  \tag{83}\\
              P(0) &=  \alpha_P  \big[ \frac{M(0)}{\gamma_P-\gamma_M} - \frac{W}{\gamma_P(\gamma_P-\gamma_M)} \big] +C \tag{84}\\
 \end{align*}$$

 so
  $$\begin{align*}
               C &= P(0)-  \frac{ \alpha_P M(0)}{\gamma_P-\gamma_M} + \frac{\alpha_P W}{\gamma_P(\gamma_P-\gamma_M)}  \tag{85}\\
               C &= P(0)+  \frac{\alpha_P}{\gamma_P-\gamma_M}  \big[ \frac{W}{\gamma_P} -  M(0)\big] \tag{85}\\
 \end{align*}$$

Finally, the set of equations is:

  $$\begin{align*}
        [M(t)]  &= [M(0)] \cdot e^{-\gamma_M \cdot t} + \frac{W}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{86}\\
                P(t) &=  \alpha_P  \big[ \frac{e^{-\gamma_M\cdot t}M(0)}{\gamma_P-\gamma_M} + \frac{W}{\gamma_M} (\frac{1}{\gamma_P}- \frac{e^{-\gamma_M\cdot t}}{\gamma_P-\gamma_M}) \big] + \big[P(0)+  \frac{\alpha_P}{\gamma_P-\gamma_M}  \big[ \frac{W}{\gamma_P} -  M(0)\big] \big]\cdot e^{-\gamma_P \cdot t}  \tag{87}
 \end{align*}$$

```julia
α_P=0.6*2
α_M=0.6
γ_M=1
γ_P=0.5
T=1.0*0.5
K=2
α_0=0.3*0
M₀=0.4*2
P₀=0.3
n=2
t=collect(0:0.1:10);
ϕ=T.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot(t,t-> M₀*exp(-γ_M*t)+W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,1))
plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ P",seriestype=:line)

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

ϕ=K.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot(t,t-> M₀*exp(-γ_M*t)+W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,1))
plot!(t,t->  α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ P",seriestype=:line)

title!("Central Dogma Repressor Hill")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))

```

```julia
T=0.2
ϕ=T.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=0.2",seriestype=:line)
T=1
ϕ=T.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=1",seriestype=:line)
T=2
ϕ=T.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=2",seriestype=:line)
T=3
ϕ=T.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=3",seriestype=:line)

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

T=0.2
ϕ=K.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=0.2",seriestype=:line)
T=1
ϕ=K.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=1",seriestype=:line)
T=2
ϕ=K.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=2",seriestype=:line)
T=3
ϕ=K.^n./(K.^n.+T.^n)
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot!(t,t-> α_P * ((M₀*exp(-γ_M*t)/(γ_P-γ_M))+W/γ_M*((1/γ_P)-(exp(-γ_M*t)/(γ_P-γ_M))))+C*exp(-γ_P*t),label="\\ T=3",seriestype=:line)

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

 for the special case that no initial protein or mRNA:

  $$\begin{align*}
        [M(t)]  &=  \frac{W}{\gamma_M} (1  -  e^{-\gamma_M \cdot t})\tag{88}\\
                P(t) &=  \alpha_P   \big[\frac{W}{\gamma_M} (\frac{\gamma_P-\gamma_M -\gamma_P e^{-\gamma_M\cdot t}}{\gamma_P(\gamma_P-\gamma_M)}) \big] +  \big[ \frac{\alpha_P W}{(\gamma_P-\gamma_M)\gamma_P} \big]\cdot e^{-\gamma_P \cdot t}  \tag{89}\\
                P(t) &=  \frac{\alpha_P W}{\gamma_P(\gamma_P-\gamma_M)}  \big[\frac{ \gamma_P-\gamma_M -\gamma_P e^{-\gamma_M\cdot t}}{\gamma_M}  +    e^{-\gamma_P \cdot t}\big]\cdot  \tag{90}\\
                 P(t) &=  \frac{\alpha_P W}{\gamma_P(\gamma_P-\gamma_M)}  \big[\frac{ \gamma_P(1- e^{-\gamma_M\cdot t})-\gamma_M}{\gamma_M}  +    e^{-\gamma_P \cdot t}\big]  \tag{91}\\
                  P(t) &=  \frac{\alpha_P W}{\gamma_P-\gamma_M}  \big[\frac{ 1- e^{-\gamma_M\cdot t}}{\gamma_M}   +    \frac{e^{-\gamma_P \cdot t} -1}{\gamma_P}\big] \tag{92}\\
                   P(t) &=  \frac{\alpha_P W}{\gamma_P-\gamma_M}  \big[\frac{ 1- e^{-\gamma_M\cdot t}}{\gamma_M}   -    \frac{1- e^{-\gamma_P \cdot t}}{\gamma_P}\big] \tag{93}
 \end{align*}$$

```julia
α_P=0.6
α_M=0.6
γ_M=3
γ_P=0.5
T=1.
K=2
α_0=0.
M₀=0.0
P₀=0.
n=2
t=collect(0:0.1:10);
ϕ=T.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot(t,t-> W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,0.5))
plot!(t,t-> α_P * W/(γ_P-γ_M)*(((1-exp(-γ_M*t))/γ_M)-((1-exp(-γ_P*t))/γ_P)),label="\\ P",seriestype=:line)

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

ϕ=K.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot(t,t-> W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,0.5))
plot!(t,t-> α_P * W/(γ_P-γ_M)*(((1-exp(-γ_M*t))/γ_M)-((1-exp(-γ_P*t))/γ_P)),label="\\ P",seriestype=:line)

title!("Central Dogma Repressor Hill")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

Having these two analytical solutions, one can ask many different questions, such as:
 - How the amount protein changes with the cooperativity ?
 - How the amount of protein changes with the Hill constant?

since $γ_M >> γ_P$, quite often, people tend to simplify the term that accounts for mRNA degradation

```julia
γ_M=100
γ_P=1
plot(t,t-> (1-exp(-γ_M*t))/γ_M,label="\\ mRNA term",seriestype=:line,ylims = (0,3))
plot!(t,t-> (1-exp(-γ_P*t))/γ_P,label="\\ Protein term",seriestype=:line,ylims = (0,3))
```

Therefore, we can simplify the Eq 93 as:

 $$\begin{align*}
                   P(t) &=  \frac{\alpha_P W}{-\gamma_M}  \big[ -    \frac{1- e^{-\gamma_P \cdot t}}{\gamma_P}\big] \tag{94}\\
                   P(t) &=  \frac{\alpha_P W}{\gamma_M \gamma_P}  \big[1- e^{-\gamma_P \cdot t}\big] \tag{95}
 \end{align*}$$

Therefore, the Central Dogma of Molecular Biology can be written in terms of two simple equations:

$$\begin{align*}
           M(t)  &=  \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M} \big[1  -  e^{-\gamma_M \cdot t}\big]\tag{96}\\
           P(t) &=  \alpha_P \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M \gamma_P}  \big[1- e^{-\gamma_P \cdot t}\big] \tag{97}
 \end{align*}$$

To finalize, the fact that the cell is growing while mRNA and Protein are being produced introduces an extra  dilution factor $\mu$ that reduces the concetration of `M`and `P`. If the volume is assumed to grow linearly,we can simply write this dilution factor as:

$$\begin{align*}
\frac{\mathrm{d} [M]}{\mathrm{d} t} &=  k_M [D]  [T]- \gamma_M [M] - \mu [M] = k_M [D]  [T]- (\gamma_M+\mu) [M]  \tag{98}\\
 \frac{\mathrm{d} [P]}{\mathrm{d} t} &=  \alpha_P [M]-\gamma_P[P] - \mu [P] =  \alpha_P [M]-(\gamma_P +\mu )[P] \tag{99}
 \end{align*}$$

 The final analyitical solutions can be obtained following the same strategy:

$$\begin{align*}
           M(t)  &=  \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M+\mu} \big[1  -  e^{-(\gamma_M+\mu) \cdot t}\big]\tag{100}\\
           P(t) &=  \alpha_P \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{(\gamma_M+\mu) (\gamma_P+\mu)}  \big[1- e^{-(\gamma_P+\mu) \cdot t}\big] \tag{101}
 \end{align*}$$

 In real systems `mRNA` natural degradation is orders of magnitude larger that the effect of dilution. Therefore $\gamma_M+\mu \approx \gamma_M$, and the final equations are:

 $$\begin{align*}
           M(t)  &=  \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M} \big[1  -  e^{-\gamma_M \cdot t}\big]\tag{102}\\
           P(t) &=  \alpha_P \frac{\alpha_0 + \alpha_M \frac{[K^n]}{K^n+[T^n]}}{\gamma_M (\gamma_P+\mu)}  \big[1- e^{-(\gamma_P+\mu) \cdot t}\big] \tag{103}
 \end{align*}$$

```julia
α_P=0.6
α_M=0.6
γ_M=3
γ_P=0.5
μ=0.5
T=1.
K=2
α_0=0.
M₀=0.0
P₀=0.
n=2
t=collect(0:0.1:10);
ϕ=T.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P1=plot(t,t-> W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,0.5))
plot!(t,t-> α_P * W/(γ_M*(γ_P+μ))*(1-exp(-γ_P*t)),label="\\ P",seriestype=:line)

title!("Central Dogma Activator Hill")
xaxis!("Time")
yaxis!("Concentration")

ϕ=K.^n./(K.^n.+T.^n)
α_M=k_M*D
W=α_0 + α_M * ϕ
C=P₀ + α_P / (γ_P-γ_M) * ((W/γ_P)-M₀)
P2=plot(t,t-> W/γ_M*(1-exp(-γ_M*t)),label="\\ M",seriestype=:line,ylims = (0,0.5))
plot!(t,t-> α_P * W/(γ_M*(γ_P+μ))*(1-exp(-γ_P*t)),label="\\ P",seriestype=:line)

title!("Central Dogma Repressor Hill")
xaxis!("Time")
yaxis!("Concentration")

plot(P1,P2,layout=(1,2),legend=true,size = (800, 500))
```

## Numerical solution of the Central Dogma

Another way to solve this type of models when they become too complex is using nuymerical simulations

```julia
using DifferentialEquations
using ParameterizedFunctions
tspan = (0.0,10)
```

    (0.0, 10)

```julia
CentralDogma4_DSL! = @ode_def ab begin
   dM = α_0-γ_M*M+α_M*T^n/(K^n +T^n)
   dP =   α_P * M - γ_P * P
    end α_0 α_M α_P γ_M γ_P T K n

CentralDogma5_DSL! = @ode_def ab begin
   dM = α_0-γ_M*M+α_M*K^n/(K^n +T^n)
   dP =   α_P * M - γ_P * P
    end α_0 α_M α_P γ_M γ_P T K n
```

    (::ab{var"#89#93",var"#90#94",var"#91#95",Nothing,Nothing,var"#92#96",Expr,Expr}) (generic function with 2 methods)

```julia
p=[α_0,α_M,α_P,γ_M,γ_P,T,K,n];
u₀ = [M₀,P₀]
prob3 = ODEProblem(CentralDogma4_DSL!,u₀,tspan,p)
prob4 = ODEProblem(CentralDogma5_DSL!,u₀,tspan,p)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 10.0)
    u0: [0.0, 0.0]

```julia
sol3 = solve(prob3)
sol4 = solve(prob4)
P3=plot(sol3,label=["mRNA","Protein"],ylims = (0,0.5))
title!("Central Dogma activator Hill function")
xlabel!("Time [s]")
ylabel!("Concentration [nM]")
P4=plot(sol4,label=["mRNA","Protein"],ylims = (0,0.5))
title!("Central Dogma repressor Hill function")
xlabel!("Time [s]")
ylabel!("Concentration [nM]")
plot(P3,P4,layout=(1,2),legend=true,size = (800, 500))
```

which are equivalent to the dynamcis predicte in the analytical soloution

```julia
function CentralDogma_activator_parameters(α_0,α_M,α_P,γ_M,γ_P,T,K,n)
     p=[α_0,α_M,α_P,γ_M,γ_P,T,K,n];
     prob3 = ODEProblem(CentralDogma4_DSL!,u₀,tspan,p)
     sol3 = solve(prob3)
     x=("T = $(T)")
     plot!(sol3,vars=(2),label=x)
end

function CentralDogma_repressor_parameters(α_0,α_M,α_P,γ_M,γ_P,T,K,n)
     p=[α_0,α_M,α_P,γ_M,γ_P,T,K,n];
     prob4 = ODEProblem(CentralDogma5_DSL!,u₀,tspan,p)
     sol4 = solve(prob4)
     x=("T = $(T)")
     plot!(sol4,vars=(2),label=x)
end
```

    CentralDogma_repressor_parameters (generic function with 1 method)

```julia
plot()
for T in [0.2,1,2,3]
   CentralDogma_activator_parameters(α_0,α_M,α_P,γ_M,γ_P,T,K,n)
end
title!("Central Dogma activator different T concentrations")
xlabel!("Time [s]")
ylabel!("Concentration [a.u]")
```

```julia
plot()
for T in [0.2,1,2,3]
   CentralDogma_repressor_parameters(α_0,α_M,α_P,γ_M,γ_P,T,K,n)
end
title!("Central Dogma Repressor different T concentrations")
xlabel!("Time [s]")
ylabel!("Concentration [a.u]")
```

## Conclusions

In conclusion, both the activator form and the repressor forms of the Hill function can be used to model numerically the transctiption and translation of mRNA into protein regulated by transcriptional activators and represors, respectively. The dynamics is similar, but the dependence on the amount of transcription factor `T` is reversed.

The Hill functions are widely used in Biochemistry, Physiology, pharmacology and Genetics.

It is physically unrealistic:
all ligands have to bind simultaneously
exact in conditions of extremely high cooperativity
it is based on equilibrium considerations (Biology is far from equilibrium)

Sequential or independent binding are more realistic, but they represent often a small correction versus the highly simple Hill equation.

Therefore, the simplicity of the Hill equation wins and it is widely used in many biological contexts (as we will see…)