# Enzyme Kinetics
We now study common reaction mechanisms that describe enzyme catalysis. Enzymes can dramatically accelerate the rate of biochemical reactions inside and outside living cells. The absolute rates of biochemical reactions are key biological design variables because they can evolve from a very low rate as determined by the mass action kinetics based on collision frequencies, to a very high and specific reaction rate as determined by appropriately-evolved enzyme properties. We first describe the procedure used to derive enzymatic rate laws, which we then apply to the Michaelis-Menten reaction mechanism, then to the Hill model, and finally to the symmetry model. The first is used to describe plain chemical transformations, while the latter two are used to describe regulatory effects.

**MASSpy** will be used to demonstrate some of the topics in this chapter.

```python
from mass import (
    MassModel, MassMetabolite, MassReaction, Simulation, MassSolution)
from mass.visualization import plot_time_profile, plot_phase_portrait
```

Other useful packages are also imported at this time.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
```

## Enzyme Catalysis
Enzymes are catalysts that accelerate biochemical transformations in cells. Almost all enzymes are proteins. There are also catalytically active ribonucleic acids, called "ribozymes." The fundamental properties of enzyme catalysis are described in this section.

### Enzymatic activity
The activity of an enzyme is measured by determining the increase in the reaction rate relative to the absence of the enzyme. In other words we compare the reaction rate of the un-catalyzed reaction to the catalyzed rate. The ratio can be thought of as an acceleration factor and this number can be quite high, sometimes by many million-fold.

### Reaction and substrate specificity
Enzymes are usually very specific both with respect to the type of reaction being catalyzed (reaction specificity) and with respect to the reactants (the "substrates") that they act on. Highly specific enzymes catalyze the cleavage of only one type of a chemical bond, and only in one substrate. Other enzymes may have a narrow reaction specificity, but broad substrate specificity, i.e., they act on a number of chemically similar substrates. Rare enzymes exist that have both low reaction specificity and low substrate specificity.

**Figure 5.1:** Basic principles of enzyme catalysis. From (Koolman, 2005).

### Catalysis
As discussed in Chapter 2 (Figure 2.4), two molecules can only react with each other if they collide in a favorable orientation. Such collisions may be rare, and thus the reaction rate is slow. An un-catalyzed reaction starts with a favorable collision as shown in Figure 5.1a. Before the products are formed, the collision complex A-B has to pass through what is called a _transition state_. Its formation requires _activation energy_. Since activation energies can be quite high, only a few A-B complexes have this amount of energy, and thus a productive transition state arises only for a fraction of favorable collisions. As a result, conversion only happens occasionally even when the reaction is thermodynamically feasible; i.e., when the net change in Gibbs free energy is negative ($\Delta G < 0$).

Enzymes can facilitate the probability of a favorable collision and lower the activation energy barrier, see Figure 5.1b,c. Enzymes are able to bind their substrates in the catalytic site. As a result, the substrates are favorably oriented relative to one another, greatly enhancing the probability that productive A-B complexes form. The transition state is stabilized leading to a lowered activation energy barrier.

### Information on enzymes
Detailed information is available on a large number of enzymes. This include structural information, the organism source, and other characteristics. An example is shown in Figure 5.2. Many online sources of such information exist.

**Figure 5.2:** Detailed information on enzymes is available. From PDB.

## Deriving Enzymatic Rate Laws
The chemical events underlying the catalytic activities of enzymes are described by a reaction mechanism.  A reaction mechanism is comprised of the underlying elementary reactions that are believed to take place. A rate law is then formulated to describe the rate of reaction.

A rate law describes the conversion of a substrate $(x_1)$ by an enzyme into a product $(x_2)$:

$$\begin{equation} x_1 \stackrel{v}{\rightarrow} x_2 \tag{5.1} \end{equation}$$

where $v$ is a function of the concentrations of the chemical species involved in the reaction. The steps involved in the development and analysis of enzymatic rate laws are illustrated in Figure 5.3 and they are as follows:

**Figure 5.3:** The process of formulating enzymatic rate laws. QSSA represents the quasi-steady state assumption and QEA represents the quasi-equilibrium assumption.

* Formulate the dynamic mass balances based on the elementary reactions in the postulated reaction mechanism,

* Identify time invariants, or conservation relationships,

* Reduce the dynamic dimension of the reaction mechanism by eliminating dynamically dependent variables using the conservation relationships,

* Apply commonly used simplifying kinetic assumptions to formulate a rate law, representing a reduction in the dynamic dimension of the kinetic model,

* Apply mathematical and numerical analysis to determine when the simplifying assumptions are valid and the reaction rate law can be used; and

* Identify key dimensionless parameter ratios. This last step is optional and used by those interested in deeper mathematical analysis of the properties of the rate laws.

The use of enzymatic rate laws in dynamic network models is hampered by their applicability in vivo based on in vitro measurements. From a practical standpoint, with the numerical simulation capacity that is now routinely available, applying simplifying assumptions may no longer be needed for computational simplification and convenience. However, it is useful to help understand the historical origin of enzymatic rate laws, the simplifications on which they are based, and when it may be desirable to use them.

## Michaelis-Menten Kinetics
The simplest enzymatic reaction mechanism, first proposed by Henri (Henri, 1903) but named after Michaelis and Menten (Michaelis, 1913) is;

$$\begin{equation} S + E \underset{k_{-1}}{\stackrel{k_1}{\rightleftharpoons}} X \stackrel{k_2}{\rightarrow} E + P \tag{5.2} \end{equation}$$

where a substrate, $S$, binds reversibly to the enzyme, $E$, to form the intermediate, $X$, which can break down to give the product, $P$, and regenerate the enzyme. Note that it is similar to the reaction mechanism of two connected reversible bi-linear reactions (Eq. (4.38)) with $x_5 = x_2$, as one of the original reactants $(E)$ is regained in the second step. Historically speaking, the Michaelis-Menten scheme is the most important enzymatic reaction mechanism. A detailed account of the early history of Michaelis-Menten kinetics is found in (Segal, 1959).

### Step 1: Dynamic mass balances for Michaelis-Menten kinetics
Applying the law of mass action to the Michaelis-Menten reaction mechanism, one obtains four differential equations that describe the dynamics of the concentrations of the four chemical species involved in the reaction mechanism:

$$\begin{align} \frac{ds}{dt} &= -k_1es + k_{-1}x, & s(t = 0) = s_0 \\ \frac{dx}{dt} &= k_1es - (k_{-1} + k_2)x, & x(t = 0) = 0 \\ \frac{de}{dt} &= -k_1es + (k_{-1} + k_2)x, & e(t = 0) = e_0 \\ \frac{dp}{dt} &= k_2x, & p(t = 0) = 0 \\ \end{align} \tag{5.3}$$

where the lower case letters denote the concentrations of the corresponding chemical species. The initial conditions shown are for typical initial rate experiments where substrate and free enzyme are mixed together at time $t=0$. $e_0$ and $s_0$ denote the initial concentration of enzyme and substrate, respectively. No mass exchange occurs with the environment.

### Step 2: Finding the time invariants for Michaelis-Menten kinetics
Using $\textbf{x} = (s, e, x, p)$ and $\textbf{v} = (k_1es, \ k_{-1}x, \  k_2x)$ the stoichiometrix matrix is

$$\begin{equation} \textbf{S} = \begin{pmatrix} {-1} & {1} & {0} \\ {-1} & {1} & {1} \\ {1} & {-1} & {-1} \\ {0} & {0} & {1} \\ \end{pmatrix} \end{equation} \tag{5.4}$$

It has a rank of 2 and thus there are two conservation quantities. They are the total concentration of the enzyme and total concentration of the substrate:

$$\begin{align} e_0 & = e + x \tag{5.5} \\ s_0 &= s + x  + p \tag{5.6} \end{align}$$

### Step 3: Reducing the dynamic description for Michaelis-Menten kinetics
As a consequence of the two conservation relationships, only two of equations 5.3 are dynamically independent. Choosing the substrate, $s$, and the intermediate complex, $x$, concentrations as the two independent variables, the reaction dynamics are described by:

$$\begin{align} \frac{ds}{dt} &= -k_1e_0s + (k_1s + k_{-1})x, \ &s(t = 0)=s_0 \tag{5.7} \\ \frac{dx}{dt} &= k_1e_0s - (k_1s + k_{-1} + k_2)x, \ &x(t = 0)=0 \tag{5.8} \\ \end{align}$$

The major problem with this mass action kinetic model is that it is mathematically intractable (Hommes, 1962). Equations 5.7 and 5.8 can be reduced to an Abel type differential equation whose solution cannot be obtained in a closed form.

### Step 4: Applying kinetic assumptions for Michaelis-Menten kinetics
A closed form analytical solution to the mass action kinetic equations, 5.7 and 5.8, is only attainable by using simplifying kinetic assumptions. Two assumptions are used: the _quasi-steady state assumption_ (QSSA) and the _quasi-equilibrium assumption_ (QEA).

#### The quasi-steady state assumption:
The rationale behind the quasi-steady state assumption (Briggs, 1925) is that, after a rapid transient phase, the intermediate, $X$, reaches a quasi-stationary state in which its concentration does not change appreciably with time. Applying this assumption to Eq. (5.8) (i.e., $dx/dt=0$) gives the concentration of the intermediate complex as:

$$\begin{equation} x_{qss} = \frac{e_0s}{K_m + s} \tag{5.9} \end{equation}$$

where $K_m = (k_{-1} + k_2)/k_1$ is the well-known Michaelis constant. Substituting $x_{qss}$ into the differential equation for the substrate (Eq. (5.7)) gives the rate law

$$\begin{equation} \frac{ds}{dt} = \frac{-k_2e_0s}{K_m + s} \tag{5.10} \end{equation}$$

which is the well-known Michaelis-Menten equation, where $v_m$ is the maximum reaction rate (or reaction velocity).

Initially, the quasi-steady state assumption was justified based on physical intuition, but justification for its applicability is actually found within the theory of singular perturbations (Bowen, 1963) Eq. (5.10) can be shown to be the first term in an asymptotic series solution derived from singular perturbation theory (Heineken,1967), (Meiske, 1978); see review in (Palsson, 1984).

#### The quasi-equilibrium assumption:
Here, one assumes that the binding step quickly reaches a quasi-equilibrium state (Henri, 1903), (Michaelis, 1913) where

$$\begin{equation} \frac{se}{x} = \frac{s(e_0 - x)}{x} = \frac{k_{-1}}{k_1} = K_d, \ \text{or} \ x_{qe} = \frac{e_0s}{K_d + s} \tag{5.11} \end{equation}$$

holds. $K_d$ is the disassociation equilibrium constant. Note the similarity to Eq. (5.9). Hence, one obtains the rate law

$$\begin{equation} \frac{dp}{dt} = \frac{k_2e_0s}{K_d + s} \tag{5.12} \end{equation}$$

by using Eq. (5.11) in the differential equation for the product $P$.

### Step 5: Numerical solutions for Michaelis-Menten kinetics
The full dynamic description of the kinetics of the reaction (Eq. (5.7) and (5.8)) can be obtained by direct numerical integration. The results are most conveniently shown on a phase portrait along with the transient response of the concentrations on both the fast and slow time scales, see Figure 5.4.

#### QSSA Solution for Michaelis-Menten kinetics

```python
t0 = 0
tf = 1e3
# QSSA Assumption
# Define function to integrate
def qssa(t, s, *params):
    k2, e0, Km = params
    dsdt = (-k2*e0*s)/(Km + s)
    return dsdt

# Define initial conditions and parameters for integration
s0 = 1

e0 = (1/100)
k2 = 1
Km = 1
params = [k2, e0, Km]

# Obtain numerical solutions
sol_obj = solve_ivp(fun=lambda t, s: qssa(t, s, *params),
                    t_span=(t0, tf), y0=[s0])
t, s_sol = (sol_obj.t, sol_obj.y)
x_sol = np.array([(e0 * val)/(Km + val) for val in s_sol])
# Store solutions into Solution Objects
qssa_sol = MassSolution(
    "QSSA", solution_type="Conc",
    data_dict={"s": s_sol[0], "x": x_sol[0]},
    time=t, interpolate=False)
```

#### Numerical Solution for Michaelis-Menten kinetics

```python
model = MassModel('Michaeli_Menten')
## Define metabolites
s = MassMetabolite("s")
e = MassMetabolite("e")
x = MassMetabolite("x")
p = MassMetabolite("p")
# Define reactions
v1 = MassReaction("v1")
v2 = MassReaction("v2", reversible=False)
v1.add_metabolites({s: -1, e: -1, x: 1})
v2.add_metabolites({x: -1, e: 1, p: 1})
model.add_reactions([v1, v2])
## Define parameters
v1.kf = 2
v1.Keq = 2
v2.kf = 1
# Define initial conditions
model.update_initial_conditions({s: s0, e: e0, x: 0, p: 0})

# Solve
MM_simulation = Simulation(model, verbose=True)
conc_sol, flux_sol = MM_simulation.simulate(model, (t0, tf))
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Michaeli_Menten' into RoadRunner.

```python
fig_5_4 = plt.figure(figsize=(9, 7))
gs = fig_5_4.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1.5],
                          height_ratios=[1, 1])

ax1 = fig_5_4.add_subplot(gs[0, 0])
ax2 = fig_5_4.add_subplot(gs[0, 1])
ax3 = fig_5_4.add_subplot(gs[1, 1])
# Phase portrait of both solutions' substrate vs. intermediate
plot_phase_portrait(
    conc_sol, x=s, y=x, ax=ax1, legend=["Numerical Solution"],
    annotate_time_points="endpoints",
    annotate_time_points_color=["r", "b"],
    annotate_time_points_labels=True);
plot_phase_portrait(
    qssa_sol, x=s, y=x, ax=ax1, legend=["QSSA", "lower outside"],
    xlabel=s.id, ylabel=x.id, linestyle=["--"],
    title=("(a) Phase Portrait", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

# Time profile of solutions' substrate concentration
plot_time_profile(conc_sol, observable=s, ax=ax2);
plot_time_profile(
    qssa_sol, observable=s, ax=ax2,
    xlabel="Time", ylabel="Concentration",
    title=("(b) Substrate Concentration", {"size": "x-large"}),
    linestyle=["--"]);
# Time profile of solutions' intermediate concentration
plot_time_profile(conc_sol, observable=x, ax=ax3);
plot_time_profile(
    qssa_sol, observable=x, ax=ax3,
    xlabel="Time", ylabel="Concentration",
    title=("(c) Intermediate Concentration", {"size": "x-large"}),
    linestyle=["--"]);
fig_5_4.tight_layout()
```

**Figure 5.4:** The transient response of the Michaelis-Menten reaction mechanism, for $k_2 = k_{-1}, \ 100e_0 = K_m \ \text{and} \ s_0 = K_m$. (a) The phase portrait . (b) The substrate concentrations. (c) The intermediate concentrations. The solid and dashed line represent the quasi-steady state and the full numerical solution respectively.

* _The phase portrait_. The phase portrait is shown in of Figure 5.4a and it shows how the reaction rapidly approaches the quasi-steady state line and then moves along that line towards the equilibrium in the origin where the reaction has gone to completion.

* _The fast motion_. With the slider on moved to the left, Figure 5.4 shows the changes in the concentrations during the faster time scale. The intermediate concentration exhibits a significant fast motion, while the substrate does not move far from its initial value.

* _The slow motion_. The changes in the concentrations during the slower time scale are shown when the slider is on the right of Figure 5.4. Both the substrate and the intermediate complex decay towards zero. During the decay process, the complex is in a quasi-stationary state and the motion of the substrate drives the reaction dynamics. The quasi-steady state solution gives a good description of the motion on the slower time scale.

### Step 6: Identification of dimensionless parameters for Michaelis-Menten kinetics
Simulation studies suggests that there are three dimensionless parameters of interest:

$$\begin{equation} a = k_2/k_{-1}, \ b = e_0/K_m, \ c = s_0/K_m \tag{5.13} \end{equation}$$

This result is also found by rigorous mathematical analysis (Palsson, 1984). The dynamic behavior of the reaction is determined by three dimensionless groups: a ratio of kinetic constants and the two initial conditions scaled to $K_m$.

1. The first dimensionless group, a, is a ratio consisting only of kinetic constants, $k_2/k_{-1}$. This ratio has been called the 'stickiness number'  (Palsson, 1984), (Palsson, 1984a), since a substrate is said to stick well to an enzyme if $k_2 > k_{-1}$. Once $X$ is formed it is more likely to break down to yield the product than to revert back to substrate.

2. The second dimensionless number, $e_0/K_m$, is a dimensionless concentration parameter - the total enzyme concentration relative to the Michaelis constant. This quantity varies from one situation to another and takes particularly different values under _in vitro_ and _in vivo_ conditions. The enzyme concentrations used _in vitro_ are several orders of magnitude lower than the $K_m$ values (Masters, 1977), (Srere, 1967), (Srere, 1970). In vivo enzyme concentrations can approach the same order of magnitude as $K_m$.

3. The third dimensionless ratio, $s_0/K_m$, is the initial condition for the substrate concentration. Typical values for this ratio _in vivo_ is on the order of unity.

```python
# Define new initial conditions and parameters for integration
s0 = (1/100)

e0 = (1/100)
k2 = 1
Km = 1
params = [k2, e0, Km]

# Obtain numerical solutions
sol_obj = solve_ivp(fun=lambda t, s: qssa(t, s, *params),
                    t_span=(t0, tf), y0=[s0])
s_sol = sol_obj.y
x_sol = np.array([(e0 * val)/(Km + val) for val in s_sol])
# Store solutions into MassSolution Objects
qssa_sol = MassSolution(
    "QSSA", solution_type="Conc",
    data_dict={"s": s_sol[0], "x": x_sol[0]},
    time=sol_obj.t, interpolate=False)

# Update initial conditions for MassModel
model.update_initial_conditions({s: s0})

# Solve
MM_simulation = Simulation(model, verbose=True)
conc_sol, flux_sol = MM_simulation.simulate(model, (t0, tf))
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Michaeli_Menten' into RoadRunner.

```python
fig_5_5 = plt.figure(figsize=(9, 7))
gs = fig_5_5.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1.5],
                          height_ratios=[1, 1])

ax1 = fig_5_5.add_subplot(gs[0, 0])
ax2 = fig_5_5.add_subplot(gs[0, 1])
ax3 = fig_5_5.add_subplot(gs[1, 1])
# Phase portrait of both solutions' substrate vs. intermediate
plot_phase_portrait(
    conc_sol, x=s, y=x, ax=ax1, legend=["Numerical Solution"],
    annotate_time_points="endpoints",
    annotate_time_points_color=["r", "b"],
    annotate_time_points_labels=True);
plot_phase_portrait(
    qssa_sol, x=s, y=x, ax=ax1, legend=["QSSA", "lower outside"],
    xlabel=s.id, ylabel=x.id, linestyle=["--"],
    title=("(a) Phase Portrait", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

# Time profile of solutions' substrate concentration
plot_time_profile(conc_sol, observable=s, ax=ax2);
plot_time_profile(qssa_sol, observable=s, ax=ax2,
                xlabel="Time", ylabel="Concentration",
                title=("(b) Substrate Concentration", {"size": "x-large"}),
                linestyle=["--"]);
# Time profile of solutions' intermediate concentration
plot_time_profile(conc_sol, observable=x, ax=ax3);
plot_time_profile(qssa_sol, observable=x, ax=ax3,
                xlabel="Time", ylabel="Concentration",
                title=("(c) Intermediate Concentration", {"size": "x-large"}),
                linestyle=["--"]);
fig_5_5.tight_layout()
```

**Figure 5.5:** The transient response of the Michaelis-Menten reaction mechanism, for $k_2 = k_{-1},\ 100e_0 = K_m \ \text{and} \ s_0 = K_m$. (a) The phase portrait . (b) The substrate concentrations. (c) The slow transients. The solid and dashed line represent the quasi-steady state and the full numerical solution respectively.

### Comment on the criterion $e_0 << s_0$
Historically, the commonly accepted criterion for the applicability of the quasi-steady state assumption is that the initial concentration of the enzyme must be much smaller than that of the substrate. The actual criterion is $e_0 << K_m, \ \text{or} \ b << 1$ (Palsson, 1984). Figure 5.5 shows the reaction dynamics for $e_0 = K_m, \ e_0 = 100s_0, \ 100k_2 = k_{-1}$, which is analogous to Figure 5.4, except the initial substrate concentration is now a hundred times smaller than $K_m$. In other words, we have $e_0 = s_0 << K_m$ and, as demonstrated in Figure 5.5, the quasi-steady state assumption is applicable.

## Hill-kinetics for Enzyme Regulation
### Regulated enzymes
Enzyme activity is regulated by the binding of small molecules to the enzyme resulting in an altered enzymatic activity. Such binding can inhibit or activate the catalytic activities of the enzyme. The regulation of enzymes such regulators represents a 'tug of war' between the functional states of the enzyme, see Figure 5.6. A simple extension of the oldest reaction mechanisms for ligand binding to oligomeric protein, i.e., oxygen binding to hemoglobin, is commonly used to obtain simple rate laws for regulated enzymes (Hill, 1910).

**Figure 5.6:** An example of a regulated multimeric enzyme. The T form of the enzyme created by inhibitor binding is inactive, where as the R form, where no inhibitor is bound, is catalytically active. From (Koolman, 2005) (reprinted with permission).

### The reaction mechanism for Hill-kinetics
The Hill reaction mechanism is based on two reactions: a catalytic conversion and the sequestration of the enzyme in an inactive form. It assumes that the catalyzed reaction is an irreversible bi-molecular reaction between the substrate, $S$, and the enzyme, $E$, to form the product,$P$, and the free enzyme in a single elementary reaction:

$$\begin{equation} S + E \stackrel{k}{\rightarrow} E + P \tag{5.14} \end{equation}$$

The enzyme in turn can be put into a catalytically inactive state, $X$, through binding simultaneously and reversibly to $\nu$ molecules of an inhibitor, $I$:

$$\begin{equation} E + {\nu}I \underset{k_{-i}^-}{\stackrel{k_{i}^+}{\rightleftharpoons}} X \tag{5.15} \end{equation}$$

Numerical values for $\nu$ often exceed unity. Thus, the regulatory action of $I$ is said to be lumped in the simple $E$ to $X$ transformation, as values of $\nu$ greater than 1 are chemically unrealistic. Numerical values estimated from data show that the best fit values for $\nu$ are not integers; for instance $\nu$ is found to be around 2.3 to 2.6 for $O_2$ binding to hemoglobin. Section 5.5 describes more realistic reaction mechanisms of serial binding of an inhibitor to a regulated enzyme to sequester it in an inactive form.

### Step 1: Dynamic mass balances for Hill-kinetics
The mass action kinetic equations are

$$\begin{equation} \frac{ds}{dt} = -v_1, \ \frac{de}{dt} = -v_2 + v_3, \ \frac{dp}{dt} = v_1, \ \frac{di}{dt} = -\nu (v_2 - v_3), \ \frac{ds}{dt} = v_2 - v_3 \end{equation}$$

where the reaction rates are

$$\begin{equation} v_1 = kse, \ v_2 = k_i^+i^{\nu}e, \ v_3 = k_i^-x \tag{5.16} \end{equation}$$

### Step 2: Finding the time invariants for Hill-kinetics
We define $\textbf{x} = (s, e, p, i, x) \ \text{and} \ \textbf{v} = (ks, k_i^+i^{\nu}e, k_i^-x)$. The stoichiometric matrix is then

$$\begin{equation} \textbf{S} = \begin{pmatrix} {-1} & {0} & {0} \\ {0} & {-1} & {1} \\ {1} & {0} & {0} \\ {0} & {-\nu} & {\nu} \\ {0} & {1} & {-1} \\ \end{pmatrix} \end{equation}$$
$$\tag{5.17}$$

and has a rank of two. The conservation quantities are a balance on the substrate, the enzyme, and the inhibitor:

$$\begin{equation} s_0 = s + p, \ e_0 = e + x, \ i_0 = i + \nu x \tag{5.18} \end{equation}$$

### Step 3: Reducing the dynamic description for Hill-kinetics
We need two differential equations to simulate the dynamic response and then the remaining three variables can be computed from the conservation relationships. We can choose the substrate, $s$, and the concentration of the enzyme, $e$:

$$\begin{equation} \frac{ds}{dt} = kse, \ \frac{de}{dt} = -k_i^+i^{\nu}e + k_i^-x \tag{5.19} \end{equation}$$

then $p$, $x$ and $i$ are computed from Eq. (5.18).

### Step 4: Applying simplifying kinetic assumptions for Hill-kinetics

If we assume that the binding of the inhibitor is fast, so that a quasi-equilibrium  forms for the reaction of Eq. (5.15), we have

$$\begin{equation} v_2 = v_3, \ \text{thus} \ x = (k_i^+/k_i^-)i^{\nu}e = (i/K_i)^{\nu}e, \ \text{and} \ \frac{de}{dt} = \frac{dx}{dt} = \frac{di}{dt} = 0 \tag{5.20} \end{equation}$$

where $K_i$ is a "per-site" dissociation constant for Eq. (5.15). The enzyme is in one of two states, so that we have the mass balance

$$\begin{equation} e_0 = e + x = (1 + (i/K_i)^{\nu})e \ \text{or} \ e(i) = \frac{e_0}{1 + (i/K_i)^{\nu}} \tag{5.21} \end{equation}$$

where $e_0$ is the total concentration of the enzyme. Using the mass balance and the quasi-equilibrium assumption gives the flux through the regulated reaction as

$$\begin{equation} v(i) = ke(i)s = \frac{ke_0s}{1 + (i/K_i)^{\nu}} = \frac{v_m}{1 + (i/K_i)^{\nu}} \tag{5.22} \end{equation}$$

with $v_m = ke_0s$. The Hill model has three parameters: 1) $\nu$, the degree of cooperativity,  2) $K_i$, the dissociation constant for the inhibitor and, 3) $v_m$, the maximum reaction rate or the capacity of the enzyme. We note that

$$\begin{equation} f_e = \frac{e(i)}{e_0} = \frac{1}{1 + (i/K_i)^{\nu}} \tag{5.23} \end{equation}$$

represents the fraction of the enzyme that is in the active state. Note that $f_e \lt 1$ for any finite concentration of the inhibitor.

```python
t0 = 0
tf = 10

def hill(t, state_vars, *params):
    s, p, e, i, x = state_vars
    k1, k_plus,k_minus, nu = params
    # Reaction Rates
    v1 = k1 * s * e
    v2 = k_plus * i**nu * e
    v3 = k_minus * x
    # Differential equations
    diffeqs =[-v1,          # ds/dt
              v1,           # dp/dt
              -v2 + v3,     # de/dt
              -nu*(v2 - v3), # di/dt
              v2 - v3]      # dx/dt
    return diffeqs

# Define initial conditions
s0, p0, e0, i0, x0 = (1, 0, 1, 1, 0)

# Define paramters
k1 = 1
k_plus, k_minus = (100, 100)
nu = 2
params = [k1, k_plus, k_minus, nu]

# Obtain numerical solutions
sol_obj = solve_ivp(
    fun=lambda t, state_vars: hill(t, state_vars, *params),
    t_span=(t0, tf), y0=[s0, p0, e0, i0, x0])
# Store solutions into Solution Objects
sol_dict = dict(zip(["s", "p", "e", "i", "x"], sol_obj.y))
hill_sol = MassSolution(
    "Hill", solution_type="Conc", data_dict=sol_dict,
    time=sol_obj.t, interpolate=False)
```

```python
fig_5_7 = plt.figure(figsize=(9, 8))
gs = fig_5_7.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1],
                          height_ratios=[1, 1])

ax1 = fig_5_7.add_subplot(gs[0, 0])
ax2 = fig_5_7.add_subplot(gs[0, 1])
ax3 = fig_5_7.add_subplot(gs[1, 0])
ax4 = fig_5_7.add_subplot(gs[1, 1])

plot_phase_portrait(
    hill_sol, x="s", y="e", ax=ax1, xlabel="s", ylabel="e",
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(a) Phase Portrait of s vs. e", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);
plot_phase_portrait(
    hill_sol, x="e", y="x", ax=ax2, xlabel="e", ylabel="x",
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(b) Phase Portrait of e vs. x", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

plot_phase_portrait(
    hill_sol, x="i", y="x", ax=ax3, xlabel="i", ylabel="x",
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(a) Phase Portrait of i vs. x", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);
plot_time_profile(
    hill_sol, ax=ax4, legend="right outside",
    title=("(d) Concentration Profiles", {"size": "x-large"}));
fig_5_7.tight_layout()
```

**Figure 5.7:** The transient response of the Hill reaction mechanism, for $k_i^+ = k_i^- = 100$, $k = 1$, $\nu = 2$, $x_0 = 0$ and $e_0 = s_0 = i_0 = 1$. (a) The phase portraits of $s$ and $e$. (b) The phase portraits of $e$ and $x$. (c) The phase portraits of $i$ and $x$. (d) The concentration profiles.

### Step 5: Numerical solutions for Hill-kinetics
The dynamic response of the Hill reaction mechanism is shown in Figure 5.7. The trace in the $s$ vs. $e$ phase portrait is L-shaped, showing a rapid initial equilibration of the enzyme to the inhibitor (the vertical line), followed by the slower conversion of the product (the horizontal line). These two reactions are naturally (stoichiometrically) decoupled and separated in time for the numerical values of the kinetic constants used.

The phase portraits for $e$ vs. $x$ and $i$ vs. $x$ are straight lines as given by the conservation Eq. (5.18), see Figure 5.7b,c. The two phase transient responses in Figure 5.7d shows the rapid equilibration of the enzyme and the slow conversion of substrate. Under these parameter conditions, the QEA should give good results.

### Step 6: Estimating key parameters
There are two features of the Hill rate law that are of interest:

#### Applicability of the quasi-equilibrium assumption.
Given the fact that the two reactions have characteristic times scales, their relative magnitude is of key concern when it comes to the justification of the QEA:

$$\begin{equation} a = (\frac{\text{characteristic binding time of the inhibitor}}{\text{characteristic turnover time of the substrate}}) = \frac{k}{k_i^+} \tag{5.24} \end{equation}$$

If $a$ is much smaller than unity, we would expect the QEA to be valid. In Figure 5.7, $a$ is 0.01.

#### Regulatory characteristics
The Hill rate law has a sigmoidal shape with sensitivity of the reaction rate to the end product concentration as

$$\begin{equation} v_i = \frac{\partial v}{\partial i} = \frac{-\nu v_m}{i} \frac{(i/K_i)^{\nu}}{[1 + (i/K_i)^{\nu}]^2} \tag{5.25} \end{equation}$$

which has a maximum

$$\begin{equation} v_i^* = -\frac{v_m}{K_i}N(\nu) \ \text{where} \ N(\nu) = \frac{1}{4\nu}(\nu - 1)^{1 - 1/\nu}(\nu + 1)^{1 + 1/\nu} \tag{5.26} \end{equation}$$

at the inflection point

$$\begin{equation} i^* = K_i(\frac{\nu - 1}{\nu + 1})^{1/\nu} \tag{5.27} \end{equation}$$

For plausible values of $\nu$, the function $N(\nu)$ is on the order of unity (Table 5.1), and hence the maximum sensitivity $v_i^*$ is on the order of $(-v_m/K_i)$. The ratio $(K_i/v_m)$ can be interpreted as a time constant characterizing the inhibition process;

$$\begin{equation} t_i = \frac{K_i}{v_m} = [\frac{\text{concentration}}{\text{concentration/time}}]\tag{5.28} \end{equation}$$

This estimate represents an upper bound since the steady state concentration of $i$ can be different from $i^*$. The turnover of the substrate happens on a time scale defined by the rate constant $t_s = 1/k$. Thus, a key dimensionless property is

$$\begin{equation} b = \frac{t_s}{t_i} = \frac{1/k}{K_i/v_m} = \frac{v_m}{kK_i} = \frac{e_t}{K_i} \tag{5.29} \end{equation}$$

Therefore, the dimensionless parameter $b$ can be interpreted as a ratio of time constants or as a ratio of concentration ranges.

**Table 5.1:** The values of the function $N(\nu)$ and $i^*/K_i$ at the inflection point.

```python
def N(nu): # N(v)
    return (1/(4*nu))*((nu-1)**(1-1/nu))*((nu+1)**(1+1/nu))

def i_Ki(nu): # i*/Ki
    return ((nu-1)/(nu+1))**(1/nu)

cols = [nu for nu in np.linspace(2, 5, 4)]
tab_5_1 = pd.DataFrame([[round(N(nu), 2) for nu in cols],
                        [round(i_Ki(nu), 2) for nu in cols]],
                      index=['N($\\nu$)', '$i^{*}$$/K_{i}$'], columns=cols)
tab_5_1.index.rename('$\\nu$', inplace=True)
tab_5_1
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>2.0</th>
      <th>3.0</th>
      <th>4.0</th>
      <th>5.0</th>
    </tr>
    <tr>
      <th>$\nu$</th>
      <th></th>
      <th></th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>N($\nu$)</th>
      <td>0.65</td>
      <td>0.84</td>
      <td>1.07</td>
      <td>1.30</td>
    </tr>
    <tr>
      <th>$i^{*}$$/K_{i}$</th>
      <td>0.58</td>
      <td>0.79</td>
      <td>0.88</td>
      <td>0.92</td>
    </tr>
  </tbody>
</table>
</div>

## The Symmetry Model
The regulatory molecules are often chemically quite different than the substrate molecule. They thus often have a different binding site on the protein molecule than the catalytic site. It called an _allosteric site_. One of the earliest enzyme kinetic models that accounted for allosterism was the symmetry model (Monod, 1965), named after certain assumed symmetry properties of the subunits of the enzyme. It is a mechanistically realistic description of regulatory enzymes. An example of a multimeric regulatory enzyme is given in Figure 5.6.

### The reaction mechanism for the symmetry model
The main chemical conversion in the symmetry model is as before and is described by Equation (5.14). The symmetry model postulates that the regulated enzyme lies naturally in two forms, $E$ and $X$, and is converted between the two states simply as

$$\begin{equation} E \underset{k_{-}}{\stackrel{k_+}{\rightleftharpoons}} X  \tag{5.30} \end{equation}$$

The equilibrium constant for this reaction,

$$\begin{equation} L = k_+/k_- = x/e \tag{5.31} \end{equation}$$

has a special name, the allosteric constant. Then $\nu$ molecules of an inhibitor, $I$, can bind sequentially to $X$ as

$$\begin{equation} \begin{matrix} {X} & {+} & {I} & {\underset{k_i^-}{\stackrel{\nu k_i^+}{\rightleftharpoons}}} & {X_1} \\ {X_1} & {+} & {I} & {\underset{2 k_i^-}{\stackrel{(\nu-1) k_i^+}{\rightleftharpoons}}} & {X_2} \\ {\vdots} & {} & {} & {} & {\vdots} \\ {X_{\nu - 1}} & {+} & {I} & {\underset{\nu k_i^-}{\stackrel{k_i^+}{\rightleftharpoons}}} & {X_{\nu}} \\ \end{matrix}\end{equation} \tag{5.32}$$

where the binding steps have the same dissociation constant, $K_i = k_i^- / k_i^+$. We will discuss the most common case of a tetramer here, i.e., $\nu = 4$, see Figure 5.8.

**Figure 5.8:** The reaction mechanisms for the symmetry model. The enzyme has four binding sites for the inhibitor.

### Step 1: Dynamic mass balances for the symmetry model
The conversion rate of the substrate is

$$\begin{equation} v = kse \tag{5.33} \end{equation}$$

whereas the enzyme sequestration is characterized by the reaction rates

$$\begin{equation} \begin{matrix} {v_1 = k^+e,} & {v_2 = k^-x,} & {v_3 = 4k_i^+xi,} \\ {v_4 = k_i^-x_1,} & {v_5 = 3 k_i^+x_1i,} & {v_6 = 2k_i^-x_2,} \\ {v_7 = k_i^+x_2i,} & {v_8 = k_i^-x_3,} & {v_9 = k_i^+x_3i,} \\ {} & {v_{10} = 4k_i^-x_4} & {} \\ \end{matrix} \end{equation} \tag{5.34}$$

The dynamic mass balances on the various states of the enzyme are:

$$\begin{align} \frac{de}{dt} &= -v_1 + v_2\\ \frac{dx}{dt}   &= v_1 - v_2 - v_3 + v_4\\ \frac{di}{dt}   &= -v_3 + v_4 - v_5 + v_6 - v_7 + v_8 - v_9 + v_{10} \\ \frac{dx_1}{dt} &= v_3 - v_4 - v_5 + v_6 \\ \frac{dx_2}{dt} &= v_5 - v_6 - v_7 + v_8 \\ \frac{dx_3}{dt} &= v_7 - v_8 - v_9 + v_{10} \\ \frac{dx_4}{dt} &= v_9 - v_{10} \\ \end{align} \tag{5.35}$$

### Step 2: Finding the time invariants for the symmetry model
The stoichiometric matrix for $\textbf{x} = (e, x, i, x_1, x_2, x_3, x_4)$ is a 7x10 matrix:

$$\begin{equation} \textbf{S} = \begin{pmatrix} {-1} & {1} & {0} & {0} & {0} & {0} & {0} & {0} & {0} & {0} \\ {1} & {-1} & {-1} & {1} & {0} & {0} & {0} & {0} & {0} & {0} \\ {0} & {0} & {-1} & {1} & {-1} & {1} & {-1} & {1} & {-1} & {1} \\ {0} & {0} & {1} & {-1} & {-1} & {1} & {0} & {0} & {0} & {0} \\ {0} & {0} & {0} & {0} & {1} & {-1} & {-1} & {1} & {0} & {0} \\ {0} & {0} & {0} & {0} & {0} & {0} & {1} & {-1} & {-1} & {1} \\ {0} & {0} & {0} & {0} & {0} & {0} & {0} & {0} & {1} & {-1} \\ \end{pmatrix} \end{equation} \tag{5.36}$$

that has a rank of 5. Thus, there are two conservation relationships, for the enzyme: $e_0 = e + x + x_1 + x_2 + x_3 + x_4$;, and, for the inhibitor: $i_0 = i + x_1 + 2x_2 + 3x_3 + 4x_4$. If the dynamic mass balances on the substrate and product are taken into account, a third conservation $s_0 = s + p$ appears.

### Step 3: Reducing the dynamic description for the symmetry model
We leave it to the reader to pick two dynamic variables from the full kinetic model as the dependent variables and then eliminate them from the dynamic description using the conservation relationships. The impetus for doing so algebraically becomes smaller as the number of differential equations grows. Most standard software packages will integrate a dynamically redundant set of differential equations and such substitution is not necessary to obtain the numerical solutions.

### Step 4: Using simplifying kinetic assumptions to derive a rate law  for the symmetry model
The serial binding of an inhibitor to X that has four binding sites is shown in Figure 5.8. The derivation of the rate law is comprised of four basic steps:

1. Mass balance on enzyme:

$$\begin{equation} e_0 = e + x + x_1 + x_2 + x_3 + x_4 \tag{5.37} \end{equation}$$

2. QEA for binding steps:

$$\begin{align} 4k_i^+ix   =  k_i^-x_1 \ &\Rightarrow \ x_1 = \frac{4}{1}x  (i/K_i) = 4x(i/K_i) \\ 3k_i^+ix_1 = 2k_i^-x_2 \ &\Rightarrow \ x_2 = \frac{3}{2}x_1(i/K_i) = 6x(i/K_i)^2 \\ 2k_i^+ix_2 = 3k_i^-x_3 \ &\Rightarrow \ x_3 = \frac{2}{3}x_2(i/K_i) = 4x(i/K_i)^3 \\ k_i^+ix_3 = 4k_i^-x_4 \ &\Rightarrow \ x_4 = \frac{1}{4}x_3(i/K_i) =  x(i/K_i)^4 \\ \end{align} \tag{5.38}$$

3. Combine 1 and 2:

$$\begin{align} e_0 &= e + x + 4x(i/K_i) + 6x(i/K_i)^2 + 4x(i/K_i)^3 + x(i/K_i)^4 \\ &= e + x(1 + (i/K_i))^4 \ \text{where} \ x=Le \\ &= e(1 + L(1 + (i/K_i)))^4  \end{align} \tag{5.39}$$

4. Form the rate law: The reaction rate is given by: $v = kse$. We can rewrite the last part of Eq. (5.39) as:

$$\begin{equation} e = \frac{e_0}{1 + L(1 + i/K_i)^4} \tag{5.40} \end{equation}$$

leading to the rate law:

$$\begin{equation} v(s, i)  = \frac{ke_0s}{1 + L(1 + i/K_i)^4} \tag{5.41} \end{equation}$$

This rate law generalizes to:

$$\begin{equation} v(s, i)  = \frac{ke_0s}{1 + L(1 + i/K_i)^{\nu}} = \frac{v_m}{1 + L(1 + i/K_i)^{\nu}} \tag{5.42} \end{equation}$$

for any $\nu$. The reader can find the same key dimensionless groups as for the Hill rate law. Note again the fraction

$$\begin{equation} f_e = \frac{e}{e_0} = \frac{1}{1 + L(1 + i/K_i)^{\nu}} \tag{5.43} \end{equation}$$

that describes the what fraction of the enzyme is in the catalytically active state.

```python
t0 = 0
tf = 15

def symmetry(t, state_vars, *params):
    s, p, e, i, x, x1, x2, x3, x4 = state_vars
    k1, k_plus, k_minus, ki_plus, ki_minus = params
    # Enzyme Reaction Rates
    v1 = k_plus * e;           v2 = k_minus * x;
    v3 = 4 * ki_plus * i * x;  v4 = ki_minus * x1;
    v5 = 3 * ki_plus * i * x1; v6 = 2 * ki_minus * x2;
    v7 = 2 * ki_plus * i * x2; v8 = 3 * ki_minus * x3;
    v9 = ki_plus * i * x3;    v10 = 4 * ki_minus * x4;
    # Differential equations to integrate
    diffeqs = [-k1 * s * e,                             #  ds/dt
               k1 * s * e,                              #  dp/dt
               -v1 + v2,                                #  de/dt
               -v3 + v4 - v5 + v6 - v7 + v8 - v9 + v10, #  di/dt
               v1 - v2 - v3 + v4,                       #  dx/dt
               v3 - v4 - v5 + v6,                       # dx1/dt
               v5 - v6 - v7 + v8,                       # dx2/dt
               v7 - v8 - v9 + v10,                      # dx3/dt
               v9 - v10]                                # dx4/dt
    return diffeqs

# Define initial conditions
s0, p0, e0, i0, x0 = (1, 0, 1, 1, 0)
x1_0, x2_0, x3_0, x4_0 = (0, 0, 0, 0)

# Define paramters
k1 = 1;
k_plus, k_minus = (100, 100)
ki_plus, ki_minus = (2, 2)
params = [k1, k_plus,k_minus, ki_plus, ki_minus]

# Obtain numerical solutions
sol_obj = solve_ivp(fun=lambda t, state_vars: symmetry(t, state_vars, *params),
                    t_span=(t0, tf), y0=[s0, p0, e0, i0, x0, x1_0, x2_0, x3_0, x4_0])
# Store solutions into Solution Objects
sol_dict = dict(zip(["s", "p", "e", "i", "x", "x1", "x2", "x3", "x4"], sol_obj.y))

x_total = sum(sol_dict[k] for k in ["x", "x1", "x2", "x3", "x4"])
i_bound = sum(i*sol_dict[k] for i, k in zip([1, 2, 3, 4], ["x1", "x2", "x3", "x4"]))

sol_dict.update({"x_total": x_total, "i_bound": i_bound})

symmetry_sol = MassSolution(
    "Symmetry", solution_type="Conc", data_dict=sol_dict,
    time=sol_obj.t, interpolate=False)
```

```python
fig_5_9 = plt.figure(figsize=(10, 8))
gs = fig_5_9.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1],
                          height_ratios=[1, 1])

ax1 = fig_5_9.add_subplot(gs[0, 0])
ax2 = fig_5_9.add_subplot(gs[0, 1])
ax3 = fig_5_9.add_subplot(gs[1, 0])
ax4 = fig_5_9.add_subplot(gs[1, 1])

plot_phase_portrait(
    symmetry_sol, x="s", y="e", ax=ax1, xlabel="s", ylabel="e",
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(a) Phase Portrait of s vs. e", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

plot_phase_portrait(
    symmetry_sol, x="e", y="x_total", ax=ax2,
    xlabel="e", ylabel='x + x1 + x2 + x3 + x4',
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(b) Phase Portrait of e vs. x_total", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

plot_phase_portrait(
    symmetry_sol, x="i", y="i_bound", ax=ax3,
    xlabel="i", ylabel='1*x1 + 2*x2 + 3*x3 + 4*x4',
    xlim=(-0.05, 1.05), ylim=(-0.05, 1.05),
    title=("(a) Phase Portrait of i vs. x", {"size": "x-large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True);

plot_time_profile(
    symmetry_sol, observable=list(
        k for k in symmetry_sol.keys() if k not in [
            "x", "x1", "x2", "x3", "x4"]),
    ax=ax4, legend="right outside",
    title=("(d) Concentration Profiles", {"size": "x-large"}));
fig_5_9.tight_layout()
```

**Figure 5.9:**  The transient response of the symmetry model, for $k^+ = k^- = 100$, $k_i^+ = k_i^- = 2$, $k = 1$, $\nu = 4$, $x_0 = x_{1, 0} = x_{2, 0} = x_{3, 0} = x_{4, 0} = 0$ and $e_0 = s_0 = i_0 = 1$. (a) The phase portraits of $s$ and $e$. (b) The phase portraits of $e$ and $(x + x_1 + x_2 + x_3 + x_4)$. (c) The phase portraits of $i$ and $(x_1 + 2x_2 + 3x_3 + 4x_4)$. (d) Concentration and pool profiles.

### Step 5: Numerical solutions for the symmetry model
These equations can be simulated. Typically the conformational changes between $E$ and $X$ are fast as are the inhibitor binding steps relative to the catalysis rate. Numerical simulations were carried out for this situation and the results are plotted in Figure 5.9.

* Figure 5.9a shows how the substrate-enzyme phase portrait is L-shaped showing that the sequestration of the enzyme in the inhibited form (the vertical line) is faster than the conversion of the substrate (the horizontal line).

* Figure 5.9b shows the redistribution of the total enzyme among the active and inactive forms, that is, $e$ vs. $(x + x_1 + x_2 + x_3 + x_4)$. The fraction of the enzyme in the inactive form is about 0.29.

* Figure 5.9c shows the redistribution of the inhibitor between the free and bound form; $i$ vs. $(x_1 + 2x_2 + 3x_3 + 4x_4)$. This panel shows that the fraction the inhibitor that is bound is high, 0.70.

* Finally, Figure 5.9d show the transient changes in the concentrations and pools on the fast and slow time scales. Note that two natural aggregation variables appear: the total enzyme in the inactive form, and the total number of inhibitor molecules bound to the enzyme.

## Scaling Dynamic Descriptions
The analysis of simple equations requires the "proper frame of mind." In step 6 of the process of formulating rate laws, this notion is translated into quantitative measures. We need to scale the variables with respect to intrinsic reference scales and thereby cast our mathematical descriptions into appropriate coordinate systems. All parameters then aggregate into dimensionless property ratios that, if properly interpreted, have a clear physical significance.

### The scaling process:
The examples above illustrate the decisive role of time constants and their use to analyze simple situations and to elucidate intrinsic reference scales. Identification of unimportant terms is sometimes more difficult and familiarity with a formal scaling procedure is useful. This procedure basically consists of four steps:

1. Identify logical reference scales. This step is perhaps the most difficult. It relies partly on physical intuition, and the use of time constants is surprisingly powerful even when analyzing steady situations.

2. Introduce reference scales into the equations and make the variables dimensionless.

3. Collect the parameters into dimensionless property ratios. The number of dimensionless parameters is always the same and it is given by the well-known Buckingham Pi theorem.

4. Interpret the results. The dimensionless groups that appear can normally be interpreted as ratios of the time constants, such as those discussed above.

Scaling of equations is typically only practiced for small models and for analysis purposes only. Numerical simulations of complex models are essentially always performed with absolute values of the variables.

### The importance of intrinsic reference scales
The process by which the equations are made dimensionless is not unique. The 'correct' way of putting the equations into a dimensionless form, where judgments of relative orders of magnitude can be made, is called _scaling_. The scaling process is defined by Lin and Segel (Segel, 1974) as:

"...select intrinsic reference quantities so that each term in the dimensional equations transforms into a product of a constant dimensional factor which closely estimates the term's order of magnitude and a dimensionless factor of unit order of magnitude."

In other words, if one has an equation which is a sum of terms $T_i$ as:

$$\begin{equation} T_1 + T_2 + \dots = 0 \tag{5.44} \end{equation}$$

one tries to scale the _variables_ involved so that they are of unit order of magnitude or

$$\begin{equation} t_i = \frac{\text{variable}_i}{\text{intrinistic reference scale}_i} \approx \text{unit order of magnitude} \tag{5.45} \end{equation}$$

Introducing these dimensionless variables into equation (5.44)  results in the dimensionless form:

$$\begin{equation} \pi_1 t_1 + \pi_2 t_2 + \dots = 0 \tag{5.44} \end{equation}$$

where the dimensionless multipliers, $\pi_i$ are the dimensionless groups and they will indicate the order of magnitude of the product, $\pi_it_i$. Once the equations are in this form, order of magnitude judgements can be made based on the dimensionless groups.

## Summary

* Enzymes are highly specialized catalysts that can dramatically accelerate the rates of biochemical reactions.

* Reaction mechanisms are formulated for the chemical conversions carried out by enzymes in terms of elementary reactions.

* Rate laws for enzyme reaction mechanisms are derived based on simplifying assumptions.

* Two simplifying assumptions are commonly used: the quasi-steady state (QSSA) and the quasi-equilibrium assumptions (QEA).

* The validity of the simplifying assumptions can be determined using scaling of the equations followed by mathematical and numerical analysis.

* A number of rate laws have been developed for enzyme catalysis and for the regulation of enzymes. Only three reaction mechanisms were described in this chapter.

$\tiny{\text{© B. Ø. Palsson 2011;}\ \text{This publication is in copyright.}\\ \text{Subject to statutory exception and to the provisions of relevant collective licensing agreements,}\\ \text{no reproduction of any part may take place without the written permission of Cambridge University Press.}}$


# Lab Assignment 1

## Important notes

**Submission deadline:**
* **Regular problems: last lab session before or on Monday, 21.10.19**
* **Bonus problems: deadline for Lab Assignment 2**

**Points: 11 + 5 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

The assignment is in the form of a Jupyter notebook. We will be using [Google Colab](https://colab.research.google.com) to solve it. Below you will find a "Setup" section. Follow instructions from this paragraph to download the notebook and open it using [Google Colab](https://colab.research.google.com).

Your goal is to solve problems posted below. Whenever possible, add your solutions to the notebook.

Please email us about any problems with it - we will try to correct them quickly. Also, please do not hesitate to use GitHubâ€™s pull requests to send us corrections!

## Setup

### 1. Open the notebook using Google Colab

1. From Github: Click on "View in Colaboratory", then save to your Google Drive.
2. Alternatively upload manually to Drive:
  1. Download the notebook or clone https://github.com/janchorowski/ml_uwr.
  2. Go to  [Google Colab](https://colab.research.google.com).
  3. Go to "UPLOAD" tab and select a local copy of the notebook that you downloaded in point 1.

Colab Tips:
1. Set tab width to 4 spaces under `Tools` â†’ `Preferences`.

### 2. Open the notebook offline using Jupyter/IPython

This notebook can be opened using Jupyter notebook. Simply install a scientific Python distribution on your computer (e.g. [Anaconda](https://www.anaconda.com/) or [WinPython](http://winpython.github.io/)), clone the repository https://github.com/janchorowski/nn_assignments and run `jupyter notebook`.

###   3. Install required dependencies, download data and import packages

Run cells below. To run a cell either click it and click a run button or press "shift + enter"

```python
# Please note that this code needs only to be run in a fresh runtime.
# However, it can be rerun afterwards too.
!pip install -q gdown httpimport
![ -e mnist.npz ] || gdown 'https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU' -O mnist.npz
```

    Downloading...
    From: https://drive.google.com/uc?id=1QPaC3IKB_5tX6yIZgRgkpcqFrfVqPTXU
    To: /content/mnist.npz
    55.4MB [00:00, 134MB/s]

```python
# Standard IPython notebook imports
%matplotlib inline

import os

import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook
import scipy.stats as sstats
import math
import cmath
import random

import seaborn as sns
from sklearn import datasets

# In this way we can import functions straight from github
with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
     from common.plotting import plot_mat

sns.set_style('whitegrid')
```

### 4. Follow the notebook and solve problems posted below

## Problems

### Problem 0 [0p]

1. To learn more about Jupyter,  read [Jupyter tutorial from Data Analysis in Biological Sciences course at Caltech](http://bebi103.caltech.edu/2015/tutorials/t0b_intro_to_jupyter_notebooks.html) (which itself can be downloaded as a Jupyter notebook). Feel free to skip the tutorial if you have some prior experience with Jupyter notebook.
2. To learn more about basic Google Colab features, go to [Google Colab](https://colab.research.google.com) and select "Overview of Colaboratory Features" in "EXAMPLES" tab. To learn more about / set up useful keyboard shortcuts (e.g. to add a new cell without clicking ""+ code"), go to "Tools --> Keyboard shortcuts"

### Problem 1: NumPy [2p]

First, get familiar with Python at https://docs.python.org/3/tutorial/. Then, get
to know the capabilities of NumPy, the prime numerical library of Python http://www.numpy.org/, for instance with the tutorial at http://wiki.scipy.org/Tentative_NumPy_Tutorial. Finally, look into Pandas at https://pandas.pydata.org/pandas-docs/stable/getting_started/tutorials.html.

You might also need:
  1. another intro to NumPy,
http://people.duke.edu/~ccc14/pcfb/numerics.html
  2. a better interactive shell for Python
http://ipython.org/
  3. a plotting library for Python
http://matplotlib.org/
  4. nice statistical plots for matplotlib https://seaborn.pydata.org/.

**a) Declare variables:**
1. $a=10$,
2. $b=2.5\times 10^{23}$,
3. $c=2+3i$, where $i$ is an imaginary unit,
4. $d=e^{i2\pi/3}$, where $i$ is an imaginary unit, $e$ is the Euler's number (use `exp`, `pi`).

```python
a = 10
b = 2.5 * 10 ** 23
c = 2 + 3j
d = cmath.exp((1j * 2 * math.pi) / 3)

a,b,c,d
```

    (10, 2.5e+23, (2+3j), (-0.4999999999999998+0.8660254037844387j))

**b) Declare vectors:**
1. $aVec=\begin{bmatrix} 3.14 & 15 & 9 & 26 \end{bmatrix}$,
2. $bVec=\begin{bmatrix} 5 & 4.8 & \cdots & -4.8 & -5 \end{bmatrix}$ (vector of numbers from $5$ to $-5$ decreasing by $0.2$),
3. $cVec=\begin{bmatrix} 10^0 & 10^{0.01} & \cdots & 10^{0.99} & 10^1 \end{bmatrix}$ (logarithmically spaced numbers from 1 to 10, use `logspace` and make sure, that the result has correct length!),
4. $dVec=Hello$ ($eVec$ is a string of characters, thus a vector).

```python
aVec = np.array([3.14, 15, 9, 26])
bVec = np.array([x for x in np.arange(5, -5.2, -0.2)])
cVec = np.logspace(0, 1, num=100)

dVec = np.array(['Hello'])
dVec2 = np.array(['H', 'e', 'l', 'l', 'o'])

aVec, bVec, cVec, dVec, dVec2
```

    (array([ 3.14, 15.  ,  9.  , 26.  ]),
     array([ 5.0000000e+00,  4.8000000e+00,  4.6000000e+00,  4.4000000e+00,
             4.2000000e+00,  4.0000000e+00,  3.8000000e+00,  3.6000000e+00,
             3.4000000e+00,  3.2000000e+00,  3.0000000e+00,  2.8000000e+00,
             2.6000000e+00,  2.4000000e+00,  2.2000000e+00,  2.0000000e+00,
             1.8000000e+00,  1.6000000e+00,  1.4000000e+00,  1.2000000e+00,
             1.0000000e+00,  8.0000000e-01,  6.0000000e-01,  4.0000000e-01,
             2.0000000e-01, -4.4408921e-15, -2.0000000e-01, -4.0000000e-01,
            -6.0000000e-01, -8.0000000e-01, -1.0000000e+00, -1.2000000e+00,
            -1.4000000e+00, -1.6000000e+00, -1.8000000e+00, -2.0000000e+00,
            -2.2000000e+00, -2.4000000e+00, -2.6000000e+00, -2.8000000e+00,
            -3.0000000e+00, -3.2000000e+00, -3.4000000e+00, -3.6000000e+00,
            -3.8000000e+00, -4.0000000e+00, -4.2000000e+00, -4.4000000e+00,
            -4.6000000e+00, -4.8000000e+00, -5.0000000e+00]),
     array([ 1.        ,  1.02353102,  1.04761575,  1.07226722,  1.09749877,
             1.12332403,  1.149757  ,  1.17681195,  1.20450354,  1.23284674,
             1.26185688,  1.29154967,  1.32194115,  1.35304777,  1.38488637,
             1.41747416,  1.45082878,  1.48496826,  1.51991108,  1.55567614,
             1.59228279,  1.62975083,  1.66810054,  1.70735265,  1.7475284 ,
             1.78864953,  1.83073828,  1.87381742,  1.91791026,  1.96304065,
             2.009233  ,  2.05651231,  2.10490414,  2.15443469,  2.20513074,
             2.25701972,  2.3101297 ,  2.36448941,  2.42012826,  2.47707636,
             2.53536449,  2.59502421,  2.65608778,  2.71858824,  2.7825594 ,
             2.84803587,  2.91505306,  2.98364724,  3.05385551,  3.12571585,
             3.19926714,  3.27454916,  3.35160265,  3.43046929,  3.51119173,
             3.59381366,  3.67837977,  3.76493581,  3.85352859,  3.94420606,
             4.03701726,  4.1320124 ,  4.22924287,  4.32876128,  4.43062146,
             4.53487851,  4.64158883,  4.75081016,  4.86260158,  4.97702356,
             5.09413801,  5.21400829,  5.33669923,  5.46227722,  5.59081018,
             5.72236766,  5.85702082,  5.9948425 ,  6.13590727,  6.28029144,
             6.42807312,  6.57933225,  6.73415066,  6.8926121 ,  7.05480231,
             7.22080902,  7.39072203,  7.56463328,  7.74263683,  7.92482898,
             8.11130831,  8.30217568,  8.49753436,  8.69749003,  8.90215085,
             9.11162756,  9.32603347,  9.54548457,  9.77009957, 10.        ]),
     array(['Hello'], dtype='<U5'),
     array(['H', 'e', 'l', 'l', 'o'], dtype='<U1'))

**c) Declare matrices:**
1. $aMat=\begin{bmatrix}
                    2      & \cdots & 2 \\
                    \vdots & \ddots & \vdots \\
                    2      & \cdots & 2
                \end{bmatrix}$,

matrix $9\times 9$ filled with 2s (use `ones` or `zeros`),
2. $bMat=\begin{bmatrix}
                    1      & 0      & \cdots &        & 0      \\
                    0      & \ddots & 0      &        & 0      \\
                    \vdots & 0      & 5      & 0      & \vdots \\
                           &        & 0      & \ddots & 0      \\
                    0      &        & \cdots & 0      & 1
                \end{bmatrix}$,

matrix $9\times 9$ filled with zeros, with $\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \end{bmatrix}$ on its diagonal (use `zeros`, `diag`),
3. $cMat=\begin{bmatrix}
                    1      & 11     & \cdots & 91     \\
                    2      & 12     & \cdots & 92     \\
                    \vdots & \vdots & \ddots & \vdots \\
                    10     & 20     & \cdots & 100
                \end{bmatrix}$,

matrix $10\times 10$, columns of which form the vector $1:100$ (use `reshape`),
4. $dMat=\begin{bmatrix}
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN \\
                    NaN & NaN & NaN & NaN
                \end{bmatrix}$,

matrix $3\times 4$ filled with `NaN`s (use... `NaN`),
5. $eMat=\begin{bmatrix}
                    13  & -1  & 5  \\
                    -22 & 10  & -87
                \end{bmatrix}$,
6. $fMat$ of shape $3\times 3$ filled with random natural numbers from $[-3,3]$ (use `rand` and `floor` or `ceil`).

```python
aMat = np.zeros((9, 9)) + 2

bMat = np.zeros((9, 9))
np.fill_diagonal(bMat,[i for i in range(1,6)] + [i for i in range(4,0,-1)])
# a = (9 + 1)//2
# bMat[range(a), range(a)] = range(1,a + 1)
# bMat[range(a, 9), range(a, 9)] = range(a-1,0,-1)

cMat = np.arange(1,101).reshape(10,10).T
dMat = np.empty((3,4))
dMat[:] = np.nan

eMat = np.array([13, -1, 5, -22, 10, -87]).reshape(2,3)

fMat = np.array([random.randint(-3,3) for _ in range(9)]).reshape(3,3)

aMat, bMat, cMat, dMat, eMat, fMat
```

    (array([[2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.],
            [2., 2., 2., 2., 2., 2., 2., 2., 2.]]),
     array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
            [0., 2., 0., 0., 0., 0., 0., 0., 0.],
            [0., 0., 3., 0., 0., 0., 0., 0., 0.],
            [0., 0., 0., 4., 0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 5., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0., 4., 0., 0., 0.],
            [0., 0., 0., 0., 0., 0., 3., 0., 0.],
            [0., 0., 0., 0., 0., 0., 0., 2., 0.],
            [0., 0., 0., 0., 0., 0., 0., 0., 1.]]),
     array([[  1,  11,  21,  31,  41,  51,  61,  71,  81,  91],
            [  2,  12,  22,  32,  42,  52,  62,  72,  82,  92],
            [  3,  13,  23,  33,  43,  53,  63,  73,  83,  93],
            [  4,  14,  24,  34,  44,  54,  64,  74,  84,  94],
            [  5,  15,  25,  35,  45,  55,  65,  75,  85,  95],
            [  6,  16,  26,  36,  46,  56,  66,  76,  86,  96],
            [  7,  17,  27,  37,  47,  57,  67,  77,  87,  97],
            [  8,  18,  28,  38,  48,  58,  68,  78,  88,  98],
            [  9,  19,  29,  39,  49,  59,  69,  79,  89,  99],
            [ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100]]),
     array([[nan, nan, nan, nan],
            [nan, nan, nan, nan],
            [nan, nan, nan, nan]]),
     array([[ 13,  -1,   5],
            [-22,  10, -87]]),
     array([[ 0,  0,  0],
            [-2,  2, -3],
            [-1, -3, -1]]))

**d) Declare a multiplication table**
as a $10\times 10$ matrix `mulMat`. Use matrix/vector multiplication.

```python
a = np.array([i for i in range(1,11)])[np.newaxis]
b = a.copy().T
mulMat = b.dot(a)
mulMat
```

    array([[  1,   2,   3,   4,   5,   6,   7,   8,   9,  10],
           [  2,   4,   6,   8,  10,  12,  14,  16,  18,  20],
           [  3,   6,   9,  12,  15,  18,  21,  24,  27,  30],
           [  4,   8,  12,  16,  20,  24,  28,  32,  36,  40],
           [  5,  10,  15,  20,  25,  30,  35,  40,  45,  50],
           [  6,  12,  18,  24,  30,  36,  42,  48,  54,  60],
           [  7,  14,  21,  28,  35,  42,  49,  56,  63,  70],
           [  8,  16,  24,  32,  40,  48,  56,  64,  72,  80],
           [  9,  18,  27,  36,  45,  54,  63,  72,  81,  90],
           [ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100]])

**e) Compute element-wise using values from b).**

For instance, the first element of $xVec[0]$ should be equal to

\begin{equation}
1/(\sqrt{2\pi2.5^2}) e^{-cVec[0]^2 / (2\cdot\pi 2.5^2)}.
\end{equation}

1. $xVec=1/(\sqrt{2\pi2.5^2}) e^{-cVec^2 / (2\cdot\pi 2.5^2)}$
2. $yVec=\log_{10}(1/cVec)$, using `log10`

```python
xVec = np.array([1 / math.sqrt(2 * math.pi * 2.5 ** 2) * math.exp(-(x ** 2) / (2 * math.pi * 2.5 ** 2)) for x in cVec])
yVec = np.array([math.log10(1/x) for x in cVec])

xVec, yVec
```

    (array([0.15556462, 0.15537611, 0.15517887, 0.1549725 , 0.1547566 ,
            0.15453075, 0.15429449, 0.15404737, 0.15378891, 0.15351861,
            0.15323595, 0.15294038, 0.15263135, 0.15230828, 0.15197056,
            0.15161756, 0.15124863, 0.1508631 , 0.15046027, 0.1500394 ,
            0.14959976, 0.14914057, 0.14866103, 0.1481603 , 0.14763754,
            0.14709187, 0.14652238, 0.14592813, 0.14530818, 0.14466153,
            0.14398717, 0.14328408, 0.14255119, 0.14178742, 0.14099168,
            0.14016284, 0.13929975, 0.13840127, 0.13746622, 0.13649342,
            0.13548169, 0.13442982, 0.13333663, 0.13220091, 0.13102149,
            0.1297972 , 0.12852688, 0.12720941, 0.12584368, 0.12442865,
            0.12296331, 0.12144669, 0.11987792, 0.11825618, 0.11658075,
            0.11485099, 0.1130664 , 0.11122656, 0.10933122, 0.10738026,
            0.10537373, 0.10331187, 0.10119508, 0.09902401, 0.0967995 ,
            0.09452265, 0.0921948 , 0.08981757, 0.08739287, 0.08492289,
            0.08241014, 0.07985745, 0.07726798, 0.07464522, 0.07199301,
            0.06931553, 0.06661729, 0.06390316, 0.06117833, 0.05844828,
            0.0557188 , 0.05299598, 0.0502861 , 0.04759569, 0.04493142,
            0.04230011, 0.03970863, 0.03716387, 0.03467267, 0.03224177,
            0.02987771, 0.02758678, 0.02537494, 0.02324774, 0.02121026,
            0.01926701, 0.01742191, 0.01567817, 0.01403829, 0.01250398]),
     array([ 0.        , -0.01010101, -0.02020202, -0.03030303, -0.04040404,
            -0.05050505, -0.06060606, -0.07070707, -0.08080808, -0.09090909,
            -0.1010101 , -0.11111111, -0.12121212, -0.13131313, -0.14141414,
            -0.15151515, -0.16161616, -0.17171717, -0.18181818, -0.19191919,
            -0.2020202 , -0.21212121, -0.22222222, -0.23232323, -0.24242424,
            -0.25252525, -0.26262626, -0.27272727, -0.28282828, -0.29292929,
            -0.3030303 , -0.31313131, -0.32323232, -0.33333333, -0.34343434,
            -0.35353535, -0.36363636, -0.37373737, -0.38383838, -0.39393939,
            -0.4040404 , -0.41414141, -0.42424242, -0.43434343, -0.44444444,
            -0.45454545, -0.46464646, -0.47474747, -0.48484848, -0.49494949,
            -0.50505051, -0.51515152, -0.52525253, -0.53535354, -0.54545455,
            -0.55555556, -0.56565657, -0.57575758, -0.58585859, -0.5959596 ,
            -0.60606061, -0.61616162, -0.62626263, -0.63636364, -0.64646465,
            -0.65656566, -0.66666667, -0.67676768, -0.68686869, -0.6969697 ,
            -0.70707071, -0.71717172, -0.72727273, -0.73737374, -0.74747475,
            -0.75757576, -0.76767677, -0.77777778, -0.78787879, -0.7979798 ,
            -0.80808081, -0.81818182, -0.82828283, -0.83838384, -0.84848485,
            -0.85858586, -0.86868687, -0.87878788, -0.88888889, -0.8989899 ,
            -0.90909091, -0.91919192, -0.92929293, -0.93939394, -0.94949495,
            -0.95959596, -0.96969697, -0.97979798, -0.98989899, -1.        ]))

**f) Compute with matrix/vector operations.**

**NOTE:** Every multiplication (and power) in this subtask is a [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication).
1. $xMat=[0,  1, ..., 6][0, 10, 20, ..., 60]^T$,
2. $yMat=[0, 10, 20, ..., 60]^T[0,  1, ..., 6]$

(remember, that matrix multiplication is not commutative).

```python
a = np.array([i for i in range(7)])[np.newaxis]
b = np.array([i for i in range(0,70,10)])[np.newaxis].T
xMat = (a).dot(b)

yMat = b.dot(a)

xMat, yMat
```

    (array([[910]]), array([[  0,   0,   0,   0,   0,   0,   0],
            [  0,  10,  20,  30,  40,  50,  60],
            [  0,  20,  40,  60,  80, 100, 120],
            [  0,  30,  60,  90, 120, 150, 180],
            [  0,  40,  80, 120, 160, 200, 240],
            [  0,  50, 100, 150, 200, 250, 300],
            [  0,  60, 120, 180, 240, 300, 360]]))

**g) Declare `ismagic(A)` function**
which checks if matrix $A$ is a [magic square](https://en.wikipedia.org/wiki/Magic_square) and returns a boolean.

```python
def ismagic(A):
    b1 = len(np.unique(A)) == np.prod(A.shape)

    row_sums = np.sum(A, axis=1)
    b2 = all(x == row_sums[0] for x in row_sums)

    column_sums = np.sum(A, axis=0)
    b3 = all(x == column_sums[0] for x in column_sums)

    return b1 and b2 and b3

assert not ismagic(np.array([[1,1], [2,2]]))
assert ismagic(np.array([[2,7,6],[9,5,1],[4,3,8]]))
```

### Problem 2: Pandas and Seaborn [2p]

1. Load the IRIS Data into a `DataFrame`

```python
iris_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
# Use read_csv to load the data. Make sure you get 150 examples!
iris_df = pd.read_csv(iris_url)

# Set the column names to
# 'sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'target'
iris_df.columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'target']

# Print the first 10 entries
iris_df.sample(10)
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      target

      57
      6.6
      2.9
      4.6
      1.3
      Iris-versicolor

      59
      5.0
      2.0
      3.5
      1.0
      Iris-versicolor

      102
      6.3
      2.9
      5.6
      1.8
      Iris-virginica

      92
      5.0
      2.3
      3.3
      1.0
      Iris-versicolor

      24
      5.0
      3.0
      1.6
      0.2
      Iris-setosa

      28
      4.7
      3.2
      1.6
      0.2
      Iris-setosa

      103
      6.5
      3.0
      5.8
      2.2
      Iris-virginica

      21
      4.6
      3.6
      1.0
      0.2
      Iris-setosa

      25
      5.0
      3.4
      1.6
      0.4
      Iris-setosa

      10
      4.8
      3.4
      1.6
      0.2
      Iris-setosa

```python
# Show numerical summary of the data, using DataFrame.describe()
iris_df.describe()
```

      sepal_length
      sepal_width
      petal_length
      petal_width

      count
      149.000000
      149.000000
      149.000000
      149.000000

      mean
      5.848322
      3.051007
      3.774497
      1.205369

      std
      0.828594
      0.433499
      1.759651
      0.761292

      min
      4.300000
      2.000000
      1.000000
      0.100000

      25%
      5.100000
      2.800000
      1.600000
      0.300000

      50%
      5.800000
      3.000000
      4.400000
      1.300000

      75%
      6.400000
      3.300000
      5.100000
      1.800000

      max
      7.900000
      4.400000
      6.900000
      2.500000

```python
# Plot the data using seaborn's pairplot
iris = sns.load_dataset("iris")
sns.pairplot(iris, hue="species")
```

The Iris data is in a so-called 'wide' format, in which each column corresponds to a variable and each row of the DataFrame corresponds to one observation. Turn it into a 'long' format in which each row is a measurement.

Specifically, change the data layout of the IRIS dataFrame so that it has 3 columns:
- variable (one of sepal_length, sepal_width, petal_length, petal_width)
- value
- target

If you would like to learn more, [Tidy Data](https://www.jstatsoft.org/index.php/jss/article/view/v059i10/v59i10.pdf) by [Hadley Wickham](http://hadley.nz/) provides a very nice explanation of best practices for data formating.

Hint: look at reshaping functions in http://pandas.pydata.org/Pandas_Cheat_Sheet.pdf

```python
iris_df_long = pd.melt(iris_df, id_vars=['target'], value_vars=['sepal_length'])
iris_df_long.head()
```

      target
      variable
      value

      0
      Iris-setosa
      sepal_length
      4.9

      1
      Iris-setosa
      sepal_length
      4.7

      2
      Iris-setosa
      sepal_length
      4.6

      3
      Iris-setosa
      sepal_length
      5.0

      4
      Iris-setosa
      sepal_length
      5.4

Now create a box-plot of values that each variable takes, split by the target species.

```python
# Hint: use a `catplot`
sns.catplot(x="target", y="sepal_length", kind="box", data=iris_df)
# sns.catplot(x="target", y="sepal_width", kind="box", data=iris_df)

# TODO: create two more plots, using a boxenplot and a swarmplot.

sns.catplot(x="target", y="sepal_length", kind="boxen", data=iris_df)
sns.catplot(x="target", y="sepal_length", kind="swarm", data=iris_df)
```

### k-Nearest Neighbors

We will use the loaded Iris data describing iris flowers
and shows relations between their length and petal width for three
species (namely: setosa, versicolor, virginica).

For this exercise we will restrict our analysis to just two variables: **petal length** and **petal width**.

```python
unknown_df = pd.DataFrame(
    [[1.5, 0.3, 'unknown'],
     [4.5, 1.2, 'unknown'],
     [5.1, 1.7, 'unknown'],
     [5.5, 2.3, 'unknown']],
     columns=['petal_length', 'petal_width', 'target'])

sns.scatterplot(x='petal_length', y='petal_width', hue='target', data=iris_df)
sns.scatterplot(x='petal_length', y='petal_width', color='gray', marker='v',
                 label='unknown', s=70, data=unknown_df)
```

Based on these two features, it is easy to distinguish iris setosa from the two remaining species. Yet iris versicolor and virginica remain mixed together.

Looking closely at the plot, we might estimate the species of the selected unknown irises (gray triangles). For three of them the answer seems obvious â€“ they belong in uniformly-colored areas covered by one species only. Yet unknown iris flower in (5.1, 1.7) is troublesome â€“ it lays on the boundary of versicolor and virginica clusters. We can assume, that its species is the one of the closest one to it, coming from the training set (and so having a label).

K-Nearest Neighbors method (http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm) solves the classification problem, i.e. sets class labels (species in case of irises) of a previously unseen sample by choosing the most common class among the top k neighbors of the sample in question (for instance according to the Euclidean distance). Thus, the k-Nearest Neighbors algorithm works as follows. For each unlabeled sample x:
1. Find k nearest neighbors among the labeled samples.
2. Set the most common label among them as label of x.

#### Problem 3 [3p]

##### Implement the k-Nearest Neighbors algorithm [1p].

Take advantage of matrix calculus rather than using for loops.

**Tip:** What is computed by \begin{equation} \sqrt{(X - Y)^T (X - Y)} \end{equation} when both X and Y are vectors?

**Tip:** Try to use broadcasting (NumPy: http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) and built-ins sort, numpy.sort, numpy.argsort (sorting), scipy.stats.mode (choosing the most common element of the set).

```python
def KNN(train_X, train_Y, test_X, ks, verbose=False):
    """
    Compute predictions for various k
    Args:
        train_X: array of shape Ntrain x D
        train_Y: array of shape Ntrain
        test_X: array of shape Ntest x D
        ks: list of integers
    Returns:
        preds: dict k: predictions for k
    """
    # Cats data to float32
    train_X = train_X.astype(np.float32)
    test_X = test_X.astype(np.float32)

    # Alloc space for results
    preds = {}

    if verbose:
        print("Computing distances... ")
    #
    # TODO: fill in an efficient distance matrix computation
    # https://medium.com/@souravdey/l2-distance-matrix-vectorization-trick-26aa3247ac6c
    dists = -2 * np.dot(train_X, test_X.T) + np.sum(test_X**2, axis=1) + np.sum(train_X**2, axis=1)[:, np.newaxis]

    if verbose:
        print("Sorting... ")

    # TODO: findes closest trainig points
    # Hint: use argsort
    closest = np.argsort(dists, axis=0)

    if verbose:
        print("Computing predictions...")

    # closest trainig points - > closest trainig targets
    targets = train_Y[closest]

    for k in ks:
        predictions = sstats.mode(targets[:k])[0] #take k closest targets
        predictions = predictions.ravel()
        preds[k] = predictions
    if verbose:
        print("Done")
    return preds
```

```python
# Now classify the 4 unknown points
iris_x = np.array(iris_df[['petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])

unknown_x = np.array(unknown_df[['petal_length', 'petal_width']])
# print(unknown_x.shape)
print(KNN(iris_x, iris_y, unknown_x, [1, 3, 5, 7]))

# print(iris_x.shape)
# print(iris_y.shape)
# print(unknown_x.shape)
```

    {1: array(['Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-virginica'], dtype=object), 3: array(['Iris-setosa', 'Iris-versicolor', 'Iris-versicolor',
           'Iris-virginica'], dtype=object), 5: array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica',
           'Iris-virginica'], dtype=object), 7: array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica',
           'Iris-virginica'], dtype=object)}

##### Plot the Decision boundary [1p]

Use meshgrid to generate the points in the space spanned by data
Than map the classes to numbers 0, 1, 2 and make a contour plot with the
decision boundary

```python
iris_x = np.array(iris_df[['petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])

X_space = np.linspace(0,7,200)
Y_space = np.linspace(0,3,200)
mesh_x, mesh_y = np.meshgrid(X_space, Y_space)

#use np.unique with suitable options to map the class names to numbers
target_names, iris_y_ids = np.unique(iris_y, return_inverse=True)
# print(target_names, iris_y_ids)

mesh_data = np.hstack([mesh_x.reshape(-1, 1), mesh_y.reshape(-1, 1)])
colors = ['red', 'blue', 'green']

preds = KNN(iris_x, iris_y_ids, mesh_data, [1, 3, 5, 7])
for k, preds_k in preds.items():
    plt.figure()
    plt.title(f"Decision boundary for k={k}")
    plt.contourf(mesh_x, mesh_y, preds_k.reshape(200,200), levels=2, cmap='Pastel1')
    plt.scatter(iris_x[:,0], iris_x[:,1], c=iris_y_ids, s=10, cmap='Dark2')

#     for p, c in zip(iris_x, iris_y_ids):
#       plt.scatter(p[0], p[1], color=colors[c], s=10)

```

##### Estimate performance for various ks [1p]
Consider the following experiment:
1. We scramble the data and split it into two parts - training set (66.6% of all samples) and test set (33.4%).
2. Based on the training set, we use the k-NN algorithm to predict the labels on the test set.
3. We then check the number of errors and write it down.

Do this 500 times for k âˆˆ {1, 3, 5, ..., 19}. Plot a function of the average number of errors as the function of k. It should be similar to the one below.

```python
#TODO: write a function to compute error rates
def err_rates(preds, test_Y):
    ret = {}
    for k, preds_k in preds.items():
        # TODO: fill in error count computation
        assert(len(test_Y) == len(preds_k))
        ret[k] = np.sum(preds_k != test_Y) / test_Y.shape[0]
    return ret
```

```python
iris_x = np.array(iris_df[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']])
iris_y = np.array(iris_df['target'])
N = len(iris_y)

ks = range(1, 20, 2)
results = []

for _rep in tqdm_notebook(range(1000)):
    #TODO
    # Use np.split and np.permutation to get training and testing indices
    train_idx, test_idx = np.split(np.random.permutation(N), [math.floor(N * 0.666)])

    #TODO: apply your kNN classifier to data subset
    preds = KNN(iris_x[train_idx], iris_y[train_idx], iris_x[test_idx], ks)
    errs = err_rates(preds, iris_y[test_idx])

    for k, errs_k in errs.items():
        results.append({'K':k, 'err_rate': errs_k})

# results_df will be a data_frame in long format
results_df = pd.DataFrame(results)

```

    HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))

```python
# Plot a function of the average number of errors as the function of k. It should be similar to the one below.
plt.figure()
sns.regplot(x='K', y=results_df.err_rate, data=results_df, x_estimator=np.mean, order=2)
# plt.figure()
# sns.regplot(...)

# results_df.head()

# gb = results_df.groupby(['K', 'err_rate'])
# counts = gb.size().to_frame(name='counts')
# counts.head()

```

#### Problem 5 [2p + 2p bonus]

Download a categorical dataset from UCI and try to find the most predictive variables

```python
columns = [
 "target", "cap-shape", "cap-surface", "cap-color", "bruises?", "odor",
 "gill-attachment", "gill-spacing", "gill-size", "gill-color", "stalk-shape",
 "stalk-root", "stalk-surface-above-ring", "stalk-surface-below-ring",
 "stalk-color-above-ring", "stalk-color-below-ring", "veil-type", "veil-color",
 "ring-number", "ring-type", "spore-print-color", "population", "habitat", ]

# Use read_csv to load the data.
url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.data'
mushroom_df = pd.read_csv(url, header=None, names=columns)
mushroom_df.head()
```

      target
      cap-shape
      cap-surface
      cap-color
      bruises?
      odor
      gill-attachment
      gill-spacing
      gill-size
      gill-color
      ...
      stalk-surface-below-ring
      stalk-color-above-ring
      stalk-color-below-ring
      veil-type
      veil-color
      ring-number
      ring-type
      spore-print-color
      population
      habitat

      0
      p
      x
      s
      n
      t
      p
      f
      c
      n
      k
      ...
      s
      w
      w
      p
      w
      o
      p
      k
      s
      u

      1
      e
      x
      s
      y
      t
      a
      f
      c
      b
      k
      ...
      s
      w
      w
      p
      w
      o
      p
      n
      n
      g

      2
      e
      b
      s
      w
      t
      l
      f
      c
      b
      n
      ...
      s
      w
      w
      p
      w
      o
      p
      n
      n
      m

      3
      p
      x
      y
      w
      t
      p
      f
      c
      n
      n
      ...
      s
      w
      w
      p
      w
      o
      p
      k
      s
      u

      4
      e
      x
      s
      g
      f
      n
      f
      w
      b
      k
      ...
      s
      w
      w
      p
      w
      o
      e
      n
      a
      g

5 rows × 23 columns

Implement the function `entropy` to compute the entropy of a columnt of the dataset.

The [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of a discrete variable is defined to be:

$$H(X) = -\sum_x p_X(x) \log_2 p_X(x)$$.

A good in tutorial is given by Chris Olah: https://colah.github.io/posts/2015-09-Visual-Information/.

When $X$ is a discrete random variables, we can estimate the probabilities with counts:

$$p_X(x) = \frac{\text{number of instances where }X=x}{\text{total number of instances}}$$.

Hint: the following `pandas` functions may be useful:
- `count`
- `value_count`

Then use the datafranme's `apply` function to compute the entropy of all columns.

```python
def entropy(series):
    p = series.value_counts(normalize=True)
    return - np.sum(p * np.log2(p))

mushroom_df.apply(entropy)
```

    target                      0.999068
    cap-shape                   1.652889
    cap-surface                 1.575486
    cap-color                   2.510143
    bruises?                    0.979327
    odor                        2.319414
    gill-attachment             0.173129
    gill-spacing                0.637878
    gill-size                   0.892256
    gill-color                  3.030433
    stalk-shape                 0.986927
    stalk-root                  1.822922
    stalk-surface-above-ring    1.221348
    stalk-surface-below-ring    1.399135
    stalk-color-above-ring      1.936809
    stalk-color-below-ring      1.978163
    veil-type                  -0.000000
    veil-color                  0.196238
    ring-number                 0.420680
    ring-type                   1.535121
    spore-print-color           2.203227
    population                  2.003398
    habitat                     2.274747
    dtype: float64

Implement the conditional entropy computation

$$H(Y|X) = \sum_x p_X(x) H(Y|x) = \sum_x p_X(x) \sum_y p_Y(y|x) \log_2 p_Y(y|x)$$

Hint 1: the above formlua can be computed as follows:
1. split the data by the values of $X$
2. for each value $x$ that $X$ takes, compute the entropy of $Y$
3. average the entropies, weighting them by how frequent the $x$ value ocurred.

Hint 2: helpful pandas constructs are:
- `groupby` and `agg`
- you can aggregate a grouping using your own custom functions

```python
def cond_entropy(df, X, Y):
    """Compute the conditional H(X|Y) in dataframe df
    Args:
        df: a dataframe
        X: the name of the conditioning columt
        Y: the name of the column whose entropy we wish to compute
    """
    grouped = df.groupby(X)[Y]
    return grouped.agg(entropy).agg(np.average, weights=grouped.agg('count'))

```

```python
# Now for each column C compute the conditional entropy H(target|C)
# Knowing which variable tells us the most about the target?
for cname in mushroom_df.columns:
    print(f"{cname}: {cond_entropy(mushroom_df, cname, 'target')}")
```

    target: 0.0
    cap-shape: 0.9502711949370876
    cap-surface: 0.9704776640986875
    cap-color: 0.9630186138962564
    bruises?: 0.8066884111112408
    odor: 0.0929929194884606
    gill-attachment: 0.9849028696218441
    gill-spacing: 0.8981847128758902
    gill-size: 0.7689135217244144
    gill-color: 0.582090373456329
    stalk-shape: 0.9915511243027961
    stalk-root: 0.8642502592451845
    stalk-surface-above-ring: 0.7143422976539759
    stalk-surface-below-ring: 0.7271734234797138
    stalk-color-above-ring: 0.7452227234102207
    stalk-color-below-ring: 0.7576523303448939
    veil-type: 0.9990678968724603
    veil-color: 0.9752508807515435
    ring-number: 0.9606152276293699
    ring-type: 0.6810463860789229
    spore-print-color: 0.5183629791875449
    population: 0.7971098778056079
    habitat: 0.8422342922673685

Bonus questions:
- **[1p]** Implement computation of [Mutual Information ](https://en.wikipedia.org/wiki/Mutual_information)
- **[1p]** Add an ID column, that assigns a unique ID to each observation (row). Comompute the $H(target|ID)$ and the mutual information between target and ID. How to interpret the results? Do you think the ID is important in predicting the target?

```python
# def mutual_information(df,X,Y):
#     '''
#     Straightforward implementation of formula
#     '''
#     if X == Y:
#         return entropy(df[X])

#     new_df = df[[X,Y]]
#     A_X = new_df.groupby([X]).agg('count')
#     A_Y = new_df.groupby([Y]).agg('count')
#     A_XY = df.groupby([X,Y]).agg('count')
#     res = 0
#     N = new_df.shape[0]

#     for XY in A_XY.index:
#         P_XY = A_XY.loc[XY][0] / N
#         P_X = A_X.loc[XY[0]][0] / N
#         P_Y = A_Y.loc[XY[1]][0] / N
#         res += P_XY * np.log2(P_XY / (P_X * P_Y))

#     return res

def mutual_information(df,X,Y):
    '''
    I(X, Y) = H(X) - H(X | Y)
    '''
    return entropy(df[X]) - cond_entropy(df, X, Y)
```

```python
# for cname in mushroom_df.columns:
#     print(f"1 -> {cname}: {mutual_information(mushroom_df, cname, 'target')}")

# for cname in mushroom_df.columns:
#     print(f" -> {cname}: {mutual_information2(mushroom_df, cname, 'target')}")

mushroom_df["ID"] = mushroom_df.index
print(f"H(target|ID): {cond_entropy(mushroom_df,'ID','target')}, \
      MI(target,ID): {mutual_information(mushroom_df,'target','ID')}, \
      H(ID|target): {cond_entropy(mushroom_df,'target','ID')}")
```

    H(target|ID): 0.0,       MI(target,ID): 0.9990678968725724,       H(ID|target): 11.988906627423692

#### Problem 6 [2p]

Apply the K-Nearest Neighbors (K-NN) algorithm to the MNIST dataset.

The MNIST (http://yann.lecun.com/exdb/mnist/) dataset consists of normalized (centered and stretched) scans of hand-written digits. Specifically, each element of the dataset is a 28 Ã— 28 grayscale image, thus having 764 8-bit pixels.

1. Display a few objects from each of the classes, paying attention to aesthetics and clarity of your presentation. **Note:** You already downloaded the dataset in "Setup" section. Please use the code below to get started.

2. **[2p]** Apply a k-NN classifier to the MNIST dataset. First, divide the training set into two parts, which we will call training and validation. On MNIST use the first 50000 samples for training and the last 10000 for validation. Then find the optimal number of neighbors by assessing the accuracy on the validation set. You do not need to repeat this experiment multiple times. Finally, compute the accuracy on the test set obtained with the best previously chosen number of neighbors. On MNIST you should get about 3% errors. Pick a few mislabeled samples from the test dataset and plot them along with the correct ones. **Note:**
  * MNIST is much larger than the Iris dataset. A good implementation may need a few minutes depending on your runtime type. Please optimize your algorithm:
  * Compute the distances only once, then test for different values of k.
  * Use vectorized expressions to compute the distance. It is possible to compute all distances between the training and testing points in one expression. Hint: think about the vectorized expression \begin{equation}(X - Y)^T (X - Y)\end{equation}
  * You can use single precision numbers in computation.
  * If your code is taking a long time to execute, please save its results before the lab session.

**Note:** in NumPy, matrices have its own data type (dtype), which is retained during
calculations. Please pay attention to it. I particular, do not subtract values of data types not
having the sign bit, do not divide integers, etc. Results of such operations will not be
automatically casted to types having the required precision.

```python
with np.load('mnist.npz') as data:
    mnist_full_train_data_uint8 = data['train_data']
    mnist_full_train_labels_int64 = data['train_labels']
    mnist_test_data_uint8 = data['test_data']
    mnist_test_labels_int64 = data['test_labels']

# Split train data into train and validation sets
mnist_train_data_uint8 = mnist_full_train_data_uint8[:50000]
mnist_train_labels_int64 = mnist_full_train_labels_int64[:50000]
mnist_valid_data_uint8 = mnist_full_train_data_uint8[50000:]
mnist_valid_labels_int64 = mnist_full_train_labels_int64[50000:]
```

```python
plot_mat(mnist_train_data_uint8[:20, None], cmap='gray')
```

```python
# MNIST is large.
# Implement a batched KNN classifier, which processes the test data in small batches
# and returns the error rates

# The code should not run for more than a couple of minutes on the Colab runtime,
# If it is slower, optimize the distance computation in KNN

def batched_KNN(train_X, train_Y, test_X, ks, verbose=False, batch_size=200):
    all_preds = {k: [] for k in ks}
    for i in range(0, test_X.shape[0], batch_size):
        batch_X = test_X[i:i + batch_size]
        if verbose:
            print(f"Processing batch {i}:{i + batch_X.shape[0]}... ")

        # TODO: run KNN on the batch and save the predictions
            for k in all_preds.keys():
               preds = KNN(train_X, train_Y, batch_X, ks)
        # TODO: combine predictions from batches
               all_preds[k] = np.concatenate((all_preds[k], preds[k]))
    return all_preds
```

```python
# Now find the best k on the validation set
ks = [1, 3, 5, 7, 9]
mnist_validation_preds = batched_KNN(
    mnist_train_data_uint8.astype('float32').reshape(-1, 28*28),
    mnist_train_labels_int64,
    mnist_valid_data_uint8.astype('float32').reshape(-1, 28*28),
    ks, verbose=True)

mnist_validation_errs = err_rates(mnist_validation_preds, mnist_valid_labels_int64)
plt.plot(ks, [mnist_validation_errs[k] for k in ks])
```

```python
# Now use the best k to compute the test error

best_K = 3

mnist_test_preds = batched_KNN(
    mnist_full_train_data_uint8.astype('float32').reshape(-1, 28*28),
    mnist_full_train_labels_int64,
    mnist_test_data_uint8.astype('float32').reshape(-1, 28*28),
    [best_K], verbose=True)

mnist_test_errs = err_rates(mnist_test_preds, mnist_test_labels_int64)
print(f"\n\nWhen k={best_K} the test error rate is {mnist_test_errs[best_K] * 100.0:.1f}%%")
```

    Processing batch 0:200...
    Processing batch 200:400...
    Processing batch 400:600...
    Processing batch 600:800...
    Processing batch 800:1000...
    Processing batch 1000:1200...
    Processing batch 1200:1400...
    Processing batch 1400:1600...
    Processing batch 1600:1800...
    Processing batch 1800:2000...
    Processing batch 2000:2200...
    Processing batch 2200:2400...
    Processing batch 2400:2600...
    Processing batch 2600:2800...
    Processing batch 2800:3000...
    Processing batch 3000:3200...
    Processing batch 3200:3400...
    Processing batch 3400:3600...
    Processing batch 3600:3800...
    Processing batch 3800:4000...
    Processing batch 4000:4200...
    Processing batch 4200:4400...
    Processing batch 4400:4600...
    Processing batch 4600:4800...
    Processing batch 4800:5000...
    Processing batch 5000:5200...
    Processing batch 5200:5400...
    Processing batch 5400:5600...
    Processing batch 5600:5800...
    Processing batch 5800:6000...
    Processing batch 6000:6200...
    Processing batch 6200:6400...
    Processing batch 6400:6600...
    Processing batch 6600:6800...
    Processing batch 6800:7000...
    Processing batch 7000:7200...
    Processing batch 7200:7400...
    Processing batch 7400:7600...
    Processing batch 7600:7800...
    Processing batch 7800:8000...
    Processing batch 8000:8200...
    Processing batch 8200:8400...
    Processing batch 8400:8600...
    Processing batch 8600:8800...
    Processing batch 8800:9000...
    Processing batch 9000:9200...
    Processing batch 9200:9400...
    Processing batch 9400:9600...
    Processing batch 9600:9800...
    Processing batch 9800:10000...

    When k=3 the test error rate is 2.9%%

### Locality sensitive hashing

Problem 5 was about speeding up the inference using loops implicitly present in matrix multiplication instead of explicit loops in Python. In this problem, we will explore a strategy to truly reduce the total number of computations required to find nearest neighbors without sacrificing too much accuracy.

To speed up nearest neighbor search we will employ *Locality Sensitive Hashing (LSH)* functions. For a given distance metric, the locality sensitive hash should put items that are similar into the same bucket. Notice that this is essentially a design choice opposite to traditional cryptographic hash functions that should amplify the difference of similar inputs (typically we want that small perturbations of data result in large changes to the hash value).

One of the simplest implementations of LSH approximates the cosine distance. Let $x\in \mathbb{R}^N$ and $y\in \mathbb{R}^N$ be two vectors. Their cosine distance is defined as:

\begin{equation}
    d_\text{cos}(x,y) = \frac{x \cdot y}{\|x\| \|y\|} = \cos\left(\theta(x,y)\right),
\end{equation}
where $\theta(x,y)$ is the unsigned angle between $x$ and $y$.

We will construct a family $H$ of hash functions that are an LSH for angle distances (an approximation to cosine distance). Assume $p\in \mathbb{R}^N$ is a random vector (components are sampled from the normal distribution) of length 1. Then define the hash function $h(x) = \text{sgn}(x\cdot p)$, where $\text{sgn()}$ is the sign function. It can be proven that:

\begin{equation}
    p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}.
\end{equation}

The equation means that the probability of a hash collision grows as the the angle between two vectors gets smaller. Therefore, vectors that are close according to the cosine distance will be put with high probability into the same bin (we use the fact that for small $\theta$ we can approximate $\cos(\theta) = 1 - \theta/\pi$.

We will say that a family of randomly chosen hash functions $H$ is $(d_1, d_2, p_1, p_2)$-sensitive with respect to a distance metric $d$ if for any $x$ and $y$:
1. If $d(x,y) \leq d_1$ then $p_{h\in H}[h(x)=h(y)] \geq p_1$.
2. If $d(x,y) \geq d_2$ then $p_{h\in H}[h(x)=h(y)] \leq p_2$.

For example, our family of randomly chosen hyperplanes is $(d_1, d_2, (1-d_1)/\pi, (1-d_2)/\pi)$-sensitive.

Ideally, vectors should be placed into the same bin with a high probability if their distance is smaller than a threshold, and with a low probability if their distance is larger that the threshold. By combining hashing functions we can get closer to this ideal sensitivity.

Given a family of hash functions $H$ with sensitivity $(d_1, d_2, p_1, p_2)$ we can construct a new family $H'$ by combining $r$ functions from $H$:
1. AND: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\forall_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, (p_1)^r, (p_2)^r)$-sensitive.
2. OR: let $h=[h_1, h_2, \ldots, h_r] \in H'$ and $h(x)=h(y)$ if and only if $\exists_i h_i(x)=h_i(y)$. Then $H'$ is $(d_1, d_2, 1-(1-p_1)^r, 1-(1-p_2)^r)$-sensitive.

AND makes all probabilities shrink, but properly choosing $r$ we can make the lower probability approach 0 while the higher does not. Conversely, OR makes all probabilities grow, we can make the upper probability approach 1 while the lower does not.

#### Problem 7 [1-3p bonus]

1. **[1bp for exercises list]** **Note:** you can show sketches of proofs for this assignment.
    1. Show that angle between vectors is a metric (https://en.wikipedia.org/wiki/Metric_(mathematics)).

    2. Show that $p_{h\in H}[h(x)=h(y)] = 1 - \frac{\theta(x,y)}{\pi}$ for $h$ computed using a randomly chosen hyperplane.

    3. Show the properties of either AND or OR boosting of LSH.

  Please show the solution to this problem dirung the Session for Homework 1, the bonus point will also be added to the points from Homework 1.

3. **[1-3bp]** Reimplement k-Nearest Neighbors for MNIST classification using the cosine distance instead of the Euclidean distance. Choose a sensible value of $k$. Use Locality Sensitive Hashing to achieve an error rate no greater than $150\%$ of the original error rate with at least a $90\%$ speedup (i.e., by considering on average at most 5000 training samples per query image). For a few settings plot the speedup-vs-accuracy relation.

  **Note:** points will be awarded based on ingenuity of your solution. Feel free to explore your own ideas!

```python

```