# Open Systems
All of the examples in the previous two chapters were closed systems. The ultimate state of closed, chemically reacting systems is chemical equilibrium. Living systems are characterized by mass and energy flow across their boundaries that keep them away from equilibrium. We now discuss open systems, which allow molecules to enter and leave. Open systems ultimately reach a steady state that is often close to a homeostatic state of interest.

**MASSpy** will be used to demonstrate some of the topics in this chapter.

```python
from mass import (
    MassModel, MassMetabolite, MassReaction, Simulation, MassSolution)
from mass.visualization import plot_time_profile, plot_phase_portrait
```

Other useful packages are also imported at this time.

```python
import matplotlib.pyplot as plt
```

## Basic Concepts
There are several fundamental concepts that need to be understood when one considers open systems. We discuss the more significant ones in this section.

### The system boundary
Implicit in the term 'open system' is the notion of an inside and an outside, the division of the world into two domains. The separation between the two is the _system boundary,_ which thus defines what is inside a system and belongs to it, and what is outside.

The definition of a boundary can be physical. An example of a physical boundary may be the cell wall, that clearly defines what is inside and what is outside. Similarly, the outer membrane of the mitochondria can serve as a clearly defined physical system boundary. Thus, systems can have hard, immovable boundaries or soft, flexible ones. In the latter case, the volume of the system may be changing.

The definition of a boundary can also be virtual. For instance, we can define a pathway, such as the TCA cycle, as a system, or the amino acid biosynthetic pathways in a cell as a system. In both cases we might draw dashed lines around it on the metabolic map to indicate the system boundary.

### Crossing the boundary: inputs and outputs
Once a system boundary has been established, we can identify and define interactions across it. They are the _inputs_ and _outputs_ of the system. In most cases, we will be considering mass flows in and out of a system; molecules coming and going. Such flows are normally amenable to experimental determination.

There may be other quantities crossing the system boundary. For instance, if we are considering photosynthesis, photons are crossing the system boundary providing a net influx of energy. Forces can also act on the system boundary. For instance, if the number of molecules coming into and leaving a system through a coupled transport mechanism is not balanced, then osmotic pressure may be generated across the system boundary. Cells have sodium-potassium pumps that displace an uneven number of two types of cations to deal with osmotic imbalances.

### Perturbations and boundaries
In most analyses of living systems, we are concerned about changes in the environment. The temperature may change, substrate availability may change, and so on. Such changes in the environment can be thought of as forcing functions to which the living system responds. These are one-way interactions, from the environment to the system. In most cases, we consider the system to be small (relative to the environment) and that the environment buffers the activities of the system. For instance, the environment may provide an infinite sink for cellular waste products.

In other cases, the activities of the system influence the state of the environment. In batch fermentation, for example, the metabolic activities will substantially change the chemical composition of the medium during the course of the fermentation. On a global scale, we are now becoming more concerned about the impact that human activities have on the climate and the larger environment of human socio-economic activities. In such cases, there are two-way interactions between the system and the environment that need to be described.

### Inside the boundary: the internal network
The definition of a system boundary determines what is inside. Once we know what is inside the system, we can determine the network of chemical transformations that takes place. This network has topology and its links have kinetic properties.

In most cases, it is hard to measure the internal state of a system. There typically are only a few non-invasive measurements available. Occasionally, there are probes that we can use to observe the internal activities. Tracers, such as $^{13}C$ atoms strategically placed in substrates, can be used to trace input to output and allow us to determine pieces of the internal state of a system. Tearing a system apart to enumerate its components is of course possible, but the system is destroyed in the process. Thus, we are most often in the situation where we cannot fully experimentally determine the internal state of a system, and may have to be satisfied with only partial knowledge.

### From networks to system models
Mathematical models may help us to simulate or estimate the internal state of the system. Sometimes we are able to bracket its state based on the inputs and outputs and our knowledge of the internal network.

Full dynamic simulation requires extensive knowledge about the internal properties of a system. Detailed models that describe the dynamic state of a system require the definition of the system boundary, the inputs and outputs, the structure of the internal network, and the kinetic properties of the links in the network. We can then simulate the response of the system to various perturbations, such as changes in the environmental conditions.

### The functional state
Once a system model has been formulated, it can be used to compute the functional state of the system for a given set of conditions. Closed systems will eventually go to chemical equilibrium. However, open systems are fundamentally different. Due to continuous interactions with the environment, they have internal states that are either dynamic or steady. A system that has fast internal time constants relative to changes in the environment will reach a steady state, or a quasi-steady state. For a biological system, such functional states are called _homeostatic states_. Such states are maintained through energy dissipation. The flow of energy in has to exceed the energy leaving the system. This difference allows living systems to reach a functional homeostatic state.

**Figure 6.1:** Open systems. (a) example production of ATP from pyruvate by mitochondria (Prepared by Nathan Lewis). (b) Basic definitions associated with an open system.

## Reversible reaction in an Open Environment
In an open environment, the simple reaction of Eq. (4.1)  has an inflow $(b_1)$ of $x_1$ and an outflow $(b_2)$ of $x_2$:

$$\begin{equation} \stackrel{b_1}{\rightarrow} x_1 \underset{v_{-1}}{\stackrel{v_1}{\rightleftharpoons}} x_2 \stackrel{b_2}{\rightarrow} \tag{6.1} \end{equation}$$

The input is fixed by the environment and there is a first-order rate for the product to leave the system and thus we have

$$\begin{equation} b_1 = \text{constant and}\ b_2 = k_2x_2 \end{equation}$$

This defines the system boundary, and the inputs and outputs.

The stoichiometric matrix is

$$\begin{equation} \textbf{S} = \begin{pmatrix} {1} & {-1} & {1} & {0} \\ {0} & {1} & {-1} & {-1} \\ \end{pmatrix} \end{equation} \tag{6.2}$$

where $\textbf{x} = (x_1,\  x_2)$ and $\textbf{v} = (b_1, \ k_1x_1, \  k_{-1}x_2, \  k_2x_2)$ The stoichiometric matrix has a rank of 2 and is thus a two-dimensional dynamic system. The differential equations that will need to be solved are:

$$\begin{equation} \frac{dx_1}{dt} = b_1 - k_1x_1 + k_{-1}x_2, \
\frac{dx_2}{dt} = k_1x_1 - k_{-1}x_2 - k_2x_2 \tag{6.3} \end{equation}$$

There are no conservation quantities.

### The steady states
There are three properties of the steady state of interest. We can find first the steady state fluxes and second the steady state concentrations. Third, we can determine the difference between the steady state and the equilibrium state of this open system.

#### Steady state fluxes:
The steady state of the fluxes is given by

$$\begin{equation} \textbf{Sv}_{ss} = 0 \tag{6.4} \end{equation}$$

and thus $\textbf{v}_{ss}$ resides in the null space of $\textbf{S}$. For this matrix, the null space is two dimensional; $\text{Dim}(\text{Null}(\textbf{S}))=n-r=4-2=2$, where $n=4$ is the number of fluxes and the rank is $r=2$. The null space is spanned by two pathway vectors: (1,1,0,1) and (0,1,1,0). The former is the path through the system while the latter corresponds to the reversible reaction. These are known as type 1 and type 3 extreme pathways, respectively (_Systems Biology: Properties of Reconstructed Networks_). All steady state flux states of the system are a non-negative combination of these two vectors.

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ k_1x_{1, ss}, \ k_{-1}x_{2, ss}, \ k_2x_{2, ss})  \tag{6.5} \\ &= a(1, 1, 0, 1) + b(0, 1, 1, 0), \ a \geq 0, \ b \geq 0 \tag{6.6} \end{align}$$

#### Steady state concentrations:
The concentrations in the steady state can be evaluated from

$$\begin{equation} 0 = b_1 - k_1x_1 + k_{-1}x_2, \ 0 = k_1x_1 - k_{-1}x_2 - k_2x_2 \end{equation}$$ $$\tag{6.7}$$

If we add these equations we find that $x_{2, ss} = b_1 / k_2$. This concentration can then be substituted into either of the two equations to show that $x_{1, ss} = (1 + k_{-1}/k_2)(b_1/k_1)$. Thus, the steady state concentration vector is

$$\begin{equation} \textbf{x}_{ss} = \begin{pmatrix} {x_{1, ss}} \\ {x_{2, ss}} \end{pmatrix} = (\frac{b_1}{k_2})\begin{pmatrix} {\frac{k_2 + k_{-1}}{k_1}} \\ {1} \end{pmatrix} \end{equation}$$ $$\tag{6.8}$$

These steady state concentrations can be substituted into the steady state flux vector to get

$$\begin{align} \textbf{v}_{ss} &= b_1(1, 1+(\frac{k_{-1}}{k_2}), \ (\frac{k_{-1}}{k_2}), \ 1)  \tag{6.9} \\ &= a(1, 1, 0, 1) + b(0, 1, 1, 0), \ a = b_1, \ b = (\frac{b_1k_{-1}}{k_2}) \end{align}$$

Therefore, the steady state flux distribution is a summation of the straight through pathway and the discounted flux through the reversible reaction. The key quantity is $k_{-1} / k_2$ that measures the relative rate of $x_2$ reacting back to form $x_1$ versus the rate at which it leaves the system.

#### The distance from the equilibrium state:
The difference between the steady state and the equilibrium state can be measured by:

$$\begin{equation} \frac{x_{2, ss}/x_{1, ss}}{x_{2, eq}/x_{1, eq}} = \frac{1}{1 + k_2/k_{-1}} \stackrel{k_2 << k_{-1}}{\longrightarrow} 1 \tag{6.11} \end{equation}$$

Thus, when $k_2 << k_{-1}$, the steady state approaches the equilibrium state (recall that $(x_{2, ss}/x_{1, ss})/(x_{2, eq}/x_{1, eq}) = \Gamma/K_{eq}$ Section 2.2). If the exchange with the environment is slow relative to the internal reaction rates, the internal system approaches that of an equilibrium state.

### Dynamic states for reversible reactions
The dynamic states are computed from the dynamic mass balances for a given condition. We are interested in two dynamic states: the approach to the steady state, and the response to a change in the input flux, $b_1$

#### A two-phase transient response:
Simulation of this system with an input rate of $b_1 = 0.01$ and a slow removal rate, $k_2 = 0.1$, from an initial state of $x_1(0) = 1.0$ and $x_2(0) = 0$ is shown in Figure 6.2. There are two discernible time scales: a rapid motion of the equilibrating reaction, and a slow removal of $x_2$ from the system, Figure 6.2b. A phase portrait shows a rapid movement along a line with a negative slope of 1, showing the existence of a conservation quantity $(x_1 + x_2)$ on a fast time scale, followed by a slow motion down a quasi-equilibrium line with a slope of 1/2 to a steady state point. Note the difference from Figure 4.4c.

```python
# Create MassModel
model = MassModel('Linear_Reversible_Open')
# Generate the MassMetabolites
x1 = MassMetabolite("x1")
x2 = MassMetabolite("x2")

# Generate the MassReactions
b1 = MassReaction("b1", reversible=False)
v1 = MassReaction("v1")
b2 = MassReaction("b2", reversible=False)
# Add metabolites to the reaction, add reaction to the model
b1.add_metabolites({x1: 1})
v1.add_metabolites({x1: -1, x2: 1})
b2.add_metabolites({x2: -1})
model.add_reactions([b1, v1, b2])
# Set parameters
b1.kf = 0.01
v1.kf, v1.kr = (1, 2)
b2.kf = 0.1

# Set initial conditions for model
x1.ic = 1
x2.ic = 0
# Utilize type 2 rate law for kf and kr parameters defined
model.get_rate_expressions(rate_type=2, update_reactions=True)

# Set a custom rate for b1 to remove substrate concentration dependence
model.add_custom_rate(reaction=b1, custom_rate=b1.kf_str)
```

```python
t0 = 0
tf = 150

sim = Simulation(model, verbose=True)
conc_sol, flux_sol = sim.simulate(model, time=(t0, tf),
                                  interpolate=True,
                                  verbose=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Linear_Reversible_Open' into RoadRunner.
    Getting time points
    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Simulating 'Linear_Reversible_Open'
    Simulation for 'Linear_Reversible_Open' successful
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_2 = plt.figure(figsize=(13, 5))
gs = fig_6_2.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_2.add_subplot(gs[0, 0])
ax2 = fig_6_2.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x=x1, y=x2, ax=ax1,
    xlabel=x1.id, ylabel=x2.id, xlim=(-.05, 1.2), ylim=(-.05, 0.5),
    title=("(a) Phase portrait of x1 vs. x2", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

ax1.annotate(
    'motion towards\n  a qe state',
    xy=(conc_sol[x1.id](0.5), conc_sol[x2.id](0.5)),
    xytext=(conc_sol[x1.id](0.3), conc_sol[x2.id](0.3)));

ax1.annotate(
    'motion of a\n  qe state',
    xy=(conc_sol[x1.id](tf), conc_sol[x2.id](tf)),
    xytext=(conc_sol[x1.id](tf) + 0.05, conc_sol[x2.id](tf) + 0.15));

plot_time_profile(
    conc_sol, ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Concentration Profiles", {"size": "large"}));
fig_6_2.tight_layout()
```

**Figure 6.2:** The concentration time profiles for the reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ for $k_1 = 1$, $k_{-1} = 2$, $x_1(0) = 0$, and $b_1 = 0.01$. (a) The phase portrait of $x_1$ and $x_2$ (b) The concentrations as a function of time.

The same pools can be formed as defined in Eq. (4.9):

$$\begin{equation} \begin{pmatrix} {p_1} \\ {p_2} \end{pmatrix} = \begin{pmatrix} {1} & {-1/K_1} \\ {1} & {1} \end{pmatrix} = \begin{pmatrix} {x_1} \\ {x_2} \end{pmatrix} \end{equation} \tag{6.12}$$

This matrix can be used to post-process the concentrations and the results can be graphed, Figure 6.3. The pool transformation leads to dynamic decoupling and is clearly illustrated in this figure. The disequilibrium pool relaxes very quickly, while the conservation pool moves slowly, Figure 6.3b. The phase portrait formed by the pools thus has an L shape (Figure 6.3a), illustrating the dynamic decoupling. Note that:

* There is first a vertical motion where $x_1 + x_2$ is essentially a constant, followed by a slow horizontal motion where the disequilibrium variable $x_1 - x_2 /K_1$ is a constant.

* These two separate motions correspond to forming the pathways found in the steady state.

* The disequilibrium aggregate is not zero, as it is forced away from equilibrium by the input and settling in a steady state.

```python
# Define pools
pools = ["x1 - x2 / Keq_v1", "x1 + x2"]
for i, equation_str in enumerate(pools):
    pool_id = "p" + str(i + 1)
    conc_sol.make_aggregate_solution(
        pool_id, equation=equation_str,
        parameters={v1.Keq_str: v1.kf/v1.kr}, update=True)

fig_6_3 = plt.figure(figsize=(11, 4))
gs = fig_6_3.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_3.add_subplot(gs[0, 0])
ax2 = fig_6_3.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x="p2", y="p1", ax=ax1,
    xlabel="p2 (inventory)", ylabel="p1 (distance from equilibrium)",
    xlim=(-.05, 2.1), ylim=(-.05, 1.05),
    title=("(a) Phase portrait of p2 vs. p1", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

ax1.annotate(
    'motion towards\n  a qe state',
    xy=(conc_sol["p2"](0.5), conc_sol["p1"](0.5)),
    xytext=(conc_sol["p2"](0.3), conc_sol["p1"](0.3)));

ax1.annotate(
    'motion of a\n  qe state',
    xy=(conc_sol["p2"](tf), conc_sol["p1"](tf)),
    xytext=(conc_sol["p2"](tf) + 0.05, conc_sol["p1"](tf) + 0.1));

plot_time_profile(
    conc_sol, observable=["p1", "p2"], ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Pool Profiles", {"size": "large"}));
fig_6_3.tight_layout()
```

**Figure 6.3:** The time profiles of the pools involved in reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ for the same conditions as in Figure 6.2. (a) The phase portrait of $p_2$ and $p_1$. (b) The pools as a function of time.

#### External disturbance:
The previous simulation represents a biologically unrealistic situation. An internal concentration cannot suddenly deviate from its value independent of what else happens in the system. A much more realistic situation is one where we start out at a steady state and an environmental change is observed. In our case, the only environmental parameter is $b_1$.

In Figure 6.4 we change the input flux, $b_1$, from 0.01 to 0.02 at time zero, when the system is initially in a steady state (the endpoint in Figure 6.4). We make three observations:

* We see that the fast motion is not activated.

* The 'inventory' or the pool of $x_1 + x_2$ moves from one steady state to another. It increases, as the forcing function was stepped up.

* The 'distance from equilibrium'

$$\begin{equation} x_{1, ss} - \frac{x_{2, ss}}{K_1} = x_{1, ss} - \frac{x_{2, ss}}{x_{2, eq}/x_{1, eq}} = x_{1, ss}(1 - \frac{\Gamma}{K_1}) \tag{6.13} \end{equation}$$

is close to zero in both steady states. The higher throughput, however, does push the system farther from equilibrium.

```python
# Ensure model starts simulation at a steady state.
sim.find_steady_state(model, strategy="simulate", update_values=True, verbose=True);

# Simulate model with disturbance from steady state
conc_sol, flux_sol = sim.simulate(
    model, time=(t0, tf), perturbations={"kf_b1": 0.02},
    interpolate=True, verbose=True)

# Determine pools
for i, equation_str in enumerate(pools):
    pool_id = "p" + str(i + 1)
    conc_sol.make_aggregate_solution(
        pool_id, equation=equation_str,
        parameters={v1.Keq_str: v1.kf/v1.kr}, update=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Setting output selections
    Getting time points
    Simulating 'Linear_Reversible_Open'
    Found steady state for 'Linear_Reversible_Open'.
    Updating 'Linear_Reversible_Open' values
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Getting time points
    Parsing perturbations
    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Simulating 'Linear_Reversible_Open'
    Simulation for 'Linear_Reversible_Open' successful
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_4 = plt.figure(figsize=(11, 4))
gs = fig_6_4.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_4.add_subplot(gs[0, 0])
ax2 = fig_6_4.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x="p2", y="p1", ax=ax1,
    xlabel="p2 (inventory)", ylabel="p1 (distance from equilibrium)",
    xlim=(0, 1), ylim=(0, 0.03),
    title=("(a) Phase portrait of p2 vs. p1", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

plot_time_profile(
    conc_sol,  observable=["p1", "p2"], ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Pool Profiles", {"size": "large"}));
fig_6_4.tight_layout()
```

**Figure 6.4:** The time profiles of the pools involved in reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ where the system starts out at equilibrium and the input rate, $b_1$, is changed from 0.01 to 0.02 at time zero. (a) The phase portrait of $p_2$ and $p_1$; (b) The pools as a function of time. Note the small numerical change in $p_1$ relative to $p_2$. Parameter values as in Figures 6.2 and Figure 6.3.

## Michaelis-Menten Kinetics in an Open Environment
We now consider the case when the Michaelis-Menten reaction mechanism operates in an open environment (Figure 6.5). The substrate enters the system and the product leaves. The enzyme stays internal to the system.

**Figure 6.5:** The Michaelis-Menten reaction mechanisms in an open setting. The substrate and the product enter and leave the system, while the enzyme stays inside.

### Dynamic description
The mass action kinetic model is

$$\begin{align} \frac{ds}{dt} &= b_1 - k_1es + k_{-1}x, \ &s(t=0) &= s_0 \\ \frac{dx}{dt} &= k_1es - (k_{-1} + k_2)x, \ &x(t=0) &= x_0 \\ \frac{de}{dt} &= -k_1es + (k_{-1} + k_2)x, \ &e(t=0) &= e_0 \\ \frac{dp}{dt} &= k_2x + k_3p, \ &p(t=0) &= p_0 \\ \end{align} \tag{6.14}$$

The initial conditions would normally be the steady state conditions. The stoichiometric matrix is

$$\begin{equation} \textbf{S} = \begin{pmatrix} {1} & {-1} & {1} & {0} & {0} \\ {0} & {-1} & {1} & {1} & {0} \\ {0} & {1} & {-1} & {-1} & {0} \\ {0} & {0} & {0} & {1} & {-1} \\ \end{pmatrix} \end{equation} \tag{6.15}$$

where $\textbf{x}=(s,\ e, \ x, \ p)$ and $\textbf{v} = (b_1, \ k_1es, \ k_{-1}x, \ k_2x, \ k_3p)$

### The steady state
As in the previous section, we can compute the steady state fluxes and concentrations. We can also compute how the parameters determine the distance from equilibrium, and here we also run into an additional issue: capacity constraints that result from a conservation quantity.

#### The steady state fluxes:
The rank of $\textbf{S}$ is 3, thus the dimension of the null space is 5-3=2. The null space of $\textbf{S}$ is spanned by two vectors, (1,1,0,1,1) and (0,1,1,0,0), that correspond to a pathway through the system and an internal reversible reaction. The steady state fluxes are given by

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ k_1e_{ss}s_{ss}, \ k_{-1}x_{ss}, \ k_2x_{ss}, \ k_3p_{ss}) \tag{6.16} \\ &= a(1, \ 1, \ 0, \ 1, \ 1) + b(0, 1, 1, 0, 0), \ a \geq 0, \ b \geq 0 \tag{6.17} \\ \end{align}$$

#### The steady state concentrations:
The dimension of the left null space is 4-3=1. The left null space has one conservation quantity ($e+x$), which can readily be seen from the fact that the second and third row of $\textbf{S}$ add up to zero. The steady state flux balances for this system are

$$\begin{equation} b_1 = v_1 - v_{-1} = v_2 = v_3 \tag{6.18} \end{equation}$$

Thus, the incoming flux and the kinetic parameters immediately set the concentrations for $X$, $P$, and $E$ as

$$\begin{equation} x_{ss} = b_1/k_2, \ p_{ss} = b_1/k_3, \ \text{and} \ e_{ss} = e_t - x_{ss} = e_t -  b_1/k_2 \tag{6.19} \end{equation}$$

and the steady state substrate concentration can be determined. The steady state concentration of the substrate is given by $k_1s_{ss}e_{ss} = b_1 + k_{-1}x_{ss}$ that can be solved to give

$$\begin{equation} s_{ss} = (\frac{k_2}{k_1})(\frac{k_{-1}/k_2 + 1}{e_tk_2/b_1 - 1}) \tag{6.20} \end{equation}$$

The steady state flux vector can now be computed

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ b_1(k_{-1}/k_2 + 1), \ b_1k_{-1}/k_2, \ b_1, \ b_1) \tag{6.21} \\ &= a(1, \ 1, \ 0, \ 1, \ 1) + b(0, 1, 1, 0, 0), \ a=b_1, \ b=b_1k_{-1}/k_2 \tag{6.22} \\ \end{align}$$

The distance from equilibrium can now be computed as in the previous section.

#### Internal capacity constraints:
Since $e_{ss} \geq 0$, the maximum input is

$$\begin{equation} b_1 \leq b_{1, max} = k_2e_t = v_m \tag{6.23} \end{equation}$$

which is the maximum reaction rate for the Michaelis-Menten mechanism. The total amount of the enzyme and the turnover rate set this flux constraint.

```python
model = MassModel('Michaelis_Menten_Open')
## Define metabolites
s = MassMetabolite("s")
e = MassMetabolite("e")
x = MassMetabolite("x")
p = MassMetabolite("p")

# Define reactions
b1 = MassReaction("b1", reversible=False)
v1 = MassReaction("v1")
v2 = MassReaction("v2", reversible=False)
v3 = MassReaction("v3", reversible=False)

b1.add_metabolites({s: 1})
v1.add_metabolites({s: -1, e: -1, x: 1})
v2.add_metabolites({x: -1, e: 1, p: 1})
v3.add_metabolites({p: -1})
model.add_reactions([b1, v1, v2, v3])

## Define parameters
b1.kf = 0.025
v1.kf, v1.kr = (1, 0.5)
v2.kf = 0.5
v3.kf = 1

# Set initial conditions for model
s.ic = 1
e.ic = 0.05
x.ic = 0.05
p.ic = 0

# Utilize type 2 rate law for kf and kr parameters defined
model.get_rate_expressions(rate_type=2, update_reactions=True)

# Set a custom rate for b1 to remove substrate concentration dependence
model.add_custom_rate(reaction=b1, custom_rate=b1.kf_str)
```

```python
t0 = 0
tf = 2e3

sim = Simulation(model, verbose=True)
# Simulate model with disturbance from steady state
conc_sol, flux_sol = sim.simulate(
    model, time=(t0, tf), perturbations={"kf_b1": 0.04},
    interpolate=True, verbose=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Michaelis_Menten_Open' into RoadRunner.
    Getting time points
    Parsing perturbations
    Setting output selections
    Setting simulation values for 'Michaelis_Menten_Open'
    Simulating 'Michaelis_Menten_Open'
    Simulation for 'Michaelis_Menten_Open' successful
    Adding 'Michaelis_Menten_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_6 = plt.figure(figsize=(8, 6))
gs = fig_6_6.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1.5],
                          height_ratios=[1, 1])

ax1 = fig_6_6.add_subplot(gs[0, 0])
ax2 = fig_6_6.add_subplot(gs[0, 1])
ax3 = fig_6_6.add_subplot(gs[1, 1])

plot_phase_portrait(
    conc_sol, x=s, y=x, ax=ax1,
    xlabel=s.id, ylabel=x.id,
    xlim=(-0.1, 6), ylim=(0, 0.105),
    title=("(a) Phase Portrait of s vs. x", {"size":"large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True)

plot_time_profile(
    conc_sol, observable=x, ax=ax2, legend="best",
    ylim=(0.03, 0.09), xlabel="Time", ylabel="Concentration",
    title=("(b) Time Response of x", {"size": "large"}));

plot_time_profile(
    conc_sol, observable=s, ax=ax3, legend="best",
    ylim=(0, 6), xlabel="Time", ylabel="Concentration",
    title=("(c) Time Response of s", {"size": "large"}));
fig_6_6.tight_layout()
```

**Figure 6.6:** The time profiles for the transient response of the Michaelis-Menten mechanisms for $b_1 = 0.025$ changed to 0.04 at time zero. The kinetic parameters are; $e_t = 0.1$,  $k_1 = 1$, $k_{-1} = 0.5$, $k_2 = 0.5$, $k_3 = 0.1$. (a) The phase portrait of $s$ vs. $x$. (b) The time response of $x$. (c) The time response of $s$.

### Dynamic states for Michaelis-Menten kinetics
The response of this system to a change in the input rate starting from a steady state is of greatest interest, see Figure 6.6. For the kinetic parameters given in the figure, the steady state concentrations of $s$ and $x$ are 1.0 and 0.5 respectively. The input rate is changed from 0.025 to 0.04 at time zero and the concentrations of $s$ and $x$ go to 4.0 and 0.8, respectively, as time goes to infinity. The maximum flux rate is 0.05.

The change in the input rate triggers an internal motion that basically follows the quasi-steady state line (Figure 6.6c). Figure 6.6b and Figure 6.6c shows that since the internal steps are rapid relative to the exchange rates, there are no rapid transients produced by a perturbation in the input rate.

If the input is increased towards $v_m$, the substrate concentration builds up to a very high value and most of the enzyme is found in the intermediate state. If the input rate exceeds $v_m$ there will be no steady state as the enzyme cannot convert the substrate to the product at the same rate as it is entering the system.

The dynamic properties of this open system can be analyzed using pool formation.

## Summary
* Open systems eventually reach a steady state, which is different from the equilibrium state of a closed system. Such steady states can be thought of as homeostatic, living states.

* Living cells are open systems that continually exchange mass and energy with their environment. The continual net throughput of mass and concomitant energy dissipation is what allows steady states to form and differentiates them from equilibrium states.

* The relative rates of the internal network to that of the exchanges across the system boundary are important. Time scale separation between internal and exchange processes can form. Rapid internal transients lead to pool formation.

* Open systems are most naturally in a steady state and respond to external stimuli. It is normally not possible to suddenly change the internal state of the system since it is in balance with the environment.

* If the internal dynamics are fast, they do not excite when external stimuli are experienced, and thus accurate information about the fast kinetics may not be needed. It may be enough to know that they are "fast."

* Non-exchanged moieties form dynamic invariants. They can set internal capacity constraints.

$\tiny{\text{© B. Ø. Palsson 2011;}\ \text{This publication is in copyright.}\\ \text{Subject to statutory exception and to the provisions of relevant collective licensing agreements,}\\ \text{no reproduction of any part may take place without the written permission of Cambridge University Press.}}$


## Enable rendering OpenAI Gym environments from CoLab

In this assignemnt, We will use [OpenAI Gym](https://gym.openai.com/) for rendering game envionment for our agent to play and learn. It is possible and important to visualize the game your agent is playing, even on Colab. This section imports the necessary package and functions needed to generate a video in Colab. The video processing steps credit to [here](https://colab.research.google.com/drive/1flu31ulJlgiRL1dnN2ir8wGh9p7Zij2t).

```
# You will need to run this block twice to make it effective
!apt-get update > /dev/null 2>&1
!apt-get install cmake > /dev/null 2>&1
!pip install --upgrade setuptools 2>&1
!pip install ez_setup > /dev/null 2>&1
!pip install gym[atari] > /dev/null 2>&1
!pip install box2d-py > /dev/null 2>&1
!pip install gym[Box_2D] > /dev/null 2>&1
```

    Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (60.10.0)

```
!pip install gym pyvirtualdisplay > /dev/null 2>&1
!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1
```

Import openAI gym and define the functions used to show the video.

```
import gym
from gym.wrappers import Monitor
import glob
import io
import base64
from IPython.display import HTML
from pyvirtualdisplay import Display
from IPython import display as ipythondisplay

display = Display(visible=0, size=(1400, 900))
display.start()

"""
Utility functions to enable video recording of gym environment
and displaying it.
To enable video, just do "env = wrap_env(env)""
"""

def show_video():
  mp4list = glob.glob('video/*.mp4')
  if len(mp4list) > 0:
    mp4 = mp4list[0]
    video = io.open(mp4, 'r+b').read()
    encoded = base64.b64encode(video)
    ipythondisplay.display(HTML(data=''''''.format(encoded.decode('ascii'))))
  else:
    print("Could not find video")

def wrap_env(env):
  env = Monitor(env, './video', force=True)
  return env
```

Import other packages:

We will use Pytorch for building and learning our DQN network.

```
import torch
from torch import nn
import copy
from collections import deque
import random
from tqdm import tqdm
import matplotlib.pyplot as plt

random.seed(42)
```

## Run the game with random agent.

```
from torch import randint
from time import sleep

env = wrap_env(gym.make('CartPole-v1'))
reward_arr = []
episode_count = 20
for i in tqdm(range(episode_count)):
    obs, done, rew = env.reset(), False, 0
    env.render()
    while not done:
        A = randint(0, env.action_space.n, (1,))
        obs, reward, done, info = env.step(A.item())
        rew += reward
        sleep(0.01)
    reward_arr.append(rew)
print("average reward per episode :", sum(reward_arr) / len(reward_arr))
env.close()
show_video()
```

The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center. The video is short (< 1s) because the pole loses balance immediately.

You can see that a random agent is having trouble balancing the CartPole, just like you. However, a difficult game for human may be very simple to a computer. Let's see how we can use DQN to train a agent.

## Experience Replay

The technique of experience replay was first proposed in to resolve temporal correlation in the input data by mixing recent experiences as well past experiences, essentially forcing the input to become independent and identically distributed (i.i.d.). It has been shown that this greatly stabilizes
and improves the DQN training procedure.

```
class ExperienceReplay(object):
      def __init__(self, length):
        self.experience_replay = deque(maxlen=length)

      def collect(self, experience):
        self.experience_replay.append(experience)
        return

      def sample_from_experience(self, sample_size):
        if len(self.experience_replay) < sample_size:
            sample_size = len(self.experience_replay)
        sample = random.sample(self.experience_replay, sample_size)
        state = torch.tensor([exp[0] for exp in sample]).float()
        action = torch.tensor([exp[1] for exp in sample]).float()
        reward = torch.tensor([exp[2] for exp in sample]).float()
        next_state = torch.tensor([exp[3] for exp in sample]).float()
        return state, action, reward, next_state

```

## Build our DQN Network

We will use a simple multi-layer neural network to learn the optimal actions. We will use Adam Optimizor and MSE loss for training. **Notice that the loss function and gamma is given to you in the class attribute.**

```
class DQN_Network:

    def __init__(self, layer_size_list, lr, seed=1423):
        torch.manual_seed(seed)
        self.policy_net = self.create_network(layer_size_list)
        self.target_net = copy.deepcopy(self.policy_net)

        self.loss_fn = torch.nn.MSELoss() # the loss function
        self.optimizer = torch.optim.Adam(self.policy_net.parameters(), lr=lr)

        self.step = 0
        self.gamma = torch.tensor(0.95).float()
        return

    def create_network(self, layer_size_list):
        assert len(layer_size_list) > 1

        layers = []
        for i in range(len(layer_size_list) - 1):
            linear = nn.Linear(layer_size_list[i], layer_size_list[i + 1])

            if i < len(layer_size_list) - 2:
              activation = nn.Tanh()
            else:
              activation = nn.Identity()

            layers += (linear, activation)
        return nn.Sequential(*layers)

    def load_pretrained_model(self, model_path):
        self.policy_net.load_state_dict(torch.load(model_path))

    def save_trained_model(self, model_path="cartpole-dqn.pth"):
        torch.save(self.policy_net.state_dict(), model_path)

```

## **[Your task]**: complete the function that chooses the next action

Choose next action based on **$\epsilon$-greedy**:

\begin{align}\text{where} \quad \mathcal{a_{t+1}} = \begin{cases}
     \text{argmax}_{a}Q(a, s)  & \text{with probability }: 1 - \epsilon, \text{exploitation}\\
     \text{Uniform}\{a_{1},...,a_{n}\} & \text{with probability}:   \epsilon, \text{exploration} \\
   \end{cases}\end{align}

```
def get_action(model, state, action_space_len, epsilon):
    # We do not require gradient at this point, because this function will be used either
    # during experience collection or during inference

    with torch.no_grad():
        Qp = model.policy_net(torch.from_numpy(state).float())

    ## TODO: select and return action based on epsilon-greedy

    pass
```

### **[Your task]**: complete the function that train the network for one step

Here, you can find an ``train`` function that performs a
single step of the optimization.

For our training update rule, the loss you are trying to minimize is:

\begin{align}\text{loss} = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

```
def train(model, batch_size):
    state, action, reward, next_state = memory.sample_from_experience(sample_size=batch_size)

    # TODO: predict expected return of current state using main network

    # TODO: get target return using target network

    # TODO: compute the loss
    loss =
    model.optimizer.zero_grad()
    loss.backward(retain_graph=True)
    model.optimizer.step()

    model.step += 1
    if model.step % 5 == 0:
        model.target_net.load_state_dict(model.policy_net.state_dict())

    return loss.item()
```

### **[Your task]**: Finish the training loop

In this part, you can play around with ```exp_replay_size```, ```episode```, ```epsilon``` and the "episodo decay" logic to train your model. **If you have done correctly, you will observe that the training time for the latter episodes is longer than the early episodes. This is because your agent is getting better and better at playing the game and thus each episode takes longer**

```
# Create the model
env = gym.make('CartPole-v0')
input_dim = env.observation_space.shape[0]
output_dim = env.action_space.n
agent = DQN_Network(layer_size_list=[input_dim, 64, output_dim], lr=1e-3)

# Main training loop
losses_list, reward_list, episode_len_list, epsilon_list = [], [], [], []

# TODO: try different values, it normally takes more than 6k episodes to train
exp_replay_size =
memory = ExperienceReplay(exp_replay_size)
episodes =
epsilon = 1 # episilon start from 1 and decay gradually.

# initiliaze experiance replay
index = 0
for i in range(exp_replay_size):
    obs = env.reset()
    done = False
    while not done:
        A = get_action(agent, obs, env.action_space.n, epsilon=1)
        obs_next, reward, done, _ = env.step(A.item())
        memory.collect([obs, A.item(), reward, obs_next])
        obs = obs_next
        index += 1
        if index > exp_replay_size:
            break

index = 128
for i in tqdm(range(episodes)):
    obs, done, losses, ep_len, rew = env.reset(), False, 0, 0, 0
    while not done:
        ep_len += 1
        A = get_action(agent, obs, env.action_space.n, epsilon)
        obs_next, reward, done, _ = env.step(A.item())
        memory.collect([obs, A.item(), reward, obs_next])

        obs = obs_next
        rew += reward
        index += 1

        if index > 128:
            index = 0
            for j in range(4):
                loss = train(agent, batch_size=16)
                losses += loss

    # TODO: add epsilon decay rule here!

    losses_list.append(losses / ep_len), reward_list.append(rew)
    episode_len_list.append(ep_len), epsilon_list.append(epsilon)

print("Saving trained model")
agent.save_trained_model("cartpole-dqn.pth")
```

## Last Step: evaluate your trained model! **Make sure to include your visualizations (plot+video) in the notebook for your submission!**

First we can plot the reward vs. episode. **If you have done correctly, you should see the reward can stabilize at 200 in later episodes**\

```
def plot_reward(r):
    plt.figure(2)
    plt.clf()
    plt.title('Result')
    plt.xlabel('Episode')
    plt.ylabel('Reward')
    plt.plot(r)

plot_reward(reward_list)
```

Next let check out how well your agent plays the game. **If you have done correctly, you should see a relatively longer video (> 3~4s) with a self-balancing pole.**

```
env = wrap_env(gym.make('CartPole-v1'))

input_dim = env.observation_space.shape[0]
output_dim = env.action_space.n
model_validate = DQN_Network(layer_size_list=[input_dim, 64, output_dim], lr=1e-3)
model_validate.load_pretrained_model("cartpole-dqn.pth")

reward_arr = []
for i in tqdm(range(200)):
    obs, done, rew = env.reset(), False, 0
    env.render()
    while not done:
        A = get_action(model_validate, obs, env.action_space.n, epsilon=0)
        obs, reward, done, info = env.step(A.item())
        rew += reward
        # sleep(0.01)

    reward_arr.append(rew)
print("average reward per episode :", sum(reward_arr) / len(reward_arr))
env.close()
show_video()
```

```

```


```python
#remove cell visibility
from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)
```

Toggle cell visibility here.

```python
%matplotlib inline
import control
import numpy
import sympy as sym
from IPython.display import display, Markdown
import ipywidgets as widgets
import matplotlib.pyplot as plt

#print a matrix latex-like
def bmatrix(a):
     """Returns a LaTeX bmatrix - by Damir Arbula (ICCT project)

     :a: numpy array
     :returns: LaTeX bmatrix as a string
     """
     if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
     lines = str(a).replace('[', '').replace(']', '').splitlines()
     rv = [r'\begin{bmatrix}']
     rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
     rv +=  [r'\end{bmatrix}']
     return '\n'.join(rv)

# Display formatted matrix:
def vmatrix(a):
    if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{vmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{vmatrix}']
    return '\n'.join(rv)

#matrixWidget is a matrix looking widget built with a VBox of HBox(es) that returns a numPy array as value !
class matrixWidget(widgets.VBox):
    def updateM(self,change):
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.M_[irow,icol] = self.children[irow].children[icol].value
                #print(self.M_[irow,icol])
        self.value = self.M_

    def dummychangecallback(self,change):
        pass

    def __init__(self,n,m):
        self.n = n
        self.m = m
        self.M_ = numpy.matrix(numpy.zeros((self.n,self.m)))
        self.value = self.M_
        widgets.VBox.__init__(self,
                             children = [
                                 widgets.HBox(children =
                                              [widgets.FloatText(value=0.0, layout=widgets.Layout(width='90px')) for i in range(m)]
                                             )
                                 for j in range(n)
                             ])

        #fill in widgets and tell interact to call updateM each time a children changes value
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
                self.children[irow].children[icol].observe(self.updateM, names='value')
        #value = Unicode('example@example.com', help="The email value.").tag(sync=True)
        self.observe(self.updateM, names='value', type= 'All')

    def setM(self, newM):
        #disable callbacks, change values, and reenable
        self.unobserve(self.updateM, names='value', type= 'All')
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].unobserve(self.updateM, names='value')
        self.M_ = newM
        self.value = self.M_
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].observe(self.updateM, names='value')
        self.observe(self.updateM, names='value', type= 'All')

                #self.children[irow].children[icol].observe(self.updateM, names='value')

#overlaod class for state space systems that DO NOT remove "useless" states (what "professor" of automatic control would do this?)
class sss(control.StateSpace):
    def __init__(self,*args):
        #call base class init constructor
        control.StateSpace.__init__(self,*args)
    #disable function below in base class
    def _remove_useless_states(self):
        pass
```

## Aircraft taxi trajectory control

The rotational dynamics of an aircraft moving on the ground can be represented as:

$$
J_z\ddot{\psi} = bF_1\delta + F_2\dot{\psi} \, ,
$$

where $J_z = 11067000$ kg$\text{m}^2$, $b = 15$ , $F_1 = 35000000$ Nm, $F_2 = 500000$ kg$\text{m}^2$/$\text{s}$, $\psi$ is the rotational angle (in rad), or yaw angle, with respect to the vertical axis, and $\delta$ is the front wheel steering angle (in rad). When the aircraft is following a straight line with longitudinal linear velocity $V$ (in m/s), the aircraft lateral velocity $V_y$ (in m/s) is approximately linearly proportional to the yaw angle: $V_y = \dot{p_y} = V\psi$.

The goal is to design a regulator for the aircraft lateral position $p_y$, with longitudinal velocity $V$ set to 35 km/h, using the front wheel steering angle $\delta$ as plant input, according to the following specifications:
- settling time for 5% tolerance band of less than 4 seconds;
- zero steady-state error in response to a desired lateral position request;
- no or minimal overshoot;
- steering angle does not exceed $\pm8$ degrees when following a lateral position step of 5 meters.

The dynamic equations in state space form are:

\begin{cases}
    \dot{x} = \begin{bmatrix} \frac{F_2}{J_z} & 0 & 0 \\ 1 & 0 & 0 \\ 0 & V & 0 \end{bmatrix}x + \begin{bmatrix} \frac{bF_1}{J_z} \\ 0 \\ 0 \end{bmatrix}u \\
    y = \begin{bmatrix} 0 & 0 & 1 \end{bmatrix}x \, ,
\end{cases}

where $x=\begin{bmatrix} x_1 & x_2 & x_3 \end{bmatrix}^T = \begin{bmatrix} \dot{\psi} & \psi & p_y \end{bmatrix}^T$ and $u=\delta$.

The poles of the system are $0$, $0$ and $\frac{F_2}{J_z} \simeq 0.045$, therefore the system is unstable.

### Regulator design
#### Controller design

To meet the requirements of zero steady-state error we add a new state:
$$
\dot{x_4} = p_y-y_d = x_3 - y_d
$$
The resulting augmented system is thus:

\begin{cases}
    \dot{x_a} = \begin{bmatrix} \frac{F_2}{J_z} & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 \\ 0 & V & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}x_a + \begin{bmatrix} \frac{bF_1}{J_z} & 0 \\ 0 & 0 \\ 0 & 0 \\ 0 & -1 \end{bmatrix}\begin{bmatrix} u \\ y_d \end{bmatrix} \\
    y_a = \begin{bmatrix} 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}x_a,
\end{cases}

where $x_a = \begin{bmatrix} x_1 & x_2 & x_3 & x_4 \end{bmatrix}^T$ and the second output is added in order to maintain the observability. The system remains controllable with the input $u$ and thus we can design the state feedback with this input. A possible solution is to place all the poles in $-2$.

#### Observer design

Even if states $x_3$ and $x_4$ can be obtained from measurements, and we need to estimate $x_2$ and $x_3$ only, it is convenient to work with the full 4x4 system and to design a state observer of order 4 with all poles in $-10$.

### How to use this notebook?
- Verify, if the requirements are met in presence of errors in the initial state estimate.

```python
# Preparatory cell

X0 = numpy.matrix('0.0; 0.0; 0.0; 0.0')
K = numpy.matrix([0,0,0,0])
L = numpy.matrix([[0,0],[0,0],[0,0],[0,0]])

X0w = matrixWidget(4,1)
X0w.setM(X0)
Kw = matrixWidget(1,4)
Kw.setM(K)
Lw = matrixWidget(4,2)
Lw.setM(L)

eig1c = matrixWidget(1,1)
eig2c = matrixWidget(2,1)
eig3c = matrixWidget(1,1)
eig4c = matrixWidget(2,1)
eig1c.setM(numpy.matrix([-2.]))
eig2c.setM(numpy.matrix([[-2.],[-0.]]))
eig3c.setM(numpy.matrix([-2.]))
eig4c.setM(numpy.matrix([[-2.],[-0.]]))

eig1o = matrixWidget(1,1)
eig2o = matrixWidget(2,1)
eig3o = matrixWidget(1,1)
eig4o = matrixWidget(2,1)
eig1o.setM(numpy.matrix([-10.]))
eig2o.setM(numpy.matrix([[-10.],[0.]]))
eig3o.setM(numpy.matrix([-10.]))
eig4o.setM(numpy.matrix([[-10.],[0.]]))
```

```python
# Misc

#create dummy widget
DW = widgets.FloatText(layout=widgets.Layout(width='0px', height='0px'))

#create button widget
START = widgets.Button(
    description='Test',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Test',
    icon='check'
)

def on_start_button_clicked(b):
    #This is a workaround to have intreactive_output call the callback:
    #   force the value of the dummy widget to change
    if DW.value> 0 :
        DW.value = -1
    else:
        DW.value = 1
    pass
START.on_click(on_start_button_clicked)

# Define type of method
selm = widgets.Dropdown(
    options= ['Set K and L', 'Set the eigenvalues'],
    value= 'Set the eigenvalues',
    description='',
    disabled=False
)

# Define the number of complex eigenvalues
selec = widgets.Dropdown(
    options= ['0 complex eigenvalues', '2 complex eigenvalues', '4 complex eigenvalues'],
    value= '0 complex eigenvalues',
    description='Eig controller:',
    disabled=False
)
seleo = widgets.Dropdown(
    options= ['0 complex eigenvalues', '2 complex eigenvalues', '4 complex eigenvalues'],
    value= '0 complex eigenvalues',
    description='Eig observer:',
    disabled=False
)

#define type of ipout
selu = widgets.Dropdown(
    options=['impulse', 'step', 'sinusoid', 'square wave'],
    value='step',
    description='Type of reference:',
    style = {'description_width': 'initial'},
    disabled=False
)
# Define the values of the input
u = widgets.FloatSlider(
    value=5,
    min=0,
    max=10,
    step=0.1,
    description='Reference [m]:',
    style = {'description_width': 'initial'},
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.1f',
)
v = widgets.FloatSlider(
    value=9.72,
    min=1,
    max=20,
    step=0.1,
    description=r'$V$ [m/s]:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.2f',
)
period = widgets.FloatSlider(
    value=0.5,
    min=0.001,
    max=10,
    step=0.001,
    description='Period: ',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.3f',
)

simTime = widgets.FloatText(
    value=5,
    description='',
    disabled=False
)
```

```python
# Support functions

def eigen_choice(selec,seleo):
    if selec == '0 complex eigenvalues':
        eig1c.children[0].children[0].disabled = False
        eig2c.children[1].children[0].disabled = True
        eig3c.children[0].children[0].disabled = False
        eig4c.children[0].children[0].disabled = False
        eig4c.children[1].children[0].disabled = True
        eigc = 0
    if seleo == '0 complex eigenvalues':
        eig1o.children[0].children[0].disabled = False
        eig2o.children[1].children[0].disabled = True
        eig3o.children[0].children[0].disabled = False
        eig4o.children[0].children[0].disabled = False
        eig4o.children[1].children[0].disabled = True
        eigo = 0
    if selec == '2 complex eigenvalues':
        eig1c.children[0].children[0].disabled = False
        eig2c.children[1].children[0].disabled = False
        eig3c.children[0].children[0].disabled = False
        eig4c.children[0].children[0].disabled = True
        eig4c.children[1].children[0].disabled = True
        eigc = 2
    if seleo == '2 complex eigenvalues':
        eig1o.children[0].children[0].disabled = False
        eig2o.children[1].children[0].disabled = False
        eig3o.children[0].children[0].disabled = False
        eig4o.children[0].children[0].disabled = True
        eig4o.children[1].children[0].disabled = True
        eigo = 2
    if selec == '4 complex eigenvalues':
        eig1c.children[0].children[0].disabled = True
        eig2c.children[1].children[0].disabled = False
        eig3c.children[0].children[0].disabled = True
        eig4c.children[0].children[0].disabled = False
        eig4c.children[1].children[0].disabled = False
        eigc = 4
    if seleo == '4 complex eigenvalues':
        eig1o.children[0].children[0].disabled = True
        eig2o.children[1].children[0].disabled = False
        eig3o.children[0].children[0].disabled = True
        eig4o.children[0].children[0].disabled = False
        eig4o.children[1].children[0].disabled = False
        eigo = 4
    return eigc, eigo

def method_choice(selm):
    if selm == 'Set K and L':
        method = 1
        selec.disabled = True
        seleo.disabled = True
    if selm == 'Set the eigenvalues':
        method = 2
        selec.disabled = False
        seleo.disabled = False
    return method
```

```python
F1 = 35000000
F2 = 500000
b = 15
V = 35/3.6
Jz = 11067000

A = numpy.matrix([[F2/Jz, 0, 0, 0],
                  [1, 0, 0, 0],
                  [0, V, 0, 0],
                  [0, 0, 1, 0]])
Bu = numpy.matrix([[b*F1/Jz],[0],[0],[0]])
Bref = numpy.matrix([[0],[0],[0],[-1]])
C = numpy.matrix([[0,0,1,0],[0,0,0,1]])

def main_callback2(v, X0w, K, L, eig1c, eig2c, eig3c, eig4c, eig1o, eig2o, eig3o, eig4o, u, period, selm, selec, seleo, selu, simTime, DW):
    eigc, eigo = eigen_choice(selec,seleo)
    method = method_choice(selm)

    A = numpy.matrix([[F2/Jz, 0, 0, 0],
                  [1, 0, 0, 0],
                  [0, v, 0, 0],
                  [0, 0, 1, 0]])

    if method == 1:
        solc = numpy.linalg.eig(A-Bu*K)
        solo = numpy.linalg.eig(A-L*C)
    if method == 2:
        #for better numerical stability of place
        if eig1c[0,0]==eig2c[0,0] or eig1c[0,0]==eig3c[0,0] or eig1c[0,0]==eig4c[0,0]:
            eig1c[0,0] *= 1.01
        if eig2c[0,0]==eig3c[0,0] or eig2c[0,0]==eig4c[0,0]:
            eig3c[0,0] *= 1.015
        if eig1o[0,0]==eig2o[0,0] or eig1o[0,0]==eig3o[0,0] or eig1o[0,0]==eig4o[0,0]:
            eig1o[0,0] *= 1.01
        if eig2o[0,0]==eig3o[0,0] or eig2o[0,0]==eig4o[0,0]:
            eig3o[0,0] *= 1.015

        if eigc == 0:
            K = control.acker(A, Bu, [eig1c[0,0], eig2c[0,0], eig3c[0,0], eig4c[0,0]])
            Kw.setM(K)
        if eigc == 2:
            K = control.acker(A, Bu, [eig3c[0,0],
                                      eig1c[0,0],
                                      numpy.complex(eig2c[0,0], eig2c[1,0]),
                                      numpy.complex(eig2c[0,0],-eig2c[1,0])])
            Kw.setM(K)
        if eigc == 4:
            K = control.acker(A, Bu, [numpy.complex(eig4c[0,0], eig4c[1,0]),
                                      numpy.complex(eig4c[0,0],-eig4c[1,0]),
                                      numpy.complex(eig2c[0,0], eig2c[1,0]),
                                      numpy.complex(eig2c[0,0],-eig2c[1,0])])
            Kw.setM(K)
        if eigo == 0:
            L = control.place(A.T, C.T, [eig1o[0,0], eig2o[0,0], eig3o[0,0], eig4o[0,0]]).T
            Lw.setM(L)
        if eigo == 2:
            L = control.place(A.T, C.T, [eig3o[0,0],
                                         eig1o[0,0],
                                         numpy.complex(eig2o[0,0], eig2o[1,0]),
                                         numpy.complex(eig2o[0,0],-eig2o[1,0])]).T
            Lw.setM(L)
        if eigo == 4:
            L = control.place(A.T, C.T, [numpy.complex(eig4o[0,0], eig4o[1,0]),
                                      numpy.complex(eig4o[0,0],-eig4o[1,0]),
                                      numpy.complex(eig2o[0,0], eig2o[1,0]),
                                      numpy.complex(eig2o[0,0],-eig2o[1,0])]).T
            Lw.setM(L)

    sys = sss(A,numpy.hstack((Bu,Bref)),[[0,0,1,0],[0,0,0,1],[0,0,0,0]],[[0,0],[0,0],[0,1]])
    syse = sss(A-L*C,numpy.hstack((Bu,Bref,L)),numpy.eye(4),numpy.zeros((4,4)))
    sysc = sss(0,[0,0,0,0],0,-K)
    sys_append = control.append(sys,syse,sysc)
    try:
        sys_CL = control.connect(sys_append,
                                 [[1,8],[3,8],[5,1],[6,2],[7,4],[8,5],[9,6],[10,7],[4,3]],
                                 [2],
                                 [1,8])
    except:
        sys_CL = control.connect(sys_append,
                                 [[1,8],[3,8],[5,1],[6,2],[7,4],[8,5],[9,6],[10,7],[4,3]],
                                 [2],
                                 [1,8])

    X0w1 = numpy.zeros((8,1))
    X0w1[4,0] = X0w[0,0]
    X0w1[5,0] = X0w[1,0]
    X0w1[6,0] = X0w[2,0]
    X0w1[7,0] = X0w[3,0]
    if simTime != 0:
        T = numpy.linspace(0, simTime, 10000)
    else:
        T = numpy.linspace(0, 1, 10000)

    if selu == 'impulse': #selu
        U = [0 for t in range(0,len(T))]
        U[0] = u
        T, yout, xout = control.forced_response(sys_CL,T,U,X0w1)
    if selu == 'step':
        U = [u for t in range(0,len(T))]
        T, yout, xout = control.forced_response(sys_CL,T,U,X0w1)
    if selu == 'sinusoid':
        U = u*numpy.sin(2*numpy.pi/period*T)
        T, yout, xout = control.forced_response(sys_CL,T,U,X0w1)
    if selu == 'square wave':
        U = u*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        T, yout, xout = control.forced_response(sys_CL,T,U,X0w1)

    try:
        step_info_dict = control.step_info(sys_CL[0,0],SettlingTimeThreshold=0.05,T=T)
        print('Step info: \n\tRise time =',step_info_dict['RiseTime'],'\n\tSettling time (5%) =',step_info_dict['SettlingTime'],'\n\tOvershoot (%)=',step_info_dict['Overshoot'])
        print('Max u value (% of 8deg)=', max(abs(yout[1]))/(8*numpy.pi/180)*100)
    except:
        print("Error in the calculation of step info.")

    fig = plt.figure(num='Simulation1', figsize=(14,12))

    fig.add_subplot(221)
    plt.title('Output response')
    plt.ylabel('Output')
    plt.plot(T,yout[0],T,U,'r--')
    plt.xlabel('$t$ [s]')
    plt.legend(['$y$','Reference'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(222)
    plt.title('Input')
    plt.ylabel('$u$ [deg]')
    plt.plot(T,yout[1]*180/numpy.pi)
    plt.plot(T,[8 for i in range(len(T))],'r--')
    plt.plot(T,[-8 for i in range(len(T))],'r--')
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(223)
    plt.title('States response')
    plt.ylabel('States')
    plt.plot(T,xout[0],
             T,xout[1],
             T,xout[2],
             T,xout[3])
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.legend(['$x_{1}$','$x_{2}$','$x_{3}$','$x_{4}$'])
    plt.grid()

    fig.add_subplot(224)
    plt.title('Estimation errors')
    plt.ylabel('Errors')
    plt.plot(T,xout[4]-xout[0])
    plt.plot(T,xout[5]-xout[1])
    plt.plot(T,xout[6]-xout[2])
    plt.plot(T,xout[7]-xout[3])
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.legend(['$e_{1}$','$e_{2}$','$e_{3}$','$e_{4}$'])
    plt.grid()
    #plt.tight_layout()

alltogether2 = widgets.VBox([widgets.HBox([selm,
                                          selec,
                                          seleo,
                                          selu]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.HBox([widgets.Label('K:',border=3), Kw,
                                                        widgets.Label('Eigenvalues:',border=3),
                                                        widgets.HBox([eig1c,
                                                                      eig2c,
                                                                      eig3c,
                                                                      eig4c])])]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.VBox([widgets.HBox([widgets.Label('L:',border=3), Lw, widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Eigenvalues:',border=3),
                                          eig1o,
                                          eig2o,
                                          eig3o,
                                          eig4o,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('X0 est.:',border=3), X0w]),
                                          widgets.Label(' ',border=3),
                                                       widgets.HBox([
                                          widgets.VBox([widgets.Label('Simulation time [s]:',border=3)]),
                                          widgets.VBox([simTime])])]),
                                          widgets.Label(' ',border=3)]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([u,
                                          v,
                                          period,
                                          START])])
out2 = widgets.interactive_output(main_callback2, {'v':v, 'X0w':X0w, 'K':Kw, 'L':Lw,
                                                   'eig1c':eig1c, 'eig2c':eig2c, 'eig3c':eig3c, 'eig4c':eig4c,
                                                   'eig1o':eig1o, 'eig2o':eig2o, 'eig3o':eig3o, 'eig4o':eig4o,
                                                   'u':u, 'period':period, 'selm':selm, 'selec':selec, 'seleo':seleo, 'selu':selu, 'simTime':simTime, 'DW':DW})
out2.layout.height = '860px'
display(out2, alltogether2)
```

    Output(layout=Layout(height='860px'))

    VBox(children=(HBox(children=(Dropdown(index=1, options=('Set K and L', 'Set the eigenvalues'), value='Set the…

```python

```