The associated codes to run this notebook can be found [here](https://github.com/prodipta/tictactoe). Clone the repo, and open this notebook (notebooks/tic_tac_toe.ipynb).

## Reinforcement Learning: An Introduction

Reinforcement learning is a machine learning algorithm for general purpose decision making. But before we understand and appreciate this, let's review our decision making process

### Exhibit 1: Evaluating decisions based on outcome

`World T20 Final 2007`: Ace spinner **Harbhajan Singh** had an over left but Dhoni chose medium-pacer **Joginder Sharma** to bowl the last over. **Misbah-ul-Haq** was strong at crease.

`World Cup 2011 India vs South Africa`: Ace spinner **Harbhajan Singh** had an over left but Dhoni chose medium-pacer **Ashish Nehra** to bowl the last over. **Robin Peterson** was strong at crease.

Sports critics and media absolutely loved the first decision, and similarly totally rejected the second one.

Too often we evaluate our past decisions based on the outcomes. We completely forget the situations we were in when the decisions were taken, and the information we had till at the point. The above two decisions are, arguably, very similar. They had very different outcome. And were judged very differently by the public.

### Exhibit 2: Short Term or Long term Rewards?

`The Standford Marshmallow Experiment`: The [experiment](https://en.wikipedia.org/wiki/Stanford_marshmallow_experiment#Original_Stanford_experiment) was first carried out in 1972 in Standford University. 32 children, under isolation with little distractions, were offered a treat (cookies or five pretzel sticks). The researchers explained to the children that they could eat the treat now and leave the room. But if they waited 15 minutes without eating the immedeate treat, they would get another.

Often the rewards are realized in the future, but we need to act now. This is known as `delayed gratification` in economics and behavioural science. The above is a simple two period case. In real life, everyday we face a continuous choice of multi-period version of this game

- Stock has rallied (sold-off) significantly, shall I sell now at a small profit (loss) now, or wait for larger profit in future?
- Shall I drink 10 rupess tea now, or let go for a month and pay off my loan shark?

Since rewards can realize over different time horizon, we need a way to compare them consistently **now**. A standard way of doing it is through discounting (usually time-consistent discounting i.e. `exponential` discounting, but in real life we also see `hyperbolic` discounting). This is also known as time-preference or temporal preference in economics or behavioural science. This time discouting is the bedrock of all our financial system - NPV

- bonus: what is the fundamental underlying assumption for discounting in financial markets (e.g. asset pricing, discount cashflow valuation, NPV project valuation etc.)?

### Exhibit 3: Risk vs Reward

You are managing a risky project<sup>[1]</sup> at work. If it works out well, you can get a good promotion. The chances are, if it gets derailed, you may get sidelined. How you make your decisions?

- In a knock-out round of a poker tournament, do you jam your small pocket pairs against a button raiser? What if you go bust?
- how to balance risk vs. reward

The balancing of risk and rewards under uncertainty is another major criteria for a good decision making process

### Exhibit 4: Multi-armed bandits

A gambler is deciding to play a row of slot machines - each with unknown, but different characteristics (win probability, payout etc.). The problem is how to distribute time and capital among different machines to try and find the best. This is a classifc decision making problem - exploration vs. exploitation

- To settle down with the current person you are dating or look for another?
- To select the current candidate who is good enough, or interview more for a better fit

### How to make a good decisions

- Evaluate (both prospectively and retrospectively) your decisions based on `information you have at the time of decision`
- For multi-period delayed rewards, define a `discouting function` to be able to compare and chose temoporally distributed rewards
- For uncertain environments, define a `reward function` to balance risk and reward.
- For uncertain environments, device a strategy to balance `exploration` against `exploitation`.

## General Problem Statement

You are interacting with a `system`. At each `state` of the system, you must choose an allowed `action`. After each such `action`, you may or may not receive an immediate `reward` (or penalty, a negative reward). After an uncertain n `steps` (or `state` changes), the `system` reaches its final `state` and you realize the final `reward` (or penalty).

- In financial investing, the `system` is the market itself. Your `action` is either buy, sell or hold of the stocks in the universe, and rewards are profit and loss.

- In a game of poker, the `system` is your opponents collectively, (only one in heads-up poker, a mathematically easier version). Your `actions` are fold, bet, call, raise or re-raise. Your rewards are realized only after each hands (or at the end of the whole game if it is a tournament)

- In a game of chess, the `system` is the board and the opponent. The `state` is the current configuration of the board. Your `actions` are defined by the chess rules and your remaining chess pieces. The `reward` is received only when the game is over.

All these decision making problems are similar in nature, although they vary greatly with each parameters - `state`, `actions` and `rewards`.

For example, in tic-tac-toe, the `state` are finite and relatively small. In chess, `states` are much more numerous, but they are still somewhat manageable. In poker, `states` are never known fully and can only be guessed - you do not get to see your opponents hand. In markets `state` is infinite - an endless combinations of varius market factors, prices and sentiments etc.

We aim to find a generalized approach to solve such problems. We try to crack the tic-tac-toe game.

## Tic-Tac-Toe: A demo for reinforcement learning

from [wikipedia](https://en.wikipedia.org/wiki/Tic-tac-toe): Tic-tac-toe (American English), noughts and crosses (Commonwealth English), or Xs and Os is a paper-and-pencil game for two players, X and O, who take turns marking the spaces in a 3×3 grid. The player who succeeds in placing three of their marks in a horizontal, vertical, or diagonal row is the winner.

### The decision making problem

Suppose we start with the 'X'. The decision problem is choose a series of action that leads to a potential win. The `reward` - win, loss, or a draw, comes at the end of the game. To get there, we need to make a series of decisions where we do not get immedeate clue if our decision was correct or wrong. What are the way(s) to create an algorithm to win?

### A common man's approach

Start with a random choice, say top corner. Then, each time it is our turn to play, do the following in order:

- block an impending triad completion of our opponent, if any
- Else, we put a 'X' to that makes `possible for us to complete a triad in the next move` in case our opponent does not notice it.

This is probably be a decent strategy for a beginner.

### A mathametical approach

Proposed in 1972, by [Newell and Simon](https://en.wikipedia.org/wiki/Tic-tac-toe#Strategy). It is a collection of heuristics that relies on combinatorics. The [game complexity](https://en.wikipedia.org/wiki/Game_complexity#:~:text=For%20tic%2Dtac%2Dtoe%2C,have%20a%20row%20of%20three.) is relatively mild - total 19,683 possible configuration - reduces drastically if we remove invalid configurations (e.g.  five crosses and no noughts).

### What about variations

This strategies quickly become more complicated as we expand the game - say `nxn` board with first `k` to finish is a winner. It can also be complicated, e.g. instead of only two characters what if we have 8, each with different rules (chess!). Or say the two players are playing on private boards not visible to each other (not a very good example in this case, these types of games are called `incomplete information`).

### Can we come up with something more general

Remeber the theme of `ML and AI`. We do not want to solve each problems with specialized algorithms. Rather we want to use standardized algorithms to solve many problems (of similar class) with different inputs. And this class of problems are general decision making problems with delayed reward

## The one-size-fits-all ML pipeline

What if we use the standardized ML approach. We generate random simulations for many many games and tabulate each board configuration and the action taken, and label it if it was positive or negative. Then we use this labelled dataset to fit a supervised model. Then use the model to decide for us.

### Design questions

- how to capture the data? How we capture the board configuration (it is a 3X3 matrix)
- how to label the data? We do not know the immedeate reward at each action step
- how to select features?
- how to choose the algorithm? What should be a good loss function? What should be the estimator (that fits the data to the observations, e.g. a `random forest`, or `svm` model or a `deep NN`?

## Generating Input data

How can we generate input data for our current problem? If we know what data we need, we can look for a vendor that supplies such data. What data we need?

### What data we need

For our problem we need labelled set of data that tells us what was the board configuration at a certain step when we had the turn, what actions we took, and what was the reward. No one unfortunately sells such data. The good news is, we can easily generate it ourselves by simulation

### Developing the Simulation

To simulate a tic-tac-toe game, we can code the game in a Python class. The features that we are looking for is something like below:

```python
class Board():
    """
        A board class to simulate our tic-tac-toe game. We need to identify the players, say 'X' is the first player, 'O' is the second player.
    """
    def __init__(self, player1='X', player2='O', blank=' '):
        pass

    @property
    def state(self):
        """ returns the current configuration of the board. """
        pass

    @property
    def winner(self):
        """ returns the winner if any, or `None`. """
        pass

    @property
    def moves(self):
        """ returns the current available moves left in the board."""
        pass

    @property
    def game_over(self):
        """ if the game is over. """
        pass

    def step(self, player, move):
        """ simulate the game, taking input from `player` and generating random move from the other player. """
        pass
```

We have already coded such a class in the `board.py` file. So let's import and simulate games to generate some data. We need to store the input board configuration, the action and the reward. The board can be represented as a three lines of 3 cells each. We will store the configuration as `flattened` version of this 3x3 matrix (so 3x3 matrix becomes a 1x9 vector). Similarly, action identifies a particular cell to mark - it is a combination of a (row, column) `tuple`. We will store this as a 1x2 vector. So these two stacked together horizontally, our features vector become 1x11.

For the rewards, we will mark +100 if we won in that step, -100 if we lost in that step, otherwise (game not over, or it is a draw) we will mark 0.

Let's see the simulation code. We start with setting our paths and importing the classes we will need.

```python
import os
import sys
import_path = os.path.abspath(os.path.join('..'))
if import_path not in sys.path:
    sys.path.append(import_path)

import numpy as np
import pandas as pd
from tictactoe.board import Board
from tictactoe.agent import Agent
```

## Play some games

Let's simulate some games using the `Board` class we just generated. Below we instantiate a `Board` object (the `board` variable). We choose to play player `X` (first mover). Then we `step` through the board simulation till the game is over.

In the next cell, we generate the data as we discussed before.

```python
board = Board('X','O', ' ')
player = 'X'
while not board.game_over:
    board.step(player, None)

print('the winner is {}\n'.format(board.winner))
print(board)
```

    the winner is O

      X  |  X  |  O
    -----+-----+-----
      X  |  O  |  O
    -----+-----+-----
         |  X  |  O

## A quick benchmark

Before we start creating the data and model for our ML tic-tac-toe player, let's establish a benchmark to evaluate it against. If we play randomly (and the board plays against us randomly as well), we should have approx 1/3 changes of win, losses and draws. Since we are the first-mover, we will have slight advantage in a game like tic-tac-toe (and chess and similar full-information games). Let's simulate a games and score it when both players play randomly.

```python
## CAUTION: it may take a while to run, we simulate a large number of games

player = 'X'
N = 10000
wins = draws = losses = 0

for i in range(N):
    board = Board('X','O', ' ')
    if player != 'X':
        board.make_random_move()

    while not board.game_over:
        choice = board.random_move()
        _, reward, terminated = board.step(player, choice)
        if terminated:
            if reward == 0:
                draws = draws + 1
            elif reward > 0:
                wins = wins + 1
            else:
                losses = losses + 1

print(round(wins/N,2), round(draws/N,2), round(losses/N,2))
```

    0.33 0.35 0.32

So, the random bench mark gives us a slight advantange (33% win against 32% loss). Let's see if we can do better with our ML model.

```python
### Generate the data
player = 'X'
N = 10000
X = np.zeros((N,(9+2)))
y = np.zeros(N)

for i in range(N):
    board = Board('X','O', ' ')
    if player != 'X':
        board.make_random_move()

    while not board.game_over:
        config = np.zeros(board.board.shape)
        config[board.board=='X'] = 1
        config[board.board=='O'] = 0
        config[board.board==' '] = -1
        action = board.random_move()
        _, reward, _ = board.step(player, action)
        X[i,0:9] = config.flatten()
        X[i,9:12] = np.array(action).flatten()
        y[i] = np.sign(reward)

print(f'X is of shape {X.shape}, y is of shape {y.shape}')
```

    X is of shape (10000, 11), y is of shape (10000,)

## Create the model

Next step is to create the model. Notice we have a `categorical` reward function. The rewards 100, -100 or 0 probably indicates levels (or classes) rather than actual (monetary) value. If this assumption is true, we have a case of `classification` problem at hand. We abhor losses more than anything, so we mark negative reward as -1, and a win or a draw is +1. This is a case of `binary classification`.

We choose a [binary cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) loss function, along with a `Random Forest` classifier. We could also have chosen a `DNN`. We split the whole data set to 70-30 as our train vs. test data.

```python
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
rf = RandomForestClassifier(n_estimators=500, max_depth=5, random_state=42)
rf.fit(X_train,y_train)
rf.score(X_test, y_test)
```

    0.6326666666666667

## Evaluate the model

Now that we have our model ready, we can run game scenarios to evaluate. We simulate each game to completion, and each turn, evaluate the predicted reward based on all possible actions (the last two in the features columns) and choose the one that does the best. Then over `N` different games we score how many we lost, won or drew.

```python
## CUATION: This may take a long time to run!

player = 'X'
N = 100
wins = draws = losses = 0

for i in range(N):
    board = Board('X','O', ' ')
    if player != 'X':
        board.make_random_move()

    while not board.game_over:
        config = np.zeros(board.board.shape)
        config[board.board=='X'] = 1
        config[board.board=='O'] = 0
        config[board.board==' '] = -1

        moves = board.moves
        choice = board.random_move()
        features = np.hstack([config.flatten(), np.array(choice).flatten()])
        best = rf.predict(features.reshape(1,11))
        for move in moves:
            features = np.hstack([config.flatten(), np.array(move).flatten()])
            predict = rf.predict(features.reshape(1,11))
            if predict > best:
                best = predict
                choice = move

        _, reward, terminated = board.step(player, choice)
        if terminated:
            if reward == 0:
                draws = draws + 1
            elif reward > 0:
                wins = wins + 1
            else:
                losses = losses + 1

print(round(wins/N,2), round(draws/N,2), round(losses/N,2))
```

    0.56 0.27 0.17

This is definitely better. But is it good enough? Can we do better?

## What is the problem here?

It is not nearly good enough. Given the fact that our opponent is playing randomly, we expected to do a lot better than just 56% wins! The main problem here is our decisions are quite `myopic`, we have trained the model thus. It is not a good way to solve decision problems with delayed rewards. All our input samples were independentl. But in reality, our decisions form a chain and the reward is at the end of it. All actions in a particular game are related. We failed to capture that. This motivates us to look at these decision making problems differently

## Building a decision making framework

We take a fresh approach to solve this problem. Recall our three points in good decision making:

- we take decision based on the currently available information
- we use discounting to normalize delayed rewards
- we use risk-reward balance for risky games and
- we follow a strategy to balance exploitation vs. exploration

What should be our objective for decision making

### Maximize the value of our decision

One good approach is to choose a series of `actions` that try and maximize the expected `reward` (expected immediate and future rewards based on current information, or mathematically, conditioned on current `state`)

\begin{align}
V_{0}^\pi(s) = \mathbb E[R_{0}(s_{0},a_{0}) + \gamma.R_{1}(s_{1},a_{1}) + \gamma^2.R_{2}(s_{2},a_{2}) ....|s_{0}=s,\pi]
\end{align}

Or in other words

\begin{align}
V_{0}^\pi(s) = \mathbb E\Bigl[\sum_n(\gamma^n.R_{n}(s_{n},a_{n}))|s_{0}=s,\pi\Bigr]
\end{align}

We aim to find `policy` such that we maximize this expression. One issue with the above expression is that it tells us what is the expected worth of our actions, but does not tell us what to do. So let's recast the above such that the `action` is a parameter as well, not just `state`.

\begin{align}
V_{0}^\pi(s,a) = \mathbb E\Bigl[\sum_n(\gamma^n.R_{n}(s_{n},a_{n}))|s_{0}=s, a_{0}=a,\pi\Bigr]
\end{align}

This makes the first term deterministic, so we take this out and the expression becomes

\begin{align}
Q_{0}^\pi(s,a) = R_{0}(s,a) + \mathbb E\Bigl[\sum_{n=1}(\gamma^n.R_{n}(s_{n},a_{n}))|s_{0}=s, a_{0}=a,\pi\Bigr]
\end{align}

### The Bellman Equation

Next, we do two simple changes - first, we change the subscript of the value Q to variable time, to make the equation dynamic explicitly. Then, we notice that the second part of the above equation is just gamma times the value Q itself. So we recast a dynamic version of the above equation as follows:

\begin{align}
Q_{t}^\pi(s,a) = R_{t}(s,a) + \gamma.Q_{t+1}^\pi(s,a)
\end{align}

From the above, and from Bellman's [principle of optimality](https://en.wikipedia.org/wiki/Bellman_equation), we claim that to maximize the value we must choose Q such that:

\begin{align}
Q_{t}^\pi(s,a) = R_{t}(s,a) + \gamma.Max(Q_{t+1}^\pi(s,a))
\end{align}

### Q-learning

From the above, we get an interesting insight. An algorithm can be developed on the following line:

- We create a table (called `Q-table`) that will map our evaluations of each allowed `actions` at each `state`. We start with all zero values.

- Next, at initial state, to chose our action `a`, we randomly try all allowed actions and chose the one that provides maximum immediate benifit in the next step.

- We remember this favourable action by making an entry in the Q-table, and take the chosen action to move to the next step

- Repeat

If we do it large enough number of times, we will have enough samples for each `(s,a)` combinations that will allow us average the value computation over and get an expected value. And if these values converge to a limit, we are done. We have created a Q-table, that has seen too many games and memorized all moves to make best judgement best on its memory.

What we have done so far is: we have taken care of our first two principles of good decision making process - using the current information to best use, and using discounting function (`gamma`) to compare rewards across time.

### The problem - being myopic

Notice in the above example we consider only the immediate rewards. Since all actions in a game are connected, that means we may have chosen to leave behind some paths that had low immediate rewards but high final rewards. We were too exploitive and forgot to explore. This risks settling down on a local minima instead of a global one.

One way to avoid this is to introduce a new parameter `epsilon`. It is a fraction between 0 to 1. While choosing the next action, now we will not always choose the best on immediate rewards. With a probability of `epsilon`, we will now choose a random action. And only with probability `(1-epsilon)` we will choose the best. This gives us enough randomness in search to explore good paths that reward later. Also, we can have a high value of `epsilon` at the beginning of learning (`explorative`), and reduce (more `expoliting`) towards the end. A popular way to do this is to introduce an exponential decay of `epslion` between each batch of training.

Finally, we can also have a parameter `nu` or `learning-rate`, that we use to balance between the old and new values of updates of the Q-table. Instead of overwriting the past learned value each time, we can use a fraction `nu` of the new value and `(1-nu)` of the old value for the final update. This smoothens out our learning process.

How to apply all these to our tic-tac-toe problem.

### The Game-Agent Model

The best way to implement this is the agent mode. We already modelled the game. Now we model an `agent` or a player. We store the Q-table and let the `agent` interact with the environment to learn. In each leaning episode, the `agent` play a complete game, choosing actions based on the algorithm above and updating Q-table at each step. In the next episode, it starts with the existing Q-table and carries out further updates and learnings. Under certain [cicumstances](https://en.wikipedia.org/wiki/Markov_decision_process) this gurantess a convergence.

We have already implemented an agent in the file `agent.py`. Let's run a training on this.

```python
a = Agent()
a.train(player='X', n=100000)
```

```python
a.score()
```

    (0.9, 0.02, 0.08)

This is a whooping improvement than the last. Our win rate is 90% compared to 56% based on the brute-force machine learning!

### So where is the catch!

Reinforcement learning is a great method. What we discussed today is a particular version called `Q-learning`. If you followed the algorithm, you would have figured our already where we may have difficulties

- First, we made assumption on convergece. This is not a big issue as such and we can always test for it. The problem is as the total `state space` (all possible states or configuration of the game) becomes larger, this also gets more difficult to ascertain

- We assumed complete information about the `state space` and current `state`. This is true for tic-tac-toe. This is even true for chess. But in a game like poker, it is difficult to evaluate the states and the rewards as a large and important parts (opponets' hands) are hidden. We have only a limited visibility to the actual `state` at any given point in time.

- At least in poker, the states may be hidden, but finite. That is not the case for financial markets. There is an endless possible combinations of market factors, and the `state space` is essentially infinite, in addition of being partially hidden.

In such scenarios, we cannot simply update the Q-table following the above method. However, if make parameterized version of it (i.e. moving from a table to a parameterized functions), we can train a ML estimator (like a deep neural network) learn the parameters and predict the Q-value updates.

If these predictions are good enough, we can then follow the rest of the algorithm to develop our decision bot. When we combine the `DNN` with `Q-learning`, we get what is known as `Deep Q-learning`.

### The butterfly effect

Note, we cannot pick and fit any ML model to parametrize the Q-values. If a linear parametrization fits the bill, it is all good. But for delayed rewards scenario, they will hardly do. Using non-linear parameterization can cause stability issues.

Q-learning is essentially operates on a dynamic system. A characteristic of non-linear dynamic program is what is called chaos. A small change in Q can significantly change subsequent learnings and policy. This can be due to correlations between observations, and that between Q and the target (we are learning both, in a way). This can also be due to inherent random noise in a noise system (a random poker player, and of course, the market.

### Tic-tac-toe vs Trading

As we noted, application of Q-learning or reinforcement learning in general, in finance is possible but not straight forward.

The first thing we need to do is to figure out a way to represent our `state space`. We can never have a complete description, like in tic-tac-toe. But we can choose a set of features (say, price moves, technical indicators and fundamental ratios, sentiments and economic data) to represent the market set.

The second thing we need to do is to define a reward function. Playing tic-tac-toe is arguably not a risky business. And games may last a few rounds. So we had a low discount rate, and considered only the rewards, not any risk (say, variance in rewards). In finance, doing that we can go bust. So it is very important to design a proper reward function. Possibly candidates are percentage returns (no risk consideration), Sharpe ratio, Return to variance ration (Kelly ratio) etc.

Finally, we need to define what is an episode, i.e. what completes a `game`. Unlike tic-tac-toe, there is no natural concept of `end` here. Depending on our investment objective and horizon, we need to define the episode. It can be time-based, i.e. say a week of trading. It can be event-based, e.g. hitting a predefined profit or stoploss. There are other possibilities too.

For a more complete discussion on reinforcement learning in trading, check out the upcoming course on reinforcement learning on [Quantra](https://quantra.quantinsti.com/)

_______________________________________________________________________________________________________________________________

- [1] from upcoming Reinforcement Learning course on [Quantra](https://quantra.quantinsti.com/)
- [2]. All images sourced from google
- [3]. Disclaimer: [Quantra](https://quantra.quantinsti.com/) is from [Quantinsti](https://www.quantinsti.com/), where I work.


```python
#remove cell visibility
from IPython.display import HTML
tag = HTML('''
Toggle cell visibility <a href="javascript:code_toggle()">here</a>.''')
display(tag)
```

Toggle cell visibility here.

```python
%matplotlib inline
import control
import numpy
import sympy as sym
from IPython.display import display, Markdown
import ipywidgets as widgets
import matplotlib.pyplot as plt

#print a matrix latex-like
def bmatrix(a):
     """Returns a LaTeX bmatrix - by Damir Arbula (ICCT project)

     :a: numpy array
     :returns: LaTeX bmatrix as a string
     """
     if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
     lines = str(a).replace('[', '').replace(']', '').splitlines()
     rv = [r'\begin{bmatrix}']
     rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
     rv +=  [r'\end{bmatrix}']
     return '\n'.join(rv)

# Display formatted matrix:
def vmatrix(a):
    if len(a.shape) > 2:
         raise ValueError('bmatrix can at most display two dimensions')
    lines = str(a).replace('[', '').replace(']', '').splitlines()
    rv = [r'\begin{vmatrix}']
    rv += ['  ' + ' & '.join(l.split()) + r'\\' for l in lines]
    rv +=  [r'\end{vmatrix}']
    return '\n'.join(rv)

#matrixWidget is a matrix looking widget built with a VBox of HBox(es) that returns a numPy array as value !
class matrixWidget(widgets.VBox):
    def updateM(self,change):
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.M_[irow,icol] = self.children[irow].children[icol].value
                #print(self.M_[irow,icol])
        self.value = self.M_

    def dummychangecallback(self,change):
        pass

    def __init__(self,n,m):
        self.n = n
        self.m = m
        self.M_ = numpy.matrix(numpy.zeros((self.n,self.m)))
        self.value = self.M_
        widgets.VBox.__init__(self,
                             children = [
                                 widgets.HBox(children =
                                              [widgets.FloatText(value=0.0, layout=widgets.Layout(width='90px')) for i in range(m)]
                                             )
                                 for j in range(n)
                             ])

        #fill in widgets and tell interact to call updateM each time a children changes value
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
                self.children[irow].children[icol].observe(self.updateM, names='value')
        #value = Unicode('example@example.com', help="The email value.").tag(sync=True)
        self.observe(self.updateM, names='value', type= 'All')

    def setM(self, newM):
        #disable callbacks, change values, and reenable
        self.unobserve(self.updateM, names='value', type= 'All')
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].unobserve(self.updateM, names='value')
        self.M_ = newM
        self.value = self.M_
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].value = self.M_[irow,icol]
        for irow in range(0,self.n):
            for icol in range(0,self.m):
                self.children[irow].children[icol].observe(self.updateM, names='value')
        self.observe(self.updateM, names='value', type= 'All')

                #self.children[irow].children[icol].observe(self.updateM, names='value')

#overlaod class for state space systems that DO NOT remove "useless" states (what "professor" of automatic control would do this?)
class sss(control.StateSpace):
    def __init__(self,*args):
        #call base class init constructor
        control.StateSpace.__init__(self,*args)
    #disable function below in base class
    def _remove_useless_states(self):
        pass
```

## Krmiljenje glave trdega diska

Ročica trdega diska, ki podpira bralno glavo, se vrti na viskoznem ležaju, poganja pa jo poganja elektromotor; povratna vzmet s torzijske togosti $ k $ nasprotuje njenemu gibanju. Ležaj ima rahlo viskozno trenje, ki je sorazmerno hitrosti vrtenja glave, s konstanto trenja $ b $. Aktuator (ročica) lahko ustvari navor $ T $ linearno sorazmeren s tokom $ I $, ki teče znotraj tuljave: $ T = k_iI $. Vztrajnost ročice je $ J $. Senzor meri kotni položaj ročice $ \ theta $.

Konstante sistema so:
- $J = 0.001$ kg$\text{m}^2$,
- $k_i = 500$ Nm/A,
- $k = 1$ Nm/rad,
- $b = 0.04$ Nm/(rad/s).

Cilj je načrtovali regulator pomika ročice ob upoštevanju danih zahtev:
- brez prenihaja,
- čas ustalitve manj kot 50 ms (dosežena vrednost izhoda naj se razlikuje od tiste v stacionarnem stanju za **2%**).
- brez odstopka v stacionarnem stanju v odzivu na spremembo zahtevanega pomika ročice z uporabo koračne funkcije,
- vhodni el. tok ne sme presegati 5 A v odzivu na vhodno koračno funkcijo z vrednostjo 30 stopinj.

Enačba sistema je:
$$
J\ddot{\theta} = k_iI - b\dot{\theta} - k\theta
$$

Ob upoštevanju vektorja stanj $x=\begin{bmatrix}x_1 & x_2\end{bmatrix}^T=\begin{bmatrix}\theta & \dot{\theta}\end{bmatrix}^T$ in $u=I$, lahko zgornjo enačbo zapišemo v obliki prostora stanj:

\begin{cases}
\dot{x} = \begin{bmatrix} 0 & 1 \\ -k & -b \end{bmatrix}x + \begin{bmatrix} 0 \\ k_i \end{bmatrix}u \\
y = \begin{bmatrix} 1 & 0 \end{bmatrix}x
\end{cases}

Z namenom dosega ničelnega odstopka v stacionarnem stanju, razširimo objekt z novo spremenljivko stanja $\dot{x_3}=y-y_d=x_1-y_d$. Posledično sistem popišemo z naslednjo enačbo:

\begin{cases}
\dot{x_a} = \begin{bmatrix} 0 & 1 & 0 \\ -k & -b & 0 \\ 1 & 0 & 0 \end{bmatrix}x_a + \begin{bmatrix} 0 & 0 \\ k_i & 0 \\ 0 & -1 \end{bmatrix}\begin{bmatrix} u \\ y_d \end{bmatrix} \\
y = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix}x
\end{cases}

Vodljivostna matrika $\mathcal{C}$, ob upoštevanju zgolj vhoda $u$, je

```python
A = numpy.matrix('0 1 0; -1 -0.04 0; 1 0 0')
B = numpy.matrix('0 0; 500 0; 0 -1')
C = numpy.matrix('1 0 0; 0 0 1')
D = numpy.matrix('0 0; 0 0')

CM = control.ctrb(A,B[:,0])
display(Markdown(bmatrix(CM)))
# print(numpy.linalg.matrix_rank(CM))
```

\begin{bmatrix}
  0. & 500. & -20.\\
  500. & -20. & -499.2\\
  0. & 0. & 500.\\
\end{bmatrix}

in ima rang enak 3, kar pomeni, da je sistem vodljiv.
Spoznavnostna matrika $\mathcal{O}$ je

```python
OM = control.obsv(A,C)
display(Markdown(bmatrix(OM)))
# print(numpy.linalg.matrix_rank(OM))
```

\begin{bmatrix}
  1. & 0. & 0.\\
  0. & 0. & 1.\\
  0. & 1. & 0.\\
  1. & 0. & 0.\\
  -1. & -0.04 & 0.\\
  0. & 1. & 0.\\
\end{bmatrix}

in ima rang enak 3, kar pomeni, da je sistem spoznaven.

### Načrtovanje regulatorja
#### Načrtovanje krmilnika

Ojačanje povratne zveze stanj določimo zgolj na podlagi vhoda $u$, medtem ko za referenco uporabimo $y_d$. Zahtevam, povezanim s prehodnim pojavom, lahko zadostimo, če dominantna pola zaprtozančnega sistema razporedimo v $-100.02\pm101i$, tretji pol npr. v $-100$ (slednji pol naj vsaj ne bo počasnejši od dominantnih dveh). To dosežemo z naslednjo matriko ojačanj:
$$
K = \begin{bmatrix} 80.416 & 0.6 & 4041.0 \end{bmatrix}.
$$

#### Načrtovanje spoznavalnika
Z namenom dosega dobre ocene stanja in s tem hitro konvergenco napake ocene stanj k vrednosti 0, razporedimo pole spoznavalnika v $-151$, $-152$ in $-153$ (poli so hitrejši od polov krmiljenega sistema).
Matrika ojačanj spoznavalnika je tako:
$$
L = \begin{bmatrix} 300.96 & 3.028e-12 \\
 2.264e+04 & 4.55e-10 \\
 1.0 & 150.0 \end{bmatrix}.
$$

Opomba: z razširjenim vektorjem stanj zagotovimo, da je ojačanje zaprtozančnega sistema vedno enako 1.

### Kako upravljati s tem interaktivnim primerom?
- Poizkusi izboljšati odziv sistema v primeru prisotnosi majhne začetne napake (npr. 0.05), s spreminjanjem prvih dve elementov začetnih pogojev, vezanih na spoznavalnik (gl. parameter X0 est.).
- **Opomba:** Vključitev začetne napake v tretjo spremenljivko ocenjenega stanja nima smisla, saj tretje stanje izraža dinamiko, ki smo jo sami določili. Spoznavalnik bi lahko načrtovalu tudi samo za $ x_2 $, saj $ x_1 $ merimo, $ x_3 $ pa poznamo.

```python
# Preparatory cell

X0 = numpy.matrix('0.0; 0.0; 0.0')
K = numpy.matrix([8/15,-4.4,-4])
L = numpy.matrix([[300, 3.028E-12],[2.264E+04, 4.55E-10],[1, 150]])

Aw = matrixWidget(3,3)
Aw.setM(A)
Bw = matrixWidget(3,1)
Bw.setM(B)
Cw = matrixWidget(1,3)
Cw.setM(C)
X0w = matrixWidget(3,1)
X0w.setM(X0)
Kw = matrixWidget(1,3)
Kw.setM(K)
Lw = matrixWidget(3,2)
Lw.setM(L)

eig1c = matrixWidget(1,1)
eig2c = matrixWidget(2,1)
eig3c = matrixWidget(1,1)
eig1c.setM(numpy.matrix([-100.]))
eig2c.setM(numpy.matrix([[-100.02],[-101.]]))
eig3c.setM(numpy.matrix([-100.]))

eig1o = matrixWidget(1,1)
eig2o = matrixWidget(2,1)
eig3o = matrixWidget(1,1)
eig1o.setM(numpy.matrix([-150.]))
eig2o.setM(numpy.matrix([[-151.],[0.]]))
eig3o.setM(numpy.matrix([-152.]))
```

```python
# Misc

#create dummy widget
DW = widgets.FloatText(layout=widgets.Layout(width='0px', height='0px'))

#create button widget
START = widgets.Button(
    description='Test',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Test',
    icon='check'
)

def on_start_button_clicked(b):
    #This is a workaround to have intreactive_output call the callback:
    #   force the value of the dummy widget to change
    if DW.value> 0 :
        DW.value = -1
    else:
        DW.value = 1
    pass
START.on_click(on_start_button_clicked)

# Define type of method
selm = widgets.Dropdown(
    options= ['Nastavi K in L', 'Nastavi lastne vrednosti'],
    value= 'Nastavi lastne vrednosti',
    description='',
    disabled=False
)

# Define the number of complex eigenvalues
sele = widgets.Dropdown(
    options= ['brez kompleksnih lastnih vrednosti', 'dve kompleksni lastni vrednosti'],
    value= 'dve kompleksni lastni vrednosti',
    description='Kompleksne lastne vrednosti:',
    style = {'description_width': 'initial'},
    disabled=False
)

#define type of ipout
selu = widgets.Dropdown(
    options=['impulzna funkcija', 'koračna funkcija', 'sinusoidna funkcija', 'kvadratni val'],
    value='koračna funkcija',
    description='Vhod:',
    style = {'description_width': 'initial'},
    disabled=False
)
# Define the values of the input
u = widgets.FloatSlider(
    value=30*numpy.pi/180,
    min=0,
    max=45*numpy.pi/180,
    step=0.05,
    description='Referenca:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.1f',
)
period = widgets.FloatSlider(
    value=0.5,
    min=0.001,
    max=10,
    step=0.001,
    description='Perioda: ',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='.2f',
)

gain_w2 = widgets.FloatText(
    value=1.,
    description='',
    disabled=True
)

simTime = widgets.FloatText(
    value=0.5,
    description='',
    disabled=False
)
```

```python
# Support functions

def eigen_choice(sele):
    if sele == 'brez kompleksnih lastnih vrednosti':
        eig1c.children[0].children[0].disabled = False
        eig2c.children[1].children[0].disabled = True
        eig1o.children[0].children[0].disabled = False
        eig2o.children[1].children[0].disabled = True
        eig = 0
    if sele == 'dve kompleksni lastni vrednosti':
        eig1c.children[0].children[0].disabled = True
        eig2c.children[1].children[0].disabled = False
        eig1o.children[0].children[0].disabled = True
        eig2o.children[1].children[0].disabled = False
        eig = 2
    return eig

def method_choice(selm):
    if selm == 'Nastavi K in L':
        method = 1
        sele.disabled = True
    if selm == 'Nastavi lastne vrednosti':
        method = 2
        sele.disabled = False
    return method
```

```python
numU = 1

def main_callback2(Aw, Bw, X0w, K, L, eig1c, eig2c, eig3c, eig1o, eig2o, eig3o, u, period, selm, sele, selu, simTime, DW):
    eige = eigen_choice(sele)
    method = method_choice(selm)

    if method == 1:
        solc = numpy.linalg.eig(A-B[:,0:numU]*K)
        solo = numpy.linalg.eig(A-L*C)
    if method == 2:
        if eige == 0:
            K = control.acker(A, B[:,0:numU], [eig1c[0,0], eig2c[0,0], eig3c[0,0]])
            Kw.setM(K)
            L = control.place(A.T, C.T, [eig1o[0,0], eig2o[0,0], eig3o[0,0]]).T
            Lw.setM(L)
        if eige == 2:
            K = control.acker(A, B[:,0:numU], [eig3c[0,0],
                                     numpy.complex(eig2c[0,0],eig2c[1,0]),
                                     numpy.complex(eig2c[0,0],-eig2c[1,0])])
            Kw.setM(K)
            L = control.place(A.T, C.T, [eig3o[0,0],
                                         numpy.complex(eig2o[0,0],eig2o[1,0]),
                                         numpy.complex(eig2o[0,0],-eig2o[1,0])]).T
            Lw.setM(L)

    sys = control.ss(A,B,numpy.vstack((C,numpy.zeros((B.shape[1],C.shape[1])))),numpy.vstack((D,numpy.eye(B.shape[1]))))
    sysC = control.ss(numpy.zeros((1,1)),
                      numpy.zeros((1,numpy.shape(A)[0])),
                      numpy.zeros((numpy.shape(B[:,0:numU])[1],1)),
                      -K)

    sysE = control.ss(A-L*C,
                      numpy.hstack((L,B-L*D)),
                      numpy.eye(numpy.shape(A)[0]),
                      numpy.zeros((A.shape[0],C.shape[0]+B.shape[1])))

    sys_append = control.append(sys, sysE, sysC, control.ss(A,B,numpy.eye(A.shape[0]),numpy.zeros((A.shape[0],B.shape[1]))))
    Q = []
    # y in ingresso a sysE
    for i in range(C.shape[0]):
        Q.append([B.shape[1]+i+1, i+1])
    # u in ingresso a sysE
    for i in range(B.shape[1]):
        Q.append([B.shape[1]+C.shape[0]+i+1, C.shape[0]+i+1])
    # u in ingresso a sys
    for i in range(B[:,0:numU].shape[1]):
        Q.append([i+1, C.shape[0]+B.shape[1]+A.shape[0]+i+1])
    # u in ingresso al sistema che ha come uscite gli stati reali
    for i in range(B.shape[1]):
        Q.append([2*B.shape[1]+C.shape[0]+A.shape[0]+i+1, C.shape[0]+i+1])
    # xe in ingresso a sysC
    for i in range(A.shape[0]):
        Q.append([2*B.shape[1]+C.shape[0]+i+1, C.shape[0]+B.shape[1]+i+1])

    inputv = [i+numU+1 for i in range(B[:,numU:].shape[1])]
    outputv = [i+1 for i in range(numpy.shape(sys_append.C)[0])]
    # in order to avoid singular value exception
    try:
        sys_CL = control.connect(sys_append, Q, inputv, outputv)
    except:
        sys_CL = control.connect(sys_append, Q, inputv, outputv)

    dcgain = control.dcgain(sys_CL[0,0])
    gain_w2.value = dcgain
    if dcgain != 0:
        u1 = u/gain_w2.value
    else:
        print('Nastavljena vrednost vnaprejšnjega ojačanja je bila nastavljena na 0 in je bila zato sprememnjena na 1')
        u1 = u/1
    print('Statično ojačanje zaprtozančnega sistema (od vhoda do izhoda) je: %.5f' %dcgain)

    X0w1 = numpy.zeros((A.shape[0],1))
    for j in range(A.shape[0]):
        X0w1 = numpy.vstack((X0w1,X0w[j]))
    X0w1 = numpy.vstack((X0w1,numpy.zeros((A.shape[0],1))))
    if simTime != 0:
        T = numpy.linspace(0, simTime, 10000)
    else:
        T = numpy.linspace(0, 1, 10000)

    if selu == 'impulzna funkcija': #selu
        U = [0 for t in range(0,len(T))]
        U[0] = u
        U1 = [0 for t in range(0,len(T))]
        U1[0] = u1
        T, yout, xout = control.forced_response(sys_CL[:,0],T,U1,X0w1)
    if selu == 'koračna funkcija':
        U = [u for t in range(0,len(T))]
        U1 = [u1 for t in range(0,len(T))]
        T, yout, xout = control.forced_response(sys_CL[:,0],T,U1,X0w1)
    if selu == 'sinusoidna funkcija':
        U = u*numpy.sin(2*numpy.pi/period*T)
        U1 = u1*numpy.sin(2*numpy.pi/period*T)
        T, yout, xout = control.forced_response(sys_CL[:,0],T,U1,X0w1)
    if selu == 'kvadratni val':
        U = u*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        U1 = u1*numpy.sign(numpy.sin(2*numpy.pi/period*T))
        T, yout, xout = control.forced_response(sys_CL[:,0],T,U1,X0w1)
    # N.B. i primi 3 stati di xout sono quelli del sistema, mentre gli ultimi 3 sono quelli dell'osservatore

    step_info_dict = control.step_info(sys_CL[0,0],SettlingTimeThreshold=0.02,T=T)
    print('Informacija o odzivu sistema: \n\tČas vzpona [s] =',step_info_dict['RiseTime'],'\n\tČas ustalitve (2%) [s] =',step_info_dict['SettlingTime'],'\n\tPrenihaj [%]=',step_info_dict['Overshoot'])
    print('Maksimalna vrednost U (delež od 5 A)=', max(abs(yout[C.shape[0]]))/(5)*100)

    fig = plt.figure(num='Simulacija 1', figsize=(14,12))

    fig.add_subplot(221)
    plt.title('Odziv sistema')
    plt.ylabel('Izhod')
    plt.plot(T,yout[0],T,U,'r--')
    plt.xlabel('$t$ [s]')
    plt.legend(['$y$','Referenca'])
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(222)
    plt.title('Vhod')
    plt.ylabel('$u$')
    plt.plot(T,yout[C.shape[0]],T,[5 for i in range(len(T))],'r--')
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.grid()

    fig.add_subplot(223)
    plt.title('Odziv stanj')
    plt.ylabel('Stanje')
    plt.plot(T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]],
             T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]+1],
             T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]+2])
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.legend(['$x_{1}$','$x_{2}$','$x_{3}$'])
    plt.grid()

    fig.add_subplot(224)
    plt.title('Napaka ocene stanj')
    plt.ylabel('Napaka ocene stanj')
    plt.plot(T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]]-yout[C.shape[0]+B.shape[1]],
             T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]+1]-yout[C.shape[0]+B.shape[1]+1],
             T,yout[C.shape[0]+B.shape[1]+B[:,numU:].shape[1]+A.shape[0]+2]-yout[C.shape[0]+B.shape[1]+2])
    plt.xlabel('$t$ [s]')
    plt.axvline(x=0,color='black',linewidth=0.8)
    plt.axhline(y=0,color='black',linewidth=0.8)
    plt.legend(['$e_{1}$','$e_{2}$','$e_{3}$'])
    plt.grid()
    #plt.tight_layout()

alltogether2 = widgets.VBox([widgets.HBox([selm,
                                          sele,
                                          selu]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('K:',border=3), Kw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Lastne vrednosti:',border=3),
                                          eig1c,
                                          eig2c,
                                          eig3c,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('X0 est.:',border=3), X0w]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.Label('L:',border=3), Lw,
                                          widgets.Label(' ',border=3),
                                          widgets.Label(' ',border=3),
                                          widgets.Label('Lastne vrednosti:',border=3),
                                          eig1o,
                                          eig2o,
                                          eig3o,
                                          widgets.Label(' ',border=3)]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([widgets.VBox([widgets.Label('Inverzna vrednost ojačanja reference:',border=3),
                            widgets.Label('Simulacijski čas [s]:',border=3)]),
                            widgets.VBox([gain_w2,simTime])]),
                            widgets.Label(' ',border=3),
                            widgets.HBox([u,
                                          period,
                                          START])])
out2 = widgets.interactive_output(main_callback2, {'Aw':Aw, 'Bw':Bw, 'X0w':X0w, 'K':Kw, 'L':Lw,
                                                 'eig1c':eig1c, 'eig2c':eig2c, 'eig3c':eig3c, 'eig1o':eig1o, 'eig2o':eig2o, 'eig3o':eig3o,
                                                 'u':u, 'period':period, 'selm':selm, 'sele':sele, 'selu':selu, 'simTime':simTime, 'DW':DW})
out2.layout.height = '870px'
display(out2, alltogether2)
```

    Output(layout=Layout(height='870px'))

    VBox(children=(HBox(children=(Dropdown(index=1, options=('Nastavi K in L', 'Nastavi lastne vrednosti'), value=…

```python

```


# Open Systems
All of the examples in the previous two chapters were closed systems. The ultimate state of closed, chemically reacting systems is chemical equilibrium. Living systems are characterized by mass and energy flow across their boundaries that keep them away from equilibrium. We now discuss open systems, which allow molecules to enter and leave. Open systems ultimately reach a steady state that is often close to a homeostatic state of interest.

**MASSpy** will be used to demonstrate some of the topics in this chapter.

```python
from mass import (
    MassModel, MassMetabolite, MassReaction, Simulation, MassSolution)
from mass.visualization import plot_time_profile, plot_phase_portrait
```

Other useful packages are also imported at this time.

```python
import matplotlib.pyplot as plt
```

## Basic Concepts
There are several fundamental concepts that need to be understood when one considers open systems. We discuss the more significant ones in this section.

### The system boundary
Implicit in the term 'open system' is the notion of an inside and an outside, the division of the world into two domains. The separation between the two is the _system boundary,_ which thus defines what is inside a system and belongs to it, and what is outside.

The definition of a boundary can be physical. An example of a physical boundary may be the cell wall, that clearly defines what is inside and what is outside. Similarly, the outer membrane of the mitochondria can serve as a clearly defined physical system boundary. Thus, systems can have hard, immovable boundaries or soft, flexible ones. In the latter case, the volume of the system may be changing.

The definition of a boundary can also be virtual. For instance, we can define a pathway, such as the TCA cycle, as a system, or the amino acid biosynthetic pathways in a cell as a system. In both cases we might draw dashed lines around it on the metabolic map to indicate the system boundary.

### Crossing the boundary: inputs and outputs
Once a system boundary has been established, we can identify and define interactions across it. They are the _inputs_ and _outputs_ of the system. In most cases, we will be considering mass flows in and out of a system; molecules coming and going. Such flows are normally amenable to experimental determination.

There may be other quantities crossing the system boundary. For instance, if we are considering photosynthesis, photons are crossing the system boundary providing a net influx of energy. Forces can also act on the system boundary. For instance, if the number of molecules coming into and leaving a system through a coupled transport mechanism is not balanced, then osmotic pressure may be generated across the system boundary. Cells have sodium-potassium pumps that displace an uneven number of two types of cations to deal with osmotic imbalances.

### Perturbations and boundaries
In most analyses of living systems, we are concerned about changes in the environment. The temperature may change, substrate availability may change, and so on. Such changes in the environment can be thought of as forcing functions to which the living system responds. These are one-way interactions, from the environment to the system. In most cases, we consider the system to be small (relative to the environment) and that the environment buffers the activities of the system. For instance, the environment may provide an infinite sink for cellular waste products.

In other cases, the activities of the system influence the state of the environment. In batch fermentation, for example, the metabolic activities will substantially change the chemical composition of the medium during the course of the fermentation. On a global scale, we are now becoming more concerned about the impact that human activities have on the climate and the larger environment of human socio-economic activities. In such cases, there are two-way interactions between the system and the environment that need to be described.

### Inside the boundary: the internal network
The definition of a system boundary determines what is inside. Once we know what is inside the system, we can determine the network of chemical transformations that takes place. This network has topology and its links have kinetic properties.

In most cases, it is hard to measure the internal state of a system. There typically are only a few non-invasive measurements available. Occasionally, there are probes that we can use to observe the internal activities. Tracers, such as $^{13}C$ atoms strategically placed in substrates, can be used to trace input to output and allow us to determine pieces of the internal state of a system. Tearing a system apart to enumerate its components is of course possible, but the system is destroyed in the process. Thus, we are most often in the situation where we cannot fully experimentally determine the internal state of a system, and may have to be satisfied with only partial knowledge.

### From networks to system models
Mathematical models may help us to simulate or estimate the internal state of the system. Sometimes we are able to bracket its state based on the inputs and outputs and our knowledge of the internal network.

Full dynamic simulation requires extensive knowledge about the internal properties of a system. Detailed models that describe the dynamic state of a system require the definition of the system boundary, the inputs and outputs, the structure of the internal network, and the kinetic properties of the links in the network. We can then simulate the response of the system to various perturbations, such as changes in the environmental conditions.

### The functional state
Once a system model has been formulated, it can be used to compute the functional state of the system for a given set of conditions. Closed systems will eventually go to chemical equilibrium. However, open systems are fundamentally different. Due to continuous interactions with the environment, they have internal states that are either dynamic or steady. A system that has fast internal time constants relative to changes in the environment will reach a steady state, or a quasi-steady state. For a biological system, such functional states are called _homeostatic states_. Such states are maintained through energy dissipation. The flow of energy in has to exceed the energy leaving the system. This difference allows living systems to reach a functional homeostatic state.

**Figure 6.1:** Open systems. (a) example production of ATP from pyruvate by mitochondria (Prepared by Nathan Lewis). (b) Basic definitions associated with an open system.

## Reversible reaction in an Open Environment
In an open environment, the simple reaction of Eq. (4.1)  has an inflow $(b_1)$ of $x_1$ and an outflow $(b_2)$ of $x_2$:

$$\begin{equation} \stackrel{b_1}{\rightarrow} x_1 \underset{v_{-1}}{\stackrel{v_1}{\rightleftharpoons}} x_2 \stackrel{b_2}{\rightarrow} \tag{6.1} \end{equation}$$

The input is fixed by the environment and there is a first-order rate for the product to leave the system and thus we have

$$\begin{equation} b_1 = \text{constant and}\ b_2 = k_2x_2 \end{equation}$$

This defines the system boundary, and the inputs and outputs.

The stoichiometric matrix is

$$\begin{equation} \textbf{S} = \begin{pmatrix} {1} & {-1} & {1} & {0} \\ {0} & {1} & {-1} & {-1} \\ \end{pmatrix} \end{equation}$$
$$\tag{6.2}$$

where $\textbf{x} = (x_1,\  x_2)$ and $\textbf{v} = (b_1, \ k_1x_1, \  k_{-1}x_2, \  k_2x_2)$ The stoichiometric matrix has a rank of 2 and is thus a two-dimensional dynamic system. The differential equations that will need to be solved are:

$$\begin{equation} \frac{dx_1}{dt} = b_1 - k_1x_1 + k_{-1}x_2, \ \frac{dx_2}{dt} = k_1x_1 - k_{-1}x_2 - k_2x_2 \tag{6.3} \end{equation}$$

There are no conservation quantities.

### The steady states
There are three properties of the steady state of interest. We can find first the steady state fluxes and second the steady state concentrations. Third, we can determine the difference between the steady state and the equilibrium state of this open system.

#### Steady state fluxes:
The steady state of the fluxes is given by

$$\begin{equation} \textbf{Sv}_{ss} = 0 \tag{6.4} \end{equation}$$

and thus $\textbf{v}_{ss}$ resides in the null space of $\textbf{S}$. For this matrix, the null space is two dimensional; $\text{Dim}(\text{Null}(\textbf{S}))=n-r=4-2=2$, where $n=4$ is the number of fluxes and the rank is $r=2$. The null space is spanned by two pathway vectors: (1,1,0,1) and (0,1,1,0). The former is the path through the system while the latter corresponds to the reversible reaction. These are known as type 1 and type 3 extreme pathways, respectively (_Systems Biology: Properties of Reconstructed Networks_). All steady state flux states of the system are a non-negative combination of these two vectors.

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ k_1x_{1, ss}, \ k_{-1}x_{2, ss}, \ k_2x_{2, ss})  \tag{6.5} \\ &= a(1, 1, 0, 1) + b(0, 1, 1, 0), \ a \geq 0, \ b \geq 0 \tag{6.6} \end{align}$$

#### Steady state concentrations:
The concentrations in the steady state can be evaluated from

$$\begin{equation} 0 = b_1 - k_1x_1 + k_{-1}x_2, \ 0 = k_1x_1 - k_{-1}x_2 - k_2x_2 \end{equation}$$ $$\tag{6.7}$$

If we add these equations we find that $x_{2, ss} = b_1 / k_2$. This concentration can then be substituted into either of the two equations to show that $x_{1, ss} = (1 + k_{-1}/k_2)(b_1/k_1)$. Thus, the steady state concentration vector is

$$\begin{equation} \textbf{x}_{ss} = \begin{pmatrix} {x_{1, ss}} \\ {x_{2, ss}} \end{pmatrix} = (\frac{b_1}{k_2})\begin{pmatrix} {\frac{k_2 + k_{-1}}{k_1}} \\ {1} \end{pmatrix} \end{equation}$$ $$\tag{6.8}$$

These steady state concentrations can be substituted into the steady state flux vector to get

$$\begin{align} \textbf{v}_{ss} &= b_1(1, 1+(\frac{k_{-1}}{k_2}), \ (\frac{k_{-1}}{k_2}), \ 1)  \tag{6.9} \\ &= a(1, 1, 0, 1) + b(0, 1, 1, 0), \ a = b_1, \ b = (\frac{b_1k_{-1}}{k_2}) \end{align}$$

Therefore, the steady state flux distribution is a summation of the straight through pathway and the discounted flux through the reversible reaction. The key quantity is $k_{-1} / k_2$ that measures the relative rate of $x_2$ reacting back to form $x_1$ versus the rate at which it leaves the system.

#### The distance from the equilibrium state:
The difference between the steady state and the equilibrium state can be measured by:

$$\begin{equation} \frac{x_{2, ss}/x_{1, ss}}{x_{2, eq}/x_{1, eq}} = \frac{1}{1 + k_2/k_{-1}} \stackrel{k_2 << k_{-1}}{\longrightarrow} 1 \tag{6.11} \end{equation}$$

Thus, when $k_2 << k_{-1}$, the steady state approaches the equilibrium state (recall that $(x_{2, ss}/x_{1, ss})/(x_{2, eq}/x_{1, eq}) = \Gamma/K_{eq}$ Section 2.2). If the exchange with the environment is slow relative to the internal reaction rates, the internal system approaches that of an equilibrium state.

### Dynamic states for reversible reactions
The dynamic states are computed from the dynamic mass balances for a given condition. We are interested in two dynamic states: the approach to the steady state, and the response to a change in the input flux, $b_1$

#### A two-phase transient response:
Simulation of this system with an input rate of $b_1 = 0.01$ and a slow removal rate, $k_2 = 0.1$, from an initial state of $x_1(0) = 1.0$ and $x_2(0) = 0$ is shown in Figure 6.2. There are two discernible time scales: a rapid motion of the equilibrating reaction, and a slow removal of $x_2$ from the system, Figure 6.2b. A phase portrait shows a rapid movement along a line with a negative slope of 1, showing the existence of a conservation quantity $(x_1 + x_2)$ on a fast time scale, followed by a slow motion down a quasi-equilibrium line with a slope of 1/2 to a steady state point. Note the difference from Figure 4.4c.

```python
# Create MassModel
model = MassModel('Linear_Reversible_Open')
# Generate the MassMetabolites
x1 = MassMetabolite("x1")
x2 = MassMetabolite("x2")

# Generate the MassReactions
b1 = MassReaction("b1", reversible=False)
v1 = MassReaction("v1")
b2 = MassReaction("b2", reversible=False)
# Add metabolites to the reaction, add reaction to the model
b1.add_metabolites({x1: 1})
v1.add_metabolites({x1: -1, x2: 1})
b2.add_metabolites({x2: -1})
model.add_reactions([b1, v1, b2])
# Set parameters
b1.kf = 0.01
v1.kf, v1.kr = (1, 2)
b2.kf = 0.1

# Set initial conditions for model
x1.ic = 1
x2.ic = 0
# Utilize type 2 rate law for kf and kr parameters defined
model.get_rate_expressions(rate_type=2, update_reactions=True)

# Set a custom rate for b1 to remove substrate concentration dependence
model.add_custom_rate(reaction=b1, custom_rate=b1.kf_str)
```

    Set parameter Username

```python
t0 = 0
tf = 150

sim = Simulation(model, verbose=True)
conc_sol, flux_sol = sim.simulate(model, time=(t0, tf),
                                  interpolate=True,
                                  verbose=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Linear_Reversible_Open' into RoadRunner.
    Getting time points
    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Simulating 'Linear_Reversible_Open'
    Simulation for 'Linear_Reversible_Open' successful
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_2 = plt.figure(figsize=(13, 5))
gs = fig_6_2.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_2.add_subplot(gs[0, 0])
ax2 = fig_6_2.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x=x1, y=x2, ax=ax1,
    xlabel=x1.id, ylabel=x2.id, xlim=(-.05, 1.2), ylim=(-.05, 0.5),
    title=("(a) Phase portrait of x1 vs. x2", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

ax1.annotate(
    'motion towards\n  a qe state',
    xy=(conc_sol[x1.id](0.5), conc_sol[x2.id](0.5)),
    xytext=(conc_sol[x1.id](0.3), conc_sol[x2.id](0.3)));

ax1.annotate(
    'motion of a\n  qe state',
    xy=(conc_sol[x1.id](tf), conc_sol[x2.id](tf)),
    xytext=(conc_sol[x1.id](tf) + 0.05, conc_sol[x2.id](tf) + 0.15));

plot_time_profile(
    conc_sol, ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Concentration Profiles", {"size": "large"}));
fig_6_2.tight_layout()
```

**Figure 6.2:** The concentration time profiles for the reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ for $k_1 = 1$, $k_{-1} = 2$, $x_1(0) = 0$, and $b_1 = 0.01$. (a) The phase portrait of $x_1$ and $x_2$ (b) The concentrations as a function of time.

The same pools can be formed as defined in Eq. (4.9):

$$\begin{equation}\begin{pmatrix} {p_1} \\ {p_2} \end{pmatrix} = \begin{pmatrix} {1} & {-1/K_1} \\ {1} & {1} \end{pmatrix} = \begin{pmatrix} {x_1} \\ {x_2} \end{pmatrix} \end{equation}$$
$$\tag{6.12}$$

This matrix can be used to post-process the concentrations and the results can be graphed, Figure 6.3. The pool transformation leads to dynamic decoupling and is clearly illustrated in this figure. The disequilibrium pool relaxes very quickly, while the conservation pool moves slowly, Figure 6.3b. The phase portrait formed by the pools thus has an L shape (Figure 6.3a), illustrating the dynamic decoupling. Note that:

* There is first a vertical motion where $x_1 + x_2$ is essentially a constant, followed by a slow horizontal motion where the disequilibrium variable $x_1 - x_2 /K_1$ is a constant.

* These two separate motions correspond to forming the pathways found in the steady state.

* The disequilibrium aggregate is not zero, as it is forced away from equilibrium by the input and settling in a steady state.

```python
# Define pools
pools = ["x1 - x2 / Keq_v1", "x1 + x2"]
for i, equation_str in enumerate(pools):
    pool_id = "p" + str(i + 1)
    conc_sol.make_aggregate_solution(
        pool_id, equation=equation_str,
        parameters={v1.Keq_str: v1.kf/v1.kr}, update=True)

fig_6_3 = plt.figure(figsize=(11, 4))
gs = fig_6_3.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_3.add_subplot(gs[0, 0])
ax2 = fig_6_3.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x="p2", y="p1", ax=ax1,
    xlabel="p2 (inventory)", ylabel="p1 (distance from equilibrium)",
    xlim=(-.05, 2.1), ylim=(-.05, 1.05),
    title=("(a) Phase portrait of p2 vs. p1", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

ax1.annotate(
    'motion towards\n  a qe state',
    xy=(conc_sol["p2"](0.5), conc_sol["p1"](0.5)),
    xytext=(conc_sol["p2"](0.3), conc_sol["p1"](0.3)));

ax1.annotate(
    'motion of a\n  qe state',
    xy=(conc_sol["p2"](tf), conc_sol["p1"](tf)),
    xytext=(conc_sol["p2"](tf) + 0.05, conc_sol["p1"](tf) + 0.1));

plot_time_profile(
    conc_sol, observable=["p1", "p2"], ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Pool Profiles", {"size": "large"}));
fig_6_3.tight_layout()
```

**Figure 6.3:** The time profiles of the pools involved in reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ for the same conditions as in Figure 6.2. (a) The phase portrait of $p_2$ and $p_1$. (b) The pools as a function of time.

#### External disturbance:
The previous simulation represents a biologically unrealistic situation. An internal concentration cannot suddenly deviate from its value independent of what else happens in the system. A much more realistic situation is one where we start out at a steady state and an environmental change is observed. In our case, the only environmental parameter is $b_1$.

In Figure 6.4 we change the input flux, $b_1$, from 0.01 to 0.02 at time zero, when the system is initially in a steady state (the endpoint in Figure 6.4). We make three observations:

* We see that the fast motion is not activated.

* The 'inventory' or the pool of $x_1 + x_2$ moves from one steady state to another. It increases, as the forcing function was stepped up.

* The 'distance from equilibrium'

$$\begin{equation} x_{1, ss} - \frac{x_{2, ss}}{K_1} = x_{1, ss} - \frac{x_{2, ss}}{x_{2, eq}/x_{1, eq}} = x_{1, ss}(1 - \frac{\Gamma}{K_1}) \tag{6.13} \end{equation}$$

is close to zero in both steady states. The higher throughput, however, does push the system farther from equilibrium.

```python
# Ensure model starts simulation at a steady state.
sim.find_steady_state(model, strategy="simulate", update_values=True, verbose=True);

# Simulate model with disturbance from steady state
conc_sol, flux_sol = sim.simulate(
    model, time=(t0, tf), perturbations={"kf_b1": 0.02},
    interpolate=True, verbose=True)

# Determine pools
for i, equation_str in enumerate(pools):
    pool_id = "p" + str(i + 1)
    conc_sol.make_aggregate_solution(
        pool_id, equation=equation_str,
        parameters={v1.Keq_str: v1.kf/v1.kr}, update=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Setting output selections
    Getting time points
    Simulating 'Linear_Reversible_Open'
    Found steady state for 'Linear_Reversible_Open'.
    Updating 'Linear_Reversible_Open' values
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Getting time points
    Parsing perturbations
    Setting output selections
    Setting simulation values for 'Linear_Reversible_Open'
    Simulating 'Linear_Reversible_Open'
    Simulation for 'Linear_Reversible_Open' successful
    Adding 'Linear_Reversible_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_4 = plt.figure(figsize=(11, 4))
gs = fig_6_4.add_gridspec(nrows=1, ncols=2, width_ratios=[1, 1.5])

ax1 = fig_6_4.add_subplot(gs[0, 0])
ax2 = fig_6_4.add_subplot(gs[0, 1])

plot_phase_portrait(
    conc_sol, x="p2", y="p1", ax=ax1,
    xlabel="p2 (inventory)", ylabel="p1 (distance from equilibrium)",
    xlim=(0, 1), ylim=(0, 0.03),
    title=("(a) Phase portrait of p2 vs. p1", {"size": "large"}),
    annotate_time_points=[t0, 1e0, 1e1, tf],
    annotate_time_points_color=["red"],
    annotate_time_points_labels=True);

plot_time_profile(
    conc_sol,  observable=["p1", "p2"], ax=ax2, legend="right outside",
    xlabel="Time", ylabel="Concentrations",
    title=("(b) Pool Profiles", {"size": "large"}));
fig_6_4.tight_layout()
```

**Figure 6.4:** The time profiles of the pools involved in reaction system $\rightarrow x_1 \rightleftharpoons x_2 \rightarrow$ where the system starts out at equilibrium and the input rate, $b_1$, is changed from 0.01 to 0.02 at time zero. (a) The phase portrait of $p_2$ and $p_1$; (b) The pools as a function of time. Note the small numerical change in $p_1$ relative to $p_2$. Parameter values as in Figures 6.2 and Figure 6.3.

## Michaelis-Menten Kinetics in an Open Environment
We now consider the case when the Michaelis-Menten reaction mechanism operates in an open environment (Figure 6.5). The substrate enters the system and the product leaves. The enzyme stays internal to the system.

**Figure 6.5:** The Michaelis-Menten reaction mechanisms in an open setting. The substrate and the product enter and leave the system, while the enzyme stays inside.

### Dynamic description
The mass action kinetic model is

$$\begin{align} \frac{ds}{dt} &= b_1 - k_1es + k_{-1}x, \ &s(t=0) &= s_0 &\tag{6.14a} \\ \frac{dx}{dt} &= k_1es - (k_{-1} + k_2)x, \ &x(t=0) &= x_0 &\tag{6.14b}\\ \frac{de}{dt} &= -k_1es + (k_{-1} + k_2)x, \ &e(t=0) &= e_0 &\tag{6.14c} \\ \frac{dp}{dt} &= k_2x + k_3p, \ &p(t=0) &= p_0 &\tag{6.14d} \\ \end{align}$$

The initial conditions would normally be the steady state conditions. The stoichiometric matrix is

$$\begin{equation} \textbf{S} = \begin{pmatrix} {1} & {-1} & {1} & {0} & {0} \\ {0} & {-1} & {1} & {1} & {0} \\ {0} & {1} & {-1} & {-1} & {0} \\ {0} & {0} & {0} & {1} & {-1} \\ \end{pmatrix} \end{equation}$$
$$\tag{6.15}$$

where $\textbf{x}=(s,\ e, \ x, \ p)$ and $\textbf{v} = (b_1, \ k_1es, \ k_{-1}x, \ k_2x, \ k_3p)$

### The steady state
As in the previous section, we can compute the steady state fluxes and concentrations. We can also compute how the parameters determine the distance from equilibrium, and here we also run into an additional issue: capacity constraints that result from a conservation quantity.

#### The steady state fluxes:
The rank of $\textbf{S}$ is 3, thus the dimension of the null space is 5-3=2. The null space of $\textbf{S}$ is spanned by two vectors, (1,1,0,1,1) and (0,1,1,0,0), that correspond to a pathway through the system and an internal reversible reaction. The steady state fluxes are given by

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ k_1e_{ss}s_{ss}, \ k_{-1}x_{ss}, \ k_2x_{ss}, \ k_3p_{ss}) \tag{6.16} \\ &= a(1, \ 1, \ 0, \ 1, \ 1) + b(0, 1, 1, 0, 0), \ a \geq 0, \ b \geq 0 \tag{6.17} \\ \end{align}$$

#### The steady state concentrations:
The dimension of the left null space is 4-3=1. The left null space has one conservation quantity ($e+x$), which can readily be seen from the fact that the second and third row of $\textbf{S}$ add up to zero. The steady state flux balances for this system are

$$\begin{equation} b_1 = v_1 - v_{-1} = v_2 = v_3 \tag{6.18} \end{equation}$$

Thus, the incoming flux and the kinetic parameters immediately set the concentrations for $X$, $P$, and $E$ as

$$\begin{equation} x_{ss} = b_1/k_2, \ p_{ss} = b_1/k_3, \ \text{and} \ e_{ss} = e_t - x_{ss} = e_t -  b_1/k_2 \tag{6.19} \end{equation}$$

and the steady state substrate concentration can be determined. The steady state concentration of the substrate is given by $k_1s_{ss}e_{ss} = b_1 + k_{-1}x_{ss}$ that can be solved to give

$$\begin{equation} s_{ss} = (\frac{k_2}{k_1})(\frac{k_{-1}/k_2 + 1}{e_tk_2/b_1 - 1}) \tag{6.20} \end{equation}$$

The steady state flux vector can now be computed

$$\begin{align} \textbf{v}_{ss} &= (b_1, \ b_1(k_{-1}/k_2 + 1), \ b_1k_{-1}/k_2, \ b_1, \ b_1) \tag{6.21} \\ &= a(1, \ 1, \ 0, \ 1, \ 1) + b(0, 1, 1, 0, 0), \ a=b_1, \ b=b_1k_{-1}/k_2 \tag{6.22} \\ \end{align}$$

The distance from equilibrium can now be computed as in the previous section.

#### Internal capacity constraints:
Since $e_{ss} \geq 0$, the maximum input is

$$\begin{equation} b_1 \leq b_{1, max} = k_2e_t = v_m \tag{6.23} \end{equation}$$

which is the maximum reaction rate for the Michaelis-Menten mechanism. The total amount of the enzyme and the turnover rate set this flux constraint.

```python
model = MassModel('Michaelis_Menten_Open')
## Define metabolites
s = MassMetabolite("s")
e = MassMetabolite("e")
x = MassMetabolite("x")
p = MassMetabolite("p")

# Define reactions
b1 = MassReaction("b1", reversible=False)
v1 = MassReaction("v1")
v2 = MassReaction("v2", reversible=False)
v3 = MassReaction("v3", reversible=False)

b1.add_metabolites({s: 1})
v1.add_metabolites({s: -1, e: -1, x: 1})
v2.add_metabolites({x: -1, e: 1, p: 1})
v3.add_metabolites({p: -1})
model.add_reactions([b1, v1, v2, v3])

## Define parameters
b1.kf = 0.025
v1.kf, v1.kr = (1, 0.5)
v2.kf = 0.5
v3.kf = 1

# Set initial conditions for model
s.ic = 1
e.ic = 0.05
x.ic = 0.05
p.ic = 0

# Utilize type 2 rate law for kf and kr parameters defined
model.get_rate_expressions(rate_type=2, update_reactions=True)

# Set a custom rate for b1 to remove substrate concentration dependence
model.add_custom_rate(reaction=b1, custom_rate=b1.kf_str)
```

```python
t0 = 0
tf = 2e3

sim = Simulation(model, verbose=True)
# Simulate model with disturbance from steady state
conc_sol, flux_sol = sim.simulate(
    model, time=(t0, tf), perturbations={"kf_b1": 0.04},
    interpolate=True, verbose=True)
```

    WARNING: No compartments found in model. Therefore creating compartment 'compartment' for entire model.

    Successfully loaded MassModel 'Michaelis_Menten_Open' into RoadRunner.
    Getting time points
    Parsing perturbations
    Setting output selections
    Setting simulation values for 'Michaelis_Menten_Open'
    Simulating 'Michaelis_Menten_Open'
    Simulation for 'Michaelis_Menten_Open' successful
    Adding 'Michaelis_Menten_Open' simulation solutions to output
    Updating stored solutions

```python
fig_6_6 = plt.figure(figsize=(8, 6))
gs = fig_6_6.add_gridspec(nrows=2, ncols=2, width_ratios=[1, 1.5],
                          height_ratios=[1, 1])

ax1 = fig_6_6.add_subplot(gs[0, 0])
ax2 = fig_6_6.add_subplot(gs[0, 1])
ax3 = fig_6_6.add_subplot(gs[1, 1])

plot_phase_portrait(
    conc_sol, x=s, y=x, ax=ax1,
    xlabel=s.id, ylabel=x.id,
    xlim=(-0.1, 6), ylim=(0, 0.105),
    title=("(a) Phase Portrait of s vs. x", {"size":"large"}),
    annotate_time_points="endpoints",
    annotate_time_points_labels=True)

plot_time_profile(
    conc_sol, observable=x, ax=ax2, legend="best",
    ylim=(0.03, 0.09), xlabel="Time", ylabel="Concentration",
    title=("(b) Time Response of x", {"size": "large"}));

plot_time_profile(
    conc_sol, observable=s, ax=ax3, legend="best",
    ylim=(0, 6), xlabel="Time", ylabel="Concentration",
    title=("(c) Time Response of s", {"size": "large"}));
fig_6_6.tight_layout()
```

**Figure 6.6:** The time profiles for the transient response of the Michaelis-Menten mechanisms for $b_1 = 0.025$ changed to 0.04 at time zero. The kinetic parameters are; $e_t = 0.1$,  $k_1 = 1$, $k_{-1} = 0.5$, $k_2 = 0.5$, $k_3 = 0.1$. (a) The phase portrait of $s$ vs. $x$. (b) The time response of $x$. (c) The time response of $s$.

### Dynamic states for Michaelis-Menten kinetics
The response of this system to a change in the input rate starting from a steady state is of greatest interest, see Figure 6.6. For the kinetic parameters given in the figure, the steady state concentrations of $s$ and $x$ are 1.0 and 0.5 respectively. The input rate is changed from 0.025 to 0.04 at time zero and the concentrations of $s$ and $x$ go to 4.0 and 0.8, respectively, as time goes to infinity. The maximum flux rate is 0.05.

The change in the input rate triggers an internal motion that basically follows the quasi-steady state line (Figure 6.6c). Figure 6.6b and Figure 6.6c shows that since the internal steps are rapid relative to the exchange rates, there are no rapid transients produced by a perturbation in the input rate.

If the input is increased towards $v_m$, the substrate concentration builds up to a very high value and most of the enzyme is found in the intermediate state. If the input rate exceeds $v_m$ there will be no steady state as the enzyme cannot convert the substrate to the product at the same rate as it is entering the system.

The dynamic properties of this open system can be analyzed using pool formation.

## Summary
* Open systems eventually reach a steady state, which is different from the equilibrium state of a closed system. Such steady states can be thought of as homeostatic, living states.

* Living cells are open systems that continually exchange mass and energy with their environment. The continual net throughput of mass and concomitant energy dissipation is what allows steady states to form and differentiates them from equilibrium states.

* The relative rates of the internal network to that of the exchanges across the system boundary are important. Time scale separation between internal and exchange processes can form. Rapid internal transients lead to pool formation.

* Open systems are most naturally in a steady state and respond to external stimuli. It is normally not possible to suddenly change the internal state of the system since it is in balance with the environment.

* If the internal dynamics are fast, they do not excite when external stimuli are experienced, and thus accurate information about the fast kinetics may not be needed. It may be enough to know that they are "fast."

* Non-exchanged moieties form dynamic invariants. They can set internal capacity constraints.

$\tiny{\text{© B. Ø. Palsson 2011;}\ \text{This publication is in copyright.}\\ \text{Subject to statutory exception and to the provisions of relevant collective licensing agreements,}\\ \text{no reproduction of any part may take place without the written permission of Cambridge University Press.}}$