# Markov chains and stochastic recurrence relations

Some recurrant dynamic systems are naturally *stochastic* (or - in other words - involve a bit of randomness).  In this post - continuing our discussion of [recurrence relations](https://jermwatt.github.io/control-notes/posts/recurrence_relations/Recurrence_relations.html) - we introduce the basic version of such a model via its most popular application - as a model of written text.  This kind of dynamic system is often referred to as a *Markov Chain*.

You can skip around this document to particular subsections via the hyperlinks below.

-  [Natural ordering in text](#text-natural-order)
-  [Stochastic choices, histograms, and Markov chains](#stochastic-choices)
-  [Markov chains on the character-level](#character-level)
-  [The mathematics of a Markov chain](#modeling)
-  [Examples and `Python` implementation](#code)
-  [Fixed order and limited memory](#limited-memory)
-  [Markov chains with one-hot encoded vectors](#one-hot)
-  [Markov chains with unlimited memory](#unlimited)

```python
# This code cell will not be shown in the HTML version of this notebook
# imports from custom library for animations
from library import word_level_markov_model
from library import markov_words_demo
from library import markov_chars_demo
from library import text_parsing_utils

# import standard libs
import numpy as np
import pandas as pd

# path to data
datapath = '../../datasets/markov_chains/'

# This is needed to compensate for matplotlib notebook's tendancy to blow up images when plotted inline
%matplotlib notebook
from matplotlib import rcParams
rcParams['figure.autolayout'] = True

%load_ext autoreload
%autoreload 2
```

## Natural ordering in text

Below we animate a simple sentence "my dog runs".  Each *word* in this sentence does - intuitively - seem to follow its predecessor in an natural, orderly, and predictable way.  In the animation we walk through the sentence word-by-word, highlighting the fact that each word follows its immediate predecessor.

 Figure 1:  Each word in a simple English sentence follows its predecessor in an natural, orderly, and predictive fashion.

This kind natural ordering holds with text more generally as well, and (in the context of dynamic systems) provokes a natural question: can text be modeled as a *recurrence relation*?  As [discussed in a previous post](https://jermwatt.github.io/control-notes/posts/recurrence_relations/Recurrence_relations.html), limited memory recurrence relations take in a window of $D$ elements of an input sequence $x_{p-1},x_{p-2},...,x_{p-D}$ and generate the next element (here a word) $x_p$ as

\begin{equation}
x_{p} = f\left(x_{p-1},x_{p-2},...,x_{p-D}\right).
\end{equation}

If this were possible we would of course need to define an appropriate value for $D$ as well as an appropriate form for the function $f$.  But more fundamentally, while text certainly seems to have the structure of a recurrence relation (with words in a sentence being reasonably predicted by its predecessors), text does have one attribute that we have not seen thus far: *choice*.  That is, a word or set of words does not often uniquely determine the word that follows it.

For example, in the sentence above we could imagine a range of words following the word "dog" instead of "runs", like e.g., "eats".  This would give us the sentence "my dog eats" instead of "my dog runs", which is a perfectly valid and meaningful English sentence.  Of course there are other words that could follow "dog" as well, some of which like e.g., the word "sighs" that while valid would be less common than "eats".  However some words, like e.g., "dracula", making the sentence "my dog dracula" do not make any sense at all.  Thus with text while we do have *choices* for each follow-up word, some choices are more likely to occur than others.  These choices are *stochastic* in nature - meaning each arises with a different *probability* of occurrence.

 Figure 2:  Many words could potentially follow the word "dog" in this sentence.  However some words - like "runs" or "eats" - are far more likely than others like "sighs".  Further, some words like e.g., "dracula" do not make any sense.

## Stochastic choices, histograms, and Markov chains

To illustrate this point, below we show the four choices of words following "dog" in the animation shown above assigning a probability of occurrence to each.  We made up for these probabilities for this simple example as well as the examples that follow, but in practice these can be computed *empirically*.  More specifically this can be done by taking a large body of text (like e.g., a long book), scanning it for all occurrences of each of its words, and for each forming a normalized histogram of probabilities of its follow-up words.  In any case, here we suppose the word "runs" is the most probable, "eats" the next, and so on.  The probabilities shown do not add up to $1$ because there could be other reasonable words that could follow "dog" other than those shown here (like "barks", for example).

This set of words following "dog", along with their respective probabilities of occurrence (that is, the probability that they occur following the word "dog"), can be nicely represented as a *normalized histogram* (also known as a discrete probability distribution) as shown in the right panel below.  This normalized histogram visualizes the probability of each follow-up word as a vertical bar whose height is proportional to a word's probability of occurrence in following the word "dog".

 Figure 3:  (left panel) Here we show a selection of words that could possibly follow the word "dog", along with a probability of the word occurring (which here we assign ourselves via our intuition).  In this case the word "runs" has the highest probability of occurring after "dog", then "eats", and so on.  The probabilities do not add up to $1$ because there are other words that could reasonably follow "dog" that we do not list here. (right panel) The set of possible words following the word "dog" shown in the left viewed as a *histogram*, where the height of each bar is proportional to each following word's probability of occurrence.

Notice that while many words *can* follow the word "dog", if we had to make a single *educated guess* as to the next best word to follow-up the word "dog" based on this discrete probability distribution we would choose the highest probably follow-up word "run".  This sort of model for text, from the capturing of its stochastic nature of follow-up words via a discrete probability distribution to the probability-maximizing "educated guess" for the next best word, is called a *stochastic recurrence relation* or (more popularly) a *Markov chain*.

Notice that we can also think of each word in a sentence following logically based not just on its immediate predecessor, but on several words (in other words, we can use a larger window).  Text certainly is generally structured like this - with preceding words in a sentence generally determining those that follow.   In the Figure below we show an illustration that mirrors that of Figure 3 above using a window size (or *order*) $D = 2$.  Now the probabilities of occurrence for each follow-up word reflects how frequently each word follows the phrase "my dog".  Once again if we wanted to make an educated guess - based on this discrete probability - of what word likely follows the phrase "my dog" we would choose the most probable follow-up word "sleeps".

 Figure 4:  An illustration mirroring Figure 3 above using a window size (or *order*) of $D = 2$.  Here we look at the statistics of follow-up words for the phrase "my dog" precisely as we did in the order $D = 1$ case - by forming a normalized histogram / discrete probability distribution over all the words following them.

Notice as we increase the number of words, or likewise our window size $D$, that it gets easier and easier to make an "educated guess" as to the correct follow-up word *using our probability-maximizing choice*.  This is because the phrase in our length $D$ window becomes more and more unique with each added word, and eventually only a single follow-up word will ever follow it, and the distribution of follow-up words collapses to an *impulse* or [dirac delta](https://en.wikipedia.org/wiki/Dirac_delta_function) function.  This means that once the window becomes large enough our probability-maximizing follow-up word model indeed defines a perfect recurrence relation,

We illustrate this idea with a longer (and more popular) phrase below: "a rose by any other name would smell as ".  This popular Shakespearean phrase has only a single possible follow-up word: "sweet".

 Figure 5:  As the window length $D$ is increased the phrase in our window becomes more and more unique, with the distribution of follow-up words collapsing to a unit impulse (meaning only one kind of follow-up word ever follows our input phrase).

## Markov chains on the character-level

The same thought process leading to the word-by-word model of text detailed above also leads to a similar idea: modeling text *character-by-character*.  All of the same logic developed for the word-wise model applies here as well.  If we examine a string of English text - on the level of characters this time - once again upon reflection we see a natural ordering to the characters (as in the example shown in the Figure below).  Some characters - as with words - seem to naturally follow others.  Of course just as with words, the issue here is once again at each step we have *stochasticity* or multiple choices for what the next character should be.  In other words, we have a histogram / discrete probability distribution over the full set of possible English characters that reflects the likelihood of each character occurring (with respect to a large corpus on which we build such histograms).  As we increase the order of our model - just as in the word-wise case - our generated text looks more and more like the corpus on which we compute our histograms.

 Figure 5:
Illustrations of the concepts described in the prior three Figures, only here for modeling text as a stochastic recurrence relation *character-wise* instead of word-wise.  In each case to make a reasonable prediction about the next character we form a normalized histogram / discrete probability distribution over all the characters that follow each input character(s).  Note in the window size $D = 2$ case shown here the empty circle contains the 'space' character.  As we increase the window size here - just as with the word-wise model detailed above - our generated sequence will start to look more and more like the text on which we computed our histograms.

## The mathematics of a Markov chain

Let us mathematically codify the Markov model - using words as our fundamental unit and beginning with the first simple word-wise example illustrated in Figure 3 as a jumping off point.  Note however that everything we discuss here generalizes to all word-level and character-level modeling as well.

As shown in Figure 3 above we form a histogram of follow-up words to the input word "dog", along with their probability of occurrence.  A normalized histogram is just *vector-valued* output representing the occurrence probability of each possible follow-up word to our input.  For example, we can formally jot down the histogram of possible outputs of words following "dog" as

\begin{equation}
\text{histogram}\left(\text{"dog"}\right) =
\begin{cases}
\text{"runs"} \,\,\,\,\,\,\,\,\,\,\,\, \text{with probability} \,\,\,\mathscr{y} = 0.4 \\
\text{"eats"} \,\,\,\,\,\,\,\,\,\,\,\,\, \text{with probability} \,\,\,\mathscr{y} = 0.3 \\
\text{"sighs"} \,\,\,\,\,\,\,\,\,\, \text{with probability} \,\,\,\mathscr{y} = 0.05 \\
\text{"dracula"} \,\,\,\,\, \text{with probability} \,\,\,\mathscr{y}= 0 \\
\,\,\,\,\,\, \vdots
\end{cases}
\end{equation}

Here in each case $\mathscr{y}$ stands for the *probability* of the corresponding follow-up word occurring.  As mentioned previously, the best follow-up word based on this histogram is simply the one with the *maximum probability* of occurrence.  Here suppose that word is "runs".  Translating this statement into math, we predict the word following "dog" by taking the $\text{argmax}$ over all of the choices above as

\begin{equation}
\text{(word we predict to follow "dog")} \,\,\,\,\, \text{"runs"} = \underset{\mathscr{y}}{\text{argmax}}\,\,\,\, \text{historgram}\left(\text{"dog"}\right).
\end{equation}

More generally, if we denote by $x_{p-1}$ the $\left(p-1\right)^{th}$ word in a sentence, then the choice of the next word $h_{p}$ can likewise be written in general as

\begin{equation}
h_{p} = \underset{\mathscr{y}}{\text{argmax}}\,\, \text{histogram}\left(x_{p-1}\right).
\end{equation}

Denoting by $f\left(x_{p-1}\right) = \underset{\mathscr{y}}{\text{argmax}}\,\, \text{histogram}\left(x_{p-1}\right)$ we can express our Markov chain model as a [dynamic system with limited memory](https://jermwatt.github.io/control-notes/posts/dynamic_systems_limited_memory/dynamic_systems_limited_memory.html) as

\begin{equation}
h_{p} = f\left(x_{p-1}\right).
\end{equation}

Using the same function $f$ we general order $D$ Markov model via the general update step

\begin{equation}
h_{p} = f\left(x_{p-1},...,x_{p - D}\right)
\end{equation}

only here our $\text{histogram}$ function computes the histogram of follow-up words of the input sequence $x_{p-1},...,x_{p - D}$.

Finally - as mentioned above - when $D$ is large enough this model becomes a perfect [recurrence relation](https://jermwatt.github.io/control-notes/posts/recurrence_relations/Recurrence_relations.html), since as $D$ increases the number of possible follow-up words narrows more and more, eventually diminishing to a single word.  Thus the formula above reduces to

\begin{equation}
x_{p} = f\left(x_{p-1},...,x_{p - D}\right)
\end{equation}

since indeed the output of $f$ is the next actual word $x_p$.

## Examples and `Python` implementation

#### Example 1.   Generating text word-by-word via a Markov chain

In this example we generate a Markov chain model of text using the classic novel *War of the Worlds* by H.G. Wells to define our transition probabilities (the text of which can be found legally for free online [e.g., here](https://archive.org/stream/TheWarOfTheWorlds-H.G.Wells/war-worlds_djvu.txt)).  Below we print out the first $500$ characters of the novel.  Note some pre-processing has been done here - in particular we removed any strange characters introduced when converting this book to its e-version, and lower-case-afied all alphabetical characters.

```python
csvname = datapath + "war_of_the_worlds.txt"
model = word_level_markov_model.Markov(csvname)
model.text[:500]
```

    "the war of the worlds  by h. g. wells   but who shall dwell in these worlds if they be  inhabited? are we or they lords of the  world? and how are all things made for man?    kepler (quoted in the anatomy of melancholy)  book one  the coming of the martians  chapter one  the eve of the war no one would have believed in the last years of the nineteenth century that this world was being watched keenly and closely by intelligences greater than man's and yet as mortal as his own; that as men busied "

The `Python` function used to pre-process the text is shown below - and requires minimal functionality.

```python
## load and preprocess text ##
def load_preprocess(csvname):
    # load in text dataset - lower case all
    text = open(csvname).read().lower()

    # cut out first chunk of giberish text - for this text non-giberish began at the 948th character
    text = text[947:]

    # remove some obvious tag-related gibberish throughout
    characters_to_remove = ['0','1','2','3','4','5','6','7','8','9','_','[',']','}','.  .  .','\\']
    for i in characters_to_remove:
        text = text.replace(i,'')

    # some gibberish that looks like it needs to be replaced with a ' '
    text = text.replace('\n',' ')
    text = text.replace('\r',' ')
    text = text.replace('--',' ')
    text = text.replace(',,',' ')
    text = text.replace('   ',' ')

    return text
```

To produce an order $D$ Markov model we then run through the text and  compute our discrete transiition probabilities.  To do this we first produce a dictionary `words_to_keys` that maps each word (also called a "token") to a discrete integer (called a "key"), and another dictionary `keys_to_words` for the reverse mapping (from keys back to words).  These dictionaries are computed via the `parse_words` function below - and employs a simple function called `CountVectorizer` from `sklearn`.

```python
# imports
from sklearn.feature_extraction.text import CountVectorizer

## parse a text into words - producing mapping dictionaries ##
def parse_words(text):
    # load in function from scikit learn that
    vectorizer = CountVectorizer()
    X = vectorizer.fit_transform([text])
    analyze = vectorizer.build_analyzer()

    # get all unique words in input corpus
    tokens = analyze(text)
    unique_words = vectorizer.get_feature_names()

    # unique nums to map words too
    unique_nums = np.arange(len(unique_words))

    # this dictionary is a function mapping each unique word to a unique integer
    words_to_keys = dict((i, n) for (i,n) in zip(unique_words,unique_nums))

    # this dictionary is a function mapping each unique integer to a unique word
    keys_to_words = dict((i, n) for (i,n) in zip(unique_nums,unique_words))

    # convert all of our tokens (words) to keys
    keys = [words_to_keys[a] for a in tokens]

    return tokens,keys,words_to_keys,keys_to_words
```

With these two dictionaries, our keys, and tokens in hand we can now easily generate our desired set of transition probabilities.

```python
# make transition probabilities based on discrete count of input text
def make_transition_probabilities(order):
    # get unique keys - for dimension of transition matrix
    unique_keys = np.unique(keys)
    num_unique_words = len(unique_keys)
    num_words = len(tokens)

    # generate initial zeros order O transition matrix
    # use a dictionary - or else this for sure won't scale
    # to any order > 1
    transition_matrix = {}

    # sweep through tokens list, update each individual distribution
    # as you go - each one a column
    for i in range(order,num_words):
        # grab current key, and previous order keys
        next_key = keys[i]
        prev_keys = tuple(keys[i-order:i])

        ## update transition matrix
        # we've seen current key already
        if prev_keys in transition_matrix.keys():
            if next_key in transition_matrix[prev_keys].keys():
                transition_matrix[prev_keys][next_key] += 1
            else:
                transition_matrix[prev_keys][next_key] = 1
        else:      # we haven't seen key already, so create new subdict
            transition_matrix[prev_keys] = {}
            transition_matrix[prev_keys][next_key] = 1

    return transition_matrix
```

Using an order $D = 1$ model we then pick a word randomly from the text, and start generating text.  Below we compare a chunk of $30$ words from the text following this initial input, and below it we show the result of the Markov model.  Here the input word $x_1 = \gamma$ is colored red, and the $30$ words generated using it are colored blue.  Note this means that we first plug in the word "had" into our model, which returns the word "been".  Then we return "been" and it returns "the", etc.,

```python
## This code cell will not be shown in the HTML version of this notebook
order = 1; num_words = 30;
demo = markov_words_demo.show_order(csvname,order,num_words)
```

    -------- TRUE TEXT -------
    he came up to the fence and extended handful of strawberries for his gardening was as generous as it was enthusiastic at the same time he told me of the burning

    -------- ORDER = 1 MODEL TEXT -------
    he said the martians were the martians were the martians were the martians were the martians were the martians were the martians were the martians were the martians were the martians

Clearly we can see that the Markov model, having only a single word in the past to base the next word on, does not generate anything meaningful.  However as we increase the order to e.g., $D = 2$ we can see that the generated sentence starts to make more sense, matching its original as shown below.  Here the two initial words are colored red, with the remaining generated words colored blue.  Notice this means that we first plug in the first two words (here the phrase "trucks bearing") and it returns "huge", then we plug in the next two words (here "bearing huge") and it returns "guns", etc.,

```python
## This code cell will not be shown in the HTML version of this notebook
order = 2; num_words = 30;
demo = markov_words_demo.show_order(csvname,order,num_words)
```

    -------- TRUE TEXT -------
    amount of scientific education to perceive that the grey scale of the thing was no common oxide that the yellowish white metal that gleamed in the crack between the lid and the

    -------- ORDER = 2 MODEL TEXT -------
    amount of scientific education to perceive that the martians had been at work upon my mind was blank wonder my muscles and nerves seemed drained of their houses got back to the

As we increase the order $D$ of the model the generated text will begin to match the original more and more.  For example, by the time we crank up the order to $D = 10$ the text generated by the model is identical to the original.

```python
## This code cell will not be shown in the HTML version of this notebook
order = 10; num_words = 30;
demo = markov_words_demo.show_order(csvname,order,num_words)
```

    -------- TRUE TEXT -------
    the sky and after time their talk died out and gave place to an uneasy state of anticipation several wayfarers came along the lane and of these my brother gathered such news as he could every broken answer he had

    -------- ORDER = 10 MODEL TEXT -------
    the sky and after time their talk died out and gave place to an uneasy state of anticipation several wayfarers came along the lane and of these my brother gathered such news as he could every broken answer he had

Why does this happen?  Notice that when we increase the order of the model the number of unique input sequences proliferates rapidly.  Eventually, when we increase the order enough, there remains only a single exemplar (input/output pair) in the text to construct each associated histogram (i.e., every input sequence used to determine the transition probabilities is *unique*).  Past this point we only have a single example of each input, thus only one choice for its associated output: whatever follows it in the text (with probability $\mathscr{p} = 1$).

#### Example 2.   Generating text character-by-character using a Markov chain

Just modeled text by *words* above using a Markov chain, we can likewise model it via *characters* (indeed we will not repeat the `Python` functionality introduced above for the word-wise Markov example, as it is entirely similar).  For the same intuitive reasons as discussed in the context of the text-wise modeling scheme - characters often logically follow one another in succession - we can model text as a stochastic dynamic system (a Markov chain) over characters as well.

Below we show the result of an order $D = 1$ Markov chain model using the characters instead of words, and the same text (H.G. Well's classic *War of the Worlds*) to calculate our transition probabilities.  Of course using only a single character as precedent we cannot capture much about the text, as reflected in the generated text below.  Here the single character used as input is colored red, with $300$ generated characters using the order $1$ model colored blue.  Note that this means that the first character "n" is plugged into the model and returns the second, here the space character.  This is then plugged in to generate "t", etc.,

```python
## This code cell will not be shown in the HTML version of this notebook
order = 1; num_chars = 300;
demo = markov_chars_demo.show_order(csvname,order,num_chars)
```

    -------- TRUE TEXT -------
    they saw the gaunt figures separating and rising out of the water as they retreated shoreward, and one of them raised the camera-like generator of the heat-ray.  he held it pointing obliquely downward, and a bank of steam sprang from the water at its touch.  it must have driven through the iron of th

    -------- ORDER = 1 MODEL TEXT -------
    the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the t

As we saw in the previous example, as we increase the order $D$ of the model we capture more and more about the text, and can therefore generate more and more meaningful sentences.  For example, below we show the result of an order $D = 5$ model, comparing to the similar chunk of the true text.  With this many characters we actually start to generate a number of real words.

```python
## This code cell will not be shown in the HTML version of this notebook
order = 5; num_chars = 300;
demo = markov_chars_demo.show_order(csvname,order,num_chars)
```

    -------- TRUE TEXT -------
    dawn of the great panic.  london, which had gone to bed on sunday night oblivious and inert, was awakened, in the small hours of monday morning, to a vivid sense of danger.  unable from his window to learn what was happening, my brother went down and out into the street, just as the sky between the parap

    -------- ORDER = 5 MODEL TEXT -------
    dawn grew small hours of the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham road to the street cobham ro

Increasing the order $D = 10$ we can further observe this trend.

```python
## This code cell will not be shown in the HTML version of this notebook
order = 10; num_chars = 300;
demo = markov_chars_demo.show_order(csvname,order,num_chars)
```

    -------- TRUE TEXT -------
     past eight.  we hurried across the exposed bridge, of course, but i noticed floating down the stream a number of red masses, some many feet across.  i did not know what these were there was no time for scrutiny and i put a more horrible interpretation on them than they deserved.  here again on the surrey sid

    -------- ORDER = 10 MODEL TEXT -------
     past eight, when the tragedy happened, and the strange and terrible as was the deputation.  there was a strong feeling in the streets that the martians were setting fire to everything was to be done.  in london that night another invisible to me because it was something of my schoolboy dreams of battle and h

Finally, just as in the word generating case, if we increase the order $D$ past a certain point our model will generate the text exactly.  Below we show the result of an order $D = 50$ model, which generates precisely the true text shown above it.  This happens for exactly the same reasoning given previously in the context of the word based model: as we increase the order of the model the number of unique input sequences balloons rapidly, until each input sequence of the text is *unique*.  This means that there is only one example of each used to determine the transition probabilities, i.e., precisely the one present in the text.

```python
## This code cell will not be shown in the HTML version of this notebook
order = 50; num_chars = 300;
demo = markov_chars_demo.show_order(csvname,order,num_chars)
```

    -------- TRUE TEXT -------
    g down the broad, sunlit roadway, between the tall buildings on each side.  i turned northwards, marvelling, towards the iron gates of hyde park.  i had half a mind to break into the natural history museum and find my way up to the summits of the towers, in order to see across the park.  but i decided to keep to the ground, where quick hiding was p

    -------- ORDER = 50 MODEL TEXT -------
    g down the broad, sunlit roadway, between the tall buildings on each side.  i turned northwards, marvelling, towards the iron gates of hyde park.  i had half a mind to break into the natural history museum and find my way up to the summits of the towers, in order to see across the park.  but i decided to keep to the ground, where quick hiding was p

## Fixed order and limited memory

In both our posts on [dynamic systems with limited memory](https://jermwatt.github.io/control-notes/posts/dynamic_systems_limited_memory/dynamic_systems_limited_memory.html) and deterministic [recurrence relations](https://jermwatt.github.io/control-notes/posts/recurrence_relations/Recurrence_relations.html) we discussed the impact of the finite window size on the "memory" of such systems.  In the present case the consequence of such systems being limited by their order is perhaps most clearly seen by a simple example of a Markov chain model of text.  For example, suppose we have constructed a word-based Markov model of order $D = 1$, whose transition probabilities have been determined using a large text corpus.  We the apply our model to both of the sentences shown below, to predict the word following "is".

 Figure 5:   The main shortcoming of fixed order systems is exemplified in this toy example.  Here we suppose we have an order $D = 1$ model whose transition probabilities have been determined on a large training corpus.  Here we use our order $D = 1$ to predict the next word of each sentence, following the word "is".  However since the model is order $D = 1$ the *same* word will be predicted for each sentence.  Given the different subject / context of each, this will likely mean that at least one of the sentences will not make sense.

The problem here is that - because we have used an order $D = 1$ model - the *same* word will be predicted to follow the word "is" in both sentences.  This will likely mean that at least one (if not both) of the completed sentences will not make sense, since they have completely different subjects.  Because a fixed order dynamic system is limited by its order, and cannot use any information from earlier in a sequence, this problem can arise regardless of the order $D$ we choose.

## Markov chains with one-hot encoded vectors

```python
## This code cell will not be shown in the HTML version of this notebook
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

def show_series(series,chars):
    num_chars,num_samples = series.shape

    fig, ax = plt.subplots(figsize = (8,5))
    ax = plt.subplot(111)
    a,b = np.meshgrid(np.arange(num_samples),np.arange(num_chars))

    ### x-axis Customize minor tick labels ###
    # make custom labels
    x_ticker_range = np.arange(0.5,num_samples,1).tolist()
    x_char_range = chars[:num_samples]
    ax.xaxis.tick_top()

    ## assign major or minor ticklabels? - chosen major by default
    ax.xaxis.set_major_locator(ticker.FixedLocator(x_ticker_range))
    ax.xaxis.set_major_formatter(ticker.FixedFormatter(x_char_range))

    ### y-axis Customize minor tick labels ###
    # make custom labels
    y_char_range = np.unique(chars)
    num_chars = np.size(y_char_range)
    y_ticker_range = np.arange(0.5,num_chars,1).tolist()

    ## assign major or minor ticklabels? - chosen major by default
    ax.yaxis.set_major_locator(ticker.FixedLocator(y_ticker_range))
    ax.yaxis.set_major_formatter(ticker.FixedFormatter(y_char_range))

    cdict = {
  'red'  :  ( (0.0, 0.25, .25), (0.02, .59, .59), (1., 1., 1.)),
  'green':  ( (0.0, 0.0, 0.0), (0.02, .45, .45), (1., .97, .97)),
  'blue' :  ( (0.0, 1.0, 1.0), (0.02, .75, .75), (1., 0.8, 0.45))
    }

    ax.pcolormesh(a, b, -series,cmap = 'hot',edgecolor = 'k') # hot, gist_heat, cubehelix
    plt.show()

# parse an input sequence
def window_series(x,order):
    # containers for input/output pairs
    x_in = []
    x_out = []
    T = x.size

    # window data
    for t in range(T - order):
        # get input sequence
        temp_in = x[:,t:t + order]
        x_in.append(temp_in)

        # get corresponding target
        temp_out = x[:,t + order]
        x_out.append(temp_out)

    # make array and cut out redundant dimensions
    x_in = np.array(x_in)
    x_in = x_in.swapaxes(0,1)[0,:,:].T
    x_out = np.array(x_out).T
    return x_in,x_out

# transform character-based input/output into equivalent numerical versions
def encode_io_pairs_fixed(keys,order):
    # count the number of unique characters in the text
    keys = np.array(keys)[np.newaxis,:]
    unique_keys = np.unique(keys)
    num_keys = np.size(unique_keys)

    # window series
    x,y = window_series(keys,order)

    # dimensions of windowed data
    order,num_data = x.shape

    # loop over inputs/outputs and tranform and store in x
    x_onehot = []
    for n in range(num_data):
        temp = np.zeros((order,num_keys))
        for o in range(order):
            temp[o,x[:,n][o]] = 1
        x_onehot.append(temp.flatten())
    return np.array(x_onehot).T,y

# pre-process text
csvname = datapath + "war_of_the_worlds.txt"
text = text_parsing_utils.load_preprocess(csvname)

# parse into characters
chars,keys,chars_to_keys,keys_to_chars = text_parsing_utils.parse_chars(text)
x,y = encode_io_pairs_fixed(keys,1)

x_sample = x[:,:50]
y_sample =  y[:,:50]
```

In discussing Markov chains its pretty commonplace to represent each fundamental unit (that is each word or character) by a numerical value.  For example if we were working on the character level we could represent each character by a unique integer - this is often called "token-izing" the characters of a text (we did this in our implementation above).  Another common numerical translation is the "one-hot encoding" scheme where we translated each character's unique token into a standard basis vector, as shown for a few characters below.

\begin{equation}
a \longrightarrow 1 \longrightarrow \begin{bmatrix} 1 \\ 0 \\ 0  \\ \vdots \\ 0 \end{bmatrix} \,\,\,\,  \,\,\,\,  \,\,\,\, b \longrightarrow 2 \longrightarrow \begin{bmatrix} 0 \\ 1 \\ 0  \\ \vdots \\ 0 \end{bmatrix} \,\,\,\,  \,\,\,\,  \,\,\,\, c \longrightarrow 3 \longrightarrow \begin{bmatrix} 0 \\ 0 \\ 1  \\ \vdots \\ 0 \end{bmatrix}
\end{equation}

We could then visualize the first bit of text from our previously used test corpus - The War of the Worlds by H.G. Wells - as shown below.  In this matrix the entire set of unique characters in the text are listed horizontally along the left-hand side, the first few characters of the text are listed along the top, and their one-hot encoded versions of the first few characters are shown as columns with the corresponding square of the column darkened in.

```python
## This code cell will not be shown in the HTML version of this notebook
# show one-hot encoded characters from test corpus
show_series(x_sample,chars)
```

    <IPython.core.display.Javascript object>

If we *one-hot encode* characters (or words) we can pull apart we can pull apart the Markov update formula in Equation (6) quite nicely.  First if denote by $x_p$ the integer token of the $p^{th}$ character in a text, we produce its equivalent one-hot encoded vector $\text{x}_p$ via an $\text{encoder}$ function

\begin{equation}
\mathbf{x}_p = \text{encoder}\left(x_p\right).
\end{equation}

If we suppose that we have one-hot encoded all characters of a text, then the average of a numerical vector representing a $D$ length window of characters $\mathbf{x}_{p-1},\,\mathbf{x}_{p-2},\,...,\mathbf{x}_{p-D}$ as

\begin{equation}
\mathbf{a}_p = \frac{\mathbf{x}_{p-1}+\mathbf{x}_{p-2} +\,...\,+\mathbf{x}_{p-D}}{D}
\end{equation}

uniquely defines this sequence of one-hot encoded characters.

The histogram of one-hot encoded charaters following this sequence / uniquely related to their average $\mathbf{a}_p$ is simply an average

\begin{equation}
\mathbf{b}_p = \frac{1}{\vert\Omega\vert}\sum_{j \in \Omega}\mathbf{x}_j
\end{equation}

where $\Omega = \left\{j \,\,\vert  \,\, \text{if} \,\, \mathbf{x}_j \,\,\text{follows} \,\, \mathbf{a}_p   \right\}$ is an index set of all (one-hot encoded) characters following $\mathbf{a}_p$ in a large corpus.

To select the next most probable word we then determine the index of the *largest* value from this average - which we can express as

\begin{equation}
h_p = \underset{j}{\text{argmax}}\,\, \left(\mathbf{b}_p\right)
\end{equation}

where here the $\text{argmax}$ returns $h_p$ the index of the largest entry of the histogram $\mathbf{z}_p$.  If we want our prediction in one-hot encoded form we simply pass the index above through our encoder function as

\begin{equation}
\mathbf{h}_p = \text{encoder}\left(h_p\right).
\end{equation}

## Markov chains with unlimited memory

If we were to replace the moving average process component above $\mathbf{a}_p = \frac{\mathbf{x}_{p-1},\,\mathbf{x}_{p-2},\,...,\,\mathbf{x}_{p-D}}{D}$ with a [dynamic system with unlimited memory](https://jermwatt.github.io/control-notes/posts/dynamic_systems_unlimited_memory/dynamic_systems_unlimited_memory.html) like the simple exponential average $\mathbf{a}_p = \alpha\mathbf{a}_{p-1} + \left(1 - \alpha\right)\mathbf{x}_p$ (where $0 \leq \alpha \leq 1$ we an analogous kind of Markov chain with "unlimited memory".  Such a Markov chain - technically speaking - makes predictions about future characters based on the entire history of input characters of a text.

When $\alpha$ is set to smaller values the exponential average looks more and more like the most recent character $\mathbf{x}_p$ (and more similar to an order $D = 1$ limited memory dynamic system).  As $\alpha$ is increased a greater amount historical context is wrapped up into the summarizing variable $\mathbf{z}_p$.  Below we plot the exponential average $\mathbf{z}_p$ of the same set of one-hot encoded characters from our test corpus above.  Here we can see how the exponential average with $\alpha = 0.8$ smears out the content of this first bit of text as it progresses, dragging along historical context.

```python
## This code cell will not be shown in the HTML version of this notebook
# running mean
def running_mean(x,alpha):
    # set initial conditions of h to values of x
    h = [x[:,0]]

    # range over x and create h
    for p in range(1,np.shape(x)[1]):
        # get current point and prior hidden state
        h_p_prev = h[-1]
        x_p = x[:,p]

        # make next element and store
        h_p = alpha*h_p_prev + (1 - alpha)*x_p
        h.append(h_p)
    return np.array(h).T

h = running_mean(x_sample,alpha = 0.8)
show_series(h,chars)
```

    <IPython.core.display.Javascript object>


# mc_compute_stationary_mpmath: demonstration

(Using sympy.mpmath)

```
import numpy as np
from mc_compute_stationary_mpmath import mc_compute_stationary_mpmath
```

```
A = [[.4, .6], [.2, .8]]
mc_compute_stationary_mpmath(A)  # default precision = 17
```

    [array([mpf('0.25'), mpf('0.75')], dtype=object)]

```
type(_[0][0])
```

    sympy.mpmath.ctx_mp_python.mpf

```
mc_compute_stationary_mpmath(A, precision=100)
```

    [array([mpf('0.25'), mpf('0.75')], dtype=object)]

```
B = np.identity(2)
mc_compute_stationary_mpmath(B)
```

    [array([mpf('1.0'), mpf('0.0')], dtype=object),
     array([mpf('0.0'), mpf('1.0')], dtype=object)]

```
C = np.identity(3)
mc_compute_stationary_mpmath(C)
```

    [array([mpf('1.0'), mpf('0.0'), mpf('0.0')], dtype=object),
     array([mpf('0.0'), mpf('1.0'), mpf('0.0')], dtype=object),
     array([mpf('0.0'), mpf('0.0'), mpf('1.0')], dtype=object)]

## KMR Markov matrices

Let us try matrices generated by the KMR model
where action 1 is risk-dominant,
so that the theory says that the unique stationary distribution is close to
the distribution that concentrates on the state $N$
(where the state is the number of players playing action 1).

```
from test_mc_compute_stationary import KMR_Markov_matrix_sequential, KMR_Markov_matrix_simultaneous
```

```
P = KMR_Markov_matrix_sequential(N=3, p=1./3, epsilon=1e-14)
eigvecs_P = mc_compute_stationary_mpmath(P)
```

```
eigvecs_P
```

    [array([mpf('5.0007905383225959e-15'), mpf('7.5003952691612597e-29'),
            mpf('1.4999999999999773e-14'), mpf('0.99999999999998')], dtype=object)]

```
[np.dot(vec, P) for vec in eigvecs_P]
```

    [array([mpf('5.0007905383225958e-15'), mpf('7.5003952691612591e-29'),
            mpf('1.4999999999999772e-14'), mpf('0.99999999999998002')], dtype=object)]

```
Q = KMR_Markov_matrix_simultaneous(N=5, p=1./3, epsilon=1e-15)
mc_compute_stationary_mpmath(Q)
```

    [array([mpf('5.2710989716153529e-77'), mpf('4.7477838728799111e-61'),
            mpf('1.710569414459031e-45'), mpf('3.0814879110196131e-30'),
            mpf('2.775557561562889e-15'), mpf('0.99999999999999722')], dtype=object)]

```
mc_compute_stationary_mpmath(Q, precision=100)
```

    [array([mpf('8.8722137051968959e-45'), mpf('2.2655312650280333e-59'),
            mpf('1.710569414459031e-45'), mpf('3.0814879110196131e-30'),
            mpf('2.775557561562889e-15'), mpf('0.99999999999999722')], dtype=object)]

```
R = KMR_Markov_matrix_sequential(N=27, p=1./3, epsilon=1e-2)
eigvecs_R = mc_compute_stationary_mpmath(R)
```

```
eigvecs_R
```

    [array([mpf('1.7806708906869897e-21'), mpf('2.4159856305803426e-22'),
            mpf('1.578282070228375e-23'), mpf('6.6092213996170641e-25'),
            mpf('1.9927300702419417e-26'), mpf('4.6063107155823758e-28'),
            mpf('8.4873397144475981e-30'), mpf('1.2794987709652257e-31'),
            mpf('1.6074386201240665e-33'), mpf('1.707766490924184e-35'),
            mpf('6.1216883692955772e-33'), mpf('1.8827013498726174e-30'),
            mpf('4.9954342797262025e-28'), mpf('1.1470285634880882e-25'),
            mpf('2.282586841341555e-23'), mpf('3.9367014390337377e-21'),
            mpf('5.8755268977578531e-19'), mpf('7.5656049289364331e-17'),
            mpf('8.3641965603241664e-15'), mpf('7.8843558102845161e-13'),
            mpf('6.2759472249864753e-11'), mpf('4.1630449925743605e-9'),
            mpf('2.259398055060813e-7'), mpf('9.7743524555891698e-6'),
            mpf('0.00032418268977704092'), mpf('0.0077414826318757366'),
            mpf('0.11850423413409787'), mpf('0.87342009602538692')], dtype=object)]

```
[np.dot(vec, R) for vec in eigvecs_R]
```

    [array([mpf('1.7806708906869898e-21'), mpf('2.4159856305803426e-22'),
            mpf('1.5782820702283751e-23'), mpf('6.6092213996170639e-25'),
            mpf('1.9927300702419399e-26'), mpf('4.606310715582319e-28'),
            mpf('8.4873397144518295e-30'), mpf('1.2794987708712456e-31'),
            mpf('1.6074386243394429e-33'), mpf('1.7077650819974548e-35'),
            mpf('6.1216880743481326e-33'), mpf('1.8827013488692391e-30'),
            mpf('4.9954342797235464e-28'), mpf('1.1470285634880883e-25'),
            mpf('2.2825868413415551e-23'), mpf('3.9367014390337374e-21'),
            mpf('5.8755268977578527e-19'), mpf('7.5656049289364331e-17'),
            mpf('8.3641965603241669e-15'), mpf('7.884355810284516e-13'),
            mpf('6.2759472249864745e-11'), mpf('4.16304499257436e-9'),
            mpf('2.2593980550608133e-7'), mpf('9.7743524555891707e-6'),
            mpf('0.00032418268977704094'), mpf('0.0077414826318757358'),
            mpf('0.11850423413409787'), mpf('0.87342009602538684')], dtype=object)]

## mpmath

mc_compute_stationary_mpmath uses `mpmath.eig` from [mpmath](http://mpmath.org/).

```
import sympy.mpmath as mp
```

```
mp.__version__
```

    '0.18'

The working precision is controlled by a context object called `mp`.
See: [Setting the precision](http://mpmath.org/doc/current/basics.html#setting-the-precision)

```
print mp.mp
```

    Mpmath settings:
      mp.prec = 53                [default: 53]
      mp.dps = 15                 [default: 15]
      mp.trap_complex = False     [default: False]

The routine `eig` solves the eigenvalue problem.
See: [The eigenvalue problem](http://mpmath.org/doc/current/matrices.html#the-eigenvalue-problem)

```
E_A, EL_A = mp.eig(mp.matrix(A), left=True, right=False)
E_A, EL_A[0, :]/sum(EL_A[0, :]), EL_A[1, :]/sum(EL_A[1, :])
```

    ([mpf('0.20000000000000001'), mpf('1.0')], matrix(
     [['7.12081624598818e+15', '-7.12081624598818e+15']]), matrix(
     [['0.25', '0.75']]))

Let us see what happens if we increase the precision.
See: [Setting the precision](http://mpmath.org/doc/current/basics.html#setting-the-precision)

```
mp.mp.dps += 15
```

```
E_A, EL_A = mp.eig(mp.matrix(A), left=True, right=False)
E_A, EL_A[0, :]/sum(EL_A[0, :]), EL_A[1, :]/sum(EL_A[1, :])
```

    ([mpf('0.200000000000000024980018054066121'),
      mpf('1.00000000000000004163336342344357')],
     matrix(
     [['14411518807585573.1730559644772', '-14411518807585572.1730559644772']]),
     matrix(
     [['0.250000000000000004336808689942', '0.749999999999999995663191310058']]))

Arithemtic seems to be more accurate if the `mpf` values are created from strings.
See: [Providing correct input](http://mpmath.org/doc/current/basics.html#providing-correct-input)

```
AA = np.array(A).astype('str')
AA
```

    array([['0.4', '0.6'],
           ['0.2', '0.8']],
          dtype='|S32')

```
E_AA, EL_AA = mp.eig(mp.matrix(AA), left=True, right=False)
E_AA, EL_AA
```

    ([mpf('0.200000000000000000000000000000015'), mpf('1.0')], matrix(
     [['0.790569415042094832999723386108', '-0.790569415042094832999723386108'],
      ['0.316227766016837933199889354443', '0.94868329805051379959966806333']]))

```
EL_AA[1, :]/sum(EL_AA[1, :])
```

    matrix(
    [['0.25', '0.75']])

KMR Markov matrices:

```
E_P, EL_P = mp.eig(mp.matrix(P), left=True, right=False)
```

```
E_P, EL_P
```

    ([mpf('-3.99638918679531821285541014806953e-18'),
      mpf('0.999999999999996670663055853461674'),
      mpf('0.66666666666666329765802890809357'),
      mpf('1.00000000000000000399638918679477')],
     matrix(
     [['0.333333333333331091275264305103', '-0.999999999999993333333333333361', '0.999999999999993329336944146565', '-0.333333333333331089943134576171'],
      ['0.999999999999994999999999999925', '4.99999999999999995413265490448e-15', '-4.99999999999999764809418455726e-15', '-0.999999999999994854238180931096'],
      ['4.99999999999994929582193330112e-15', '-4.9999999999999500344731847569e-15', '-0.999999999999985000000000000254', '0.999999999999984829470157119688'],
      ['-5.0000000000001000777919432869e-15', '-7.50000000000011167190087028916e-29', '-1.50000000000000731970634277783e-14', '-0.999999999999999999999999999875']]))

```
E_Q, EL_Q = mp.eig(mp.matrix(Q), left=True, right=False)
```

```
E_Q
```

    [mpf('-1.49690297038509714586703443814896e-46'),
     mpf('1.49690297038508838775163240798528e-46'),
     mpf('-8.23548869832210537509734489159652e-47'),
     mpf('0.999999999999999946487043362145377'),
     mpf('7.00649232162408535461864791644958e-46'),
     mpf('1.00000000000000000000000000000079')]

```
E_R, EL_R = mp.eig(mp.matrix(R), left=True, right=False)
```

```
E_R
```

    [mpf('3.98920755789498887221462802905316e-18'),
     mpf('0.347402974926105442776856364857058'),
     mpf('0.401430314822682992223009748578764'),
     mpf('0.44346370835574638375517773575688'),
     mpf('0.481394103526276933318667259327033'),
     mpf('0.518514068942550355629297463648163'),
     mpf('0.555555410535722954392186901330341'),
     mpf('0.59259258931721140166631097759314'),
     mpf('0.680085877238326372501428175726829'),
     mpf('0.629629629575695062410538976338045'),
     mpf('0.735169997185588140710017801414445'),
     mpf('0.666666666665970246357253600572777'),
     mpf('0.777031877313813931845128452635375'),
     mpf('0.703703703703698035299316901128785'),
     mpf('0.740740740740740683017994145687565'),
     mpf('0.814767949255607752872064249682868'),
     mpf('0.777777777777777774151331208252009'),
     mpf('0.851850411773480265661651320106536'),
     mpf('0.814814814814814765566251882406014'),
     mpf('0.888888865223574719753626902501014'),
     mpf('0.851851851851851824978215819009429'),
     mpf('0.925925925713011059448614773704718'),
     mpf('0.962962962961977024988484907626878'),
     mpf('0.88888888888888886677964618523252'),
     mpf('0.925925925925925916778464982765757'),
     mpf('0.962962962962962979355951480225381'),
     mpf('0.999999999999998167348029338613544'),
     mpf('1.00000000000000000015687216788222')]

```

```


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

# Monte Carlo Sampling Methods

So far, we have studied analytical ways to
transform random variables and how
to augment these methods using Python. In
spite of all this, we frequently must
resort to purely numerical methods to
solve real-world problems.  Hopefully,
now that we have seen the deeper theory,
these numerical methods will feel more
concrete.  Suppose we want to generate
samples of a given density, $f(x)$,
given we already can generate samples from a
uniform distribution,
$\mathcal{U}[0,1]$.  How do we know a random sample $v$
comes from the $f(x)$
distribution?  One approach is to look at how a histogram
of samples of $v$
approximates $f(x)$. Specifically,

$$
\begin{equation}
\mathbb{P}( v \in N_{\Delta}(x) )  = f(x) \Delta x
\end{equation}
\label{eq:mc01} \tag{1}
$$

The histogram approximates the
target probability density.

which says that the probability that a sample is in some $N_\Delta$
neighborhood
of $x$ is approximately $f(x)\Delta x$. [Figure](#fig:Sampling_Monte_Carlo_000)
shows the target probability density function
$f(x)$ and a histogram that
approximates it. The histogram is generated from
samples $v$. The hatched
rectangle in the center illustrates Equation
[1](#eq:mc01). The area of this
rectangle is approximately $f(x)\Delta x$ where
$x=0$, in this case. The width
of the rectangle is $N_{\Delta}(x)$ The quality
of the approximation may be
clear visually, but to know that $v$ samples are
characterized by $f(x)$, we
need the statement of Equation [1](#eq:mc01), which
says that the proportion of
samples $v$  that fill the hatched rectangle is
approximately equal to
$f(x)\Delta x$.

Now that we know how to evaluate samples $v$ that are
characterized by the density
$f(x)$, let's consider how to create these samples
for both discrete and
continuous random variables.

## Inverse CDF Method for
Discrete Variables

Suppose we want to generate samples from a fair six-sided
die.  Our workhouse
uniform random variable is defined continuously over the
unit interval  and the
fair six-sided die is discrete. We must first create a
mapping between the
continuous random variable $u$ and the discrete outcomes of
the die. This
mapping is shown in [Figure](#fig:Sampling_Monte_Carlo_0001) where
the unit
interval is broken up into segments, each of length $1/6$. Each
individual
segment is assigned to one of the die outcomes. For example, if $u
\in
[1/6,2/6)$, then the outcome for the die is $2$. Because the die is fair,
all
segments on the unit interval are the same length. Thus, our new random
variable $v$ is derived from $u$ by this assignment.

A uniform distribution random
variable on the unit interval is assigned to the six outcomes of a fair die
using these segements.

For
example, for $v=2$, we have,

$$
\mathbb{P}(v=2) = \mathbb{P}(u\in [1/6,2/6)) = 1/6
$$

 where, in the language of the Equation [1](#eq:mc01), $f(x)=1$
(uniform
distribution), $\Delta x = 1/6$, and $N_\Delta (2)=[1/6,2/6)$.
Naturally, this
pattern holds for all the other die outcomes in
$\left\{1,2,3,..,6\right\}$.
Let's consider a quick simulation to make this
concrete. The following code
generates uniform random samples and stacks them
in a Pandas dataframe.

```python
import pandas as pd
import numpy as np
from pandas import DataFrame
u= np.random.rand(100)
df = DataFrame(data=u,columns=['u'])
```

The next block uses `pd.cut` to map the individual samples to
the set
$\left\{1,2,\ldots,6\right\}$ labeled `v`.

```python
labels = [1,2,3,4,5,6]
df['v']=pd.cut(df.u,np.linspace(0,1,7),
               include_lowest=True,labels=labels)
```

This is what the dataframe contains. The `v` column contains
the samples drawn
from the fair die.

```python
df.head()
```

      u
      v

      0
      0.596534
      4

      1
      0.410969
      3

      2
      0.416233
      3

      3
      0.171567
      2

      4
      0.716067
      5

The following is a count of the number of samples in each group. There
should
be roughly the same number of samples in each group because the die is fair.

```python
df.groupby('v').count()
```

      u

      v

      1
      11

      2
      19

      3
      18

      4
      17

      5
      16

      6
      19

So far, so good. We now have a way to simulate a fair
die from a uniformly
distributed random variable.

To extend this to unfair die, we need only make
some small adjustments to this
code.  For example, suppose that we want an
unfair die so that
$\mathbb{P}(1)=\mathbb{P}(2)=\mathbb{P}(3)=1/12$ and
$\mathbb{P}(4)=\mathbb{P}(5)=\mathbb{P}(6)=1/4$. The only change we have to
make
is with `pd.cut` as follows,

```python
df['v']=pd.cut(df.u,[0,1/12,2/12,3/12,2/4,3/4,1],
               include_lowest=True,labels=labels)
df.groupby('v').count()/df.shape[0]
```

      u

      v

      1
      0.06

      2
      0.05

      3
      0.10

      4
      0.27

      5
      0.24

      6
      0.28

where now these are the individual probabilities of each digit. You
can take
more than `100` samples to get a clearer view of the individual
probabilities
but the mechanism for generating them is the same. The method is
called the
inverse CDF [^CDF] method because the CDF
(namely,$\texttt{[0,1/12,2/12,3/12,2/4,3/4,1]}$) in the last example has been
inverted (using the `pd.cut` method) to generate the samples.
The inversion is
easier to see for continuous variables, which we consider
next.

[^CDF]:
Cumulative density function. Namely, $F(x)=\mathbb{P}(X < x)$.

## Inverse CDF
Method for Continuous Variables

The method above applies to continuous random
variables, but now we have to
squeeze the  intervals down to individual points.
In the example above, our
inverse function was a piecewise function that
operated on uniform random
samples. In this case, the piecewise function
collapses to a continuous inverse
function.  We want to generate random samples
for a CDF that is invertible.
As before, the criterion for generating an
appropriate sample $v$ is the
following,

$$
\mathbb{P}(F(x) < v < F(x+\Delta x)) =  F(x+\Delta x) - F(x) =
\int_x^{x+\Delta x} f(u) du \approx  f(x) \Delta x
$$

 which says that the probability that the sample $v$ is contained in a
$\Delta
x$ interval is approximately equal to $f(x) \Delta x$, at that point.
Once
again, the trick is to use a uniform random sample $u$ and an invertible
CDF
$F(x)$ to construct these samples.  Note that for a uniform random variable
$u
\sim \mathcal{U}[0,1]$, we have,

$$
\begin{align*}
\mathbb{P}(x < F^{-1}(u) < x+\Delta x) & = \mathbb{P}(F(x) < u
< F(x+\Delta x)) \\\
                                       & =  F(x+\Delta x) -
F(x) \\\
                                       & = \int_x^{x+\Delta x} f(p) dp
\approx  f(x) \Delta x
\end{align*}
$$

 This means that $ v=F^{-1}(u) $ is distributed according to $f(x)$,
which is
what we want.

Let's try this to generate samples from the
exponential
distribution,

$$
f_{\alpha}(x) = \alpha e^{ -\alpha x }
$$

 which has the following CDF,

$$
F(x) = 1-e^{ -\alpha x  }
$$

 and corresponding inverse,

$$
F^{-1}(u)  = \frac{1}{\alpha}\ln \frac{1}{(1-u)}
$$

 Now, all we have to do is generate some uniformly distributed
random samples
and then feed them into $F^{-1}$.

```python
from numpy import array, log
import scipy.stats
alpha = 1.   # distribution parameter
nsamp = 1000 # num of samples
# define uniform random variable
u=scipy.stats.uniform(0,1)
# define inverse function
Finv=lambda u: 1/alpha*log(1/(1-u))
# apply inverse function to samples
v = array(list(map(Finv,u.rvs(nsamp))))
```

Now, we have the samples from the exponential distribution, but how
do we know
the method is correct with samples distributed accordingly?
Fortunately,
`scipy.stats` already has a exponential distribution, so we can
check our work
against the reference using a *probability plot* (i.e.,  also
known as a
*quantile-quantile* plot).  The following code sets up the
probability plot from
`scipy.stats`.

```python
%matplotlib inline

from matplotlib.pylab import setp, subplots
fig,ax = subplots()
fig.set_size_inches((7,5))
_=scipy.stats.probplot(v,(1,),dist='expon',plot=ax)
line=ax.get_lines()[0]
_=setp(line,'color','k')
_=setp(line,'alpha',.1)
line=ax.get_lines()[1]
_=setp(line,'color','gray')
_=setp(line,'lw',3.0)
_=setp(ax.yaxis.get_label(),'fontsize',18)
_=setp(ax.xaxis.get_label(),'fontsize',18)
_=ax.set_title('Probability Plot',fontsize=18)
_=ax.grid()
fig.tight_layout()
fig.savefig('fig-probability/Sampling_Monte_Carlo_005.png')
```

```python
fig,ax=subplots()
scipy.stats.probplot(v,(1,),dist='expon',plot=ax)
```

Note that we have to supply an axes object (`ax`) for it to draw on.
The result
is [Figure](#fig:Sampling_Monte_Carlo_005). The more the samples
line match the
diagonal line, the more they match the reference distribution
(i.e., exponential
distribution in this case). You may also want to try
`dist=norm` in the code
above To see what happens when the normal distribution
is the reference
distribution.

The samples created using the
inverse cdf method match the exponential reference distribution.

## Rejection Method

In some cases, inverting the CDF may be impossible.
The *rejection*
method can handle this situation. The idea is to pick two
uniform random
variables $u_1,u_2 \sim \mathcal{U}[a,b]$ so that

$$
\mathbb{P}\left(u_1 \in N_{\Delta}(x) \bigwedge u_2 < \frac{f(u_1)}{M}
\right) \hspace{0.5em} \approx \frac{\Delta x}{b-a} \frac{f(u_1)}{M}
$$

 where we take $x=u_1$ and $f(x) < M $.  This is a two-step process.
First, draw
$u_1$ uniformly from the interval $[a,b]$. Second,  feed it into
$f(x)$ and if
$u_2 < f(u_1)/M$, then you have a valid sample for $f(x)$. Thus,
$u_1$ is the
proposed sample from $f$ that may or may not be rejected depending
on $u_2$. The
only job of the $M$ constant is to scale down the $f(x)$ so that
the $u_2$
variable can span the range.  The *efficiency* of this method is the
probability
of accepting $u_1$ which comes from integrating out the above
approximation,

$$
\int \frac{f(x)}{M(b-a)} dx = \frac{1}{M(b-a)} \int f(x)dx =\frac{1}{M(b-a)}
$$

 This means that we don't want an unecessarily large $M$ because that
makes it
more likely that samples will be discarded.

Let's try this method for a
density that does not have a continuous inverse [^normalization].
[^normalization]: Note that this example density does not *exactly* integrate
out to one like a probability density function should, but the normalization
constant for this is distracting for our purposes here.

$$
f(x) = \exp\left(-\frac{(x-1)^2}{2x} \right) (x+1)/12
$$

 where $x>0$. The following code implements the rejection plan.

```python
import numpy as np
x = np.linspace(0.001,15,100)
f= lambda x: np.exp(-(x-1)**2/2./x)*(x+1)/12.
fx = f(x)
M=0.3                          # scale factor
u1 = np.random.rand(10000)*15  # uniform random samples scaled out
u2 = np.random.rand(10000)     # uniform random samples
idx,= np.where(u2<=f(u1)/M)    # rejection criterion
v = u1[idx]
```

```python
fig,ax=subplots()
fig.set_size_inches((9,5))
_=ax.hist(v,density=True,bins=40,alpha=.3,color='gray')
_=ax.plot(x,fx,'k',lw=3.,label='$f(x)$')
_=ax.set_title('Estimated Efficency=%3.1f%%'%(100*len(v)/len(u1)),
             fontsize=18)
_=ax.legend(fontsize=18)
_=ax.set_xlabel('$x$',fontsize=24)
fig.tight_layout()
fig.savefig('fig-probability/Sampling_Monte_Carlo_007.png')
```

[Figure](#fig:Sampling_Monte_Carlo_007) shows a histogram of the
so-generated
samples that nicely fits the probability density function. The
title in the
figure shows the efficiency (the number of rejected samples),
which is poor. It
means that we threw away most of the proposed samples.  Thus,
even though there
is nothing conceptually wrong with this result, the low
efficiency must be
fixed, as a practical matter.  [Figure](#fig:Sampling_Monte_Carlo_008) shows
where the proposed samples were
rejected. Samples under the curve were retained
(i.e., $u_2 <
\frac{f(u_1)}{M}$) but the vast majority of the samples are
outside this
umbrella.

The rejection method generate
samples in the histogram that nicely match the target distribution.
Unfortunately, the efficiency is not so good.

```python
fig,ax=subplots()
fig.set_size_inches((9,5))
_=ax.plot(u1,u2,'+',label='rejected',alpha=.3,color='gray')
_=ax.plot(u1[idx],u2[idx],'.',label='accepted',alpha=.3,color='k')
_=ax.legend(fontsize=22)
fig.tight_layout()
fig.savefig('fig-probability/Sampling_Monte_Carlo_008.png')
```

The proposed samples under the
curve were accepted and the others were not. This shows the majority of samples
were rejected.

The rejection method uses $u_1$ to select
along the domain of $f(x)$ and the
other $u_2$ uniform random variable  decides
whether to accept or not. One idea
would be to choose $u_1$ so that $x$ values
are coincidentally those that are
near the peak of $f(x)$, instead of uniformly
anywhere in the domain,
especially near the tails, which are low probability
anyway. Now, the trick is
to find a new density function $g(x)$ to sample from
that has a similiar
concentration of probability density. One way it to
familiarize oneself with
the probability density functions that have adjustable
parameters and fast random
sample generators already. There are lots of places
to look and, chances are,
there is likely already such a generator for your
problem. Otherwise, the
family of $\beta$ densities is a good place to start.
To be explicit, what we want is $u_1 \sim g(x)$ so that, returning to our
earlier argument,

$$
\mathbb{P}\left( u_1 \in N_{\Delta}(x) \bigwedge u_2 < \frac{f(u_1)}{M}
\right) \approx g(x) \Delta x \frac{f(u_1)}{M}
$$

 but this is *not* what we need here. The problem is with the
second part of the
logical $\bigwedge$ conjunction. We need to put
something there that will give
us something proportional to $f(x)$.
Let us  define the following,

$$
\begin{equation}
 h(x) = \frac{f(x)}{g(x)}
\end{equation}
\label{eq:rej01}
\tag{2}
$$

 with corresponding maximum on the domain as $h_{\max}$ and
then go back and
construct the second part of the clause as

$$
\mathbb{P}\left(u_1 \in N_{\Delta}(x) \bigwedge u_2 < \frac{h(u_1)}{h_{\max}}
\right) \approx g(x) \Delta x \frac{h(u_1)}{h_{\max}} = f(x)/h_{\max}
$$

 Recall that satisfying this criterion means that $u_1=x$. As before,
we can
estimate the probability of acceptance of the $u_1$ as $1/h_{\max}$.

Now, how
to construct the $g(x)$ function in the denominator of Equation
[2](#eq:rej01)?
Here's where familarity with some standard probability densities
pays off. For
this case, we choose the $\chi^2$ distribution. The following
plots the $g(x)$
and $f(x)$ (left plot) and the corresponding $h(x)=f(x)/g(x)$
(right plot). Note
that $g(x)$ and $f(x)$ have peaks that almost coincide,
which is what we are
looking for.

```python
ch=scipy.stats.chi2(4) # chi-squared
h = lambda x: f(x)/ch.pdf(x) # h-function
```

```python
fig,axs=subplots(1,2,sharex=True)
fig.set_size_inches(12,4)
_=axs[0].plot(x,fx,label='$f(x)$',color='k')
_=axs[0].plot(x,ch.pdf(x),'--',lw=2,label='$g(x)$',color='gray')
_=axs[0].legend(loc=0,fontsize=24)
_=axs[0].set_xlabel(r'$x$',fontsize=22)
_=axs[1].plot(x,h(x),'-k',lw=3)
_=axs[1].set_title('$h(x)=f(x)/g(x)$',fontsize=24)
_=axs[1].set_xlabel(r'$x$',fontsize=22)
fig.tight_layout()
fig.savefig('fig-probability/Sampling_Monte_Carlo_009.png')
```

The plot on the right shows
$h(x)=f(x)/g(x)$ and the one on the left shows $f(x)$ and $g(x)$ separately.

 Now, let's generate  some samples from this $\chi^2$
distribution
with the rejection method.

```python
hmax=h(x).max()
u1 = ch.rvs(5000)        # samples from chi-square distribution
u2 = np.random.rand(5000)# uniform random samples
idx = (u2 <= h(u1)/hmax) # rejection criterion
v = u1[idx]              # keep these only
```

```python
fig,ax=subplots()
fig.set_size_inches((7,3))
_=ax.hist(v,density=True,bins=40,alpha=.3,color='gray')
_=ax.plot(x,fx,color='k',lw=3.,label='$f(x)$')
_=ax.set_title('Estimated Efficency=%3.1f%%'%(100*len(v)/len(u1)))
_=ax.axis(xmax=15)
_=ax.legend(fontsize=18)
fig.savefig('fig-probability/Sampling_Monte_Carlo_010.png')
```

Using the updated method, the
histogram matches the target probability density function with high
efficiency.

Using the $\chi^2$ distribution with the
rejection method results in throwing
away less than 10% of the generated samples
compared with our prior example
where we threw out at least 80%. This is
dramatically more
efficient!  [Figure](#fig:Sampling_Monte_Carlo_010) shows that
the histogram
and the probability density function match.  For completeness,
[Figure](#fig:Sampling_Monte_Carlo_011) shows the samples with the corresponding
threshold $h(x)/h_{\max}$ that was used to select them.

```python
fig,ax=subplots()
fig.set_size_inches((7,4))
_=ax.plot(u1,u2,'+',label='rejected',alpha=.3,color='gray')
_=ax.plot(u1[idx],u2[idx],'g.',label='accepted',alpha=.3,color='k')
_=ax.plot(x,h(x)/hmax,color='k',lw=3.,label='$h(x)$')
_=ax.legend(fontsize=16,loc=0)
_=ax.set_xlabel('$x$',fontsize=24)
_=ax.set_xlabel('$h(x)$',fontsize=24)
_=ax.axis(xmax=15,ymax=1.1)
fig.tight_layout()
fig.savefig('fig-probability/Sampling_Monte_Carlo_011.png')
```

Fewer proposed points were
rejected in this case, which means better efficiency.

#
Sampling Importance Resampling

An alternative to the Rejection Method that does
not involve rejecting samples
or coming up with $M$ bounds or bounding functions
is the Sampling Importance
Resampling (SIR) method.  Choose a tractable $g$
probability density function
and draw a $n$ samples from it, $\lbrace x_i
\rbrace_{i=1}^n$. Our objective is
to derive samples $f$.  Next, compute the
following,

$$
q_i = \frac{w_i}{\sum w_i}
$$

 where

$$
w_i = \frac{f(x_i)}{g(x_i)}
$$

 The $q_i$ define a probability mass function whose samples
approximate samples
from $f$. To see this, consider,

$$
\begin{align*}
\mathbb{P}(X\le a) =& \sum_{i=1}^n q_i
\mathbb{I}_{(-\infty,a]}(x_i) \\
                   =& \frac{\sum_{i=1}^n w_i
\mathbb{I}_{(-\infty,a]}(x_i)}{\sum_{i=1}^n w_i} \\
                   =&
\frac{\frac{1}{n}\sum_{i=1}^n \frac{f(x_i)}{g(x_i)}
\mathbb{I}_{(-\infty,a]}(x_i)}{\frac{1}{n}\sum_{i=1}^n \frac{f(x_i)}{g(x_i)}}
\end{align*}
$$

 Because the samples are generated from the $g$ probability distribution,
the
numerator is approximately,

$$
\mathbb{E}_g\left(\frac{f(x)}{g(x)}\right) = \int_{-\infty}^a f(x) dx
$$

 which gives

$$
\mathbb{P}(X\le a) =  \int_{-\infty}^a f(x) dx
$$

 which shows that the samples generated this way are $f$-distributed.
Note more
samples have to be generated from this probability mass function  the
further
away $g$ is from the desired function $f$. Further, because there is no
rejection step, we no longer have the issue of efficiency.

For example, let us
choose a beta distribution for $g$, as in the following code,

```python
g = scipy.stats.beta(2,3)
```

This distribution does not bear a strong resemblance to our desired
$f$
function from last section.  as shown in the
[Figure](#fig:Sampling_Monte_Carlo_012) below.  Note that we scaled the domain
of the
beta distribution to get it close to the support of $f$.

```python
fig,ax=subplots(figsize=[10,5])
g = scipy.stats.beta(2,3)
_=ax.plot(x,fx,label='$f(x)$',color='k')
_=ax.plot(np.linspace(0,1,100)*15,1/15*g.pdf(np.linspace(0,1,100)),label='$g(x)$',color='r')
_=ax.legend(fontsize=16)
_=ax.set_xlabel('$x$',fontsize=18)
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
fig.savefig('fig-probability/Sampling_Monte_Carlo_012.png')
```

Histogram of samples generated using SIR comparted to target probability
density function.

 In the next block, we sample from the $g$
distribution and compute
the weights as described above. The final step is to
sample from this new
probability mass function.  The resulting normalized
histogram is shown
compared to the target $f$ probability density function in
[Figure](#fig:Sampling_Monte_Carlo_013).

```python
xi = g.rvs(500)
w = np.array([f(i*15)/g.pdf(i) for i in xi])
fsamples=np.random.choice(xi*15,5000,p = w/w.sum())
```

```python
fig,ax=subplots(figsize=[10,5])
_=ax.hist(fsamples,rwidth=.8,density=True,bins=20,color='gray')
_=ax.plot(x,fx,label='$f(x)$',color='k')
_=ax.legend(fontsize=14)
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
fig.savefig('fig-probability/Sampling_Monte_Carlo_013.png')
```

Histogram and probability density
function using SIR.

In this section, we investigated how to
generate random samples from a given
distribution, beit discrete or continuous.
For the continuous case, the key
issue was whether or not the cumulative density
function had a continuous
inverse. If not, we had to turn to the rejection
method, and find an
appropriate related density that we could easily sample from
to use as part of
a rejection threshold. Finding such a function is an art, but
many families of
probability densities have been studied over the years that
already have fast
random number generators.

The rejection method has many
complicated extensions that involve careful
partitioning of the domains and lots
of special methods for corner cases.
Nonetheless, all of these advanced
techniques are still variations on the same
fundamental theme we illustrated
here [[dunn2011exploring]](#dunn2011exploring),
[[johnson1995continuous]](#johnson1995continuous).


# Vibrations of an elastic rod

We consider the eigenvalue problem associated to the longitudinal vibrations of an elastic rod

\begin{align}
  -\partial_{xx} u = \omega^2 u, \quad \Omega, \quad u_{|\partial \Omega} = 0
\end{align}

where $\Omega = (0, 1)$.

The weak formulation, for $u \in \mathcal{V}_h$, is given by

\begin{align}
  a(u,v) = \omega^2 b(u,v), \quad \forall v \in \mathcal{V}_h
  \label{eq:vibration-wf}
\end{align}

where $a(u,v) = \int_{\Omega} \nabla v \cdot \nabla u~d\Omega$ and $b(u,v) = \int_{\Omega} v u~d\Omega$.

The bilinear forms $a$ and $b$ give the Stiffness and Mass matrices after discretization. Their associated GLT symbols are respectivaly $\mathfrak{s}_p$ and $\mathfrak{m}_p$. Therefor, the eigenvalues will be approximated by a uniform sampling of $\frac{\mathfrak{s}_p}{\mathfrak{m}_p}$.

## Formal model

```python
from sympde.calculus import grad, dot
from sympde.topology import ScalarFunctionSpace
from sympde.topology import Domain
from sympde.topology import elements_of
from sympde.expr     import BilinearForm
from sympde.expr     import integral

DIM = 1
domain = Domain('\Omega', dim=DIM)

V = ScalarFunctionSpace('V', domain)

u,v = elements_of(V, names='u,v')

laplace = BilinearForm((u,v),
                       integral(domain, dot(grad(v), grad(u))))
mass    = BilinearForm((u,v),
                       integral(domain, u*v))
```

## Symbolic expressions

```python
# imports from gelato
from gelato.expr     import gelatize
from gelato.printing import latex
```

We shall need to defined the following symbols

```python
# imports from sympy
from sympy import Symbol
from sympy import ratsimp
from sympy import expand

tx = Symbol("tx")
nx = Symbol("nx", integer=True)
N  = Symbol("N", integer=True)
wl = Symbol('\omega_l')
```

We compute the GLT expression by calling the function **gelatize** for a given degree $p$

```python
p = 3

sl = gelatize(laplace, p)
sm = gelatize(mass, p)

expr = sl/sm
```

Print the latex expression using the **latex** function

```python
from IPython.display import Math
```

```python
Math(latex(ratsimp(expr/nx**2)))
```

$\displaystyle - \frac{- 46872 \cos{\left (\theta_x \right )} + 168 \cos{\left (3 \theta_x \right )} - 59136}{5955 \cos{\left (\theta_x \right )} + 600 \cos{\left (2 \theta_x \right )} + 5 \cos{\left (3 \theta_x \right )} + 6040} - \frac{42}{5}$

```python
expr = expr.subs({tx: wl / N})
expr = expr.subs({nx: N-p}) #N = nx + p
expr = expand(expr)
```

Let us print again our expression in Latex,

```python
Math(latex(expr))
```

$\displaystyle - \frac{N^{2} \cos{\left (\frac{\omega_l}{N} \right )}}{4 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} - \frac{2 N^{2} \cos{\left (\frac{2 \omega_l}{N} \right )}}{5 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} - \frac{N^{2} \cos{\left (\frac{3 \omega_l}{N} \right )}}{60 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} + \frac{2 N^{2}}{3 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} + \frac{3 N \cos{\left (\frac{\omega_l}{N} \right )}}{2 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} + \frac{12 N \cos{\left (\frac{2 \omega_l}{N} \right )}}{5 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} + \frac{N \cos{\left (\frac{3 \omega_l}{N} \right )}}{10 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} - \frac{4 N}{\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}} - \frac{9 \cos{\left (\frac{\omega_l}{N} \right )}}{4 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} - \frac{18 \cos{\left (\frac{2 \omega_l}{N} \right )}}{5 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} - \frac{3 \cos{\left (\frac{3 \omega_l}{N} \right )}}{20 \left(\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}\right)} + \frac{6}{\frac{397 \cos{\left (\frac{\omega_l}{N} \right )}}{840} + \frac{\cos{\left (\frac{2 \omega_l}{N} \right )}}{21} + \frac{\cos{\left (\frac{3 \omega_l}{N} \right )}}{2520} + \frac{151}{315}}$

Well this can lead to a complicated expression. Let us take the expansion of the eigenvalues with respect to $\frac{1}{N}$.

```python
# here we consider an expansion up to the forth order
order = 4
expr = expr.series(1/N, 0, order)
```

Now let us check again the eigenvalue expression

```python
Math(latex(expr))
```

$\displaystyle \frac{- 6 N \omega_l^{2} + 9 \omega_l^{2}}{N^{2}} + \omega_l^{2} + O\left(\frac{1}{N^{4}}; N\rightarrow \infty\right)$

```python
# css style
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


**Authors: Francesco Mannella and Emilio Cartoni

This manuscript describes the leaky Echo-State network together with some implementations and optimizations of the inner weights of the network.

## The leaky Echo-State network

The dynamical system is a recurrent network of leaky units with a sigmoidal output function:

$
\tau\dot{\mathbf{x}} = -\mathbf{x} + \mathbf{W}\tanh\left(\mathbf{x}\right)
$

$
\mathbf{x} \in \mathbb{R}^N,  \mathbf{W} \in \mathbb{R}^{N\times N}
$

The only constraint defining echo-state networks compared to other recurrent networks is the ***ESP*** (Echo-State Property):

$
1-\epsilon <  \rho\left(\mathbf{V}\right)  < 1
$

where $\mathbf{V}$ is the correction of the weight matrix accounting for leakyness (http://goo.gl/bqGAJu)

$
\mathbf{V} = \frac{\delta t}{\tau}\mathbf{W} + \left(1 - \frac{\delta t}{\tau}\right)\mathbf{I}
$

and $\rho\left(\mathbf{V}\right)$ is the spectral radius of the matrix

$
\rho\left(\mathbf{V}\right) = \max_i\left(\left|\lambda^\mathbf{V}_i \right|\right)
$

## The dynamic-rotation trick

Provided that we scale the spectral radius of the weight matrix $\mathbf{W}$ so that it fulfills the ESP clause we can initialize the weights to any set of values.

In particular, we can modulate the ratio between the variance of the real part of the eigenvalues of $\mathbf{W}$ and the variance of their imaginary part.

For every square matrix $\mathbf{W}$ the following holds true:

$\mathbf{W} =\frac{\mathbf{W} + \mathbf{W}^T}{2} + \frac{\mathbf{W} - \mathbf{W}^T}{2}$

where

$\mathbf{W}_{sim} =\frac{\mathbf{W} + \mathbf{W}^T}{2}$

is a simmetric matrix and

$\mathbf{W}_{skew} =\frac{\mathbf{W} - \mathbf{W}^T}{2}$

is a skew matrix.

Skew matrices have pure imaginary eigenvalues.
A dynamical system having a skew matrix as its weight matrix produces only ***infinitesimal rotations***.

On the other side simmetric matrices have pure real eigenvalues.
A dynamical system having a simmetric matrix as its weight matrix produces produces only ***infinitesimal contractions/expansions***.

Controlling the amount of infinitesimal rotation in the behaviour of a reservoir allows to increase the variability in the states of the network narrowing the distance with chaotic behaviour.
As a result the network is sensible to features at a lower time scale (Note: does it mean faster or slower time scale?) and its "memory" interval increases (Note: do we have proofs, references or examples for this?).

Knowing this we can build a new matrix

$\mathbf{W} = \alpha \mathbf{W}_{skew} + (1-\alpha)\mathbf{W}_{sim}$

where $\alpha$ is the proportion of "skewness"
of the matrix.

## Finding the inner weights $\mathbf{W}$

* [ESP: analytical method (the wrong one)](#ESP%3A-analytical-method-%28the-wrong-one%29)
* [ESP: iterative method (the right one)](#ESP%3A-iterative-method-%28the-right-one%29)
* [ESP: analytical method (the RIGHT one)](#ESP%3A-analytical-method-%28the-RIGHT-one%29)
* [Redefining our ESN according to Jaeger 2007](#Redefining-our-ESN-according-to-Jaeger-2007)

#### Common settings

```python
%matplotlib inline
from pylab import *

# costants
#-------------------------------------------------------
n = 200
dt = 0.001
tau = 0.01
epsilon = 1e-60
alpha = 0.1
target = 1-epsilon
itarget = 1.0 - (epsilon/2.0)
#-------------------------------------------------------

# utils
#-------------------------------------------------------
def get_rho_and_eig(X) :
    e = eigvals(X)
    return  max(abs(e)), e

def get_W_from_V(X) :
    return (tau/dt)*(X - (1-(dt/tau))*eye(n,n))

def get_V_from_W(X) :
    return (dt/tau)*X + (1-(dt/tau))*eye(n,n)

def modulate_rotation_contraction(X) :
    W1 = .5*(X-X.T)
    W2 = .5*(X+X.T)

    return alpha*W2 +(1-alpha)*W1

#-------------------------------------------------------
# create initial weights
W = randn(n, n)
W = modulate_rotation_contraction(W)
rho, e = get_rho_and_eig(W)
3
SSW = W.copy()
# store weights scaled tho rho=1
SW = (W/rho)
print "done"
```

    done

### ESP: analytical method (the wrong one)

* ***Init*** $\mathbf{V}$ sampling from $\mathcal{N} (m=0,\sigma=1)$
* ***Scale***  $\mathbf{V}$ so that $1 -\epsilon < \rho(\mathbf{V}) < 1$
* ***Get*** $\mathbf{W}$ from $\mathbf{V}$ : $\mathbf{W} = \frac{\tau}{\delta t}\left(\mathbf{V} - \left(1-\frac{\delta t}{\tau}\right)\mathbf{I} \right)$

#### Why it is the wrong method:
What we are really searching is a relation between the spectral radius of $\mathbf{W}$ and $\mathbf{V}$, instead we are not interested in maintaining other characteristics of $\mathbf{V}$ in $\mathbf{W}$. In particular, $\mathbf{V}$ is obtainied from $\mathbf{W}$ by scaling and translating its diagonal by a constant. Indeed multiplying a square matrix times a scalar means scaling all its eigenvalues, while trans means translating its diagonal by a costant means translating the mean of the real part of its eigenvalues. Thus the method above does a further unwanted translation $ -\frac{\tau}{\delta t}\left(1-\frac{\delta t}{\tau}\right)\mathbf{I}$. This last operation has the catastrophic effect of redefining all weights as negative.

```python
V = get_V_from_W(W)
rho, e = get_rho_and_eig(V)
V = (1-epsilon/2.0)*V/rho
W = get_W_from_V(W)

figure("Method1", figsize=(6,2))
#*****************************
m_rho, m_e = get_rho_and_eig(V)
#*****************************
subplot(121, aspect="equal")
title("eigenvalues of V")
scatter(real(m_e), imag(m_e))
xlim([-m_rho, m_rho])
ylim([-m_rho, m_rho])
#*****************************

#*****************************
rho, e = get_rho_and_eig(W)
#*****************************
subplot(122, aspect="equal")
title("eigenvalues of W")
scatter(real(e), imag(e))
xlim([-rho, rho])
ylim([-rho, rho])
#*****************************

print """
ESP: analytical method (the wrong one)"

W
--------------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}

V
--------------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}
""".format(rho, W.min(), W.max(),
           m_rho, V.min(), V.max())
```

### ESP: iterative method (the right one)

* ***Init*** $\mathbf{SW}$ sampling from $\mathcal{N} (m=0,\sigma=1)$
* ***Scale***  $\mathbf{SW}$ so that $\rho(\mathbf{SW}) = 1$
* *** Do ***
    * ***choose*** an estimate $\hat{\rho}$
    * ***scale*** to  $\mathbf{W} = \hat{\rho}\mathbf{SW}$
    * ***compute*** $\mathbf{V} = \frac{\delta t}{\tau}\mathbf{W} + \left(1 - \frac{\delta t}{\tau}\right)\mathbf{I}$
* *** while *** not $(1-\epsilon < \rho(\mathbf{V}) < 1)$

We can optimize the search using a root-finding method (e.g the [bisection](https://en.wikipedia.org/wiki/Bisection_method) method )

***PROS***: This method finds the correct $\hat{\rho} = \rho{(\mathbf{W})}$ because it searches only in the space of the spectral radii of $\mathbf{W}$. Instead, $\rho(\mathbf{V})$ is only used as a parameter of the objective function of any choosen root-finding iterative method.

***CONS***: This method can be computationally expensive. In particular, the smaller we choose $\epsilon$ the longer it takes (more iterations) to find the solution.

```python
from scipy.optimize import bisect

def dist(rho_estimate) :
    V = get_V_from_W(rho_estimate*SW)
    effective_rho_estimate,_ = get_rho_and_eig(V)
    return itarget - effective_rho_estimate

import time
start = time.clock()

# ---------------------------------------------------------------
rho_estimate = bisect(lambda x: dist(x),
                      1-epsilon, tau/dt,
                      xtol = epsilon/2.)
# ---------------------------------------------------------------

print
print
print "computed in {}s".format(time.clock() - start)
print
print

W = SW*rho_estimate
V = get_V_from_W(W)

figure("Method2")
#*****************************
m_rho, m_e = get_rho_and_eig(V)
#*****************************
subplot(121, aspect="equal")
title("eigenvalues of V")
scatter(real(m_e), imag(m_e))
xlim([-m_rho, m_rho])
ylim([-m_rho, m_rho])
#*****************************

#*****************************
rho, e = get_rho_and_eig(W)
#*****************************
subplot(122, aspect="equal")
title("eigenvalues of W")
scatter(real(e), imag(e))
xlim([-rho, rho])
ylim([-rho, rho])
#*****************************

print """
ESP: iterative method
W
--------------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}

V
--------------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}
""".format(rho, W.min(), W.max(),
           m_rho, V.min(), V.max())
```

### ESP: analytical method (the RIGHT one)

Given that

$\mathbf{V} = \frac{\delta t}{\tau}\hat{\rho}\mathbf{SW} + \left(1 - \frac{\delta t}{\tau}\right)\mathbf{I}$

where

$\left\{\mathbf{SW}\mid\mathbf{SW}\in\mathbb{R}^{n\times  n}\wedge\rho(\mathbf{SW}) = 1\right\}$

$\left\{\hat{\rho}\mid  \hat{\rho}\in \mathbb{R}_{>0}\wedge \rho(\mathbf{V}) = 1-\frac{\epsilon}{2}\right\}$

and $(\mathbf{v}\circ\mathbf{v})$ is the [Hadamard product
](https://en.wikipedia.org/wiki/Hadamard_product_%28matrices%29#Definition).

The vector of eigenvalues of $\mathbf{V}$ is

$\mathbf{\Lambda}^V = (\lambda^V_0,\lambda^V_1\dots\lambda^V_n)$

And we can write the absolute values of the eigenvalues as

$
\begin{align}
\left|\mathbf{\Lambda}^V\right| = \sqrt{ \left(h\mathbf{k}\circ\mathbf{x} + \left(1-h\right)\right)^2 + \left(h\mathbf{k} \circ\mathbf{y}\right)^2 }&
\end{align}
$

where

$h = \frac{\delta t}{\tau}$

$\mathbf{x} = \operatorname{Re}(\mathbf{\Lambda}^V)$

$\mathbf{y} = \operatorname{Im}(\mathbf{\Lambda}^V)$

$\mathbf{k} = \hat{\rho}\mathbf{1}_n$

Now we can think $\mathbf{k}$ as a vector of unknown values and
solve for $\mathbf{k}$ the following equation:

$\left|\mathbf{\Lambda}^V\right| =  (1 -\frac{\epsilon}{2})\mathbf{1}_n $

$
\left|\mathbf{\Lambda}^V\right| - (1-\frac{\epsilon}{2})\mathbf{1}_n = 0
$

$
\left(\mathbf{x}^2h^2+\mathbf{y}^2h^2\right)\circ\mathbf{k}^{\circ2} +
\left(2\mathbf{x}h-2\mathbf{x}h^2\right)\circ\mathbf{k} + \left(1+h^2-2h-
\left(1-\frac{\epsilon}{2}\right)^2\mathbf{1}_n\right) = 0
$

where $\mathbf{v}^{\circ2}$ is the [Hadamard power](https://en.wikipedia.org/wiki/Hadamard_product_%28matrices%29#Analogous_operations).

Since this is a quadratic equation we will have two sets of solutions.
We choose as the correct solution the right edge of the intersection
between all the n equations, within which interval all eigenvalue modules are less
than $1 -\frac{\epsilon}{2}$ (red area in the picture below).
The chosen solution (red dot in the figure) ensures that at least one
of the eigenvalues is exactly $1 -\frac{\epsilon}{2}$ and that
this is also the maximum absolute eigenvalue (spectral radius) since
all other values are also $1 -\frac{\epsilon}{2}$ (if they share a solution
in that point) or less than $1 -\frac{\epsilon}{2}$.

```python
from IPython.display import Image
Image('pics/intersection.png', width=500)
```

```python
target = 1 - epsilon/2.

W = SW.copy()

#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§
#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§
#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§
start = time.clock()

# --------------------------------------------------

rho, e = get_rho_and_eig(W)
x = e.real
y = e.imag
h = dt/tau
a = x**2*h**2 + y**2*h**2
b = 2*x*h - 2*x*h**2
c = 1 + h**2 - 2*h - target**2

# just get the positive solutions
sol2 = (-b + sqrt(b**2 - 4*a*c))/(2*a)
# and take the minor amongst them
correctRhoMultiplier = min(sol2)
W = correctRhoMultiplier*W
# --------------------------------------------------
print
print
print "computed in {}s".format(time.clock() - start)
print
print

#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§
#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§
#§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§§

V = get_V_from_W(W)

figure("Method3")
#*****************************
m_rho, m_e = get_rho_and_eig(V)
#*****************************
subplot(121, aspect="equal")
title("eigenvalues of M_tilde")
scatter(real(m_e), imag(m_e))
xlim([-m_rho, m_rho])
ylim([-m_rho, m_rho])
#*****************************

#*****************************
rho, e = get_rho_and_eig(W)
#*****************************
subplot(122, aspect="equal")
title("eigenvalues of W")
scatter(real(e), imag(e))
xlim([-rho, rho])
ylim([-rho, rho])
#*****************************
print """
ESP: analytical method (the RIGHT one)
W
--------------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}

V
--------------------------------
rho := {:16.12f}
min := {:8.4f}  max := {:8.4f}
""".format(rho, W.min(), W.max(),
           m_rho, V.min(), V.max())

```

### Redefining our ESN according to Jaeger 2007

Our ESN has the following update:

$\dot{x_f} = -hx_f+h(I+\rho_fWf(x_f))$

while Jaeger's has the following:

$\dot{x_j} = -a_jx_j+f(I+\rho_jWx_j)$

The presence of $h$ multiplying the activation function f in our update makes it impossible to replicate our network using Jaeger's equation, unless h = 1.
However, it is possible to define Jaeger's parameters $a_j$ and $\rho_j$ so that Jaeger's network will follow the same dynamics of our network, scaled by a factor h.

If $a_j = h$ and $\rho_j = h\rho_f$ then Jaeger's update will be:

$x_j = -hx_j+f(I+h\rho_fWx_j)$

This will produce a network with the same dynamics as our network, but the internal states $x_j$ will be $1/h$ of our network states $x_f$ (so if $h = 0.1$ they will be 1/0.1 = 10 times larger).

See code below for numerical proof.

```python
T = 100
n_test = 20
u = zeros([n,T])
u[:,:10] = np.random.rand(n,10)

x_f = np.zeros((n,T))
x_j = np.zeros((n,T))
x_f[:,0] = 0
x_j[:,0] = 0
for t in range(1,T):
    x_f[:,t] = x_f[:,t-1]-h*x_f[:,t-1]+h*(u[:,t]+W.dot(tanh(x_f[:,t-1])))
    x_j[:,t] = x_j[:,t-1]-h*x_j[:,t-1]+tanh(u[:,t]+h*W.dot(x_j[:,t-1]))

figure("ESN Comparison: same dynamics but 1/h times wider h = %f" % h, figsize=(15,5))
subplot(121)#, aspect="equal")
title("Evolution of our ESN (first %i neurons)" % n_test)
plot(x_f[0:n_test,:].T)

subplot(122)#, aspect="equal")
title("Evolution of Jaeger's ESN (first %i neurons)" % n_test)
plot(x_j[0:n_test,:].T)

tight_layout()

```