# Probabilistic Programming and Bayesian Methods for Hackers Chapter 1

    Run in Google Colab

    View source on GitHub

Original content ([this Jupyter notebook](https://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter1_Introduction/Ch1_Introduction_PyMC2.ipynb)) created by Cam Davidson-Pilon ([`@Cmrn_DP`](https://twitter.com/Cmrn_DP))

Ported to [Tensorflow Probability](https://www.tensorflow.org/probability/) by Matthew McAteer ([`@MatthewMcAteer0`](https://twitter.com/MatthewMcAteer0)) and Bryan Seybold, with help from the TFP team at  Google ([`tfprobability@tensorflow.org`](mailto:tfprobability@tensorflow.org)).

Welcome to Bayesian Methods for Hackers. The full Github repository is available at [github/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers). The other chapters can be found on the project's [homepage](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/). We hope you enjoy the book, and we encourage any contributions!

---
### Table of Contents
- Dependencies & Prerequisites
- The Philosophy of Bayesian Inference
- The Bayesian state of mind
- Bayesian Inference in Practice
- Are frequentist methods incorrect then?
- Our Bayesian framework
- Example: Mandatory coin-flip example
- Example: Bug, or just sweet, unintended feature?
- Probability Distributions
  - Discrete Case
- Continuous Case
- But what is $\lambda \;$?
  - Example: Inferring behaviour from text-message data
- Introducing our first hammer: Tensorflow Probability
- specify the joint log-density
- Specify the posterior sampler
- Execute the TF graph to sample from the posterior
- Plot the Results
- Interpretation
- Exercises
- References

### Dependencies & Prerequisites

    Tensorflow Probability is part of the colab default runtime, so you don't need to install Tensorflow or Tensorflow Probability if you're running this in the colab.

    If you're running this notebook in Jupyter on your own machine (and you have already installed Tensorflow), you can use the following

 For the most recent nightly installation: pip3 install -q tfp-nightly

 For the most recent stable TFP release: pip3 install -q --upgrade tensorflow-probability

 For the most recent stable GPU-connected version of TFP: pip3 install -q --upgrade tensorflow-probability-gpu

 For the most recent nightly GPU-connected version of TFP: pip3 install -q tfp-nightly-gpu

Again, if you are running this in a Colab, Tensorflow and TFP are already installed

```
#@title Imports and Global Variables (make sure to run this cell)  { display-mode: "form" }

try:
  # %tensorflow_version only exists in Colab.
  %tensorflow_version 2.x
except Exception:
  pass

from __future__ import absolute_import, division, print_function

#@markdown This sets the warning status (default is `ignore`, since this notebook runs correctly)
warning_status = "ignore" #@param ["ignore", "always", "module", "once", "default", "error"]
import warnings
warnings.filterwarnings(warning_status)
with warnings.catch_warnings():
    warnings.filterwarnings(warning_status, category=DeprecationWarning)
    warnings.filterwarnings(warning_status, category=UserWarning)

import numpy as np
import os
#@markdown This sets the styles of the plotting (default is styled like plots from [FiveThirtyeight.com](https://fivethirtyeight.com/)
matplotlib_style = 'fivethirtyeight' #@param ['fivethirtyeight', 'bmh', 'ggplot', 'seaborn', 'default', 'Solarize_Light2', 'classic', 'dark_background', 'seaborn-colorblind', 'seaborn-notebook']
import matplotlib.pyplot as plt; plt.style.use(matplotlib_style)
import matplotlib.axes as axes;
from matplotlib.patches import Ellipse
#%matplotlib inline
import seaborn as sns; sns.set_context('notebook')
from IPython.core.pylabtools import figsize
#@markdown This sets the resolution of the plot outputs (`retina` is the highest resolution)
notebook_screen_res = 'retina' #@param ['retina', 'png', 'jpeg', 'svg', 'pdf']
#%config InlineBackend.figure_format = notebook_screen_res

import tensorflow as tf

import tensorflow_probability as tfp
tfd = tfp.distributions
tfb = tfp.bijectors

class _TFColor(object):
    """Enum of colors used in TF docs."""
    red = '#F15854'
    blue = '#5DA5DA'
    orange = '#FAA43A'
    green = '#60BD68'
    pink = '#F17CB0'
    brown = '#B2912F'
    purple = '#B276B2'
    yellow = '#DECF3F'
    gray = '#4D4D4D'
    def __getitem__(self, i):
        return [
            self.red,
            self.orange,
            self.green,
            self.blue,
            self.pink,
            self.brown,
            self.purple,
            self.yellow,
            self.gray,
        ][i % 9]
TFColor = _TFColor()

def session_options(enable_gpu_ram_resizing=True, enable_xla=False):
    """
    Allowing the notebook to make use of GPUs if they're available.

    XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear
    algebra that optimizes TensorFlow computations.
    """
    config = tf.config
    gpu_devices = config.experimental.list_physical_devices('GPU')
    if enable_gpu_ram_resizing:
        for device in gpu_devices:
           tf.config.experimental.set_memory_growth(device, True)
    if enable_xla:
        config.optimizer.set_jit(True)
    return config

session_options(enable_gpu_ram_resizing=True, enable_xla=True)
```

    <module 'tensorflow._api.v2.config' from '/usr/local/lib/python3.7/dist-packages/tensorflow/_api/v2/config/__init__.py'>

## The Philosophy of Bayesian Inference

>You are a skilled programmer, but bugs still slip into your code. After a particularly difficult implementation of an algorithm, you decide to test your code on a trivial example. It passes. You test the code on a harder problem. It passes once again. And it passes the next, even more difficult, test too! You are starting to believe that there may be no bugs in this code...

If you think this way, then congratulations, you already are thinking Bayesian! Bayesian inference is simply updating your beliefs after considering new evidence. A Bayesian can rarely be certain about a result, but he or she can be very confident. Just like in the example above, we can never be 100% sure that our code is bug-free unless we test it on every possible problem; something rarely possible in practice. Instead, we can test it on a large number of problems, and if it succeeds we can feel more confident about our code, but still not certain. Bayesian inference works identically: we update our beliefs about an outcome; rarely can we be absolutely sure unless we rule out all other alternatives.

## The Bayesian state of mind
Bayesian inference differs from more traditional statistical inference by preserving uncertainty. At first, this sounds like a bad statistical technique. Isn't statistics all about deriving certainty from randomness? To reconcile this, we need to start thinking like Bayesians.

The Bayesian world-view interprets probability as measure of believability in an event, that is, how confident we are in an event occurring. In fact, we will see in a moment that this is the natural interpretation of probability.

For this to be clearer, we consider an alternative interpretation of probability: Frequentist, known as the more classical version of statistics, assume that probability is the long-run frequency of events (hence the bestowed title). For example, the probability of plane accidents under a frequentist philosophy is interpreted as the long-term frequency of plane accidents. This makes logical sense for many probabilities of events, but becomes more difficult to understand when events have no long-term frequency of occurrences. Consider: we often assign probabilities to outcomes of presidential elections, but the election itself only happens once! Frequentists get around this by invoking alternative realities and saying across all these realities, the frequency of occurrences defines the probability.

Bayesians, on the other hand, have a more intuitive approach. Bayesians interpret a probability as measure of belief, or confidence, of an event occurring. Simply, a probability is a summary of an opinion. An individual who assigns a belief of 0 to an event has no confidence that the event will occur; conversely, assigning a belief of 1 implies that the individual is absolutely certain of an event occurring. Beliefs between 0 and 1 allow for weightings of other outcomes. This definition agrees with the probability of a plane accident example, for having observed the frequency of plane accidents, an individual's belief should be equal to that frequency, excluding any outside information. Similarly, under this definition of probability being equal to beliefs, it is meaningful to speak about probabilities (beliefs) of presidential election outcomes: how confident are you candidate A will win?

Notice in the paragraph above, I assigned the belief (probability) measure to an individual, not to Nature. This is very interesting, as this definition leaves room for conflicting beliefs between individuals. Again, this is appropriate for what naturally occurs: different individuals have different beliefs of events occurring, because they possess different information about the world. The existence of different beliefs does not imply that anyone is wrong. Consider the following examples demonstrating the relationship between individual beliefs and probabilities:

* I flip a coin, and we both guess the result. We would both agree, assuming the coin is fair, that the probability of Heads is 1/2. Assume, then, that I peek at the coin. Now I know for certain what the result is: I assign probability 1.0 to either Heads or Tails (whichever it is). Now what is your belief that the coin is Heads? My knowledge of the outcome has not changed the coin's results. Thus we assign different probabilities to the result.

* Your code either has a bug in it or not, but we do not know for certain which is true, though we have a belief about the presence or absence of a bug.

* A medical patient is exhibiting symptoms *x*, *y* and *z*. There are a number of diseases that could be causing all of them, but only a single disease is present. A doctor has beliefs about which disease, but a second doctor may have slightly different beliefs.

This philosophy of treating beliefs as probability is natural to humans. We employ it constantly as we interact with the world and only see partial truths, but gather evidence to form beliefs. Alternatively, you have to be trained to think like a frequentist.

To align ourselves with traditional probability notation, we denote our belief about event $A$ as $P(A)$. We call this quantity the prior probability.

John Maynard Keynes, a great economist and thinker, said "When the facts change, I change my mind. What do you do, sir?" This quote reflects the way a Bayesian updates his or her beliefs after seeing evidence. Even — especially — if the evidence is counter to what was initially believed, the evidence cannot be ignored. We denote our updated belief as $P(A|X)$, interpreted as the probability of $A$ given the evidence $X$. We call the updated belief the posterior probability so as to contrast it with the prior probability. For example, consider the posterior probabilities (read: posterior beliefs) of the above examples, after observing some evidence $X$:

1. $P(A)$: the coin has a 50 percent chance of being Heads. $P(A|X)$: You look at the coin, observe a Heads has landed, denote this information  $X$, and trivially assign probability 1.0 to Heads and 0.0 to Tails.

2. $P(A)$: This big, complex code likely has a bug in it. $P(A|X)$: The code passed all $X$ tests; there still might be a bug, but its presence is less likely now.

3. $P(A)$: The patient could have any number of diseases. $P(A|X)$: Performing a blood test generated evidence $X$, ruling out some of the possible diseases from consideration.

It's clear that in each example we did not completely discard the prior belief after seeing new evidence $X$, but we re-weighted the prior to incorporate the new evidence (i.e. we put more weight, or confidence, on some beliefs versus others).

By introducing prior uncertainty about events, we are already admitting that any guess we make is potentially very wrong. After observing data, evidence, or other information, we update our beliefs, and our guess becomes less wrong. This is the alternative side of the prediction coin, where typically we try to be more right.

## Bayesian Inference in Practice
If frequentist and Bayesian inference were programming functions, with inputs being statistical problems, then the two would be different in what they return to the user. The frequentist inference function would return a number, representing an estimate (typically a summary statistic like the sample average etc.), whereas the Bayesian function would return probabilities.

For example, in our debugging problem above, calling the frequentist function with the argument "My code passed all $X$ tests; is my code bug-free?" would return a YES. On the other hand, asking our Bayesian function "Often my code has bugs. My code passed all $X$ tests; is my code bug-free?" would return something very different: probabilities of YES and NO. The function might return:

>YES, with probability 0.8; NO, with probability 0.2

This is very different from the answer the frequentist function returned. Notice that the Bayesian function accepted an additional argument: "Often my code has bugs". This parameter is the prior. By including the prior parameter, we are telling the Bayesian function to include our belief about the situation. Technically this parameter in the Bayesian function is optional, but we will see excluding it has its own consequences.

### Incorporating evidence
As we acquire more and more instances of evidence, our prior belief is washed out by the new evidence. This is to be expected. For example, if your prior belief is something ridiculous, like "I expect the sun to explode today", and each day you are proved wrong, you would hope that any inference would correct you, or at least align your beliefs better. Bayesian inference will correct this belief.

Denote $N$ as the number of instances of evidence we possess. As we gather an infinite amount of evidence, say as $N→∞,$ our Bayesian results (often) align with frequentist results. Hence for large N, statistical inference is more or less objective. On the other hand, for small $N$, inference is much more unstable: frequentist estimates have more variance and larger confidence intervals. This is where Bayesian analysis excels. By introducing a prior, and returning probabilities (instead of a scalar estimate), we preserve the uncertainty that reflects the instability of statistical inference of a small N dataset.

One may think that for large $N$, one can be indifferent between the two techniques since they offer similar inference, and might lean towards the computationally-simpler, frequentist methods. An individual in this position should consider the following quote by Andrew Gelman (2005)[[1]](#scrollTo=nDdph0r1ABCn), before making such a decision:

Sample sizes are never large. If $N$, is too small to get a sufficiently-precise estimate, you need to get more data (or make more assumptions). But once $N$, is "large enough," you can start subdividing the data to learn more (for example, in a public opinion poll, once you have a good estimate for the entire country, you can estimate among men and women, northerners and southerners, different age groups, etc.). $N$, is never enough because if it were "enough" you'd already be on to the next problem for which you need more data.

## Are frequentist methods incorrect then?
No.

Frequentist methods are still useful or state-of-the-art in many areas. Tools such as least squares linear regression, LASSO regression, and expectation-maximization algorithms are all powerful and fast. Bayesian methods complement these techniques by solving problems that these approaches cannot, or by illuminating the underlying system with more flexible modeling.

### A note on *Big Data*
Paradoxically, big data's predictive analytic problems are actually solved by relatively simple algorithms [[2]](#scrollTo=nDdph0r1ABCn)[[3]](#scrollTo=nDdph0r1ABCn). Thus we can argue that big data's prediction difficulty does not lie in the algorithm used, but instead on the computational difficulties of storage and execution on big data. (One should also consider Gelman's quote from above and ask "Do I really have big data?")

The much more difficult analytic problems involve medium data and, especially troublesome, really small data. Using a similar argument as Gelman's above, if big data problems are big enough to be readily solved, then we should be more interested in the not-quite-big enough datasets.

## Our Bayesian framework
We are interested in beliefs, which can be interpreted as probabilities by thinking Bayesian. We have a prior belief in event A, beliefs formed by previous information, e.g., our prior belief about bugs being in our code before performing tests.

Secondly, we observe our evidence. To continue our buggy-code example: if our code passes X tests, we want to update our belief to incorporate this. We call this new belief the posterior probability. Updating our belief is done via the following equation, known as Bayes' Theorem, after its discoverer Thomas Bayes:

$$ P(A|X) = \frac{P(X | A) P(A) }{P(X) } $$

$$ P(A|X) \propto{P(X | A) P(A) } $$

NOTE: ($\propto$ is "proportional to")

The above formula is not unique to Bayesian inference: it is a mathematical fact with uses outside Bayesian inference. Bayesian inference merely uses it to connect prior probabilities $P(A)$ with an updated posterior probabilities $P(A|X)$.

## Example: Mandatory coin-flip example
Every statistics text must contain a coin-flipping example, I'll use it here to get it out of the way. Suppose, naively, that you are unsure about the probability of heads in a coin flip (spoiler alert: it's 50%). You believe there is some true underlying ratio, call it p, but have no prior opinion on what p might be.

We begin to flip a coin, and record the observations: either H or T. This is our observed data. An interesting question to ask is how our inference changes as we observe more and more data? More specifically, what do our posterior probabilities look like when we have little data, versus when we have lots of data.

Below we plot a sequence of updating posterior probabilities as we observe increasing amounts of data (coin flips), while also demonstrating some of the best practices when it comes to evaluating tensors and plotting the data. First, the easy part: We define the values in our Tensorflow graph

## Flat Beta Prior

```
# Build Graph
rv_coin_flip_prior = tfp.distributions.Bernoulli(probs=0.2, dtype=tf.int32)

num_trials = tf.constant([0,1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000])

coin_flip_data = rv_coin_flip_prior.sample(num_trials[-1])

# prepend a 0 onto tally of heads and tails, for zeroth flip
coin_flip_data = tf.pad(coin_flip_data,tf.constant([[1, 0,]]),"CONSTANT")

# compute cumulative headcounts from 0 to 2000 flips, and then grab them at each of num_trials intervals
cumulative_headcounts = tf.gather(tf.cumsum(coin_flip_data), num_trials)

# display(cumulative_headcounts)
# Beta Distribution
rv_observed_heads = tfp.distributions.Beta(
    concentration0=tf.cast(1 + tf.cast(num_trials, tf.double) - tf.cast(cumulative_headcounts, tf.double), tf.float32),
    concentration1=tf.cast(1 + tf.cast(cumulative_headcounts, tf.double), tf.float32)
)

probs_of_heads = tf.linspace(start=0., stop=1., num=100, name="linspace")

observed_probs_heads = tf.transpose(rv_observed_heads.prob(probs_of_heads[:, tf.newaxis]))

# display(observed_probs_heads.numpy().shape)
#Plotting

# For the already prepared, I'm using Binomial's conj. prior.
plt.figure(figsize(16, 9))
for i in range(len(num_trials)):
    sx = plt.subplot(len(num_trials)/2, 2, i+1)
    plt.xlabel("$p$, probability of heads") \
    if i in [0, len(num_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.plot(probs_of_heads, observed_probs_heads[i],
             label="observe %d tosses,\n %d heads" % (num_trials[i], cumulative_headcounts[i]))
    plt.fill_between(probs_of_heads, 0, observed_probs_heads[i],
                     color=TFColor[3], alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities", y=1.02,
             fontsize=14)
plt.tight_layout()
```

## Informed Beta Prior

```
# Build Graph
rv_coin_flip_prior = tfp.distributions.Bernoulli(probs=0.2, dtype=tf.int32)

num_trials = tf.constant([0,1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000])

coin_flip_data = rv_coin_flip_prior.sample(num_trials[-1])

# prepend a 0 onto tally of heads and tails, for zeroth flip
coin_flip_data = tf.pad(coin_flip_data,tf.constant([[1, 0,]]),"CONSTANT")

# compute cumulative headcounts from 0 to 2000 flips, and then grab them at each of num_trials intervals
cumulative_headcounts = tf.gather(tf.cumsum(coin_flip_data), num_trials)

# display(cumulative_headcounts)
# Beta Distribution
alpha = 2.75
beta = 8
rv_observed_heads = tfp.distributions.Beta(
    concentration0=tf.cast(beta + tf.cast(num_trials, tf.double) - tf.cast(cumulative_headcounts, tf.double), tf.float32),
    concentration1=tf.cast(alpha + tf.cast(cumulative_headcounts, tf.double), tf.float32))

probs_of_heads = tf.linspace(start=0., stop=1., num=100, name="linspace")

observed_probs_heads = tf.transpose(rv_observed_heads.prob(probs_of_heads[:, tf.newaxis]))

# display(observed_probs_heads.numpy().shape)
#Plotting

# For the already prepared, I'm using Binomial's conj. prior.
plt.figure(figsize(16, 9))
for i in range(len(num_trials)):
    sx = plt.subplot(len(num_trials)/2, 2, i+1)
    plt.xlabel("$p$, probability of heads") \
    if i in [0, len(num_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.plot(probs_of_heads, observed_probs_heads[i],
             label="observe %d tosses,\n %d heads" % (num_trials[i], cumulative_headcounts[i]))
    plt.fill_between(probs_of_heads, 0, observed_probs_heads[i],
                     color=TFColor[3], alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities", y=1.02,
             fontsize=14)
plt.tight_layout()
```

## Very Opinionated Prior : Correct

```
# Build Graph
rv_coin_flip_prior = tfp.distributions.Bernoulli(probs=0.2, dtype=tf.int32)

num_trials = tf.constant([0,1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000])

coin_flip_data = rv_coin_flip_prior.sample(num_trials[-1])

# prepend a 0 onto tally of heads and tails, for zeroth flip
coin_flip_data = tf.pad(coin_flip_data,tf.constant([[1, 0,]]),"CONSTANT")

# compute cumulative headcounts from 0 to 2000 flips, and then grab them at each of num_trials intervals
cumulative_headcounts = tf.gather(tf.cumsum(coin_flip_data), num_trials)

# display(cumulative_headcounts)
# Beta Distribution
rv_observed_heads = tfp.distributions.Beta(
    concentration0=tf.cast(4000 + tf.cast(num_trials, tf.double) - tf.cast(cumulative_headcounts, tf.double), tf.float32),
    concentration1=tf.cast(1000 + tf.cast(cumulative_headcounts, tf.double), tf.float32)
)

probs_of_heads = tf.linspace(start=0., stop=1., num=100, name="linspace")

observed_probs_heads = tf.transpose(rv_observed_heads.prob(probs_of_heads[:, tf.newaxis]))

# display(observed_probs_heads.numpy().shape)
#Plotting

# For the already prepared, I'm using Binomial's conj. prior.
plt.figure(figsize(16, 9))
for i in range(len(num_trials)):
    sx = plt.subplot(len(num_trials)/2, 2, i+1)
    plt.xlabel("$p$, probability of heads") \
    if i in [0, len(num_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.plot(probs_of_heads, observed_probs_heads[i],
             label="observe %d tosses,\n %d heads" % (num_trials[i], cumulative_headcounts[i]))
    plt.fill_between(probs_of_heads, 0, observed_probs_heads[i],
                     color=TFColor[3], alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities", y=1.02,
             fontsize=14)
plt.tight_layout()
```

## Very Opinionated Prior: Wrong

```
# Build Graph
rv_coin_flip_prior = tfp.distributions.Bernoulli(probs=0.2, dtype=tf.int32)

num_trials = tf.constant([0,1, 2, 3, 4, 5, 8, 15, 50, 500, 1000, 2000])

coin_flip_data = rv_coin_flip_prior.sample(num_trials[-1])

# prepend a 0 onto tally of heads and tails, for zeroth flip
coin_flip_data = tf.pad(coin_flip_data,tf.constant([[1, 0,]]),"CONSTANT")

# compute cumulative headcounts from 0 to 2000 flips, and then grab them at each of num_trials intervals
cumulative_headcounts = tf.gather(tf.cumsum(coin_flip_data), num_trials)

# display(cumulative_headcounts)
# Beta Distribution
rv_observed_heads = tfp.distributions.Beta(
    concentration0=tf.cast(500 + tf.cast(num_trials, tf.double) - tf.cast(cumulative_headcounts, tf.double), tf.float32),
    concentration1=tf.cast(4000 + tf.cast(cumulative_headcounts, tf.double), tf.float32)
)

probs_of_heads = tf.linspace(start=0., stop=1., num=100, name="linspace")

observed_probs_heads = tf.transpose(rv_observed_heads.prob(probs_of_heads[:, tf.newaxis]))

# display(observed_probs_heads.numpy().shape)
#Plotting

# For the already prepared, I'm using Binomial's conj. prior.
plt.figure(figsize(16, 9))
for i in range(len(num_trials)):
    sx = plt.subplot(len(num_trials)/2, 2, i+1)
    plt.xlabel("$p$, probability of heads") \
    if i in [0, len(num_trials)-1] else None
    plt.setp(sx.get_yticklabels(), visible=False)
    plt.plot(probs_of_heads, observed_probs_heads[i],
             label="observe %d tosses,\n %d heads" % (num_trials[i], cumulative_headcounts[i]))
    plt.fill_between(probs_of_heads, 0, observed_probs_heads[i],
                     color=TFColor[3], alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)
    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities", y=1.02,
             fontsize=14)
plt.tight_layout()
```

Finally, we move onto plotting our tensors in matplotlib.

The posterior probabilities are represented by the curves, and our uncertainty is proportional to the width of the curve. As the plot above shows, as we start to observe data our posterior probabilities start to shift and move around. Eventually, as we observe more and more data (coin-flips), our probabilities will tighten closer and closer around the true value of $p=0.5$ (marked by a dashed line).

Notice that the plots are not always peaked at 0.5. There is no reason it should be: recall we assumed we did not have a prior opinion of what p is. In fact, if we observe quite extreme data, say 8 flips and only 1 observed heads, our distribution would look very biased away from lumping around 0.5 (with no prior opinion, how confident would you feel betting on a fair coin after observing 8 tails and 1 head?). As more data accumulates, we would see more and more probability being assigned at $p=0.5$, though never all of it.

The next example is a simple demonstration of the mathematics of Bayesian inference.

## Example: Bug, or just sweet, unintended feature?
Let $A$ denote the event that our code has no bugs in it. Let $X$ denote the event that the code passes all debugging tests. For now, we will leave the prior probability of no bugs as a variable, i.e. $P(A)=p$.

We are interested in $P(A|X)$, i.e. the probability of no bugs, given our debugging tests $X$. To use the formula above, we need to compute some quantities.

What is $P(X|A)$, i.e., the probability that the code passes $X$ tests given there are no bugs? Well, it is equal to 1, for a code with no bugs will pass all tests.

$P(X)$ is a little bit trickier: The event $X$ can be divided into two possibilities, event X occurring even though our code indeed has bugs (denoted $∼A$, spoken not $A$), or event $X$ without bugs $(A)$. $ P(X)$ can be represented as:

$$ \begin{align*}
P(A|X) &= \frac{P(X | A) P(A) }{P(X) } \\
 P(X) &= P(X \text{ and } A) + P(X \text{ and } \sim A) \\
  &= P(X|A)P(A) + P(X | \sim A)P(\sim A) \\
  &= P(X|A)p + P(X | \sim A)(1-p) \end{align*} $$

We have already computed $P(X|A)$ above. On the other hand, $P(X|\sim A)$ is subjective: our code can pass tests but still have a bug in it, though the probability there is a bug present is reduced. Note this is dependent on the number of tests performed, the degree of complication in the tests, etc. Let's be conservative and assign $P(X|\sim A)=0.5$. Then:

$$ \begin{align*}
P(A | X) &= \frac{1\cdot p}{ 1\cdot p +0.5 (1-p) } \\
&= \frac{ 2 p}{1+p} \end{align*} $$

This is the posterior probability. What does it look like as a function of our prior, $p\in[0,1]$?

```
# Defining our range of probabilities
p = tf.linspace(start=0., stop=1., num=50)

# Visualization.
plt.figure(figsize=(12.5, 6))
plt.plot(p, 2*p/(1+p), color=TFColor[3], lw=3)
plt.fill_between(p, 2*p/(1+p), alpha=.5, facecolor=["#A60628"])
plt.scatter(0.2, 2*(0.2)/1.2, s=140, c=TFColor[3])
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel(r"Prior, $P(A) = p$")
plt.ylabel(r"Posterior, $P(A|X)$, with $P(A) = p$")
plt.title(r"Are there bugs in my code?");
```

We can see the biggest gains if we observe the $X$ tests passed when the prior probability, $p$, is low. Let's settle on a specific value for the prior. I'm a strong programmer (I think), so I'm going to give myself a realistic prior of 0.20, that is, there is a 20% chance that I write code bug-free. To be more realistic, this prior should be a function of how complicated and large the code is, but let's pin it at 0.20. Then my updated belief that my code is bug-free is 0.33.

Recall that the prior is a probability: $p$ is the prior probability that there are no bugs, so $1 \text{-} p$ is the prior probability that there are bugs.

Similarly, our posterior is also a probability, with $P(A|X)$ the probability there is no bug given we saw all tests pass, hence $1 \text{-} P(A|X)$ is the probability there is a bug given all tests passed. What does our posterior probability look like? Below is a chart of both the prior and the posterior probabilities.

```
# Defining our priors and posteriors
prior = tf.constant([0.20, 0.80])
posterior = tf.constant([1./3, 2./3])

# Our Simple Visualization
plt.figure(figsize=(12.5, 4))
colours = [TFColor[0], TFColor[3]]
plt.bar([0, .7], prior, alpha=0.70, width=0.25,
        color=colours[0], label="prior distribution",
        lw="3", edgecolor=colours[0])
plt.bar([0+0.25, .7+0.25], posterior, alpha=0.7,
        width=0.25, color=colours[1],
        label=r"posterior distribution",
        lw="3", edgecolor=colours[1])

plt.xticks([0.20, .95], ["Bugs Absent", "Bugs Present"])
plt.title(r"Prior and Posterior probability of bugs present")
plt.ylabel("Probability")
plt.legend(loc="upper left");
```

Notice that after we observed $X$ occur, the probability of bugs being absent increased. By increasing the number of tests, we can approach confidence (probability 1) that there are no bugs present.

This was a very simple example of Bayesian inference and Bayes rule. Unfortunately, the mathematics necessary to perform more complicated Bayesian inference only becomes more difficult, except for artificially constructed cases. We will later see that this type of mathematical analysis is actually unnecessary. First we must broaden our modeling tools. The next section deals with probability distributions. If you are already familiar, feel free to skip (or at least skim), but for the less familiar the next section is essential.

## Probability Distributions
Let's quickly recall what a probability distribution is: Let $Z$ be some random variable. Then associated with $Z$ is a probability distribution function that assigns probabilities to the different outcomes $Z$ can take. Graphically, a probability distribution is a curve where the probability of an outcome is proportional to the height of the curve. You can see examples in the first figure of this chapter.

We can divide random variables into three classifications:

*  $Z$ is discrete: Discrete random variables may only assume values on a specified list. Things like populations, movie ratings, and number of votes are all discrete random variables. Discrete random variables become more clear when we contrast them with...

* $Z$ is continuous: Continuous random variable can take on arbitrarily exact values. For example, temperature, speed, time, color are all modeled as continuous variables because you can progressively make the values more and more precise.

* $Z$ is mixed: Mixed random variables assign probabilities to both discrete and continuous random variables, i.e. it is a combination of the above two categories.

### Discrete Case

If $Z$ is discrete, then its distribution is called a *probability mass function*, which measures the probability $Z$ takes on the value $k$, denoted $P(Z=k)$. Note that the probability mass function completely describes the random variable $Z$, that is, if we know the mass function, we know how $Z$ should behave. There are popular probability mass functions that consistently appear: we will introduce them as needed, but let's introduce the first very useful probability mass function. We say $Z$ is *Poisson*-distributed if:

$$P(Z = k) =\frac{ \lambda^k e^{-\lambda} }{k!}, \; \; k=0,1,2, \dots $$

$\lambda$ is called a parameter of the distribution, and it controls the distribution's shape. For the Poisson distribution, $\lambda$ can be any positive number. By increasing $\lambda$, we add more probability to larger values, and conversely by decreasing $\lambda$ we add more probability to smaller values. One can describe $\lambda$ as the *intensity* of the Poisson distribution.

Unlike $\lambda$, which can be any positive number, the value $k$ in the above formula must be a non-negative integer, i.e., $k$ must take on values 0,1,2, and so on. This is very important, because if you wanted to model a population you could not make sense of populations with 4.25 or 5.612 members.

If a random variable $Z$ has a Poisson mass distribution, we denote this by writing

$$Z \sim \text{Poi}(\lambda) $$

One useful property of the Poisson distribution is that its expected value is equal to its parameter, i.e.:

$$E\large[ \;Z\; | \; \lambda \;\large] = \lambda $$

We will use this property often, so it's useful to remember. Below, we plot the probability mass distribution for different $\lambda$ values. The first thing to notice is that by increasing $\lambda$, we add more probability of larger values occurring. Second, notice that although the graph ends at 15, the distributions do not. They assign positive probability to every non-negative integer.

```
# Build graph.
x = tf.range (start=0., limit=16.,dtype=tf.float32)
lambdas = tf.constant([1.5, 4.25])

poi_pmf = tfd.Poisson(
  rate=lambdas[:, tf.newaxis]).prob(x)

plt.figure(figsize=(12.5, 8))

# Display results in two different histograms, for easier comparison
colours = [TFColor[0], TFColor[3]]
for i in [0,1]:
  ax = plt.subplot(2,1,i+1)
  ax.set_autoscaley_on(False)
  plt.title("Probability mass function of a Poisson random variable");

  plt.bar(x,
          poi_pmf[i],
          color=colours[i],
          label=r"$\lambda = %.1f$" % lambdas[i], alpha=0.60,
          edgecolor=colours[i], lw="3")
  plt.xticks(x)
  plt.ylim([0, .5])
  plt.legend()
  plt.ylabel(r"probability of $k$")
  plt.xlabel(r"$k$")
```

### Continuous Case

Instead of a probability mass function, a continuous random variable has a *probability density function*. This might seem like unnecessary nomenclature, but the density function and the mass function are very different creatures. An example of continuous random variable is a random variable with *exponential density*. The density function for an exponential random variable looks like this:

$$f_Z(z | \lambda) = \lambda e^{-\lambda z }, \;\; z\ge 0$$

Like a Poisson random variable, an exponential random variable can take on only non-negative values. But unlike a Poisson variable, the exponential can take on *any* non-negative values, including non-integral values such as 4.25 or 5.612401. This property makes it a poor choice for count data, which must be an integer, but a great choice for time data, temperature data (measured in Kelvins, of course), or any other precise *and positive* variable. The graph below shows two probability density functions with different $\lambda$ values.

When a random variable $Z$ has an exponential distribution with parameter $\lambda$, we say *$Z$ is exponential* and write

$$Z \sim \text{Exp}(\lambda)$$

Given a specific $\lambda$, the expected value of an exponential random variable is equal to the inverse of $\lambda$, that is:

$$E[\; Z \;|\; \lambda \;] = \frac{1}{\lambda}$$

```
# Defining our Data and assumptions (use tf.linspace for continuous)
a = tf.range(start=0., limit=4., delta=0.04)
a = a[..., tf.newaxis]
lambdas = tf.constant([0.5, 1.])

# Now we use TFP to compute probabilities in a vectorized manner.
expo_pdf = tfd.Exponential(rate=lambdas).prob(a)

# Visualizing our results
plt.figure(figsize=(12.5, 4))
for i in range(lambdas.shape[0]):
    plt.plot(tf.transpose(a)[0], tf.transpose(expo_pdf)[i],
             lw=3, color=TFColor[i], label=r"$\lambda = %.1f$" % lambdas[i])
    plt.fill_between(tf.transpose(a)[0], tf.transpose(expo_pdf)[i],
                         color=TFColor[i], alpha=.33)
plt.legend()
plt.ylabel("PDF at $z$")
plt.xlabel("$z$")
plt.ylim(0,1.2)
plt.title(r"Probability density function of an Exponential random variable; differing $\lambda$");

```

## But what is $\lambda \;$?

**This question is what motivates statistics**. In the real world, $\lambda$ is hidden from us. We see only $Z$, and must go backwards to try and determine $\lambda$. The problem is difficult because there is no one-to-one mapping from $Z$ to $\lambda$. Many different methods have been created to solve the problem of estimating $\lambda$, but since $\lambda$ is never actually observed, no one can say for certain which method is best!

Bayesian inference is concerned with *beliefs* about what $\lambda$ might be. Rather than try to guess $\lambda$ exactly, we can only talk about what $\lambda$ is likely to be by assigning a probability distribution to $\lambda$.

This might seem odd at first. After all, $\lambda$ is fixed; it is not (necessarily) random! How can we assign probabilities to values of a non-random variable? Ah, we have fallen for our old, frequentist way of thinking. Recall that under Bayesian philosophy, we *can* assign probabilities if we interpret them as beliefs. And it is entirely acceptable to have *beliefs* about the parameter $\lambda$.

#### Example: Inferring behaviour from text-message data

Let's try to model a more interesting example, one that concerns the rate at which a user sends and receives text messages:

>  You are given a series of daily text-message counts from a user of your system. The data, plotted over time, appears in the chart below. You are curious to know if the user's text-messaging habits have changed over time, either gradually or suddenly. How can you model this? (This is in fact my own text-message data. Judge my popularity as you wish.)

```
# Defining our Data and assumptions
count_data = tf.constant([
    13,  24,   8,  24,   7,  35,  14,  11,  15,  11,  22,  22,  11,  57,
    11,  19,  29,   6,  19,  12,  22,  12,  18,  72,  32,   9,   7,  13,
    19,  23,  27,  20,   6,  17,  13,  10,  14,   6,  16,  15,   7,   2,
    15,  15,  19,  70,  49,   7,  53,  22,  21,  31,  19,  11,  18,  20,
    12,  35,  17,  23,  17,   4,   2,  31,  30,  13,  27,   0,  39,  37,
    5,  14,  13,  22,
], dtype=tf.float32)
n_count_data = tf.shape(count_data)
days = tf.range(n_count_data[0],dtype=tf.int32)

# Visualizing the Results
plt.figure(figsize=(12.5, 4))
plt.bar(days.numpy(), count_data, color="#5DA5DA")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Did the user's texting habits change over time?")
plt.xlim(0, n_count_data[0].numpy());
```

Before we start modeling, see what you can figure out just by looking at the chart above. Would you say there was a change in behaviour during this time period?

How can we start to model this? Well, as we have conveniently already seen, a Poisson random variable is a very appropriate model for this type of *count* data. Denoting day $i$'s text-message count by $C_i$,

$$ C_i \sim \text{Poisson}(\lambda)  $$

We are not sure what the value of the $\lambda$ parameter really is, however. Looking at the chart above, it appears that the rate might become higher late in the observation period, which is equivalent to saying that $\lambda$ increases at some point during the observations. (Recall that a higher value of $\lambda$ assigns more probability to larger outcomes. That is, there is a higher probability of many text messages having been sent on a given day.)

How can we represent this observation mathematically? Let's assume that on some day during the observation period (call it $\tau$), the parameter $\lambda$ suddenly jumps to a higher value. So we really have two $\lambda$ parameters: one for the period before $\tau$, and one for the rest of the observation period. In the literature, a sudden transition like this would be called a *switchpoint*:

$$\lambda =
\begin{cases} \lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\lambda_1 = \lambda_2$, then the $\lambda$s posterior distributions should look about equal.

We are interested in inferring the unknown $\lambda$s. To use Bayesian inference, we need to assign prior probabilities to the different possible values of $\lambda$. What would be good prior probability distributions for $\lambda_1$ and $\lambda_2$? Recall that $\lambda$ can be any positive number. As we saw earlier, the *exponential* distribution provides a continuous density function for positive numbers, so it might be a good choice for modeling $\lambda_i$. But recall that the exponential distribution takes a parameter of its own, so we'll need to include that parameter in our model. Let's call that parameter $\alpha$.
$$
\begin{align}
&\lambda_1 \sim \text{Exp}( \alpha ) \\
&\lambda_2 \sim \text{Exp}( \alpha )
\end{align}
$$
$\alpha$ is called a *hyper-parameter* or *parent variable*. In literal terms, it is a parameter that influences other parameters. Our initial guess at $\alpha$ does not influence the model too strongly, so we have some flexibility in our choice.  A good rule of thumb is to set the exponential parameter equal to the inverse of the average of the count data. Since we're modeling $\lambda$ using an exponential distribution, we can use the expected value identity shown earlier to get:

$$\frac{1}{N}\sum_{i=0}^N \;C_i \approx E[\; \lambda \; |\; \alpha ] = \frac{1}{\alpha}$$

An alternative, and something I encourage the reader to try, would be to have two priors: one for each $\lambda_i$. Creating two exponential distributions with different $\alpha$ values reflects our prior belief that the rate changed at some point during the observations.

What about $\tau$? Because of the noisiness of the data, it's difficult to pick out a priori when $\tau$ might have occurred. Instead, we can assign a *uniform prior belief* to every possible day. This is equivalent to saying
$$
\begin{align}
& \tau \sim \text{DiscreteUniform(1,70) }\\
& \Rightarrow P( \tau = k ) = \frac{1}{70}
\end{align}
$$
So after all this, what does our overall prior distribution for the unknown variables look like? Frankly, *it doesn't matter*. What we should understand is that it's an ugly, complicated mess involving symbols only a mathematician could love. And things will only get uglier the more complicated our models become. Regardless, all we really care about is the posterior distribution.

We next turn to [TensorFlow Probability](https://tensorflow.org/probability), a Python library for performing Bayesian analysis that is undaunted by the mathematical monster we have created.

## Introducing our first hammer: TensorFlow Probability

TensorFlow Probability (TFP) is a Python library for programming Bayesian analysis. It is intended for data scientists, statisticians, machine learning practitioners, and scientists. Since it is built on the TensorFlow (TF) stack, it brings the runtime benefits of TF to Bayesian analysis. These include write-once run-many (ability to run your development model in production) and speedups via state-of-the-art hardware (GPUs and TPUs).

Since TFP is relatively new, the TFP community is actively developing documentation,
especially docs and examples that bridge the gap between beginner and hacker. One of this book's main goals is to solve that problem, and also to demonstrate why TFP is so cool.

We will model the problem above using TFP. This type of programming is called *probabilistic programming*, an unfortunate misnomer that invokes ideas of randomly-generated code and has likely confused and frightened users away from this field. The code is not random; it is probabilistic in the sense that we create probability models using programming variables as the model's components.

B. Cronin [[4]](#scrollTo=nDdph0r1ABCn) has a very motivating description of probabilistic programming:

>   Another way of thinking about this: unlike a traditional program, which only runs in the forward directions, a probabilistic program is run in both the forward and backward direction. It runs forward to compute the consequences of the assumptions it contains about the world (i.e., the model space it represents), but it also runs backward from the data to constrain the possible explanations. In practice, many probabilistic programming systems will cleverly interleave these forward and backward operations to efficiently home in on the best explanations.

Because of the confusion engendered by the term *probabilistic programming*, I'll refrain from using it. Instead, I'll simply say *programming*, since that's what it really is.

TFP code is easy to read. The only novel thing should be the syntax. Simply remember that we are representing the model's components ($\tau, \lambda_1, \lambda_2$ ) as variables.

## Specify the joint log-density

We'll assume the data is a consequence of the following generative model:

$$\begin{align*}
\lambda_{1}^{(0)} &\sim \text{Exponential}(\text{rate}=\alpha) \\
\lambda_{2}^{(0)} &\sim \text{Exponential}(\text{rate}=\alpha) \\
\tau &\sim \text{Uniform}[\text{low}=0,\text{high}=1) \\
\text{for }  i &= 1\ldots N: \\
\lambda_i &= \begin{cases} \lambda_{1}^{(0)}, & \tau > i/N \\ \lambda_{2}^{(0)}, & \text{otherwise}\end{cases}\\
 X_i &\sim \text{Poisson}(\text{rate}=\lambda_i)
\end{align*}$$

Happily, this model can be easily implemented using TF and TFP's distributions:

This code creates a new function `lambda_`, but really we can think of it as a random variable: the random variable $\lambda$ from above. The [gather](https://https://www.tensorflow.org/api_docs/python/tf/gather) function assigns `lambda_1` or `lambda_2` as the value of `lambda_`, depending on what side of `tau` we are on. The values of `lambda_` up until `tau` are `lambda_1` and the values afterwards are `lambda_2`.

Note that because `lambda_1`, `lambda_2` and `tau` are random, `lambda_` will be random. We are **not** fixing any variables yet.

TFP performs probabilistic inference by evaluating the model parameters using a joint_log_prob function, which we'll describe more in Chapter 2.

```
def joint_log_prob(count_data, lambda_1, lambda_2, tau):
    tfd = tfp.distributions

    alpha = (1. / tf.reduce_mean(count_data))
    rv_lambda_1 = tfd.Exponential(rate=alpha)
    rv_lambda_2 = tfd.Exponential(rate=alpha)

    rv_tau = tfd.Uniform()

    lambda_ = tf.gather(
         [lambda_1, lambda_2],
         indices=tf.cast(tau * tf.cast(tf.size(count_data), dtype=tf.float32) <= tf.cast(tf.range(tf.size(count_data)), dtype=tf.float32), dtype=tf.int32))
    rv_observation = tfd.Poisson(rate=lambda_)

    return (
         rv_lambda_1.log_prob(lambda_1)
         + rv_lambda_2.log_prob(lambda_2)
         + rv_tau.log_prob(tau)
         + tf.reduce_sum(rv_observation.log_prob(count_data))
    )

# Define a closure over our joint_log_prob.
def unnormalized_log_posterior(lambda1, lambda2, tau):
    return joint_log_prob(count_data, lambda1, lambda2, tau)

```

Notice that the implementation is arguably very close to being a 1:1 translation of the mathematical model. The main difference is merely that once we've specified the probabilistic model, we return the sum of the log_probs.

## Specify the posterior sampler

```
# wrap the mcmc sampling call in a @tf.function to speed it up
@tf.function(autograph=False)
def graph_sample_chain(*args, **kwargs):
  return tfp.mcmc.sample_chain(*args, **kwargs)

num_burnin_steps = 5000
num_results = 20000

# Set the chain's start state.
initial_chain_state = [
    tf.cast(tf.reduce_mean(count_data), tf.float32) * tf.ones([], dtype=tf.float32, name="init_lambda1"),
    tf.cast(tf.reduce_mean(count_data), tf.float32) * tf.ones([], dtype=tf.float32, name="init_lambda2"),
    0.5 * tf.ones([], dtype=tf.float32, name="init_tau"),
]

# Since HMC operates over unconstrained space, we need to transform the
# samples so they live in real-space.
unconstraining_bijectors = [
    tfp.bijectors.Exp(),       # Maps a positive real to R.
    tfp.bijectors.Exp(),       # Maps a positive real to R.
    tfp.bijectors.Sigmoid(),   # Maps [0,1] to R.
]

step_size = 0.2

kernel=tfp.mcmc.TransformedTransitionKernel(
        inner_kernel=tfp.mcmc.HamiltonianMonteCarlo(
            target_log_prob_fn=unnormalized_log_posterior,
            num_leapfrog_steps=2,
            step_size=step_size,
            state_gradients_are_stopped=True),
        bijector=unconstraining_bijectors)

kernel = tfp.mcmc.SimpleStepSizeAdaptation(
    inner_kernel=kernel, num_adaptation_steps=int(num_burnin_steps * 0.8))

# Sample from the chain.
[
    lambda_1_samples,
    lambda_2_samples,
    posterior_tau,
], kernel_results = graph_sample_chain(
    num_results=num_results,
    num_burnin_steps=num_burnin_steps,
    current_state=initial_chain_state,
    kernel = kernel)

tau_samples = tf.floor(posterior_tau * tf.cast(tf.size(count_data),dtype=tf.float32))
```

```
print("acceptance rate: {}".format(
    tf.reduce_mean(tf.cast(kernel_results.inner_results.inner_results.is_accepted,dtype=tf.float32))))
print("final step size: {}".format(
    tf.reduce_mean(kernel_results.inner_results.inner_results.accepted_results.step_size[-100:])))

```

    acceptance rate: 0.5755000114440918
    final step size: 0.030197443440556526

## Plot the Results

```
plt.figure(figsize=(12.5, 15))
#histogram of the samples:

ax = plt.subplot(311)
ax.set_autoscaley_on(False)

plt.hist(lambda_1_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label=r"posterior of $\lambda_1$", color=TFColor[0], density=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables $\lambda_1,\;\lambda_2,\;\tau$""")
plt.xlim([15, 30])
plt.xlabel(r"$\lambda_1$ value")

ax = plt.subplot(312)
ax.set_autoscaley_on(False)
plt.hist(lambda_2_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label=r"posterior of $\lambda_2$", color=TFColor[6], density=True)
plt.legend(loc="upper left")
plt.xlim([15, 30])
plt.xlabel(r"$\lambda_2$ value")

plt.subplot(313)
w = 1.0 / tau_samples.shape[0] * np.ones_like(tau_samples)
plt.hist(tau_samples, bins=n_count_data[0], alpha=1,
         label=r"posterior of $\tau$",
         color=TFColor[2], weights=w, rwidth=2.)
plt.xticks(np.arange(n_count_data[0]))

plt.legend(loc="upper left")
plt.ylim([0, .75])
plt.xlim([35, len(count_data)-20])
plt.xlabel(r"$\tau$ (in days)")
plt.ylabel(r"probability");
```

## Interpretation

Recall that Bayesian methodology returns a *distribution*. Hence we now have distributions to describe the unknown $\lambda$s and $\tau$. What have we gained? Immediately, we can see the uncertainty in our estimates: the wider the distribution, the less certain our posterior belief should be. We can also see what the plausible values for the parameters are: $\lambda_1$ is around 18 and $\lambda_2$ is around 23. The posterior distributions of the two $\lambda$s are clearly distinct, indicating that it is indeed likely that there was a change in the user's text-message behaviour.

What other observations can you make? If you look at the original data again, do these results seem reasonable?

Notice also that the posterior distributions for the $\lambda$s do not look like exponential distributions, even though our priors for these variables were exponential. In fact, the posterior distributions are not really of any form that we recognize from the original model. But that's OK! This is one of the benefits of taking a computational point of view. If we had instead done this analysis using mathematical approaches, we would have been stuck with an analytically intractable (and messy) distribution. Our use of a computational approach makes us indifferent to mathematical tractability.

Our analysis also returned a distribution for $\tau$. Its posterior distribution looks a little different from the other two because it is a discrete random variable, so it doesn't assign probabilities to intervals. We can see that near day 45, there was a 50% chance that the user's behaviour changed. Had no change occurred, or had the change been gradual over time, the posterior distribution of $\tau$ would have been more spread out, reflecting that many days were plausible candidates for $\tau$. By contrast, in the actual results we see that only three or four days make any sense as potential transition points.

### Why would I want samples from the posterior, anyways?

We will deal with this question for the remainder of the book, and it is an understatement to say that it will lead us to some amazing results. For now, let's end this chapter with one more example.

We'll use the posterior samples to answer the following question: what is the expected number of texts at day $t, \; 0 \le t \le 70$ ? Recall that the expected value of a Poisson variable is equal to its parameter $\lambda$. Therefore, the question is equivalent to *what is the expected value of $\lambda$ at time $t$*?

In the code below, let $i$ index samples from the posterior distributions. Given a day $t$, we average over all possible $\lambda_i$ for that day $t$, using $\lambda_i = \lambda_{1,i}$ if $t \lt \tau_i$ (that is, if the behaviour change has not yet occurred), else we use $\lambda_i = \lambda_{2,i}$.

```
# tau_samples, lambda_1_samples, lambda_2_samples contain
# N samples from the corresponding posterior distribution

N_ = tau_samples.shape[0]
expected_texts_per_day = tf.zeros(N_,n_count_data.shape[0]) #(10000,74)

plt.figure(figsize=(12.5, 9))

day_range = tf.range(0,n_count_data[0],delta=1,dtype = tf.int32)

# expand from shape of 74 to (10000,74)
day_range = tf.expand_dims(day_range,0)
day_range = tf.tile(day_range,tf.constant([N_,1]))

# expand from shape of 10000 to 10000,74
tau_samples_per_day = tf.expand_dims(tau_samples,0)
tau_samples_per_day = tf.transpose(tf.tile(tau_samples_per_day,tf.constant([day_range.shape[1],1])))

tau_samples_per_day = tf.cast(tau_samples_per_day,dtype=tf.int32)
#ix_day is (10000,74) tensor where axis=0 is number of samples, axis=1 is day. each value is true iff sampleXday value is < tau_sample value
ix_day = day_range < tau_samples_per_day

lambda_1_samples_per_day = tf.expand_dims(lambda_1_samples,0)
lambda_1_samples_per_day = tf.transpose(tf.tile(lambda_1_samples_per_day,tf.constant([day_range.shape[1],1])))
lambda_2_samples_per_day = tf.expand_dims(lambda_2_samples,0)
lambda_2_samples_per_day = tf.transpose(tf.tile(lambda_2_samples_per_day,tf.constant([day_range.shape[1],1])))

expected_texts_per_day = ((tf.reduce_sum(lambda_1_samples_per_day*tf.cast(ix_day,dtype=tf.float32),axis=0) + tf.reduce_sum(lambda_2_samples_per_day*tf.cast(~ix_day,dtype=tf.float32),axis=0))/N_)

plt.plot(range(n_count_data[0]), expected_texts_per_day, lw=4, color="#E24A33",
         label="expected number of text-messages received")
plt.xlim(0, n_count_data.numpy()[0])
plt.xlabel("Day")
plt.ylabel("Expected # text-messages")
plt.title("Expected number of text-messages received")
plt.ylim(0, 60)
plt.bar(np.arange(len(count_data)), count_data, color="#5DA5DA", alpha=0.65,
        label="observed texts per day")

plt.legend(loc="upper left");
```

Our analysis shows strong support for believing the user's behavior did change ($\lambda_1$ would have been close in value to $\lambda_2$ had this not been true), and that the change was sudden rather than gradual (as demonstrated by $\tau$'s strongly peaked posterior distribution). We can speculate what might have caused this: a cheaper text-message rate, a recent weather-to-text subscription, or perhaps a new relationship. (In fact, the 45th day corresponds to Christmas, and I moved away to Toronto the next month, leaving a girlfriend behind.)

## Exercises

1.   Using `lambda_1_samples` and `lambda_2_samples`, what is the mean of the posterior distributions of $\lambda_1$ and $\lambda_2$?

```
display(f'Lambda 1: {tf.reduce_mean(lambda_1_samples)}' )
display(f'Lambda 2: {tf.reduce_mean(lambda_2_samples)} ')
```

    'Lambda 1: 17.750774383544922'

    'Lambda 2: 22.709463119506836 '

2.   What is the expected percentage increase in text-message rates? `hint:` compute the mean of `lambda_1_samples/lambda_2_samples`. Note that this quantity is very different from `lambda_1_samples.mean()/lambda_2_samples.mean()`

```
display(f'Percentage Increase: {tf.reduce_mean(lambda_1_samples/lambda_2_samples)}')
```

    'Percentage Increase: 22.709463119506836'

3. What is the mean of $\lambda_1$ **given** that we know $\tau$ is less than 45? That is, suppose we have been given new information that the change in behaviour occurred prior to day 45. What is the expected value of $\lambda_1$ now? (You do not need to redo the TFP part. Just consider all instances where `tau_samples < 45`.)

```
ix = tau_samples < 45

display(f'Lambda Average Give Tau < 45: {tf.reduce_mean(lambda_1_samples[ix])}')
```

    'Lambda Average Give Tau < 45: 17.750774383544922'

## References

[1] Gelman, Andrew. N.p.. Web. 22 Jan 2013. [N is never large enough](http://andrewgelman.com/2005/07/31/n_is_never_larg)

[2] Norvig, Peter. 2009. [The Unreasonable Effectiveness of Data](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf).

[3] Jimmy Lin and Alek Kolcz. Large-Scale Machine Learning at Twitter. Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data (SIGMOD 2012), pages 793-804, May 2012, Scottsdale, Arizona.

[4] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. <https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1>.


# **[HW7] Practice_5**
1. Install packages
2. Geometry of eigendecomposition
3. Eigendecomposition

선형대수 실습은, exercise 혹은 coding 문제풀이에 집중하기 보다는, 수업시간에 배웠던 개념들을 visualize 하면서 이를 폭넓게 이해하는 데에 초점을 맞추고 있습니다. 실습에서 사용한 예시 외에도, 다양한 matrix들을 visualize 하면서 개념을 익혀보시기 바랍니다.

# 1. Install packages

> 필요한 package를 설치하고 import합니다

```python
# visualization을 위한 helper code입니다.
from urllib.request import urlretrieve
URL = 'https://go.gwu.edu/engcomp4plot'
urlretrieve(URL, 'plot_helper.py')

import sys
sys.path.append('../scripts/')

# 다음 세 custom function (1)plot_vector, (2)plot_linear_transformation, (3) plot_linear_transformations
# 을 사용할 것입니다.
from plot_helper import *
```

```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import scipy as sp
import scipy.linalg
import sympy as sy

sy.init_printing()
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# 2. Geometry of eigendecomposition

> Eigendecomposition은 선형대수에서 가장 중요한 주제 중 하나입니다. 여러 행렬들의 eigenvector들을 구하고 이를 시각화하면서, 해당 성질들을 이해해보겠습니다.

## Eigenvectors of $A = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$

> 다음과 같은 행렬 $A = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$로 표현되는 linear transformation을 생각해봅시다. 아래 plot_linear_transformation(A)는 $\begin{bmatrix}1\\0\end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 $A$에 의해 transform 되는 과정을 visualize하는 custom 함수입니다.

```python
A = np.array([[2,1], [1,2]])
print(A)
plot_linear_transformation(A)
```

> $A$는 $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ 을 $\begin{bmatrix} 2 \\ 1 \end{bmatrix}$로, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$을 $\begin{bmatrix} 1 \\ 2 \end{bmatrix}$로 옮기는 linear transformation입니다. $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$, $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$ 외에 다른 vector들은 어떻게 옮겨지는 지 visualize 해봅시다.

```python
# 길이(norm)가 1인 vector들
alpha = np.linspace(0, 2*np.pi, 41)  # x축과의 각도 (0~2pi)
vectors = list(zip(np.cos(alpha), np.sin(alpha)))
vectors = np.array(vectors)

newvectors = A @ vectors.T
newvectors = newvectors.T

plot_vector(vectors)  # plot_vector는 주어진 vector들을 plot하는 custom function 입니다.
```

```python
plot_vector(newvectors)
```

> 위 $A$는 원을 타원으로 만드는 linear transformation임을 확인할 수 있습니다. 이 타원의 semi major axis (장축의 절반), semi minor axis (단축의 절반)을 찾기 위해선, transform된 newvectors의 길이가 가장 길고 짧은 두 vector를 찾으면 됩니다.

```python
lengths = np.linalg.norm(newvectors, axis=1)  # newvectos 각각의 norm
lengths
```

    array([2.236, 2.497, 2.711, 2.87 , 2.967, 3.   , 2.967, 2.87 , 2.711,
           2.497, 2.236, 1.94 , 1.628, 1.328, 1.094, 1.   , 1.094, 1.328,
           1.628, 1.94 , 2.236, 2.497, 2.711, 2.87 , 2.967, 3.   , 2.967,
           2.87 , 2.711, 2.497, 2.236, 1.94 , 1.628, 1.328, 1.094, 1.   ,
           1.094, 1.328, 1.628, 1.94 , 2.236])

```python
semi_major_index = np.argmax(lengths)  # norm이 가장 긴 것이 타원의 장축
semi_major_vector = newvectors[semi_major_index]
semi_major_length = lengths[semi_major_index]

print(semi_major_vector)  # 장축 vector
print(semi_major_length)
```

    [-2.121 -2.121]
    3.0000000000000004

```python
semi_minor_index = np.argmin(lengths)  # norm이 가장 짧은 것이 타원의 장축
semi_minor_vector = newvectors[semi_minor_index]
semi_minor_length = lengths[semi_minor_index]

print(semi_minor_vector)  # 단축 vector
print(semi_minor_length)
```

    [-0.707  0.707]
    1.0

> 장축은 $y=x$ 위에 있는 vector, 즉 x축과 45도를 이루는 vector임을 알 수 있습니다. 단축은 $y=-x$ 위에 있는 vector, 즉 x축과 $-45(=\pi/4)$도를 이루는 vector임을 알 수 있습니다. 이 두 vector $\begin{bmatrix} -2.121 \\ -2.121 \end{bmatrix}$, $\begin{bmatrix} -0.707 \\ 0.707 \end{bmatrix}$은 원래 원의 어느 vector가 linear transform된 것일까요?

> 이를 구하기 위해 $A^{-1}$을 곱해서, 원래 vector를 구해봅시다.

```python
A_inv = np.linalg.inv(A)
v1 = A_inv @ semi_major_vector
v2 = A_inv @ semi_minor_vector

print(v1)  # v1 -> A -> 장축
print(v2)  # v2 -> A -> 단축
```

    [-0.707 -0.707]
    [-0.707  0.707]

> $v_1$과 semi major vector, 그리고 $v_2$와 semi minor vector은 각각 방향이 같습니다 ($v_1$과 semi major vector는 y=x, $v_2$와 semi minor vector는 $y=-x$위에 존재). 즉 $Av_1 = 3v_1, Av_2=1v_2$임을 알 수 있습니다. 따라서, 위 $A$는 원을 타원으로 바꾸는 matrix이며, 이때 타원의 장축, 단축이 각각 eigenvector가 됨을 알 수 있습니다. \\

**(주의) 이는 모든 타원에 대해서 성립하는 것이 아닙니다. $A$가 symmetric matrix라는 점이 중요한데, 실습 마지막 부분에서 더 자세히 알아보겠습니다.**

## Scaling과 Rotation으로 A 만들기

```python
plot_vector(vectors)
```

```python
plot_vector(newvectors)
```

> $A$로 만들어진 타원(newvectors)을 보면, scaling과 rotation의 조합으로 만들 수 있는 것처럼 보입니다. 예를 들면, 반지름이 1인 원(vectors)을 (1) 가로로 3배(장축 방향의 eigenvalue가 3이니까) 늘린 다음 (i.e., y축 성분은 그냥 놔두고, x축 성분을 3배), (2) 시계 반대방향으로 45를 돌린다면 위의 타원을 만들 수 있지 않을까요? 한번 이를 visualize 해봅시다.

### Exercise

```python
vectors.shape
```

```python
S = np.array([[3,0],[0,1]]) #TODO: 가로로 3배 scaling 하는 2x2 matrix
ellipse = vectors @ S#TODO: S에 의해서 늘려진 vector들.
ellipse.shape
```

```python
# 정답 plot 예시
plot_vector(ellipse)
```

> 이제 이 가로로 3배 늘려진 ellipse를 시계 반대방향으로 45도 돌려봅시다. Rotation matrix는 $\mathbf{i}$는 $(\cos\theta, \sin\theta)$로 옮기고, $\mathbf{j}$는 $(-\sin\theta, \cos\theta)$로 옮기는 matrix입니다.

 출처: Ref. [1]

$$R = \begin{bmatrix} \cos{\theta} & -\sin{\theta} \\ \sin{\theta} & \cos{\theta} \end{bmatrix}$$

> 이제 위의 ellipse를 $R$을 통해 45도 회전시켜보도록 하겠습니다.

### Exercise

```python
theta = np.pi/4 # TODO
R = np.array([[np.cos(theta),np.sin(theta)],[-np.sin(theta),np.cos(theta)]]) # TODO: 시계방향으로 45도 회전시키는 matrix

rotated_ellipse = ellipse @ R # TODO

```

```python
# 정답 plot 예시
plot_vector(rotated_ellipse)
```

> 위의 rotated_ellipse는 처음에 $A$에 의해 변환된 아래 그림과 겉으로 보기에 같아 보입니다.

```python
plot_vector(newvectors)
```

> 즉, $R$과 $S$의 조합으로 A를 만드는 데 성공한 것처럼 보입니다. 하지만 vector 하나 하나를 transformation 시켜보면, 저 $R$과 $S$의 조합은 사실 A가 아님을 알 수 있습니다.

```python
plot_linear_transformations(S,R)
```

> 위에서 $v_1$은 matrix $A$의 eigenvector임을 알 수 있었습니다 ($Av_1 = 3v_1$). 하지만, 위 그림에선 변환 전 (Before transformation)의 초록색 vector $\mathbf{i}$가, 변환 후 (After 2 transformations)의 초록색 vector $\begin{bmatrix} \alpha \\ \alpha \end{bmatrix}$가 됨을 알 수 있습니다 ($\alpha$는 상수). 이 말은, $v_1$는 $S$@$R$의 eigenvector가 아니라는 뜻입니다. vector의 방향이 바뀌기 때문입니다.

$$A\begin{bmatrix} \alpha \\ \alpha \end{bmatrix} = 3 \begin{bmatrix} \alpha \\ \alpha \end{bmatrix} \\
RS\begin{bmatrix} \alpha \\ \alpha \end{bmatrix} \neq 3\begin{bmatrix} \alpha \\ \alpha \end{bmatrix}$$

> 그럼 $v_1$은 실제로 어떻게 바뀌고 있을까요?

### Exercise

```python
v1 = -v1  # eigenvector에 non-zero 상수를 곱하면 여전히 eigenvector이기 때문에, -1을 곱해줍니다 (좌표를 양수로 만들기 위해)
print(v1)

scaled_v1 = v1 @ S # TODO
rotated_scaled_v1 = scaled_v1 @ R # TODO
plot_vector([v1, rotated_scaled_v1])

print(rotated_scaled_v1)
```

```python

```

> 즉 v1이 크기 뿐 아니라 방향마저 바뀌었음을 알 수 있고, 눈으로 언뜻 보기와는 다르게 $A$와 $R$@$S$는 다름을 알 수 있습니다.

$$ A \neq RS$$

## A의 Eigendecomposition

> 그렇다고 해서, rotation과 scaling의 조합으로 $A$를 만들 수 없는 것은 아닙니다. 정답부터 말하자면, (1) rotation을 먼저 시계방향으로 45도로 돌린 후 (2) 이를 scaling하고 (3) 다시 반시계방향으로 45도로 돌리면 정확히 $A$를 만들 수 있습니다.

> 이를 자세히 알아보기전에, 시계방향으로 돌리는 rotation matrix는 $R (=\begin{bmatrix} \cos{\theta} & -\sin{\theta} \\ \sin{\theta} & \cos{\theta} \end{bmatrix})$에서 $\theta$를 $-\theta$로 바꿔주면 됩니다 (위의 rotation matrix에서 설명했던 것의 반대의 작용을 하는 matrix, 즉 $(\cos\theta, \sin\theta)$를  $\mathbf{i}$로 옮기고, $(-\sin\theta, \cos\theta)$를 $\mathbf{j}$로 옮기는 matrix입니다). 즉 $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$가 됩니다.

> $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$는 반시계방향으로 돌려진 matrix를 원래대로 되돌리는 역할을 하기 때문에, 당연히 $R^{-1}$이 됩니다.

```python
theta_list = [np.pi/8, np.pi/4, np.pi/2, np.pi]

for theta in theta_list:
  R = np.array([[numpy.cos(theta), -numpy.sin(theta)],  # 반시계방향으로 45도 회전하는 matrix
                  [numpy.sin(theta), numpy.cos(theta)]])
  print(R)
  R_inv = np.linalg.inv(R)
  print(R_inv)
  print()
```

    [[ 0.924 -0.383]
     [ 0.383  0.924]]
    [[ 0.924  0.383]
     [-0.383  0.924]]

    [[ 0.707 -0.707]
     [ 0.707  0.707]]
    [[ 0.707  0.707]
     [-0.707  0.707]]

    [[ 0. -1.]
     [ 1.  0.]]
    [[ 0.  1.]
     [-1.  0.]]

    [[-1. -0.]
     [ 0. -1.]]
    [[-1.  0.]
     [-0. -1.]]

> 동시에, $\begin{bmatrix} \cos{\theta} & \sin{\theta} \\ -\sin{\theta} & \cos{\theta} \end{bmatrix}$은 $R^{T}$와 같습니다. 왜냐하면 rotation은 orthogonal matrix이기 때문입니다. Orthogonal matrix란 (i) 모든 column의 norm이 1이고 (ii) 모든 column vector가 서로 orthogonal한 matrix를 말합니다. 이에 관한 자세한 설명은 wiki를 참조하시기 바랍니다 (https://en.wikipedia.org/wiki/Orthogonal_matrix). 결론적으로 다음이 성립하게 됩니다.

$$ \text{시계방향으로 } \theta\text{만큼 회전하는 matrix}  = R^{-1} = R^T$$

```python
for theta in theta_list:
  R = np.array([[numpy.cos(theta), -numpy.sin(theta)],  # 반시계방향으로 45도 회전하는 matrix
                  [numpy.sin(theta), numpy.cos(theta)]])
  R_inv = np.linalg.inv(R)

  print(R.T)
  print(R_inv)
  print()
```

    [[ 0.924  0.383]
     [-0.383  0.924]]
    [[ 0.924  0.383]
     [-0.383  0.924]]

    [[ 0.707  0.707]
     [-0.707  0.707]]
    [[ 0.707  0.707]
     [-0.707  0.707]]

    [[ 0.  1.]
     [-1.  0.]]
    [[ 0.  1.]
     [-1.  0.]]

    [[-1.  0.]
     [-0. -1.]]
    [[-1.  0.]
     [-0. -1.]]

> 다시 원래 내용으로 돌아가서, rotation과 scaling을 조합해 $A$를 만들어 봅시다.

1. 시계방향으로 45도 회전한다. ($R$)
2. scaling 한다. ($S$)
3. 반시계방향으로 45도 회전해서 원래 방향으로 되돌려준다.. ($R^T$)

```python
theta = np.pi/4
R = np.array([[np.cos(theta), -np.sin(theta)],  # 반시계방향으로 theta만큼 회전하는 matrix
              [np.sin(theta), np.cos(theta)]])
plot_linear_transformation(R @ S @ R.T)
```

> 좀 더 자세한 이해를 위해, $R, S, R^T$를 하나씩 visualize 해보겠습니다. $A$의 eigenvector $v_1, v_2$를 세 행렬로 변환시켜봅시다.

```python
v1, v2
```

    (array([-0.707, -0.707]), array([-0.707,  0.707]))

```python
plot_linear_transformation(R.T, v1, v2)
```

> $y=x, y=-x$ 위에 있는 $v_1$(파랑), $v_2$(분홍)가 시계방향으로 45도 돌아가서 $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것을 확인할 수 있습니다. 이제 scaling을 적용해 봅시다.

```python
rotated_v1 = R.T @ v1
rotated_v2 = R.T @ v2

scaled_rotated_v1 = S @ rotated_v1
scaled_rotated_v2 = S @ rotated_v2

plot_linear_transformation(S, rotated_v1, rotated_v2)
```

> $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 x축 방향으로 3배 길어져서, $\begin{bmatrix}3\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것을 확인할 수 있습니다. 이제 다시 반시계방향으로 45도를 돌려봅시다.

```python
rotated_scaled_rotated_v1 = R@scaled_rotated_v1
rotated_scaled_rotated_v2 = R@scaled_rotated_v2

plot_linear_transformation(R, scaled_rotated_v1, scaled_rotated_v2)
print(rotated_scaled_rotated_v1, rotated_scaled_rotated_v2)
```

> $\begin{bmatrix}3\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 반시계 방향으로 돌아가서, $\begin{bmatrix}2.121\\2.121 \end{bmatrix}, \begin{bmatrix}-0.707\\0.707\end{bmatrix}$이 된 것을 확인할 수 있습니다. 즉 $v_1 \rightarrow 3v_1$, $v_2 \rightarrow v_2$이 됩니다. 결론적으로

$$ v_1 \xrightarrow{rotate(-\theta)} \xrightarrow{scaling} \xrightarrow{rotate(\theta)} 3v_1 \\
v_2 \xrightarrow{rotate(-\theta)} \xrightarrow{scaling} \xrightarrow{rotate(\theta)} v_2 \\
A = RSR^T$$

> 이 됩니다. $v_1, v_2$가 rotate($-\theta$)로  $\begin{bmatrix}1\\0 \end{bmatrix}, \begin{bmatrix}0\\1\end{bmatrix}$이 된 것에서 짐작할 수 있듯, rotate(-$\theta$)은 eigenvector $v_1, v_2$를 one-hot vector로 만들게 됩니다 ($v_1$은 첫 번째, $v_2$는 두 번째 component만 non-zero). 이는 예전에 배웠던 change of basis와 연관되는데, 이를 이용해 eigendecomposition을 정리해보도록 하겠습니다.

# 3. Eigendecomposition 정리

## Symmetric Matrix

> 잠시 eigendecomposition을 정리하기 전에, symmetric matrix에 대해 알아봅시다. 지금까지 우리는 symmetric matrix인 $A$를 다뤘습니다. Symmetric matrix란 다음의 성질을 만족하는 matrix를 말합니다.

$$ A = A^T  $$

> 내일 강의에서 배우겠지만, symmetric matrix는 여러가지 유용한 성질을 갖고 있습니다. 그 중 하나는, $n \times n$ symmetric matrix의 경우 항상 n개의 orthogonal한 eigenvector를 가질 수 있다는 것입니다.

```python
F = np.random.rand(5,5)
F_symmetric = F + F.T  # A + A^T는 항상 symmetric입니다.
F_symmetric
```

    array([[1.489, 0.917, 1.516, 0.454, 1.235],
           [0.917, 0.426, 1.038, 0.774, 1.783],
           [1.516, 1.038, 0.382, 1.138, 0.559],
           [0.454, 0.774, 1.138, 0.802, 1.547],
           [1.235, 1.783, 0.559, 1.547, 1.6  ]])

```python
eigenvalues, eigenvectors = np.linalg.eig(F_symmetric)
eigenvectors
```

    array([[ 0.462,  0.291, -0.703,  0.388, -0.238],
           [ 0.422,  0.45 ,  0.14 , -0.736, -0.242],
           [ 0.376, -0.583, -0.368, -0.374,  0.494],
           [ 0.394,  0.394,  0.367,  0.315,  0.675],
           [ 0.558, -0.467,  0.465,  0.264, -0.429]])

```python
from itertools import combinations
for i,j in combinations(range(5), 2):
  print(i,j)
  print(eigenvectors[:,i] , eigenvectors[:,j])
  print(eigenvectors[:,i] @ eigenvectors[:,j])
```

    0 1
    [0.462 0.422 0.376 0.394 0.558] [ 0.291  0.45  -0.583  0.394 -0.467]
    -3.3306690738754696e-16
    0 2
    [0.462 0.422 0.376 0.394 0.558] [-0.703  0.14  -0.368  0.367  0.465]
    -1.1102230246251565e-16
    0 3
    [0.462 0.422 0.376 0.394 0.558] [ 0.388 -0.736 -0.374  0.315  0.264]
    5.551115123125783e-17
    0 4
    [0.462 0.422 0.376 0.394 0.558] [-0.238 -0.242  0.494  0.675 -0.429]
    -5.273559366969494e-16
    1 2
    [ 0.291  0.45  -0.583  0.394 -0.467] [-0.703  0.14  -0.368  0.367  0.465]
    -2.498001805406602e-16
    1 3
    [ 0.291  0.45  -0.583  0.394 -0.467] [ 0.388 -0.736 -0.374  0.315  0.264]
    -4.718447854656915e-16
    1 4
    [ 0.291  0.45  -0.583  0.394 -0.467] [-0.238 -0.242  0.494  0.675 -0.429]
    1.6653345369377348e-16
    2 3
    [-0.703  0.14  -0.368  0.367  0.465] [ 0.388 -0.736 -0.374  0.315  0.264]
    -2.185751579730777e-16
    2 4
    [-0.703  0.14  -0.368  0.367  0.465] [-0.238 -0.242  0.494  0.675 -0.429]
    2.7755575615628914e-16
    3 4
    [ 0.388 -0.736 -0.374  0.315  0.264] [-0.238 -0.242  0.494  0.675 -0.429]
    6.106226635438361e-16

> 모든 eigenvector의 dot product가 0이 되는 것으로부터, 우리는 symmetric matrix로부터 n개의 orthogonal eigenvector를 얻을 수 있음을 알 수 있습니다.

## Eigendecomposition

> 이번 실습에서 예시로 들었던 $A$는 symmetric matrix입니다. 하지만 (1) matrix $A$를 거쳐서 나온 타원의 장축, 단축이 $A$의 eigenvector가 되는 것, 그리고 (2) 모든 행렬이 rotation과 scaling의 조합으로 쪼갤 수 있는것, 두 성질은 모든 임의의 matrix에 대해 성립하는 것은 아닙니다. 지금부터는 오늘 강의에서 배웠던 것처럼, 보다 일반적으로 eigendecomposition을 유도하도록 하겠습니다.

$$
\begin{align*}
  A \mathbf{v_1} = s_1 \mathbf{v_1} \\
  A \mathbf{v_2} = s_2 \mathbf{v_2}
\end{align*}
$$

> 위 두 식은 matrix의 곱 형태로 나타낼 수 있습니다.

$$ A \begin{bmatrix}
    \mid & \mid \\
    \mathbf{v_1} & \mathbf{v_2}\\
    \mid & \mid
    \end{bmatrix}
    = \begin{bmatrix}
    A\mathbf{v_1} & A\mathbf{v_2}
    \end{bmatrix}
$$

$$ A \begin{bmatrix}
    \mid & \mid \\
    \mathbf{v_1} & \mathbf{v_2} \\
    \mid & \mid \\
    \end{bmatrix}
    = \begin{bmatrix}
    s_1\mathbf{v_1} & s_2\mathbf{v_2}
    \end{bmatrix}
$$

> $\begin{bmatrix} s_1\mathbf{v_1} & s_2\mathbf{v_2} \end{bmatrix}$는 다음과 같이 표현이 가능합니다.

$$ s_1 \mathbf{v_1} = \begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\ \mid & \mid  \end{bmatrix} \begin{bmatrix} s_1 \\ 0 \end{bmatrix} $$

$$ s_2 \mathbf{v_2} = \begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\  \mid & \mid \end{bmatrix} \begin{bmatrix} 0 \\ s_2 \end{bmatrix} $$

$$  \begin{bmatrix}s_1\mathbf{v_1} & s_2\mathbf{v_2} \end{bmatrix} =
\begin{bmatrix}  \mid & \mid \\  \mathbf{v_1} & \mathbf{v_2} \\  \mid & \mid \end{bmatrix} \begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix} $$

> $V$를 eigenvector를 column으로 갖는 matrix $\begin{bmatrix} \mid & \mid \\ \mathbf{v_1} & \mathbf{v_2} \\\mid & \mid \\\end{bmatrix}$라고 하고, $\begin{bmatrix} s_1 & 0 \\ 0 & s_2 \end{bmatrix}$를 D라고 합시다. 그렇다면 결론적으로

$$ AV = VD $$

> 를 유도할 수 있습니다. $V$의 역행렬이 존재한다면,

$$ A = VDV^{-1} \\ D = V^{-1}AV $$

> 이 됩니다. 위와 같은 형태로 A가 decompose 될 수 있으면, A를 diagonalizable이라고 합니다. 사실 우리는 이를 이전 실습에서 본 적이 있습니다. [HW5] Practice_3의 change of basis를 공부할 때, **같은 linear transformation을 새로운 basis에서 계산하려고 할 때**, 정확히 위와 같은 형태로 유도되었습니다. 이 부분이 확실하지 않으시다면, 오늘 강의와 해당 실습을 복습하시기 바랍니다.

> $V$를 change of basis로 본다면, $VDV^{-1}$는 어떤 의미가 될까요? 저번 실습에서 배웠던 것을 떠올려 본다면, (1) $V^{-1}$: standard basis ($\{ \mathbf{i}, \mathbf{j}\}$) 에서 eigenbasis ($\{ \mathbf{v_1}, \mathbf{v_2}\}$)로 basis를 change하고, (2) $D=\begin{bmatrix}s_1&0\\0&s_2\end{bmatrix}:$ $\mathbf{v_1}$ 방향의 성분 (eigenbasis에서의 좌표의 첫 번째 성분)에 $s_1$을 곱해주고, $\mathbf{v_2}$ 방향의 성분 (eigenbasis에서의 좌표의 두 번째 성분)에 $s_2$을 곱해준 다음, (i.e., elementwise scaling) (3) 다시 standard basis로 되돌리는 행위가 됩니다.

$$
  A\, \mathbf{x} = V\, D\, V^{-1}\mathbf{x}
$$

> 위와 같은 형태로 나타낼 수 있는 A, D를 **similar** 하다고 합니다. 즉 어떤 두 matrix가 같은 linear transformation을 나타내는데 basis만 다른 경우에, 두 matrix는 similar한 관계에 있다고 합니다.

> 이제 non-symmetric matrix인  $B = \begin{bmatrix} 1 & 0 \\ 1 & 3 \end{bmatrix}$의 eigendecomposition을 visualize 해봅시다.

```python
B = np.array([[1,0], [1,3]])
plot_eigen(B)
```

```python
eigenvalues, eigenvectors = np.linalg.eig(B)
eigenvalues
```

    array([3., 1.])

```python
eigenvectors
```

    array([[ 0.   ,  0.894],
           [ 1.   , -0.447]])

> 갈색, 파랑색 vector는 basis를 나타내고, 왼쪽 위의 빨간색 원은 길이가 1인 vector들의 set입니다 ($x$는 빨간색 원 위의 벡터 하나하나를 나타냅니다).

> 위 설명의 $V$를 $C$라고 하고, $v_1, v_2$는 $\mathbf{a,b}$라고 합시다 ($V=C$, $v_1=\mathbf{a}$, $v_2=\mathbf{b}$).  $C^{-1}x$는 standard basis에서 eigenbasis로 바꾸는 행위입니다. 따라서 basis $\mathbf{i,j}$는 $\mathbf{a,b}$로 바뀌었고, 빨간색 원은 바뀌지 않았습니다. 왜냐하면, change of basis는 vector를 고정시킨 채 축만 바꾸는 것이기 때문입니다.

> $DC^{-1}x$는 $C^{-1}$에 $D$로 scaling한 결과입니다. 축은 바뀌지 않았기 때문에 $\mathbf{a,b}$는 유지되고, 빨간색 원 vector의 $\mathbf{a}$ 성분은 3배, $\mathbf{b}$ 성분은 1배 (eigenvalue) 가 됩니다. 강의에서 배웠듯, matrix 계산에서 scalar 계산으로 바뀌기 때문에 (4번 계산할 것을 2번만 계산해도 됨), computational advantage가 생기게 됩니다.

> $CDC^{-1}x$는 $DC^{-1}x$의 basis를 다시 standard basis로 되돌리는 행위가 됩니다. 마찬가지로 change of basis는 vector를 고정시킨 채 축만 바꾸기 때문에 빨간색 원을 바뀌지 않고, basis만 $\mathbf{a,b}$는 $\mathbf{i,j}$로 바뀌게 됩니다.

> Eigendecomposition은 computational advantage를 비롯해, 다양한 application에 사용됩니다.

# Reference

1. https://github.com/jclosure/EngComp4_landlinear

02_Linear_Alegebra/my_sol/[HW7] LCS Practice 5 answer.ipynb


# Using DQN to cross a bridge in World of Warcraft

## The bridge
The bridge can be found at coordinates X:1669.62, Y:-3731.47, Z:148.3 in the zone Howling Fjord. The bridge have no rails thus the agent can easily fall off. The goal is to cross the bridge without falling down with the help of DQN. A trained agent should be able to cross. The bridge is also slightly tilted from the north.

For this task, Keras will be used for modeling the network.

```python
from keras.models import Sequential, load_model
from keras.optimizers import Adam, RMSprop
from keras.layers import Dense, Flatten, Dropout
from keras.callbacks import TensorBoard, EarlyStopping
from keras.initializers import RandomUniform

from collections import deque

import random
import numpy as np
import matplotlib.pyplot as plt

from shapely.geometry import Point
from shapely.geometry.polygon import Polygon

from ctypes import cdll, POINTER, c_float, c_int
from random import randint
from time import sleep

from IPython.display import clear_output
```

## Measured constants
The first step was to measure the bridge as a polygon concealed within four points. These four points are initalized below and were measured from the World of Warcraft client. The inital start position x and y is between the eastern and western starting point.

The variables k and m are the linear equation between the end points on the other side of the bridge. This linear equation is useful to determine if the agent passed the end of the bridge or not.

The step size is equal to half the width of the bridge. This will determine how many units the agent will move in every step. The bridge is contained within a polygon that will use the raycasting algorithm to decide if the agent is in or outside the polygon aka the bridge.

The image visualize the polygon that surrounds the bridge in the simulation.

```python
bridge_start_west = np.array([-3728.34, 1668.16])
bridge_start_east = np.array([-3734.89, 1668.64])

bridge_end_west = np.array([-3735.31, 1580.07])
bridge_end_east = np.array([-3741.63, 1580.53])

init_pos = np.sum([bridge_start_west, bridge_start_east], axis=0) / 2

k = (bridge_end_west[1] - bridge_end_east[1]) / (bridge_end_west[0] - bridge_end_east[0])
m = bridge_end_west[1] -(k * bridge_end_west[0])

step_size = np.abs((bridge_start_west[0] - bridge_start_east[0]) / 2)

current_angle = 0

polygon = Polygon([bridge_start_west, bridge_start_east, bridge_end_east, bridge_end_west])
```

## Determine goal

The goal line can be defined as:

\begin{align}
y = kx + m
\end{align}

If the agents current y position is less than the goals y, the agent crossed the bridge. The reason why the agent y have to be lesser is beacuse it move in a north to south direction.

```python
def is_goal(x, y):

    goal_y = (k*x) + m

    return y < goal_y
```

## Is within bridge
This is calculated with the Shapley library that implement the raycasting algorithm.

```python
def is_within_bridge(x, y):

    point = Point(x, y)

    return polygon.contains(point)
```

## Calculate distance from the right edge

To determine the distance to the right edge, the linear equation is calculated for the right sided line. This will be used to later calculate the perpendicular line. The x position in realtion to the edge can then be solved by finding what x is intersection both lines. The euclidean distance can then be used once we have the x and y on the right edge line compared to the agents current position.

The first two methods are only used to plot how far the agent reach after each episode.

```python
def get_start_edge():

    k_s = (bridge_start_west[1] - bridge_start_east[1]) / (bridge_start_west[0] - bridge_start_east[0])
    m_s = bridge_start_west[1] -(k_s * bridge_start_west[0])

    return k_s, m_s

k_s, m_s = get_start_edge()
```

```python
def get_start_edge_dist(x, y):

    k_p = -1 / k_s
    m_p = y - (k_p * x);

    x_new = (m_p - m_s) / (k_s - k_p);
    y_new = (k_s*x_new) + m_s;

    a = (x_new - x);
    b = (y_new - y);

    d = np.sqrt(a**2 + b**2);

    return d
```

```python
def get_right_edge():

    k_r = (bridge_start_west[1] - bridge_end_west[1]) / (bridge_start_west[0] - bridge_end_west[0])
    m_r = bridge_end_west[1] -(k_r * bridge_end_west[0])

    return k_r, m_r

k_r, m_r = get_right_edge()
```

```python
def get_edge_dist(x, y):

    k_p = -1 / k_r
    m_p = y - (k_p * x);

    x_new = (m_p - m_r) / (k_r - k_p);
    y_new = (k_r*x_new) + m_r;

    a = (x_new - x);
    b = (y_new - y);

    d = np.sqrt(a**2 + b**2);

    return d
```

## Avaliable actions
The agent have four avaliable actions that follows:

Adjust angle $+\frac{\pi}{8}$ radians

Adjust angle $+\frac{\pi}{4}$ radians

Adjust angle 0 radians

Adjust angle $-\frac{\pi}{8}$ radians

Adjust angle $-\frac{\pi}{4}$ radians

```python
def get_new_angle(action):

    v = np.pi / 8

    return{
        0 : current_angle + v,
        1 : current_angle + v*2,
        2 : current_angle,
        3 : current_angle - v,
        4 : current_angle - v*2,
    }[action]
```

## New position from predicted angle

Once the new angle is predicted, the new position can be determined from the current position. The figure below visualizes how the triangle can look before a step where the current angle is equal to $\alpha$. Since $sin(\alpha) = \frac{a}{c}$ and $cos(\alpha) = \frac{b}{c}$, $a$ and $b$ can be calculated from this. The new position $x'$ and $y'$ ten evaluates to: $x' = x + a$, $y' = y + b$.

```python
def do_action(action, x, y):

    v = get_new_angle(action)
    current_angle = v

    a = step_size * np.sin(v);
    b = step_size * np.cos(v);

    x_new = x - a;
    y_new = y - b;

    return x_new, y_new
```

## Rewards
These are the rewards that will determine how the agent perform.

```python
terminate_reward = 100
step_reward = -1
stuck_reward = -100

state_size = 2
action_size = 5
```

## Step
The state contain two variables, the curent angle and the distance to the right edge in realtion to the agent.

```python
def step(action, p_x, p_y):

    tmp_player_x = p_x
    tmp_player_y = p_y

    new_pos = do_action(action, p_x, p_y)

    p_x = new_pos[0]
    p_y = new_pos[1]

    reward = 0
    done = False

    if is_goal(p_x, p_y):
        done = True
        reward = terminate_reward

    elif is_within_bridge(p_x, p_y):
        reward = step_reward

    else:
        p_x = tmp_player_x
        p_y = tmp_player_y
        reward = stuck_reward;
        done = True

    state = np.reshape([current_angle, get_edge_dist(p_x, p_y)], [1, state_size])

    return state, reward, done, p_x, p_y

```

```python
def reset():

    p_x = init_pos[0]
    p_y = init_pos[1]

    current_angle = 0

    state = np.reshape([current_angle, get_edge_dist(p_x, p_y)], [1, state_size])

    return state, p_x, p_y

```

## The DQN

The agent class contain the logic for the DQN. The netowrk consist of two hidden layers with 40 neurons each. The neural net is visualized below.

```python
model_path = "my_model.h5"

class Agent:
    def __init__(self, state_size, action_size, discount, eps, eps_decay, eps_min, l_rate, decay_linear):
        self.path = model_path
        self.state_size = state_size
        self.action_size = action_size
        self.mem = deque(maxlen=2000)
        self.discount = discount
        self.eps = eps
        self.eps_decay = eps_decay
        self.decay_linear = decay_linear
        self.eps_min = eps_min
        self.l_rate = l_rate
        self.model = self.init_model()

    def load_model(self):
        return load_model(self.path)

    def save_model(self):
        self.model.save(self.path)

    def init_model(self):
        model = Sequential()
        model.add(Dense(40, input_dim=self.state_size, activation='relu'))
        model.add(Dense(40, activation='relu'))
        model.add(Dense(self.action_size, activation='linear'))
        model.compile(loss='mse', optimizer=Adam(lr=self.l_rate))

        return model

    def action(self, state):

        if np.random.rand() <= self.eps:
            return random.randrange(self.action_size)

        actions = self.model.predict(state)

        return np.argmax(actions[0])

    def remember(self, state, action, reward, next_state, terminal):
        self.mem.append((state, action, reward, next_state, terminal))

    def replay(self, batch_size):

        if len(self.mem) < batch_size:
            batch = self.mem
        else:
            batch = random.sample(self.mem, batch_size)

        for state, action, reward, next_state, terminal in batch:
            target = reward

            if not terminal:
                target += self.discount * np.amax(self.model.predict(next_state)[0])

            target_f = self.model.predict(state)
            target_f[0][action] = target

            self.model.fit(state, target_f, epochs=1, verbose=0)

        self.decay()

    def decay(self):
        if self.eps > self.eps_min:
            if self.decay_linear:
                self.eps -= self.eps_decay
            else:
                self.eps *= self.eps_decay

```

```python
agent = Agent(
            state_size = state_size,
            action_size = action_size,
            discount = 0.98,
            eps = 1,
            eps_decay = 0.001,
            eps_min = 0.001,
            l_rate = 0.001,
            decay_linear = True
            )

#Train agent
episodes = 2000
steps = 100

print_freq = 100
goalCounter = 0
goalAvg = -1
stepCounter = 0
epGoalCounter = 0
epGoalAvg = -1
epStepCounter = 0

total_ep = []
total_rewards = []
total_distance = []

for ep in range(1, episodes + 1):

    state, p_x, p_y = reset()

    player_x = p_x
    player_y = p_y

    total_reward = 0

    for st in range(1, steps):

        action = agent.action(state)

        next_state, reward, done, p_x, p_y = step(action, player_x, player_y)

        total_reward += reward

        player_x = p_x
        player_y = p_y

        agent.remember(state, action, reward, next_state, done)

        state = next_state

        if done:

            if reward > 1:
                goalCounter += 1
                epGoalCounter += 1
                epStepCounter += st
                epGoalAvg = epStepCounter / epGoalCounter
            break

    agent.replay(32)

    if ep % print_freq == 0 or ep == 1:

        total_ep.append(ep)
        total_rewards.append(total_reward)
        total_distance.append(get_start_edge_dist(player_x, player_y))

        print("episode {} done, found goal {} times with and avg step of {}, total goals: {}".format(ep, epGoalCounter, epGoalAvg, goalCounter))
        epGoalCounter = 0
        epGoalAvg = -1
        epStepCounter = 0

agent.save_model()
print("summary: total goals found: {}, an avg of {} episodes reached the goal".format(goalCounter, goalCounter/episodes))
```

    episode 1 done, found goal 0 times with and avg step of -1, total goals: 0
    episode 100 done, found goal 0 times with and avg step of -1, total goals: 0
    episode 200 done, found goal 0 times with and avg step of -1, total goals: 0
    episode 300 done, found goal 2 times with and avg step of 30.5, total goals: 2
    episode 400 done, found goal 8 times with and avg step of 30.375, total goals: 10
    episode 500 done, found goal 13 times with and avg step of 30.53846153846154, total goals: 23
    episode 600 done, found goal 25 times with and avg step of 30.96, total goals: 48
    episode 700 done, found goal 28 times with and avg step of 30.392857142857142, total goals: 76
    episode 800 done, found goal 33 times with and avg step of 30.393939393939394, total goals: 109
    episode 900 done, found goal 71 times with and avg step of 30.718309859154928, total goals: 180
    episode 1000 done, found goal 90 times with and avg step of 30.68888888888889, total goals: 270
    episode 1100 done, found goal 100 times with and avg step of 29.45, total goals: 370
    episode 1200 done, found goal 100 times with and avg step of 29.06, total goals: 470
    episode 1300 done, found goal 100 times with and avg step of 28.42, total goals: 570
    episode 1400 done, found goal 100 times with and avg step of 28.85, total goals: 670
    episode 1500 done, found goal 100 times with and avg step of 28.69, total goals: 770
    episode 1600 done, found goal 100 times with and avg step of 30.03, total goals: 870
    episode 1700 done, found goal 100 times with and avg step of 29.4, total goals: 970
    episode 1800 done, found goal 100 times with and avg step of 29.81, total goals: 1070
    episode 1900 done, found goal 100 times with and avg step of 29.14, total goals: 1170
    episode 2000 done, found goal 100 times with and avg step of 29.25, total goals: 1270
    summary: total goals found: 1270, an avg of 0.635 episodes reached the goal

```python
plt.plot(total_ep, total_rewards)
plt.title('Total reward from start')
plt.ylabel('Reward')
plt.xlabel('Episode')
plt.legend(['Reward'], loc='upper left')
plt.show()
```

```python
plt.plot(total_ep, total_distance, color='orange')
plt.title('Distance from start over time')
plt.ylabel('Distance')
plt.xlabel('Episode')
plt.legend(['Distance'], loc='upper left')
plt.show()
```

## Testing the trained network in the client
The last step is to confirm that the trained network works in the World of Warcraft client. To do this Click to move was used  to walk towards a given x and y. This is an built in function for World of Warcraft. This can be accsed by manipulating memory variables with a language such as C++. The python class ctypes allow for calls to compiled C++ classes. The libenv.so were compiled with MinGW and the source code for this library can be found at: https://github.com/Jacobth/wow_rl_environment.

```python
lib = cdll.LoadLibrary('libenv.so')

class Env(object):
    def __init__(self):
        self.obj = lib.Env()

    def reset(self, init_x, init_y):
        lib.ResetTest.argtypes = [c_float, c_float]
        lib.ResetTest(self.obj, init_x, init_y)

    def step(self, new_x, new_y):
        lib.StepTest.argtypes = [c_float, c_float]
        lib.StepTest(self.obj, new_x, new_y)

    def get_pos(self):
        lib.GetPos.restype = POINTER(c_float * 2)

        values = lib.GetPos(self.obj).contents
        x = values[0]
        y = values[1]

        return x, y
```

```python
def test_model() :

    env = Env()

    model = load_model(model_path)

    state, p_x, p_y = reset()
    env.reset(c_float(p_x), c_float(p_y))

    pl_x = p_x
    pl_y = p_y

    total_reward = 0
    total_steps = 0

    while True:

        action = np.argmax(model.predict(state)[0])
        next_state, reward, done, p_x, p_y = step(action, pl_x, pl_y)

        env.step(c_float(p_x), c_float(p_y))

        state = next_state

        pl_x = p_x
        pl_y = p_y

        total_reward += reward
        total_steps += 1

        if done:
            print("completed with reward: {} and {} steps.".format(total_reward, total_steps))
            break;

test_model()
```

    completed with reward: 72 and 29 steps.

This method can be used to test if the right distance behave correctly in the client.

```python
def test_dist() :

    env = Env()

    while True:

        x, y = env.get_pos()

        print(get_edge_dist(x, y))
        sleep(0.5)
        clear_output()

#test_dist()
```